Antipodes of Label Differential Privacy: PATE and ALIBI

1 Introduction . Sophisticated machine learning models perform well on their target tasks despite of—or thanks to—their predilection for data memorization [ 39 , 2 , 17 ] . When such models are trained on non-public inputs , privacy concerns become paramount [ 31 , 6 , 7 ] , which motivates the actively developing area of privacy-preserving machine learning . With some notable exceptions , which we discuss in the related works section , existing research has focused on an “ all or nothing ” privacy definition , where all of the training data , i.e. , both features and labels , is considered private information . While this goal is appropriate for many applications , there are several important scenarios where label-only privacy is the right solution concept . A prominent example of label-only privacy is in online advertising , where the goal is to predict conversion of an ad impression ( the label ) given a user ’ s profile and the spot ’ s context ( the features ) . The features are available to the advertising network , which trains the model , while the labels ( data on past conversion events ) are visible only to the advertiser . More generally , any two-party setting where data is vertically split between public inputs and ( sensitive ) outcomes is a candidate for training with label-only privacy . In this work we adopt the definition of differential privacy as our primary notion of privacy . The definition , introduced in the seminal work by Dwork et al . [ 13 ] , satisfies several important properties that make it well-suited for applications : composability , robustness to auxiliary information , preservation under post-processing , and graceful degradation in the presence of correlated inputs ( group privacy ) . Following Ghazi et al . [ 18 ] , we refer to the setting of label-only differential privacy as Label DP . ∗ { manimalek , imironov , krp , shilov } @ fb.com , Facebook AI . †tramer @ cs.stanford.edu , Stanford University . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Our contributions . We explore two approaches—with very different characteristics—toward Label DP . These approaches and our methodology for estimating empirical privacy loss via label memorization are briefly reviewed below : • PATE : We adapt the PATE framework ( Private Aggregation of Teacher Ensembles ) of Papernot et al . [ 28 ] to the Label DP setting by observing that PATE ’ s central assumption— availability of a public unlabeled dataset—can be satisfied for free . Indeed , a public unlabeled dataset is simply the private dataset with the labels removed ! By instantiating PATE with a state-of-the-art technique for semi-supervised learning , we demonstrate an excellent tradeoff between empirical privacy loss and accuracy . • ALIBI ( Additive Laplace with Iterative Bayesian Inference ) : We propose applying additive Laplace noise to a one-hot encoding of the label . Since differential privacy is preserved under post-processing , we can de-noise the mechanism ’ s output by performing Bayesian inference using the prior provided by the model itself , doing so continuously during the training process . ALIBI improves , particularly in a high-privacy regime , Ghazi et al. ’ s approach based on randomized response [ 18 ] . • Empirical privacy loss . We describe a memorization attack targeted at the Label DP scenario that efficiently extracts lower bounds for the privacy parameter that , in some cases , come within a factor of 2.5 from the theoretical , worst-case upper bounds against practically relevant , large-scale models . For comparison , recent , most advanced black-box attacks on DP-SGD have approximately a 10× gap between the upper and lower bounds [ 20 , 26 ] . Both algorithms , PATE and ALIBI , come with rigorous privacy analyses that , for any fixed setting of parameters , result in an ( ε , δ ) -DP bound . We emphasize that these bounds are just that—they are upper limits on an attacker ’ s ability to breach privacy ( in a precise sense ) on worst-case inputs . Even though these bounds can be pessimistic , intuitively , a smaller ε for the same δ corresponds to a more private instantiation of a mechanism , and indeed this is likely the case within an algorithmic family . Can the privacy upper bounds be used to compare diverse mechanisms with widely dissimilar analyses , such as ours ? We argue that focusing on the upper bounds alone may be misleading to the point of favoring a less private algorithm with tighter analysis over a more private one whose privacy analysis is less developed . For a uniform perspective , we estimate the empirical privacy loss of all trained models by evaluating the performance of a black-box membership inference attack [ 31 , 20 , 26 ] . Our attack directly measures memorization of deliberately mislabeled examples , or canaries [ 6 ] . Following prior work [ 26 , 20 ] , we use the success rate of our attack to compute a lower bound on the level ε of label-DP achieved by the training algorithm . To avoid the prohibitive cost of retraining a model for each canary , we propose and analyze a heuristic that consists in inserting multiple canaries into one model , and measuring an attacker ’ s success rate in guessing each canary ’ s label . In some settings , we find that the empirical privacy of PATE ( as measured by our attack ) is significantly stronger than that of ALIBI , even though the provable DP bound for PATE is much weaker . Finally , to characterize setups where label-only privacy may be applicable , it is helpful to distinguish between inference tasks , where the label is fully determined by the features , and prediction , where there is some uncertainty as to the eventual outcome , despite ( hopefully ) some predictive value of the feature set . We are mostly concerned with the latter scenario , where the outcome is not evident from the features , or protecting choices made by individuals may be mandated or desirable ( for connection between privacy and self-determination see Schwartz [ 30 ] ) . While we do use public data sets ( CIFAR-10 and CIFAR-100 ) in which it is possible to “ infer ” the label , we do so only to evaluate our approaches on standard benchmarks . We completely remove the triviality of inference by adding mislabeled canaries in our attack , thereby introducing a measure of non-determinism . 2 Notation and Definitions . Throughout the paper we consider standard supervised learning problems . The inputs are ( x , y ) pairs where x ∈ X are the features and y ∈ Y are the labels . ( If some labels are absent , the problem becomes semi-supervised learning . ) The cardinality of the set Y is the number of classes C. The task is to learn a model M parameterized with weights W that predicts y given x . To this end the training algorithm has access to a training set of n samples { ( xi , yi ) } n. Differential privacy due to Dwork et al . [ 13 ] is a rigorous , quantifiable notion of individual privacy for statistical algorithms . ( See Dwork and Roth [ 14 ] and Vadhan [ 33 ] for book-length introductions . ) Definition 2.1 ( Differential Privacy ) . We say that a randomized mechanismM : U → R satisfies ( ε , δ ) -differential privacy ( DP ) if for all adjacent inputs D , D′ ∈ U that differ in contributions of a single sample , the following holds : ∀S ⊂ R Pr [ M ( D ) ∈ S ] ≤ eε · Pr [ M ( D′ ) ∈ S ] + δ . If δ = 0 , we refer to it as ε-DP . The notion of adjacency between databases D and D′ is domain- and application-dependent . To instantiate the Label DP setting , we say that D and D′ are adjacent if they consist of the same number of examples { ( xi , yi ) } n and are identical except for a difference in a single label at index i∗ : ( xi∗ , yi∗ ) ∈ D and ( xi∗ , y′i∗ ) ∈ D′ . 3 Background and related work . DP and machine learning . Differential privacy aligns extremely well with the goal of machine learning , which is to produce valid models or hypotheses on the basis of observed data . In fact , the relationship can be made precise . An important metric in ML is generalization error , which measures the model ’ s performance on previously unseen samples . Differentially private mechanisms provably generalize [ 11 ] . For an introduction to some important early results on connections between differential privacy and learning theory see Dwork and Roth [ 14 , Chapter 11 ] . Randomized response ( RR ) . As a disclosure avoidance methodology , randomized response ( RR ) [ 36 ] precedes the advent of differential privacy . In RR ’ s basic form , with some probability the sensitive input is replaced by a sample drawn uniformly from its entire domain . RR can be implemented at the source , making it an essential building block in achieving Local DP [ 16 ] . More generally , RR is an instance of the input perturbation technique , which leaves the ML training algorithm intact and achieves differential privacy via input randomization . Input perturbations methods are particularly friendly to implementers as they require minimal changes to existing algorithms , treating them as black boxes . Additive noise mechanisms . Another important class of differentially private mechanisms are additive noise methods . Two canonical examples are additive Laplace [ 13 ] and additive Gaussian [ 12 ] that can be applied to vector-valued functions . Their noise distributions are calibrated to the function ’ s sensitivity—the maximal distance between the function ’ s outputs on adjacent inputs ( D and D′ in Definition 2.1 ) under the ` 1 or ` 2 metrics for the Laplace and Gaussian mechanisms respectively . PATE . Private Aggregation of Teacher Ensembles ( PATE ) [ 28 , 29 ] is a framework based on private knowledge aggregation of an ensemble model and knowledge transfer . PATE trains an ensemble of “ teachers ” on disjoint subsets of the private dataset . The ensemble ’ s knowledge is then transferred to a “ student ” model via differentially private aggregation of the teachers ’ votes on samples from an unlabeled public dataset . Only the student model is released as the output of the training , as it accesses sensitive data via a privacy-preserving interface . Since PATE consumes the privacy budget each time the student queries the teachers , the training algorithm must be optimized to minimize the number of queries . To this end , Papernot et al . explore the use of semi-supervised learning techniques , most notably GANs . Recent advances in semisupervised learning have been shown to boost the performance of PATE on standard benchmarks [ 5 ] . FixMatch . FixMatch [ 32 ] is a state-of-the-art semi-supervised learning algorithm , which achieves high accuracy on benchmark image datasets with very few labeled examples . It is based on the concepts of pseudo-labeling [ 23 ] and consistency regularization [ 3 ] : FixMatch uses a partially trained model to generate labels on weakly-augmented unlabeled data , and then , if the prediction is confident enough , uses the output as a label for the strongly-augmented version of the same data point . Label-only privacy . The setting of label-only DP was formally introduced by Chaudhuri and Hsu [ 8 ] , who proved lower bounds on sample complexity for private PAC-learners . Beimel et al . [ 4 ] demonstrated that the sample complexity of Label DP matches that of non-private PAC learning ( ignoring constants , dependencies on the privacy parameters , and computational efficiency ) . Wang and Xu [ 34 ] considered the linear regression problem in the local , label-only privacy setting . Most recently , Ghazi et al . [ 18 ] considered label-only privacy for deep learning . They propose a Label Privacy Multi-Stage Training ( LP-MST ) algorithm , which we revisit in Section 5 . The algorithm advances in stages ( Ghazi et al . evaluate it up to four ) , training the model on disjoint partitions of the dataset . In each stage , the private labels are protected using a randomized response scheme . The most innovative part of LP-MST happens at the point of transition between stages . Labels of the training examples of the next partition are perturbed using randomized response . The ( soft ) output of the previous stage ’ s model is interpreted as a prior , which is used to choose parameters for the randomized response algorithm separately for each example . For instance , if the current model assigns high probabilities to labels 1–10 and low probabilities to labels 11–100 , the randomized response will be constrained to outputting a label within the set 1–10 . We build on the insights of Ghazi et al . by making two observations . First , training algorithms used for deep learning ( such as SGD and its variants ) can naturally handle soft labels ( i.e. , probability distributions over the label space ) ; there is thus no need to force hard labels by sampling from the posterior . Second , by applying additive noise , we can repeatedly recompute the posterior distribution as better priors become available without consuming privacy budget . By following the “ once and done ” approach to label perturbation , we forgo the need to split training into disjoint stages . To summarize , Ghazi et al . fix the prior at the beginning of the stage , choose parameters of the randomized response algorithm on the basis of this prior , and update the model by feeding it the perturbed label . We flip the order of these operations by applying a DP mechanism ( additive Laplace ) first , and then repeatedly use Bayesian inference by combining the prior that changes with each model update and the observables that do not . To our knowledge , we are the first to explore connections between PATE and Label DP .

Summary: The authors consider label differential privacy (DP) where the goal is to satisfy DP with respect to the labels of the examples. This model was formally introduced by Chaudhuri and Hsu (COLT 2011).  They propose two models for label DP:  1) Based on ALIBI (Additive Laplace noIse coupled with Bayesian Inference)  2) Based on PATE (Private Aggregation of Teacher Ensembles)  A prominent example where label DP is quite useful is in online advertising where the goal is to, given a user's profile, predict the advert conversion probability (the label). If we assume that the training is done by the advertising network (e.g., a tech company) who has the data (the features), the goal is to make sure the advertiser (some company who wishes to advertise on a tech platform) cannot infer the labels.  They estimate the *empirical* privacy loss of the trained models via the lens of black-box membership inference attacks.  The novelty of this work, it seems, lies in the (previously unexplored) connections between PATE and label DP. Ghazi et al. also proposed randomized response mechanisms for label privacy.  They experimentally evaluate the performance of their algorithms on CIFAR-10, CIFAR-100 where they show medium to high accuracy results (e.g., 93.7% on $\epsilon = 1.6$).