Multiclass versus Binary Differentially Private PAC Learning

1 Introduction . 1 Machine learning and data analytics are increasingly deployed on sensitive information about individuals . Differential privacy [ 10 ] gives a mathematically rigorous way to enable such analyses while guaranteeing the privacy of individual information . The model of differentially private PAC learning [ 17 ] captures binary classification for sensitive data , providing a simple and broadly applicable abstraction for many machine learning procedures . Private PAC learning is now reasonably well-understood , with a host of general algorithmic techniques , lower bounds , and results for specific fundamental concept classes [ 8 , 12 , 3 , 4 , 1 , 15 , 2 , 16 ] . Beyond binary classification , many problems in machine learning are better modeled as multiclass learning problems . Here , given a training set of examples from domain X with labels from [ k ] = { 0 , 1 , . . . , k } , the goal is to learn a function h : X → [ k ] that approximately labels the data and generalizes to the underlying population from which it was drawn . Much less is presently known about differentially private multiclass learnability than is known about private binary classification , though it appears that many specific tools and techniques can be adapted one at a time . In this work , we ask : Can we generically relate multiclass to binary learning so as to automatically transfer results from the binary setting to the multiclass setting ? To illustrate , there is a simple reduction from a given multiclass learning problem to a sequence of binary classification problems . ( This reduction was described by Ben-David et al . [ 5 ] for non-private 1The full version of this paper with all of the details can be found at https : //arxiv.org/abs/2107.10870 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . learning , but works just as well in the private setting . ) Intuitively , one can learn a multi-valued label one bit at a time . That is , to learn an unknown function f : X → [ k ] , it suffices to learn the dlog2 ( k + 1 ) e binary functions fi : X → [ k ] , where each fi is the ith bit of f . Theorem 1.1 ( Informal ) . Let H be a concept class consisting of [ k ] -valued functions . If all of the binary classes Hi = { fi : f ∈ H } are privately learnable , then H is privately learnable . Beyond its obvious use for enabling the use of tools for binary private PAC learning on the classes Hi , we show that Theorem 1.1 has strong implications for relating the private learnability of H to the combinatorial properties ofH itself . Our main application of this reductive perspective is an improved sample complexity upper bound for private multiclass learning in terms of online learnability . 1.1 Online vs . Private Learnability . A recent line of work has revealed an intimate connection between differentially private learnability and learnability in Littlestone ’ s mistake-bound model of online learning [ 18 ] . For binary classes , the latter is tightly captured by a combinatorial parameter called the Littlestone dimension ; a class H is online learnable with mistake bound at most d if and only if its Littlestone dimension is at most d. The Littlestone dimension also qualitatively characterizes private learnability . If a class H has Littlestone dimension d , then every private PAC learner for H requires at least Ω ( log∗ d ) samples [ 1 ] . Meanwhile , Bun et al . [ 7 ] showed that H is privately learnable using 22 O ( d ) samples , and Ghazi et al . [ 13 ] gave an improved algorithm using Õ ( d6 ) samples . ( Moreover , while quantitatively far apart , both the upper and lower bound are tight up to polynomial factors as functions of the Littlestone dimension alone [ 15 ] . ) Jung et al . [ 14 ] recently extended this connection from binary to multiclass learnability . They gave upper and lower bounds on the sample complexity of private multiclass learnability in terms of the multiclass Littlestone dimension [ 9 ] . Specifically , they showed that if a multi-valued class H has multiclass Littlestone dimension d , then it is privately learnable using 2k O ( d ) samples and that every private learner requires Ω ( log∗ d ) samples . Jung et al. ’ s upper bound [ 14 ] directly extended the definitions and arguments from Bun et al. ’ s [ 7 ] earlier 22 O ( d ) -sample algorithm for the binary case . While plausible , it is currently unknown and far from obvious whether similar adaptations can be made to the improved binary algorithm of Ghazi et al . [ 13 ] . Instead of attacking this problem directly , we show that Theorem 1.1 , together with additional insights relating multiclass and binary Littlestone dimensions , allows us to generically translate sample complexity upper bounds for private learning in terms of binary Littlestone dimension into upper bounds in terms of multiclass Littlestone dimension . Instantiating this general translation using the algorithm of Ghazi et al . gives the following improved sample complexity upper bound . Theorem 1.2 ( Informal ) . Let H be a concept class consisting of [ k ] -valued functions and let d be the multiclass Littlestone dimension of H . Then H is privately learnable using Õ ( d6 log8 ( k + 1 ) ) samples . In addition to being conceptually simple and modular , our reduction from multiclass to binary learning means that potential future improvements for binary learning will also automatically give improvements for multiclass learning . For example , if one were able to prove that all binary classes of Littlestone dimension d are privately learnable with O ( d ) samples , this would imply that every [ k ] -valued class of multiclass Litttlestone dimension d is privately learnable with Õ ( d log3 ( k + 1 ) ) samples.2 1.2 Techniques . Theorem 1.1 shows that a multi-valued class H is privately learnable if all of the binary classes Hi are privately learnable , which in turn holds as long as we can control their ( binary ) Littlestone dimensions . So the last remaining step in order to conclude Theorem 1.2 is to show that if H has bounded multiclass Littlestone dimension , then all of the classes Hi have bounded binary Littlestone dimension . At first glance , this may seem to follow immediately from the fact that ( multiclass ) 2The nearly cubic dependence on log ( k + 1 ) follows from the fact that the accuracy of private learners can be boosted with a sample complexity blowup that is nearly inverse linear in the target accuracy [ 11 , 6 ] . Littlestone dimension characterizes ( multiclass ) online learnability – a mistake bounded learner for a multiclass problem is , in particular , able to learn each individual output bit of the function being learned . The problem with this intuition is that the multiclass learner is given more feedback from each example , namely the entire multi-valued class label , than a binary learner for each Hi that is only given a single bit . Nevertheless , we are still able to use combinatorial methods to show that multiclass online learnability of a class H implies online learnability of all of the binary classes Hi . Theorem 1.3 . Let H be a [ k ] -valued concept class with multiclass Littlestone dimension d. Then every binary class Hi has Littlestone dimension at most 6d ln ( k + 1 ) . Moreover , this result is nearly tight . In the full version of the paper , we show that for every k , d ≥ 1 there is a [ k ] -valued class with multiclass Littlestone dimension d such that at least one of the classes Hi has Littlestone dimension at least Ω ( d log ( k + 1 ) ) . In addition , it turns out that online learnability of the binary classes Hi implies multiclass online learnability of H . In the full version of the paper , we show that if the multiclass Littlestone dimension of H is d , then there is at least one binary class Hi with Littlestone dimension larger than dlog ( k+1 ) . This result is also tight . Theorem 1.3 is the main technical contribution of this work . The proof adapts techniques introduced by Ben-David et al . [ 5 ] for characterizing the sample complexity of ( non-private ) multiclass PAC learnability . Specifically , Ben-David et al . introduced a family of combinatorial dimensions , parameterized by collections of maps Ψ and called Ψ-dimensions , associated to classes of multi-valued functions . One choice of Ψ corresponds to the “ Natarajan dimension ” [ 19 ] , which was previously known to give a lower bound on the sample complexity of multiclass learnability . Another choice corresponds to the “ graph dimension ” [ 19 ] which was known to give an upper bound . Ben-David et al . gave conditions under which Ψ-dimensions for different choices of Ψ could be related to each other , concluding that the Natarajan and graph dimensions are always within an O ( log ( k+ 1 ) ) factor , and thus characterizing the sample complexity of multiclass learnability up to such a factor . Our proof of Theorem 1.3 proceeds by extending the definition of Ψ-dimension to online learning . We show that one choice of Ψ corresponds to the multiclass Littlestone dimension , while a different choice corresponds to the maximum Littlestone dimension of any binary class Hi . We relate the two quantities up to a logarithmic factor using a new variant of the Sauer-Shelah-Perles Lemma for the “ 0- cover numbers ” of a class of multi-valued functions . While we were originally motivated by privacy , we believe that Theorem 1.3 and the toolkit we develop for understanding online Ψ-dimensions may be of broader interest in the study of ( multiclass ) online learnability . Finally , we remark that Theorem 1.3 implies a qualitative converse to Lemma 1.1 . If a multi-valued class H is privately learnable , then the lower bound of [ 14 ] implies that H has finite multiclass Littlestone dimension . Theorem 1.3 then shows that all of the classes Hi have finite binary Littlestone dimension , which implies via [ 7 , 13 ] that they are also privately learnable . Societal Impact Our work is motivated by privacy-respecting data analysis . Our focus is on theoretical questions aimed at uncovering general principles about when private learning is feasible . As such , it does not negatively impact the way privacy-respecting techniques are used , but rather it clarifies their potential . 2 Background . Differential privacy . Differential privacy is a property of a randomized algorithm guaranteeing that the distributions obtained by running the algorithm on two datasets differing for one individual ’ s data are indistinguishable up to a multiplicative factor e and an additive factor δ . Formally , it is defined as follows : Definition 2.1 ( Differential privacy , [ 10 ] ) . Let n ∈ N. A randomized algorithm M : Xn → Y is ( , δ ) -differentially private if for all subsets S ⊆ Y of the output space , and for all datasets X and X ′ containing n elements of the universe X and differing in at most one element ( we call these neighbouring datasets ) , we have that Pr ( M ( X ) ∈ S ) ≤ e Pr ( M ( X ′ ) ∈ S ) + δ One useful property of differential privacy that we will use is that any output of a differentially private algorithm is closed under ‘ post-processing ’ , that is , its can not be made less private by applying any data-independent transformations . Lemma 2.2 ( Post-processing of differential privacy , [ 10 ] ) . If M : Xn → Y is ( , δ ) -differentially private , and B : Y → Z is any randomized function , then the algorithm B ◦M is ( , δ ) -differentially private . PAC learning . PAC learning [ 21 ] aims at capturing natural conditions under which an algorithm can approximately learn an hypothesis class . Definition 2.3 ( Hypothesis class ) . A hypothesis class H with input space X and output space Y ( also called the label space ) is a set of functions f mapping X to Y . Where it is clear , we will not explicitly name the input and output spaces . We can now formally define PAC learning . Definition 2.4 ( PAC learning , [ 21 ] ) . A learning problem is defined by a hypothesis class H . For any distribution P over the input space X , consider n independent draws x1 , x2 , · · ·xn from distribution P . A labeled sample of size n is the set { ( x1 , f ( x1 ) ) , ( x2 , f ( x2 ) ) , · · · , ( xn , f ( xn ) ) } where f ∈ H . We say an algorithm A taking a labeled sample X of size n is an ( α , β ) -accurate PAC learner for the hypothesis class H if for all functions f ∈ H and for all distributions P over the input space , A on being given a labeled sample of size n drawn from P and labeled by f , outputs a hypothesis h ∈ H such that with probability greater than or equal to 1− β over the randomness of the sample and the algorithm , Pr [ h ( x ) 6= f ( x ) ] ≤ α . The definition above defines PAC learning in the realizable setting , where all the functions f labeling the data are in H . Two well studied settings for PAC learning are the binary learning case , where Y = { 0 , 1 } and the multiclass learning case , where Y = [ k ] = { 0 , 1 , · · · , k } for natural numbers k > 2 . The natural notion of complexity for PAC learning is sample complexity . Definition 2.5 ( Sample complexity ) . The sample complexity SH , α , β ( A ) of algorithm A with respect to hypothesis class H is the minimum size of the sample that the algorithm requires in order to be an ( α , β ) -accurate PAC learner for H . The PAC complexity of the hypothesis class H is inf A SH , α , β ( A ) . In this work , we will be interested in generic learners , that work for every hypothesis class . Definition 2.6 ( Generic learners ) . We say that an algorithm A that additionally takes the hypothesis class as an input , is a generic ( α , β ) -accurate private PAC learner with sample complexity function SC ( H , α , β ) , if for every hypothesis class H , it is an ( α , β ) -accurate private PAC learner for H with sample complexity SC ( H , α , β ) . Differentially private PAC learning . We can now define differentially private PAC learning , by putting together the constraints imposed by differential privacy and PAC learning , respectively . Definition 2.7 ( Differentially private PAC learning [ 17 ] ) . An algorithm A is an ( , δ ) -differentially private and ( α , β ) -accurate private PAC learner for the hypothesis class H with sample complexity n if and only if : 1 . A is an ( α , β ) -accurate PAC learner for the hypothesis class H with sample complexity n. 2 . A is ( , δ ) -differentially private . In this work , we study the complexity of private PAC learning . Our work focuses on the multiclass realizable setting . Multiclass Littlestone dimension . We recall here the definition of multiclass Littlestone dimension [ 9 ] , which we will use extensively in this work . Unless stated otherwise , we will use the convention that the root of a tree is at depth 0 . As a first step , we define a class of labeled binary trees , representing possible input-output label sequences over an input space X and the label space [ k ] . Definition 2.8 ( Complete io-labeled binary tree ) . A complete io-labeled binary tree of depth b with input set X and output set [ k ] consists of a complete binary tree of depth b with the following properties : 1 . Every node of the tree other than the leaves is labeled by an example x ∈ X . 2 . The 2 edges going from any parent node to its two children are labeled by two different labels in [ k ] . 3 . The leaf nodes of the tree are unlabeled . We are interested in whether the input-ouput labelings defined by the complete io-labeled tree can be achieved by some function in the hypothesis class ; to this end , we define realizability for root-to-leaf paths . Definition 2.9 . Given a complete io-labeled binary tree of depth b , consider a root-to-leaf path described as an ordered sequence S = { ( xi , yi ) |i ∈ [ b ] } , where xi is a node label and yi is the label of the edge between xi and xi+1 , and where x0 is the root . We say that the root-to-leaf path is realized by a function f ∈ H if for every ( xi , yi ) in S , we have xi ∈ X and yi = f ( xi ) . Using this definition we can now define what it means for a hypothesis class of functions to shatter a complete io-labeled binary tree , which helps to capture how expressive the hypothesis class is . Definition 2.10 ( Shattering ) . We say that a complete io-labeled binary tree of depth b with label set [ k ] is shattered by a hypothesis class H if for all 2b root-to-leaf sequences S of the tree , there exists a function f ∈ H that realizes S. Using this definition of shattering we can finally define the multiclass Littlestone dimension . Definition 2.11 ( Multiclass Littlestone dimension , [ 9 ] ) . The multiclass Littlestone dimension of a hypothesis class H , denoted MLS ( H ) , is defined to be the maximum b such that there exists a complete io-labeled binary tree of depth b that is shattered by H . If no maximum exists , then we say that the multiclass Littlestone dimension of H is∞ .

Summary: This work studies the connection between the differential privacy of binary learning and multiclass learning. The authors first show a simple reduction from multiclass to binary to private learning, meaning that they show that for any $f$ if there is an algorithm that private learns each binary function of $f$ (meaning learns each bit of $f$) then there is a private algorithm for $f$. Furthermore, the authors show tight bounds on Littlestone of multiclass and binary learning. They prove that if a multiclass hypothesis class has Littlestone dimension $d$ then each binary restriction of the same class has Littlestone dimension bounded by $\sim 6d\log k$. The authors provide upper and lower bounds for this claim. Moreover, the authors provide an upper bound to the sample complexity or privately learning a multiclass hypothesis relatively to the Littlestone dimension of this class.