On the Sample Complexity of Privately Learning Axis-Aligned Rectangles

{ d· log |X| , d1.5· ( log∗ |X| ) 1.5 } . That is , existing constructions either require sample complexity that grows linearly with log |X| , or else it grows super linearly with the dimension d. We present a novel algorithm that reduces the sample complexity to only Õ ( d· ( log∗ |X| ) 1.5 ) , attaining a dimensionality optimal dependency without requiring the sample complexity to grow with log |X| . The technique used in order to attain this improvement involves the deletion of “ exposed ” data-points on the go , in a fashion designed to avoid the cost of the adaptive composition theorems . The core of this technique may be of individual interest , introducing a new method for constructing statistically-efficient private algorithms . 1 Introduction . Differential privacy [ Dwork et al. , 2006 ] is a mathematical definition for privacy , that aims to enable statistical analyses of databases while providing strong guarantees that individual-level information does not leak . More specifically , consider a database containing data pertaining to individuals , and suppose that we have some data analysis procedure that we would like to apply to this database . We say that this procedure preserves differential privacy if no individual ’ s data has a significant effect on the distribution of the outcome of the procedure . Intuitively , this guarantees that whatever is learned about an individual from the outcome of the computation could also be learned with her data arbitrarily modified ( or without her data ) . Formally , Definition 1.1 ( Dwork et al . [ 2006 ] ) . A randomized algorithm A is ( ε , δ ) -differentially private if for every two databases S , S′ that differ on one row ( such databases are called neighboring ) , and every set of outcomes F , we have Pr [ A ( S ) ∈ F ] ≤ eε · Pr [ A ( S′ ) ∈ F ] + δ . The definition is referred to as pure differential privacy when δ = 0 , and approximate differential privacy when δ > 0 . Over the last decade , we have witnessed an explosion of research on differential privacy , and by now it is largely accepted as a gold-standard for privacy preserving data analysis . In particular , there has been a lot of interest in designing private learning algorithms , which are learning algorithms that guarantee differential privacy for their training data . Intuitively , this guarantees that the outcome of the learner ( the identified hypothesis ) leaks very little information on any particular point from the training set . Works in this vein include [ Kasiviswanathan et al. , 2011 , Beimel et al. , 2014 , 2019b , 2016 , 2020 , Bun et al. , 2015 , Feldman and Xiao , 2015 , Bun et al. , 2019 , Beimel et al. , 2019a , Kaplan et al. , 2019 , 2020a , Alon et al. , 2020 , Kaplan et al. , 2020b , Bun et al. , 2020 , Alon et al. , 2019 ] , and much more . ∗Department of Computer Science , Ben-Gurion University of the Negev . sadigurs @ post.bgu.ac.il . †Blavatnik School of Computer Science , Tel Aviv University and Google Research . u @ uri.co.il . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , in spite of the dramatic progress made in recent years on the theory and practice of private learning , much remains unknown and answers to fundamental questions are still missing . In this work , we revisit one such fundamental open question , specifically , Question 1.2 . What is the sample complexity of learning axis-aligned rectangles with privacy ? Non-privately , learning axis aligned rectangles is one of the most simple and basic of learning tasks , often given as the first example for PAC learning in courses or teaching books . Nevertheless , somewhat surprisingly , the sample complexity of learning axis-aligned rectangles with differential privacy is not well-understood . In this work we make a significant progress towards understanding this basic question . 1.1 Existing and New Results . Recall that the VC dimension of the class of all axis-aligned rectangles over Rd is O ( d ) , and hence a sample of size O ( d ) suffices to learn axis-aligned rectangles non-privately ( we omit throughout the introduction the dependency of the sample complexity in the accuracy , confidence , and privacy parameters ) . In contrast , it turns out that with differential privacy , learning axis-aligned rectangles over Rd is impossible , even when d = 1 [ Feldman and Xiao , 2015 , Bun et al. , 2015 , Alon et al. , 2019 ] . In more detail , let X = { 1 , 2 , . . . , |X| } be a finite ( one dimensional ) grid , and consider the task of learning axis-aligned rectangles over the finite d-dimensional grid Xd ⊆ Rd . In other words , consider the task of learning axis-aligned rectangles under the promise that the underlying distribution is supported on ( a subset of ) the finite grid Xd . For pure-private learning , Feldman and Xiao [ 2015 ] showed a lower bound of Ω ( d · log |X| ) on the sample complexity of this task . This lower bound is tight , as a pure-private learner with sample complexity Θ ( d · log |X| ) can be obtained using the generic upper bound of Kasiviswanathan et al . [ 2011 ] . This should be contrasted with the non-private sample complexity , which is independent of |X| . For approximate-private learning , Beimel et al . [ 2016 ] showed that the dependency of the sample complexity in |X| can be significantly reduced . This , however , came at the cost of increasing the dependency in the dimension d. Specifically , the private learner of Beimel et al . [ 2016 ] has sample complexity Õ ( d3 · 8log∗ |X| ) . We mention that a dependency on log∗ |X| is known to be necessary [ Bun et al. , 2015 , Alon et al. , 2019 ] . Recently , Beimel et al . [ 2019a ] and Kaplan et al . [ 2020b ] studied the related problem of privately learning halfspaces over a finite grid Xd , and presented algorithms with sample complexity Õ ( d2.5 · 8log∗ |X| ) . Their algorithms can be used to privately learn axis-aligned rectangles overXd with sample complexity Õ ( d1.5 · 8log∗ |X| ) . This can be further improved using the recent results of Kaplan et al . [ 2020a ] , and obtain a differentially private algorithm for learning axis-aligned rectangles over Xd with sample complexity Õ ( d1.5 · ( log∗ |X| ) 1.5 ) . We consider this bound to be the baseline for our work , and we will elaborate on it later . To summarize , our current understanding of the task of privately learning axis-aligned rectangles over Xd gives us two kinds of upper bounds on the sample complexity : Either d · log |X| or d1.5 · ( log∗ |X| ) 1.5 . That is , current algorithms either require sample complexity that scales with log |X| , or else it scales super linearly in the dimension d. This naturally leads to the following question . Question 1.3 . Is there a differentially private algorithm for learning axis-aligned rectangles with sample complexity that scales linearly in d and asymptotically smaller than log |X| ? We answer this question in the affirmative , and present the following theorem . Theorem 1.4 ( informal ) . There exists a differentially private algorithm for learning axis-aligned rectangles over Xd with sample complexity Õ ( d · ( log∗ |X| ) 1.5 ) . 1.2 Baseline Construction using Composition . Before we present the technical ideas behind our construction ( obtaining sample complexity linear in d ) , we first elaborate on the algorithm obtaining sample complexity Õ ( d1.5 · ( log∗ |X| ) 1.5 ) , which we consider to be the baseline for this work . This baseline algorithm is based on a reduction to ( privately ) solving the following problem , called the interior point problem . Definition 1.5 ( Bun et al . 2015 ) . An algorithm A is said to solve the Interior Point Problem for domain X with failure probability β and sample complexity n , if for every m ≥ n and every database S containing m elements from X it holds that : Pr [ min ( S ) ≤ A ( S ) ≤ max ( S ) ] ≥ 1− β . That is , given a database S containing ( unlabeled ) elements from a ( one dimensional ) grid X , the interior point problem asks for an element of X between the smallest and largest elements in S. The baseline we consider for privately learning axis-aligned rectangles is as follows . Suppose that we have a differentially private algorithm A for the interior point problem over domain X with sample complexity n ( let us ignore the failure probability for simplicity ) . We now use A to construct the following algorithm B that takes a database S containing labeled elements from Xd . For simplicity , we assume that S contains “ enough ” positive elements , as otherwise we could simply return the all-zero hypothesis . 1 . For every axis i ∈ [ d ] : ( a ) Project the positive points in S onto the ith axis . ( b ) Let Ai and Bi denote the smallest n and the largest n ( projected ) points , without their labels . ( c ) Let ai ← A ( Ai ) and bi ← A ( Bi ) . 2 . Return the axis-aligned rectangle defined by the intervals [ ai , bi ] at the different axes . Now , recall that each application of algorithm A returns an interior point of its input points . Hence , for every axis i , it holds that the interval [ ai , bi ] contains ( the projection ) of all but at most 2n of the positive examples in the ith axis . Therefore , the rectangle returned in Step 2 contains all but at most 2nd of the positive points ( and it does not contain any of the negative points , because this rectangle is contained inside the target rectangle ) . So algorithm B errs on at most 2nd of its input points . Assuming that |S| 2nd , we therefore get that algorithm B has small empirical error . As the VC dimension of the class of axis-aligned rectangles isO ( d ) , this means that algorithm B is a PAC learner for this class with sample complexity O ( nd ) . The issue here is that algorithm B executes algorithm A many times ( specifically , 2d times ) . Hence , in order to argue that B is ( ε , δ ) -differentially private , standard composition theorems for differential privacy require each execution of algorithm A to be done with a privacy parameter of ≈ ε/ √ 2d . This , in turn , would mean that n ( the sample complexity of algorithm A ) needs to be at least √ 2d , which means that algorithm B errs on 2nd ≈ d1.5 input points , which translates to sample complexity of |S| d1.5 . The takeaway from this baseline learner is that in order to reduce the sample complexity to be linear in d , we want to bypass the costs incurred from composition . That is , we still want to follow the same strategy ( apply algorithm A twice on every axis ) , but we want to do it without appealing to composition arguments in the privacy analysis . We now briefly survey two intuitive attempts that fail to achieve this , but are useful for the presentation . Failed Attempt # 1 . As before , letA denote an algorithm for the interior point problem over domain X with sample complexity n. Consider the following modification to algorithm B ( marked in red ) . As before , algorithm B takes a database S containing labeled elements from Xd , where we assume for simplicity that S contains “ enough ” positive elements . 1 . For every axis i ∈ [ d ] : ( a ) Project the positive points in S onto the ith axis . ( b ) Let Ai and Bi denote the smallest n and the largest n ( projected ) points , without their labels . ( c ) Let ai ← A ( Ai ) and bi ← A ( Bi ) . ( d ) Delete from S all points ( with their labels ) that correspond to Ai and Bi . 2 . Return the axis-aligned rectangle defined by the intervals [ ai , bi ] at the different axes . The ( incorrect ) idea here is that by adding Step 1d we make sure that each datapoint from S is “ used only once ” , and hence we do not need to pay in composition . In other words , the hope is that if every execution of algorithm A is done with a privacy parameter ε , then the whole construction would satisfy differential privacy with parameter O ( ε ) . The failure point of this idea is that by deleting one point from the data , we can create a “ domino effect ” that effects ( one by one ) many of the sets Ai , Bi throughout the execution . Specifically , consider two neighboring datasets S and S′ = S ∪ { ( x′ , y′ ) } for some labeled point ( x′ , y′ ) ∈ Xd × { 0 , 1 } . Suppose that during the execution on S′ it holds that x′ ∈ A1 . So the additional point x′ participates “ only ” in the first iteration of the algorithm , and gets deleted afterwards . However , since the size of the sets Ai , Bi is fixed , during the execution on S ( without the point x′ ) it holds that a different point z gets included in A1 instead of x′ , and this point z is then deleted from S ( but it is not deleted from S′ during the execution on S′ ) . Therefore , also during the second iteration we have that S and S′ are not identical ( they still differ on one point ) and this domino effect can continue throughout the execution . That is , a single data point can affect many of the executions of A , and we would still need to pay in composition to argue privacy . Failed Attempt # 2 . In order to overcome the previous issue , one might try the following variant of algorithm B . 1 . For every axis i ∈ [ d ] : ( a ) Project the positive points in S onto the ith axis . ( b ) Let sizeAi = 2n+ Noise and let sizeBi = 2n+ Noise . ( c ) Let Ai and Bi denote the smallest sizeAi and the largest sizeBi ( projected ) points , respectively , without their labels . ( d ) Let ai ← A ( Ai ) and bi ← A ( Bi ) . ( e ) Delete from S all points ( with their labels ) that correspond to Ai and Bi . 2 . Return the axis-aligned rectangle defined by the intersection of the intervals [ ai , bi ] at the different axes . The idea now is that the noises we add to the sizes of the Ai ’ s and the Bi ’ s would “ mask ” the domino effect mentioned above . Specifically , the hope is as follows . Consider the execution of ( the modified ) algorithm B on S and on S′ = S ∪ { ( x′ , y′ ) } , and let i be the first axis such that x′ ∈ Ai ∪Bi during the execution on S′ . Suppose w.l.o.g . that x′ ∈ Bi . Now , the hope is that if during the execution on S we have that the noisy sizeBi is smaller by 1 than its value during the execution on S ′ , then this eliminates the domino effect we mentioned , because we would not need to add another point instead of x′ . Specifically , during time i , the point x′ gets deleted from S′ , and every other point is either deleted from both S , S′ or not deleted from any of them . So after time i the two executions continue identically . Thus , the hope is that by correctly “ synchronizing ” the noises between the two executions ( such that only the size of the “ correct ” set gets modified by 1 ) we can make sure that only one application of A is effected ( in the last example – only the execution of A ( Bi ) is effected ) , and so we would not need to apply composition arguments . Although very convincing , this idea fails . The ( very subtle ) issue here is that it is not clear how to synchronize the noises between the two executions . To see the problem , let us try to formalize the above argument . Fix two neighboring databases S and S′ = S ∪ { ( x′ , y′ ) } . Let us write Ai , Bi and A′i , B′i to denote these sets during the executions on S and on S′ , respectively . Aiming to synchronize the two executions , let us define a mapping π : R2d → R2d from noise vectors during the execution on S′ to noise vectors during the execution on S ( determining the values of sizeA1 , sizeB1 , . . . , sizeAd , sizeBd ) , such that throughout the execution we have that Ai = A′i and Bi = B ′ i for all i except for a single pair , say Bj 6= B′j , of neighboring sets . The straightforward way for defining such a mapping is as follows : Let j be the first time step in which the additional point x′ gets included in a set A′j or B ′ j , and say that it is included in B ′ j . Then the mapping would be to reduce ( by 1 ) the value of sizeBj ( the noisy size of Bj during the execution on S ) . This would indeed make sure that , conditioned on the noise vectors v′ and v = π ( v′ ) , the two executions differ only in a single application of the interior point algorithmA , and hence the outcome distribution of these two ( conditioned ) executions are very similar ( in the sense of differential privacy ) . That is , for any noise vector v and any event F , Pr [ B ( S′ ) ∈ F |v ] ≤ eε · Pr [ B ( S ) ∈ F |π ( v ) ] + δ . Furthermore , ( assuming an appropriate noise distribution ) we can make sure that the probability of obtaining the noise vectors v and π ( v ) are similar , with densities differing by at most an eε factor ( as is standard in the literature of differential privacy ) . Therefore , had the mapping π we defined was a bijection , for any event F we would have that Pr [ B ( S′ ) ∈ F ] = ∑ v Pr [ v ] · Pr [ B ( S′ ) ∈ F |v ] ≤ ∑ v eε · Pr [ π ( v ) ] · ( eε · Pr [ B ( S ) ∈ F |π ( v ) ] + δ ) = ∑ π ( v ) eε · Pr [ π ( v ) ] · ( eε · Pr [ B ( S ) ∈ F |π ( v ) ] + δ ) = e2ε · Pr [ B ( S ) ∈ F ] + eε · δ , which would be great . Unfortunately , the mapping π we defined is not a bijection , and hence the second-to-last equality above is incorrect . To see that it is not a bijection , suppose that d = 2 and consider a database S containing the following positively labeled points : Many copies of the point ( 0 , 0 ) , as well as 10 copies of the point ( 1 , 0 ) and 10 copies of the point ( 0 , 1 ) . The neighboring database S′ contains , in addition to all these points , also the point ( 1 2 , 1 2 ) . Now suppose that during the execution on S′ we have that |B′1| = 5 and |B′2| = 4 . That is , the additional point is included in B′1 . During the execution on S we therefore reduce ( by 1 ) the size of B1 and so |B1| = |B2| = 4 . Now suppose that during the execution on S′ we have that |B′1| = 4 and |B′2| = 5 . Here , during the execution on S we reduce the size of B2 and so , again , |B1| = |B2| = 4 . This shows that the mapping π we defined is not a bijection . In general , in d dimensions , it is only a d-to-1 mapping , which would would break our analysis completely ( it will not allow us to avoid the extra factor in d ) .

Summary: This paper studies the problem of PAC learning an axis-aligned rectangle in d-dimensions under the constraint of differential privacy (DP). The class of d-dimensional axis-aligned rectangles can be viewed as a generalization of threshold functions. Previous work have shown that the problem of privately learning thresholds is equivalent (up to constants) to the problem of privately selecting the interior point of a set of totally ordered points. A naive approach to extend this result to high-dimensional rectangles is to compute the threshold in each dimension separately and then use advanced composition. Unfortunately the dependence of the samples complexity of this approach on the dimension is $d^{1.5}$ (rather than $d$). Therefore, this problem captures one of the main technical challenges of designing DP methods, namely handling high-dimensional data.   The authors introduce a carefully constructed algorithm that applies the private interior point algorithm on each dimension, but at each step prunes some points in the dataset so that they are not used in future steps. Intuitively, this approach allows them to overcome the expensive advanced composition since not too many points will be reused. [and as a consequence, the resulting method has only linear dependence on $d$]