Privately Learning Mixtures of Axis-Aligned Gaussians

1 Introduction . The fundamental problem of distribution learning concerns the design of algorithms ( i.e. , estimators ) that , given samples generated from an unknown distribution f , output an “ approximation ” of f . While distribution learning has a long history [ 54 ] , studying it under privacy constraints is relatively new and unexplored . In this paper , we work with a rigorous and practical notion of data privacy known as differential privacy [ 36 ] . Roughly speaking , differential privacy guarantees that no single data point can influence the output of an algorithm too much . Intuitively , this provides privacy by “ hiding ” the contribution of each individual . Differential privacy is the de facto standard for modern private analysis and has seen widespread impact in both industry and government [ 12 , 22 , 28 , 29 , 39 ] . In recent years , there has been a flurry of activity in differentially private distribution learning and a number of techniques have been developed in the literature . In the pure differentially private setting , Bun et al . [ 17 ] introduced a method to learn classes of distributions that admit a finite cover , i.e . when the class of distributions is well-approximated by a finite number of distributions . They show that this is an exact characterization of distributions which can be learned under pure differential privacy in the sense that a class of distributions is learnable under pure differential privacy if and only if the class 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . admits a finite cover [ 17 , 41 ] . As a consequence of this result , they obtained pure differentially private algorithms for learning Gaussian distributions provided that the mean of the Gaussians are bounded and the covariance matrix of the Gaussians are spectrally bounded.1 Moreover , such restrictions on the Gaussians are necessary under the constraint of pure differential privacy . One way to remove the requirement of having a finite cover is to relax to a weaker notion of privacy known as approximate differential privacy . With this notion , Bun et al . [ 17 ] introduced a method to learn a class of distributions that , instead of requiring a finite cover , requires a “ locally small ” cover , i.e . a cover where each distribution in the class is well-approximated by only a small number of elements within the cover . They prove that the class of Gaussians with arbitrary mean and a fixed , known covariance matrix has a locally small cover which implies an approximate differentially private algorithm to learn this class of distributions . Later , Aden-Ali , Ashtiani , and Kamath [ 4 ] proved that the class of mean-zero Gaussians ( with no assumptions on the covariance matrix ) admits a locally small cover leading to an approximate differentially private method to learn the class of all Gaussians . It is a straightforward observation that if a class of distributions admits a finite cover then the class of its mixtures also admits a finite cover . Combined with the aforementioned work of Bun et al. , this implies a pure differentially private algorithm for learning mixtures of Gaussians with bounded mean and spectrally bounded covariance matrices . It is natural to wonder whether an analogous statement holds for locally small covers . In other words , if a class of distributions admits a locally small cover then does the class of mixtures also admit a locally small cover ? If so , this would provide a fruitful direction to design differentially private algorithms for learning mixtures of arbitrary Gaussians . Unfortunately , there are simple examples of classes of distributions that admit a locally small cover yet their mixture do not . This leaves open the question of designing private algorithms for many classes of distributions that are learnable in the non-private setting . One concrete open problem is for the class of mixtures of two arbitrary univariate Gaussian distributions . A more general problem is private learning of mixtures of k axis-aligned ( or general ) Gaussian distributions . 1.1 Main Results . We demonstrate that it is indeed possible to privately learn mixtures of unbounded univariate Gaussians . More generally , we give sample complexity upper bounds for learning mixtures of unbounded d-dimensional axis-aligned Gaussians . In the following theorem and the rest of the paper we use Õ to hide polylogarithmic factors , i.e . Õ ( f ( x ) ) means O ( f ( x ) logc f ( x ) ) for some c > 0 . Theorem 1.1 ( Informal ) . The sample complexity of learning a mixture of k d-dimensional axisaligned Gaussians to α-accuracy in total variation distance under ( ε , δ ) -differential privacy is Õ ( k2d log3/2 ( 1/δ ) α2ε ) . The formal statement of this theorem can be found in Theorem 5.1 . For technical reasons , we do require that δ ∈ ( 0 , 1/n ) for the above theorem to hold . This condition is quite standard in the differential privacy literature . Indeed , for useful privacy , δ should be “ cryptographically small ” , i.e. , δ 1/n . Even for the univariate case , our result is the first sample complexity upper bound for learning mixture of Gaussians under differential privacy for which the variances are unknown and the parameters of the Gaussians may be unbounded . In the non-private setting , it is known that Θ̃ ( kd/α2 ) samples are necessary and sufficient to learn a mixture of k axis-aligned Gaussian in Rd [ 6 , 61 ] . In the private setting , the best known sample complexity lower bound is Ω ( d/αε log ( d ) ) under ( ε , δ ) -DP when δ ≤ Õ ( √ d/n ) [ 46 ] . Obtaining improved upper or lower bounds in this setting remains an open question . If the covariance matrix of each component of the mixture is the same and known or , without loss of generality , equal to the identity matrix , then we can improve the dependence on the parameters and obtain a result that is in line with the non-private setting . 1When we say that a matrix Σ is spectrally bounded , we mean that there are 0 < a1 ≤ a2 such that a1 · I Σ a2 · I . Theorem 1.2 ( Informal ) . The sample complexity of learning a mixture of k d-dimensional Gaussians with identity covariance matrix to α-accuracy in total variation distance under ( ε , δ ) -differential privacy is Õ ( kd α2 + kd log ( 1/δ ) αε ) . We relegate the formal statement and the proof of this theorem to the supplementary materials ( see Appendix F ) . Note that the work of [ 53 ] implies an upper bound of O ( k2d3 log2 ( 1/δ ) /α2ε2 ) for private learning of the same class albeit in the incomparable setting of parameter estimation . Comparison with locally small covers . While the results in [ 4 , 17 ] for learning Gaussian distributions under approximate differential privacy do not yield finite-time algorithms , they do give strong information-theoretic upper bounds . This is achieved by showing that certain classes of Gaussians admit locally small covers . It is thus natural to ask if we can obtain sharper results by showing that mixtures of Gaussians also admit locally small covers . Unfortunately , the following simple example shows that not even mixtures of two univariate Gaussians admit locally small covers . Proposition 1.3 ( Informal version of Proposition B.6 ) . Every cover for the class of mixtures of two univariate Gaussians is not locally small . 1.2 Techniques . To prove our result , we devise a novel technique which reduces the problem of privately learning mixture distributions to the problem of private list-decodable learning of distributions . The framework of list-decodable learning was introduced by Balcan , Blum , and Vempala [ 8 ] and Balcan , Röglin , and Teng [ 9 ] in the context of clustering but has since been studied extensively in the literature in a number of different contexts [ 7 , 20 , 21 , 26 , 27 , 49 , 55 , 56 ] . The problem of list-decodable learning of distributions is as follows . There is a distribution f of interest that we are aiming to learn . However , we do not receive samples from f ; rather we receive samples from a corrupted distribution g = ( 1 − γ ) f + γh where γ ∈ ( 0 , 1 ) and h is some arbitrary distribution . In our application , γ is close to 1 , i.e . most of the samples are corrupted . The goal in list-decodable learning is to output a short list of distributions f1 , . . . , fm with the requirement that f is close to at least one of the fi ’ s . The formal definition of list-decodable learning can be found in Definition 2.6 . Informally , the reduction can be summarized by the following theorem which is formalized in Section 3 . Theorem 1.4 ( Informal ) . If a class of distributions F is privately list-decodable then mixtures of distributions from F are privately learnable . Roughly speaking , the reduction from learning mixtures of distribution to list-decodable learning works as follows . Suppose that there is an unknown distribution f which is a mixture of k distributions f1 , . . . , fk . A list-decodable learner would then receive samples from f as input and output a short list of distributions F̂ so that for every fi there is some element in F̂ that is close to fi . In particular , some mixture of distributions from F̂ must be close to the true distribution f . Since F̂ is a small finite set , the set of possible mixtures must also be relatively small . This last observation allows us to make use of private hypothesis selection which selects a good hypothesis from a small set of candidate hypotheses [ 4 , 17 ] . In Section 3 , we describe the aforementioned reduction in more detail . We note that a similar connection between list-decodable learning and learning mixture distributions was also used by Diakonikolas et al . [ 26 ] . However , our reduction is focused on the private setting . The reduction shows that to privately learn mixtures , it is sufficient to design differentially private list-decodable learning algorithms that work for ( corrupted versions of ) the individual mixture components . To devise list-decodable learners for ( corrupted ) univariate Gaussian , we utilize “ stability-based ” histograms [ 15 , 51 ] that satisfy approximate differential privacy . To design a list-decodable learner for corrupted univariate Gaussians , we follow a three-step approach that is inspired by the seminal work of Karwa and Vadhan [ 50 ] . First , we use a histogram to output a list of variances one of which approximates the true variance of the Gaussian . As a second step , we would like to output a list of means which approximate the true mean of the Gaussian . This can be done using histograms provided that we roughly know the variance of the Gaussian . Since we have candidate variances from the first step , we can use a sequence of histograms where the width of the bins of each of the histograms is determined by the candidate variances from the first step . As a last step , using the candidate variances and means from the first two steps , we are able to construct a small set of distributions one of which approximates the true Gaussian to within accuracy α . In the axis-aligned Gaussians setting , we use our solution for the univariate case as a subroutine on each dimension separately . Now that we have a list-decodable learner for axis-aligned Gaussians , we use our reduction to obtain a private learning algorithm for learning mixtures of axis-aligned Gaussians . Approaches based on constructing lists of candidate variances and means in order to learn an accurate mixture have been previously considered in the non-private setting [ 5 , 6 , 23 , 61 ] . However , it does not seem possible to directly privatize the algorithms in this line of work since they construct these candidates directly from the samples , which is a clear violation of privacy .

Summary: This paper considered the problem of learning a mixture of gaussians under DP constraint. The authors proposed an (exponential time) algorithm for learning univariate k-mixture of gaussian distributions with sample complexity k^2log^(3/2)(1/\delta)/alpha^2eps. They conjectured the extra k factor in the sample complexity is information theoretically unnecessary. The key of the algorithm is a reduction from the list-decodable learning to learning mixture distributions. Specifically, given a list decodable algorithm, one can enumerate all possible combinations of the k-mixtures and use a private hypothesis selection algorithm to find the most plausible k-mixture. They proposed a list decodable algorithm for learning gaussian distribution, which adopts a similar approach as [48].   For the multivariate gaussian setting, they considered the axis-aligned gaussian setting. They first obtain a list of distribution on each dimension, and take all possible combinations of the univariate distribution to form a large list of axis aligned gaussians. This leads to a multivariate gaussian k-mixture learning algorithm with sample complexity k^2dlog^(3/2)(1/delta)/alpha^2eps.