Iterative Methods for Private Synthetic Data: Unifying Framework and New Methods

1 Introduction . As the collection and analyses of sensitive data become more prevalent , there is an increasing need to protect individuals ’ private information . Differential privacy [ 15 ] is a rigorous and meaningful criterion for privacy preservation that enables quantifiable trade-offs between privacy and accuracy . In recent years , there has been a wave of practical deployments of differential privacy across organizations such as Google , Apple , and most notably , the U.S. Census Bureau [ 3 ] . In this paper , we study the problem of differentially private query release : given a large collection of statistical queries , the goal is to release approximate answers subject to the constraint of differential privacy . Query release has been one of the most fundamental and practically relevant problems in differential privacy . For example , the release of summary data from the 2020 U.S. Decennial Census can be framed as a query release problem . We focus on the approach of synthetic data generation—that is , generate a privacy-preserving `` fake '' dataset , or more generally a representation of a probability distribution , that approximates all statistical queries of interest . Compared to simple Gaussian or Laplace mechanisms that perturb the answers directly , synthetic data methods can provably answer an exponentially larger collection of queries with non-trivial accuracy . However , their statistical advantage also comes with a computational cost . Prior work has shown that achieving better accuracy than simple Gaussian perturbation is intractable in the worst case even for the simple query class of 2-way marginals that release the marginal distributions for all pairs of attributes [ 38 ] . ∗First two authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Despite its worst-case intractability , there has been a recent surge of work on practical algorithms for generating private synthetic data . Even though they differ substantially in details , these algorithms share the same iterative form that maintains and improves a probability distribution over the data domain : identifying a small collection of high-error queries each round and updating the distribution to reduce these errors . Inspired by this observation , we present a unifying algorithmic framework that captures these methods . Furthermore , we develop two new algorithms , GEM and PEP , and extend the former to the setting in which public data is available . We summarize our contributions below : Unifying algorithmic framework . We provide a framework that captures existing iterative algorithms and their variations . At a high level , algorithms under this framework maintain a probability distribution over the data domain and improve it over rounds by optimizing a given loss function . We therefore argue that under this framework , the optimization procedures of each method can be reduced to what loss function is minimized and how its distributional family is parameterized . For example , we can recover existing methods by specifying choices of loss functions—we rederive MWEM [ 21 ] using an entropy-regularized linear loss , FEM [ 40 ] using a linear loss with a linear perturbation , and DualQuery [ 19 ] with a simple linear loss . Lastly , our framework lends itself naturally to a softmax variant of RAP [ 5 ] , which we show outperforms RAP itself.2 Generative networks with the exponential mechanism ( GEM ) . GEM is inspired by MWEM , which attains worst-case theoretical guarantees that are nearly information-theoretically optimal [ 11 ] . However , MWEM maintains a joint distribution over the data domain , resulting in a runtime that is exponential in the dimension of the data . GEM avoids this fundamental issue by optimizing the absolute loss over a set of generative models parameterized by neural networks . We empirically demonstrate that in the high-dimensional regime , GEM outperforms all competing methods . Private Entropy Projection ( PEP ) . The second algorithm we propose is PEP , which can be viewed as a more advanced version of MWEM with an adaptive and optimized learning rate . We show that PEP minimizes a regularized exponential loss function that can be efficiently optimized using an iterative procedure . Moreover , we show that PEP monotonically decreases the error over rounds and empirically find that it achieves higher accuracy and faster convergence than MWEM . Incorporating public data . Finally , we consider extensions of our methods that incorporate prior information in publicly available datasets ( e.g. , previous data releases from the American Community Survey ( ACS ) prior to their differential privacy deployment ) . While Liu et al . [ 26 ] has established PMWPub as a state-of-the-art method for incorporating public data into private query release , we discuss how limitations of their algorithm prevent PMWPub from effectively using certain public datasets . We then demonstrate empirically that GEM circumvents such issues via simple pretraining , achieving max errors on 2018 ACS data for Pennsylvania ( at ε = 1 ) 9.23x lower than PMWPub when using 2018 ACS data for California as the public dataset . 1.1 Related work . Beginning with the seminal work of Blum et al . [ 9 ] , a long line of theoretical work has studied private synthetic data for query release [ 33 , 21 , 22 , 20 ] . While this body of work establishes optimal statistical rates for this problem , their proposed algorithms , including MWEM [ 22 ] , typically have running time exponential in the dimension of the data . While the worst-case exponential running time is necessary ( given known lower bounds [ 16 , 36 , 39 ] ) , a recent line of work on practical algorithms leverage optimization heuristics to tackle such computational bottlenecks [ 19 , 40 , 5 ] . In particular , DualQuery [ 19 ] and FEM [ 40 ] leverage integer program solvers to solve their NP-hard subroutines , and RAP [ 5 ] uses gradient-based methods to solve its projection step . In Section 3 , we demonstrate how these algorithms can be viewed as special cases of our algorithmic framework . Our work also relates to a growing line of work that use public data for private data analyses [ 6 , 4 , 7 ] . For query release , our algorithm , GEM , improves upon the state-of-the-art method , PMWPub [ 26 ] , which is more limited in the range of public datasets it can utilize . Finally , our method GEM is related to a line 2We note that Aydore et al . [ 5 ] have since updated the original version ( https : //arxiv.org/pdf/2103 . 06641v1.pdf ) of their work to include a modified version of RAP that leverages SparseMax [ 27 ] , similar to way in which the softmax function is applied in our proposed baseline , RAPsoftmax . of work on differentially private GANs [ 8 , 44 , 30 , 42 ] . However , these methods focus on generating synthetic data for simple downstream machine learning tasks rather than for query release . Beyond synthetic data , a line of work on query release studies `` data-independent '' mechanisms ( a term formulated in Edmonds et al . [ 17 ] ) that perturb the query answers with noise drawn from a data-independent distribution ( that may depend on the query class ) . This class of algorithms includes the matrix mechanism [ 25 ] , the high-dimensional matrix mechanism ( HDMM ) [ 28 ] , the projection mechanism [ 31 ] , and more generally the class of factorization mechanisms [ 17 ] . In addition , McKenna et al . [ 29 ] provide an algorithm that can further reduce query error by learning a probabilistic graphical model based on the noisy query answers released by privacy mechanisms . 2 Preliminaries . Let X denote a finite d-dimensional data domain ( e.g. , X = { 0 , 1 } d ) . Lete U be the uniform distribution over the domain X . Throughout this work , we assume a private dataset P that contains the data of n individuals . For any x ∈ X , we represent P ( x ) as the normalized frequency of x in dataset P such that ∑ x∈X P ( x ) = 1 . One can think of a dataset P either as a multi-set of items from X or as a distribution over X . We consider the problem of accurately answering an extensive collection of linear statistical queries ( also known as counting queries ) about a dataset . Given a finite set of queries Q , our goal is to find a synthetic dataset D such that the maximum error over all queries in Q , defined as maxq∈Q |q ( P ) − q ( D ) | , is as small as possible . For example , one may query a dataset by asking the following : how many people in a dataset have brown eyes ? More formally , a statistical linear query qφ is defined by a predicate function φ : X → { 0 , 1 } , as qφ ( D ) = ∑ x∈X φ ( x ) D ( x ) for any normalized dataset D. Below , we define an important , and general class of linear statistical queries called k-way marginals . Definition 1 ( k-way marginal ) . Let the data universe with d categorical attributes be X = ( X1 × . . .×Xd ) , where each Xi is the discrete domain of the ith attribute Ai . A k-way marginal query is defined by a subset S ⊆ [ d ] of k features ( i.e. , |S| = k ) plus a target value y ∈ ∏ i∈S Xi for each feature in S. Then the marginal query φS , y ( x ) is given by : φS , y ( x ) = ∏ i∈S 1 ( xi = yi ) where xi ∈ Xi means the i-th attribute of record x ∈ X . Each marginal has a total of ∏k i=1 |Xi| queries , and we define a workload as a set of marginal queries . We consider algorithms that input a dataset P and produce randomized outputs that depend on the data . The output of a randomized mechanismM : X ∗ → R is a privacy preserving computation if it satisfies differential privacy ( DP ) [ 15 ] . We say that two datasets are neighboring if they differ in at most the data of one individual . Definition 2 ( Differential privacy [ 15 ] ) . A randomized mechanism M : Xn → R is ( ε , δ ) differentially privacy , if for all neighboring datasets P , P ′ ( i.e. , differing on a single person ) , and all measurable subsets S ⊆ R we have : Pr [ M ( P ) ∈ S ] ≤ eεPr [ M ( P ′ ) ∈ S ] + δ Finally , a related notion of privacy is called concentrated differential privacy ( zCDP ) [ 14 , 10 ] , which enables cleaner composition analyses for privacy . Definition 3 ( Concentrated DP , Dwork and Rothblum [ 14 ] , Bun and Steinke [ 10 ] ) . A randomized mechanismM : Xn → R is 12 ε̃ 2-CDP , if for all neighboring datasets P , P ′ ( i.e. , differing on a single person ) , and for all α ∈ ( 1 , ∞ ) , Rα ( M ( P ) ‖ M ( P ′ ) ) ≤ 1 2 ε̃2α where Rα ( M ( P ) ‖ M ( P ′ ) ) is the Rényi divergence between the distributionsM ( P ) andM ( P ′ ) .

Summary: This paper tackles the problem of synthetic data generation for query release. First, it presents a general algorithmic framework (Adaptive measurements) for the task, and interprets a long line of prior work under this lense. At a high level, these works solve the min max problem in Equation 1.   Given this formulation, the paper proposes two algorithms, PEP and GEM. PEP selects a maximum entropy distribution to minimize the error for a given query set by solving the dual of the constrained optimization problem. They offer an iterative algorithm for this.  To avoid the exponential runtime of approaches like PEP (because they maintain a distribution over entire data domains), the paper also proposes GEM, which uses a generator network to represent this high dimensional space.  GEM can easily incorporate public data (with various missingness) by using it for pretraining.   Empirically, GEM outperforms prior work in accuracy to privacy tradeoffs on ACS and the adult datasets, and is more robust to domain shift in the public data than PMW or PEP. This gain is most pronounced when the data is high dimensional.