Relaxed Marginal Consistency for Differentially Private Query Answering

1 Introduction . A central problem in the design of differentially private algorithms is answering sets of counting queries from a database . Many proposed algorithms follow the select-measure-reconstruct paradigm : they select a set of measurement queries , they privately measure them ( using Gaussian or Laplace noise addition ) , and then they reconstruct the data or query answers from the noisy measurements . When done in a principled manner , the reconstruct phase serves a number of critical functions : it combines the noisy evidence provided by the measurement queries , it allows new unmeasured queries to be answered ( with no additional privacy cost ) , and it resolves inconsistencies in the noisy measurements to produce consistent estimates , which often have lower error . In this paper , we propose a novel , scalable , and general-purpose approach to the reconstruct step . With a principled approach to this problem , future research can focus on the challenging open problem of query selection . Most existing general-purpose methods for reconstruction can not scale to high-dimensional data , as they operate over a vectorized representation of the data , whose size is exponential in the dimensionality [ 1–6 ] . Some special purpose methods exist that have better scalability , but are only applicable within a particular mechanism or in certain special cases [ 7–11 , 5 , 12–15 ] . A recently-proposed method , PRIVATE-PGM [ 16 ] , offers the scalability of these special purpose methods and retains much of the generality of the general-purpose methods . PRIVATE-PGM can be used for the reconstruction phase whenever the measurements only depend on the data through its low-dimensional marginals . PRIVATE-PGM avoids the data vector representation in favor of a more compact graphical model representation , and was shown to dramatically improve the scalability of a number of popular mechanisms while also improving accuracy [ 16 ] . PRIVATE-PGM was used in the winning entry of the 2018 NIST differential privacy synthetic data contest [ 17 , 18 ] , as well as in both the first and second-place entry of the follow-up 2020 NIST differential privacy temporal map contest [ 19 , 20 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . While PRIVATE-PGM is far more scalable than operating over a vector representation of the data , it is still limited . In particular , its required memory and runtime depend on the structure of the underlying graphical model , which in turn is determined by which marginals the mechanism depends on . When the mechanism depends on a modest number of carefully chosen marginals , PRIVATE-PGM is extremely efficient . But , as the number of required marginals increases , the underlying graphical model becomes intractably large , and PRIVATE-PGM eventually fails to run . This is due to the inherent hardness of exact marginal inference in a graphical model . In this paper , we overcome the scalability limitations of PRIVATE-PGM by proposing a natural relaxation of the estimation objective that enforces specified local consistency constraints among marginals , instead of global ones , and can be solved efficiently . Our technical contributions may be of broader interest . We develop an efficient algorithm to solve a generic convex optimization problem over the local polytope of a graphical model , which uses a body of prior work on generalized belief propagation [ 21–31 ] and can scale to problems with millions of optimization variables . We also propose a variational approach to predict “ out-of-model ” marginals given estimated pseudo-marginals , which gives a completely variational formulation for both estimation and inference : the results are invariant to optimization details , including the approximate inference methods used as subroutines . Our new approach , APPROX-PRIVATE-PGM ( APPGM ) , offers many of the same benefits as PRIVATE-PGM , but can be deployed in far more settings , allowing effective reconstruction to be performed without imposing strict constraints on the selected measurements . We show that APPGM permits efficient reconstruction for HDMM [ 32 ] , while also improving its accuracy , allows MWEM [ 5 ] to scale to far more measurements , and improves the accuracy of FEM [ 33 ] . 2 Background . We first review background on our data model , marginals , and differential privacy , following [ 16 ] . Data Our input data represents a population of individuals , each contributing a single record x = ( x1 , . . . , xd ) where xi is the ith attribute belonging to a discrete finite domain Ωi of ni possible values . The full domain is Ω = �d i=1 Ωi and its size n = �d i=1 ni is exponential in the number of attributes . A dataset X consists of m such records X = ( x ( 1 ) , . . . , x ( m ) ) . It is often convenient to work with an alternate representation of X : the data vector or data distribution p is a vector of length n , indexed by x ∈ Ω such that p ( x ) counts the fraction of individuals with record equal to x . That is , p ( x ) = 1m �m i=1 I { x ( i ) = x } , ∀x ∈ Ω , where I { · } is an indicator function . Marginals When dealing with high-dimensional data , it is common to work with marginals defined over a subset of attributes . Let r ⊆ [ d ] be a region or clique that identifies a subset of attributes and , for x ∈ Ω , let xr = ( xi ) i∈r be the sub-vector of x restricted to r. Then the marginal vector ( or simply “ marginal on r ” ) µr , is defined by : µr ( xr ) = 1 m m� i=1 I { x ( i ) r = xr } , ∀xr ∈ Ωr : = � i∈r Ωi . ( 1 ) This marginal is the data vector on the sub-domain Ωr corresponding to the attribute set r. Its size is nr : = |Ωr| = � i∈r ni , which is exponential in |r| but may be considerably smaller than n. A marginal on r can be computed from the full data vector or the marginal for any superset of attributes by summing over variables that are not in r. We denote these ( linear ) operations by Mr and Ps→r , so µr = Mrp = Ps→rµs for any r ⊆ s. We will also consider vectors µ that combine marginals for each region in a collection C , and let MC be the linear operator such that µ = ( µr ) r∈C = MCp . Differential Privacy Differential privacy protects individuals by bounding the impact any one individual can have on the output of an algorithm . Definition 1 ( Differential Privacy [ 34 ] ) . A randomized algorithm A satisfies ( � , δ ) -differential privacy if , for any input X , any X� ∈ nbrs ( X ) , and any subset of outputs S ⊆ Range ( A ) , Pr [ A ( X ) ∈ S ] ≤ exp ( � ) Pr [ A ( X� ) ∈ S ] + δ Above , nbrs ( X ) denotes the set of datasets formed by replacing any x ( i ) ∈ X with an arbitrary new record x� ( i ) ∈ Ω . When δ = 0 we say A satisfies �-differential privacy . 3 Private-PGM . In this section we describe PRIVATE-PGM [ 16 ] , a general-purpose reconstruction method applied to differentially private measurements of a discrete dataset . There are two steps to PRIVATE-PGM : ( 1 ) estimate a representation of the data distribution given noisy measurements , and ( 2 ) infer answers to new queries given the data distribution representation . In particular , suppose an arbitrary ( � , δ ) -differentially private algorithm A is run on a discrete dataset with data vector p0 , where A only depends on p0 through its low-dimensional marginals µ0 = MCp0 for some collection of cliques C. The sample y ∼ A ( µ0 ) typically reveals noisy high-level aggregate information about the data . PRIVATE-PGM will first estimate a compact representation of a distribution p̂ that explains y well , and then answer new queries using p̂ . Estimation : Finding a Data Distribution Representation PRIVATE-PGM first estimates a data vector by finding p̂ to solve the inverse problem minp L ( MCp ) , where L ( µ ) is a convex loss function that measures how well µ explains the observations y . Since L ( MCp ) only depends on p through its marginals , it is clear we can find the optimal marginals by instead solving the following problem . Problem 1 ( Convex Optimization over the Marginal Polytope ) . Given a clique set C and convex loss function L ( µ ) , solve µ̂ ∈ argmin µ∈M ( C ) L ( µ ) , where M ( C ) = { µ : ∃ p s.t . MCp = µ } is the set of realizable marginals , known as the marginal polytope of C [ 35 ] . The solution to this problem gives marginals that are consistent with some underlying data vector and , therefore , typically provide a better estimate of the true marginals than y . In the general case , the loss function L can simply be set to the negative log likelihood , i.e. , L ( µ ) = − log Pr [ A ( µ ) = y ] ,1 however other choices are also possible . As a concrete motivating application , consider the case where the mechanism A adds Gaussian noise directly to the data marginals µ0 , i.e. , A ( µ0 ) = y where yr = µ0 , r + N ( 0 , σ2Inr ) . In this case , the log-likelihood is proportional to the squared Euclidean distance and gives the loss function L ( µ ) = �µ− y�22 , so the problem at hand is an L2 minimization problem . The theory for PRIVATE-PGM focuses on convex loss functions , but the algorithms are also used to seek local minima of Problem 1 when L is non-convex . Algorithm 1 PROX-PGM [ 16 ] Input : Convex loss function L ( µ ) Output : Marginals µ̂ , parameters θ̂ θ̂ = 0 for t = 1 , . . . , T do µ̂ =MARGINAL-ORACLE ( θ̂ ) θ̂ = θ̂ − ηt∇L ( µ̂ ) return µ̂ , θ̂ Graphical models Two remaining issues are how to solve Problem 1 and how to recover a full data vector from µ̂ . PRIVATE-PGM addresses both with graphical models . A graphical model with clique set C is a distribution over Ω where the unnormalized probability is a product of factors involving only subsets of variables , one for each clique in C. It has the form pθ ( x ) = 1 Z exp �� r∈C θr ( xr ) � . The real numbers θr ( xr ) are log-potentials or parameters . The full parameter vector θ = ( θr ( xr ) ) r∈C , xr∈Ωr matches the marginal vector µ in size and indexing , and the relationship between these two vectors is central to graphical models [ 35 ] : • A parameter vector θ determines a unique marginal vector µθ ∈ M ( C ) , defined as µθ = MCpθ , the marginals of pθ . Marginal inference is the problem of ( efficiently ) computing µθ from θ . It can be solved exactly by algorithms such as variable elimination or belief propagation with a junction tree [ 36 ] . We denote by MARGINAL-ORACLE an algorithm that outputs µθ on input θ . • For every µ ∈ M ( C ) with positive entries , there is a unique distribution pθ in the family of graphical models with cliques C that has marginals µ , and pθ has maximum entropy among all distributions with marginals µ . 1For mechanisms with continuous output values , interpret this as a negative log-density . PROX-PGM ( Algorithm 1 ) is a proximal algorithm to solve Problem 1 [ 16 ] . It returns a marginal vector µ̂ ∈ M ( C ) that minimizes L ( µ ) and parameters θ̂ such that pθ̂ has marginals µ̂ . The core of the computation is repeated calls to MARGINAL-ORACLE . The estimated graphical model pθ̂ has cliques C that coincide with the marginals measured by the privacy mechanism , and has maximum entropy among all distributions whose marginals minimize L ( µ ) . In general , there will be infinitely many distributions that have marginals µ̂ ( and hence are equally good from the perspective of the loss function L ) . PRIVATE-PGM chooses the distribution with maximum entropy , which is an appealing way to break ties that falls out naturally from the graphical model . Inference : Answering New Queries With pθ̂ in hand , PRIVATE-PGM can readily estimate new marginals µr . There are two separate cases . If r is contained in a clique of C , we say r is “ in-model ” , and we can readily calculate µr from the output of PROX-PGM . The more interesting case occurs when r is out-of-model ( is not contained in a clique of C ) : in this case , the standard way to compute µr is to perform variable elimination in the graphical model pθ . Remark 1 . The complexity of PRIVATE-PGM depends on that of MARGINAL-ORACLE , which depends critically on the structure of the cliques C measured by the privacy mechanism . In general , running time is exponential in the treewidth of the graph G induced by attribute co-occurrence within a clique of C. When G is tree-like , PRIVATE-PGM can be highly efficient and exponentially faster than working with a full data vector . When G is dense , PRIVATE-PGM may fail to run due to time or memory constraints . This limitation is not specific to the PROX-PGM algorithm : Problem 1 is as hard as marginal inference , which can be solved by minimizing the ( convex ) variational free energy over the marginal polytope [ 36 ] . A primary difficulty is the intractability of M ( C ) , which is a convex set , but in general requires a very large number of constraints to represent explicitly [ 35 ] . In two state-of-the-art mechanisms for synthetic data , MST [ 18 ] and PrivMRF [ 20 ] , C was specifically chosen to limit the treewidth and ensure tractability of Private-PGM . In other mechanisms agnostic to the limitations of PRIVATE-PGM , like HDMM and MWEM , the set C can often lead to graphs with intractable treewidths .

Summary: For the problem of differentially privacy query answering, a standard paradigm is select-measure-reconstruct. In most methods, the reconstruction step needs to represent the data in vector form, which is impossible when the data is high dimensional. Private-PGM is an approach that uses a graphical model to represent the data more compactly, and it has been shown to improve the scalability of many popular algorithms while decreasing error. However, Private-PGM works best when the graphical model is sparse, and it becomes very inefficient when the graph is dense, which is due to the hardness of exact marginal inference. Therefore, Private-PGM has limited applications for real-world settings. The authors propose a new algorithm that relaxed the marginal inference step to allow Private-PGM to be adapted to more settings. To prove that their method is more scalable, they used the FIRE dataset, which has 15 attributes and 300K rows.