Title: An Optimal and Scalable Matrix Mechanism for Noisy Marginals under Convex Loss Functions
Venue: NeurIPS 2023
Accepted: yes
Keywords: differential privacy

marginals

matrix mechanism

scalability
Abstract: Noisy marginals are a common form of confidentiality-protecting data release and are useful for many downstream tasks such as contingency table analysis, construction of Bayesian networks, and even synthetic data generation. Privacy mechanisms that provide unbiased noisy answers to linear queries (such as marginals) are known as matrix mechanisms.

We propose ResidualPlanner, a matrix mechanism for marginals with Gaussian noise that is both optimal and scalable. ResidualPlanner can optimize for many loss functions that can be written as a convex function of marginal variances (prior work was restricted to just one predefined objective function). ResidualPlanner can optimize the accuracy of marginals in large scale settings in seconds, even when the previous state of the art (HDMM) runs out of memory. It even runs on datasets with 100 attributes in a couple of minutes. Furthermore ResidualPlanner can efficiently compute variance/covariance values for each marginal (prior methods quickly run out of memory, even for relatively small datasets).

An Optimal and Scalable Matrix Mechanism for Noisy Marginals under Convex Loss Functions

Abstract

Noisy marginals are a common form of confidentiality-protecting data release and are useful for many downstream tasks such as contingency table analysis, construction of Bayesian networks, and even synthetic data generation. Privacy mechanisms that provide unbiased noisy answers to linear queries (such as marginals) are known as matrix mechanisms.

We propose ResidualPlanner, a matrix mechanism for marginals with Gaussian noise that is both optimal and scalable. ResidualPlanner can optimize for many loss functions that can be written as a convex function of marginal variances (prior work was restricted to just one predefined objective function). ResidualPlanner can optimize the accuracy of marginals in large scale settings in seconds, even when the previous state of the art (HDMM) runs out of memory. It even runs on datasets with 100 attributes in a couple of minutes. Furthermore ResidualPlanner can efficiently compute variance/covariance values for each marginal (prior methods quickly run out of memory, even for relatively small datasets).

1 Introduction

Marginals are tables of counts on a set of attributes (e.g., how many people there are for each combination of race and gender). They are one of the most common formats for the dissemination of statistical data [8, 2], studying correlations between attributes, and are sufficient statistics for loglinear models, including Bayesian networks and Markov random fields. For this reason, a lot of work in the differential privacy literature has considered how to produce a set of noisy marginals that is both privacy-preserving and accurate.

One line of work, called the matrix mechanism [32, 52, 30, 53, 37, 51, 46, 18, 42] designs algorithms for answering linear queries (such as marginals) so that the privacy-preserving noisy answers are accurate, unbiased, and have a simple distribution (e.g., multivariate normal). These crucial properties allow statisticians to work with the data, model error due to data collection (sampling error) and error due to privacy protections. It enables valid confidence intervals and hypothesis tests and other methods for quantifying the uncertainty of a statistical analysis (e.g,. [20, 29, 50, 25, 26]). Incidentally, sets of noisy marginals are also used to generate differentially private synthetic data (e.g., [54, 4, 41, 10]).

For the case of marginals, significant effort has been spent in designing optimal or nearly optimal matrix mechanisms for just a single objective function (total variance of all the desired marginals) [32, 49, 13, 52, 53, 31, 51] and each new objective function requires significant additional effort [6, 18, 42, 46]. However, existing optimal solutions do not scale and additional effort is needed to design scalable, but suboptimal, matrix mechanisms for marginals [37, 38]. This is because prior work used mathematical properties specific to their chosen objective function in order to improve runtime. Furthermore, computing the individual variances of the desired noisy marginals is a slow process and more difficult is computing the covariance between cells in the same marginal.

Contributions. Our paper addresses these problems with a novel matrix mechanisms called ResidualPlanner. It can optimize for a wide variety of convex objective functions and return solutions that are guaranteed to be optimal under Gaussian noise. It is highly scalable - running in seconds even when other scalable algorithms run out of memory. It also efficiently returns the variance and covariances of each cell of the desired marginals. It leverages the following insights. Since a dataset can be represented as a vector x of counts, and since a marginal query on a set A of attributes can be represented as a matrix Q A (with Q A x being the true answer to the marginal query), we find a new linearly independent basis that can parsimoniously represent both a marginal Q A and the 'difference' between two marginals Q A and Q A ′ (subspace spanned by the rows of Q A that is orthogonal to the rows of Q A ′ ). Using parsimonious linear bases, instead of overparametrized mechanisms, accounts for the scalability. Optimality results from a deep analysis of the symmetry that marginals impose on the optimal solution - the same linear basis is optimal for a wide variety of loss functions. Our code is available at https://github.com/dkifer/ResidualPlanner .

2 Preliminaries

The Kronecker product between a k 1 × k 2 matrix V = [ 1 , 1 1 ,k 2 . . . ··· . . . v k 1 , 1 ··· v k 1 ,k 2 ] and an ℓ 1 × ℓ 2 matrix W , denoted by V ⊗ W , is the k 1 ℓ 1 × k 2 ℓ 2 matrix that can be represented in block matrix form as: [ v 1 , 1 W ··· v 1 ,k 2 W . . . ··· . . . v k 1 , 1 W ··· v k 1 ,k 2 W ] . A dataset D = { r 1 , . . . , r n } is a collection of records (there is exactly one record for each individual in the dataset). Every record r i contains n a attributes Att 1 , . . . , Att n a and each attribute Att j can take values a ( j ) 1 , . . . , a ( j ) | Att j | . An attribute value a ( j ) i for attribute Att j can be represented as a vector using one-hot encoding. Specifically, let e ( j ) i be a row vector of size | Att j | with a 1 in component i and 0 everywhere else. In this way e ( j ) i represents the attribute value a ( j ) i . A record r with attributes Att 1 = a (1) i 1 , Att 2 = a (2) i 2 , . . . , Att n a = a ( n a ) i na can thus be represented as the Kronecker product e (1) i 1 ⊗ e (2) i 2 ⊗···⊗ e ( n a ) i na . This vector has a 1 in exactly one position and 0s everywhere else. The position of the 1 is the index of record r . With this notation, a dataset D can be represented as a vector x of integers. The value at index i is the number of times the record associated with index i appears in D . The number of components in this vector is denoted as d = ∏ n a i =1 | Att i | . Given a subset A of attributes, a marginal query on A is a table of counts: for each combination of values for the attributes in A , it provides the number of records in D having those attribute value combinations. The marginal query can be represented as a Kronecker product Q A = V 1 ⊗···⊗ V n a where V i is the row vector of all ones (i.e, 1 T | Att i | ) if Att i / ∈ A and V i is the identity matrix I | Att i | if Att i ∈ A . The answer to the marginal query is obtained by evaluating the matrix-vector product Q A x . For convenience, the notation introduced in this paper is summarized as a table in Section A in the supplementary material.

v ··· v

EXAMPLE 2.1. As a running example, consider a dataset in which there are two attributes: Att 1 with values ' yes ' and ' no ', and Att 2 with values ' low ', ' med ', ' high '. The record ( no , med ) is represented by the kron product [ 0 1 ] ⊗ [ 0 1 0 ] and the marginal query on the set A = { Att 1 } is represented as Q { Att 1 } = [ 1 0 0 1 ] ⊗ [ 1 1 1 ] . Similarly, the marginal on attribute Att 2 is represented as Q { Att 2 } = [ 11 ] ⊗ [ 1 0 0 0 1 0 0 0 1 ] . The query representing all one-way marginals is obtained by stacking them: Q 1-way = [ Q { Att 1 } Q { Att 2 } ] and Q 1-way x consists of the five query answers (number of records with Att 1 = yes , number with Att 1 = no , number with Att 2 = low, etc.).

2.1 Differential Privacy

A mechanism M is an algorithm whose input is a dataset and whose output provides privacy protections. Differential privacy is a family of privacy definitions that guide the behavior of mechanisms so that they can inject enough noise to mask the effects of any individual. There are many versions of differential privacy that support Gaussian noise, including approximate DP, zCDP, and Gaussian DP.

DEFINITION 2.2 (Differential Privacy). Let M be a mechanism. For every pair of datasets D 1 , D 2 that differ on the presence/absence of a single record and for all (measurable) sets S ⊆ range ( M ) ,

If P ( M ( D 1 ) ∈ S ) ≤ e ϵ P ( M ( D 2 ) ∈ S ) + δ then M satisfies ( ϵ, δ ) -approximate differential privacy [17];

If Φ -1 ( P ( M ( D 1 ) ∈ S )) ≤ Φ -1 ( P ( M ( D 2 ) ∈ S )) + µ , where Φ is the cdf of the standard Gaussian distribution, then M satisfies µ -Gaussian DP [15].

If the Rényi divergence D α ( M ( D 1 ) ||M ( D 2 )) between the output distributions of M ( D 1 ) and M ( D 2 ) satisfies D α ( M ( D 1 ) ||M ( D 2 )) ≤ ρα for all α > 1 , then M satisfies ρ -zCDP [7].

Queries that are linear functions of the data vector x can be answered privately using the linear Gaussian mechanism , which adds correlated Gaussian noise to a linear function of x , as follows.

DEFINITION 2.3 (Linear Gaussian Mechanism [46]). Given a m × d matrix B and m × m covariance matrix Σ , the (correlated) linear Gaussian mechanism M is defined as M ( x ) = Bx + N ( 0 , Σ ) . The privacy cost matrix of M is defined as B T Σ -1 B . The privacy cost of M , denoted by pcost ( M ) , is the largest diagonal of the privacy cost matrix and is used to compute the privacy parameters: M satisfies ρ -zCDP with ρ = pcost ( M ) / 2 [46], satisfies ( ϵ, δ ) -approximate DP with δ = Φ( √ pcost ( M ) / 2 -ϵ/ √ pcost ( M )) -e ϵ Φ( -√ pcost ( M ) / 2 -ϵ/ √ pcost ( M )) (this is an increasing function of pcost ( M ) [5]), and satisfies µ -Gaussian DP with µ = √ pcost ( M ) [15, 46].

The use of a non-diagonal covariance matrix is crucial because it will help simplify the description of the optimal choices of B and Σ . In particular, using non-diagonal covariance allows us to provide explicit formulas for the entries of the B matrices. We note that an algorithm M ∗ that releases the outputs of multiple linear Gaussian mechanisms M 1 , . . . , M k (with M i ( x ) = B i x + N ( 0 , Σ i ) ) is again a linear Gaussian mechanism. It is represented as M ∗ ( x ) = B ∗ x + N ( 0 , Σ ∗ ) with the matrix B ∗ obtained by vertically stacking the B i and with covariance Σ ∗ being a block-diagonal matrix where the blocks are the Σ i . Its privacy cost pcost ( M ∗ ) = pcost ( M 1 , . . . , M k ) is the largest diagonal entry of ∑ k i =1 B T i Σ -1 i B i .

2.2 Matrix Mechanism

The Matrix Mechanism [32, 52, 30, 53, 37, 38, 51, 46, 18, 42] is a framework for providing unbiased privacy-preserving answers to a workload of linear queries, represented by a matrix W (so that the true non-private answer to the workload queries is Wx ). The matrix mechanism framework consists of 3 steps: select , measure , and reconstruct . The purpose of the select phase is to determine what we add noise to and how much noise to use. More formally, when a user's preferred noise distribution is Gaussian, the select phase chooses a Gaussian linear mechanism M ( x ) ≡ Bx + N ( 0 , Σ ) whose noisy output can be used to estimate the true query answer Wx . Ideally, M uses the least amount of noise subject to privacy constraints (specified by a privacy definition and settings of its privacy parameters). The measure phase runs the mechanism on the data to produce (noisy) privacy-preserving outputs ω = M ( x ) . The reconstruct step uses ω to compute an unbiased estimate of Wx . The unbiased estimate is typically W ( B T Σ -1 B ) † B T Σ -1 ω , where † represents the Moore-Penrose pseudo-inverse. This is the best linear unbiased estimate of Wx that can be obtained from ω [32]. This means that the goal of the select step is to optimize the choice of B and Σ so that the reconstructed answer is as accurate as possible, subject to privacy constraints. Ideally, a user would specify their accuracy requirements using a loss function, but existing matrix mechanisms do not allow this flexibility - they hard-code the loss function. In fact, adding support for new loss function used to require significant research and new optimization algorithms [53, 46, 18] because each new algorithm was customized to specific properties of a chosen loss function. On top of this, existing optimal matrix mechanism algorithms do not scale, while scalable matrix mechanisms are not guaranteed to produce optimal solutions [37]. Additionally, the reconstruction phase should also compute the variance of each workload answer. The variances are the diagonals of W ( B T Σ -1 B ) † W T and making this computation scale is also challenging.

3 Additional Related Work

The marginal release mechanism by Barak et al. [6] predates the matrix mechanism [32, 52, 30, 53, 13, 43, 37, 51, 46, 18, 42, 38] and adds noise to the Fourier decomposition of marginals. We add noise to a different basis, resulting in the scalability and optimality properties. The SVD bound [31] is a lower bound on total matrix mechanism error when the loss function is the sum of variances. This lower bound is tight for marginals and we use it as a sanity check for our results and implementation (note ResidualPlanner provides optimal solutions even when the SVD bound is infeasible to compute).

Alternative approaches to the matrix mechanism can produce privacy preserving marginal query answers that reduce variance by adding bias. This is often done by generating differentially private synthetic data or other such data synopses from which marginals can be computed. State-of-the art approaches iteratively ask queries and fit synthetic data to the resulting answers [22, 34, 4, 19, 39, 35, 44, 56]. For such mechanisms, it is difficult to estimate error of a query answer but recently AIM [39] has made progress in upper bounding the error. PGM [41] provides a connection between the matrix mechanism and this line of work, as it can postprocess noisy marginals into synthetic data. It is a better alternative to sampling a synthetic dataset from models fit to carefully chosen marginals [54, 11, 55, 10]. Synthetic data for answering marginal queries can also be created from random projections [48], copulas [33, 3], and deep generative models [23, 1, 35].

With respect to the matrix mechanism, the reconstruction step is often one of the bottlenecks to scalability. While PGM [41] provides one solution, another proposal by McKenna et al. [40] is to further improve scalability by sacrificing some consistency (the answers to two different marginals may provide conflicting answers to submarginals they have in common). Work on differential privacy marginals has also seen extensions to hierarchical datasets, in which records form meaningful groups that need to be queried. That is, in addition to marginals on characteristics of people, marginals can be computed in different hierarchies such as geographic level (state, county, etc.) and marginals on household composition (or other groupings of people) [2, 28, 36].

4 ResidualPlanner

ResidualPlanner is our proposed matrix mechanism for optimizing the accuracy of marginal queries with Gaussian noise. It is optimal and more scalable than existing approaches. It supports optimizing the accuracy of marginals under a wide variety of loss functions and provides exact variances/covariances of the noisy marginals in closed-form. In this section, we first explain the loss functions it supports. We then describe the base mechanisms it uses to answer marginal queries. We next show how to reconstruct the marginal queries from the outputs of the base mechanisms and how to compute their variances in closed form. We then explain how to optimize these base mechanisms for different loss functions. The reason this selection step is presented last is because it depends on the closed form variance calculations. Then we analyze computational complexity. To aid in understanding, we added a complete numerical run-through of the steps in Section B of the supplementary material. All of our proofs also appear in the supplementary material.

4.1 Loss Functions Supported by ResidualPlanner

The loss functions we consider are generalizations of the sum of variances and max of variances used in prior work. Our more general class of loss functions is based on the following principle: different marginals can have different relative importance but within a marginal, its cells are equally important. That is, a loss function can express that the two-way marginal on the attribute set { Race, Marital Status } is more important (i.e., requires more accuracy) than the 1-way marginal on { EducationLevel } , but all cells within the { Race, MaritalStatus } marginal are equally important. This is a commonly accepted principle for answering differentially private marginal queries (e.g., [32, 52, 30, 53, 37, 51, 46, 18, 42, 39, 4, 34]) and is certainly true for the 2020 Census redistricting data [2].

Let Wkload = { A 1 , . . . , A k } be a workload of marginals, where each A i is a subset of attributes and represents a marginal. E.g., Wkload = {{ Race, MaritalStatus } , { EducationLevel }} consists of 2 marginals, a two-way marginal on Race/MaritalStatus, and a one-way marginal on Education. Let M be a Gaussian linear mechanism whose output can be used to reconstruct unbiased answers to the marginals in Wkload . For each A i ∈ Wkload , let V ar ( A i ; M ) be the function that returns the

variances of the reconstructed answers to the marginal on A i ; the output of V ar ( A i ; M ) is a vector v i with one component for each cell of the marginal on A i . A loss function L aggregates all of these vectors together: L ( v 1 , . . . , v k ) . We have the following regularity conditions on the loss function.

DEFINITION 4.1 (Regular Loss Function). We say the loss function L is regular if: (1) L is convex and continuous; (2) L ( v 1 , . . . , v k ) is minimized when all the v i are the 0 vectors; and (3) for any i , permuting just the components of v i does not affect the value of L ( v 1 , . . . , v k ) . This latter condition just says that cells within the same marginal are equally important.

Loss functions used on prior work are all regular. For example, weighted sum of variances [32, 52, 30, 53, 37, 51] can be expressed as L ( v 1 , . . . , v k ) = ∑ i c i 1 T v i , where the c i are the nonnegative weights that indicate the relative importance of the different marginals. Another popular loss function is maximum (weighted) variance [46, 18, 42], expressed as L ( v 1 , . . . , v k ) = max { max( v 1 ) c 1 , . . . , max( v k ) c k } . Thus, the optimization problem that the selection step needs to solve is either privacy constrained: minimize loss while keeping privacy cost (defined at the end of Section 2.1) below a threshold γ ; or utility constrained: minimize privacy cost such that the loss is at most γ .

We note that regular loss functions cover other interesting cases. For instance, suppose Alicia, Bob, and Carol wish to minimize the sum of variances on their own separate workloads. Taking the max over these three sum-of-variances as the loss function allows the data curator to minimize the largest unhappiness among the three data stakeholders.

4.2 Base Mechanisms used by ResidualPlanner

In the most common application setting, the user picks a privacy budget, a workload of marginals and a loss function L . Based on these choices, a matrix mechanism must decide what linear queries to add noise to and how much noise to add to them. Then it uses those noisy measurements to reconstruct answers to the workload queries. In the case of ResidualPlanner, the linear queries that need noise are represented by base mechanisms that are described in this section. Each base mechanism has a scalar noise parameter that determines how much noise it uses (so optimizing the loss function L is equivalent to finding a good value for the noise parameter of each base mechanism). As long as the loss function L is regular, we prove that an optimal mechanism can be constructed from the set of base mechanisms that we describe here.

To begin, we define a subtraction matrix Sub m to be an ( m -1) × m matrix where the first column is filled with 1, entries of the form ( i, i +1) are -1, and all other entries are 0. For example, Sub 3 = [ 1 -1 0 1 0 -1 ] and Sub 2 = [ 1 -1 ] . We use these subtraction matrices to define special matrices called residual matrices that are important for our algorithm.

For any subset A ⊆ { Att 1 , . . . , Att n a } of attributes, we define the residual matrix R A as the Kronecker product R A = V 1 ⊗···⊗ V n a , where V i = 1 T | Att i | if Att i / ∈ A and V i = Sub | Att i | if Att i ∈ A . Continuing Example 2.1, we have R ∅ = [ 1 1 ] ⊗ [ 1 1 1 ] , and R { Att 1 } = [ 1 -1 ] ⊗ [ 1 1 1 ] , and R { Att 2 } = [ 1 1 ] ⊗ [ 1 -1 0 1 0 -1 ] , and R { Att 1 ,Att 2 } = [ 1 -1 ] ⊗ [ 1 -1 0 1 0 -1 ] .

Using subtraction matrices, we also define the matrix Σ A as the Kronecker product ⊗ Att i ∈ A ( Sub | Att i | Sub T | Att i | ) and we note that it is proportional to R A R T A . Σ ∅ is defined as 1.

Each subset A of attributes can be associated with a 'base' mechanism M A that takes as input the data vector x and a scalar parameter σ 2 A for controlling how noisy the answer is. M A is defined as:

The residual matrices R A used by base mechanisms form a linearly independent basis that compactly represent marginals, as the next result shows.

̸

THEOREM 4.2. Let A be a set of attributes and let Q A be the matrix representation of the marginal on A . Then the rows of the matrices R A ′ , for all A ′ ⊆ A , form a linearly independent basis of the row space of Q A . Furthermore, if A ′ = A ′′ then R A ′ R T A ′′ = 0 (they are mutually orthogonal).

REMARK 4.3. To build an intuitive understanding of residual matrices, consider again Example 2.1. Both R ∅ and Q ∅ are the sum query (marginal on no attributes). The rows of R { Att 1 } span the subspace of Q { Att 1 } that is orthogonal to Q ∅ (and similarly for R { Att 2 } ). The rows of R { Att 1 ,Att 2 } span the subspace of Q { Att 1 ,Att 2 } that is orthogonal to both Q { Att 1 } and Q { Att 2 } . Hence a residual matrix spans the subspace of a marginal that is orthogonal to its sub-marginals.

Theorem 4.2 has several important implications. If we define the downward closure of a marginal workload Wkload = { A 1 , . . . , A k } as the collection of all subsets of the sets in Wkload (i.e., closure ( Wkload ) = { A ′ : A ′ ⊆ A for some A ∈ Wkload } ) then the theorem implies that the combined rows from { R A ′ : A ′ ∈ closure ( Wkload ) } forms a linearly independent basis for the marginals in the workload. In other words, it is a linearly independent bases for the space spanned by the rows of the marginal query matrices Q A for A ∈ Wkload . Thus, in order to provide privacypreserving answers to all of the marginals represented in Wkload , we need all the mechanisms M A ′ for A ′ ∈ closure ( Wkload ) - any other matrix mechanism that provides fewer noisy outputs cannot reconstruct unbiased answers to the workload marginals. This is proved in Theorem 4.4, which also states that optimality is achieved by carefully setting the σ A noise parameter for each M A .

THEOREM 4.4. Given a marginal workload Wkload and a regular loss function L , suppose the optimization problem (either Equation 1 or 2) is feasible. Then there exist nonnegative constants σ 2 A for each A ∈ closure ( Wkload ) (the constants do not depend on the data), such that the optimal linear Gaussian mechanism M opt for loss function L releases M A ( x ; σ 2 A ) for all A ∈ closure ( Wkload ) . Furthermore, any matrix mechanism for this workload must produce at least this many noise measurements during its selection phase.

M A can be evaluated efficiently, directly from the marginal of x on attribute set A , as shown in Algorithm 1. It uses the technique from [37] to perform fast multiplication between a Kronecker product and a vector (so that the Kronecker product does not need to be expanded). It also generates correlated noise from independent Gaussians. The privacy cost pcost ( M A ) of each base mechanism M A is also easy to compute and is given by the following theorem.

THEOREM 4.5. The privacy cost of M A with noise parameter σ 2 A is 1 σ 2 A ∏ Att i ∈ A | Att i |-1 | Att i | and the evaluation of M A given in Algorithm 1 is correct - i.e., the output has the distribution N ( R A x , σ 2 A Σ A ) .

4.3 Reconstruction

Next we explain how to reconstruct unbiased answers to marginal queries from the outputs of the base mechanisms and how to compute (co)variances of the reconstructed marginals efficiently, without any heavy matrix operations (inversion, pseudo-inverses, etc.). Then, given the closed form expressions for marginals and privacy cost (Theorem 4.5), we will be able to explain in Section 4.4 how to optimize the σ 2 A parameters of the base mechanisms M A to optimize regular loss functions L .

Since the base mechanisms were built using a linearly independent basis, reconstruction is unique - just efficiently invert the basis. Hence, unlike PGM and its extensions [41, 40], our reconstruction algorithm does not need to solve an optimization problem and can reconstruct each marginal independently, thus allowing marginals to be reconstructed in parallel, or as needed by users. The reconstructed marginals are consistent with each other (any two reconstructed marginals agree on their sub-marginals). Just as the subtraction matrices Sub k were useful in constructing the base mechanisms M A , their pseudo-inverses Sub † k are useful for reconstructing noisy marginals from the

noisy answers of M A . The pseudo-inverses have a closed form. For example Sub 4 = [ 1 -1 0 0 1 0 -1 0 ]

1 0 0 -1 and Sub † 4 = 1 4 [ 1 1 1 -3 1 1 1 -3 1 ] . More generally, they are expressed as follows:

LEMMA 4.6. For any Att i , let ℓ = | Att i | . The matrix Sub ℓ has the following block matrix, with dimensions ℓ × ( ℓ -1) , as its pseudo-inverse (and right inverse): Sub † ℓ = 1 ℓ [ 1 T ℓ -1 1 ℓ -1 1 T ℓ -1 -ℓ I ℓ -1 ] .

Each mechanism M A , for A ∈ closure ( Wkload ) , has a noise scale parameter σ 2 A and a noisy output that we denote by ω A . After we have obtained the noisy outputs ω A for all A ∈ closure ( Wkload ) , we can proceed with the reconstruction phase. The reconstruction of an unbiased noisy answer for any marginal on an attribute set A ∈ closure ( Wkload ) is obtained using Algorithm 2. We note that to reconstruct a marginal on attribute set A , one only needs to use the noisy answers ω A ′ for A ′ ∈ closure ( A ) . In other words, if we want to reconstruct a marginal on attribute set { Att 1 , Att 2 } , we only need the outputs of M ∅ , M { Att 1 } , M { Att 2 } , and M { Att 1 ,Att 2 } no matter how many other attributes are in the data and no matter what other marginals are in the Wkload . We emphasize again, the reconstruction phase does not run the base mechanisms anymore, it is purely post-processing.

THEOREM 4.7. Given a marginal workload Wkload and positive numbers σ 2 A for each A ∈ closure ( Wkload ) , let M be the mechanism that outputs {M A ( x ; σ 2 A ) : A ∈ closure ( Wkload ) } and let { ω A : A ∈ closure ( Wkload ) } denote the privacy-preserving noisy answers (e.g., ω A = M A ( x , σ 2 ) ). Then for any marginal on an attribute set A ∈ closure ( Wkload ) , Algorithm 2 returns the unique linear unbiased estimate of Q A x (i.e., answers to the marginal query) that can be computed from the noisy differentially private answers.

The variances V ar ( A ; M ) of all the noisy cell counts of the marginal on A is the vector whose components are all equal to ∑ A ′ ⊆ A ( σ 2 A ′ ∏ Att i ∈ A ′ | Att i |-1 | Att i | ∗ ∏ Att j ∈ ( A / A ′ ) 1 | Att j | 2 ) . The covariance between any two noisy answers of the marginal on A is ∑ A ′ ⊆ A ( σ 2 A ′ ∏ Att i ∈ A ′ -1 | Att i | ∗ ∏ Att j ∈ ( A / A ′ ) 1 | Att j | 2 ) .

To see an example of how the choices of the σ 2 A affect the variance of different marginals, see Section C in the supplementary material.

4.4 Optimizing the Base Mechanism Selection

We now consider how to find the optimal Gaussian linear mechanism M ∗ that solves the optimization problems in Equations 1 or 2. Given a workload on marginals Wkload , the optimization involves V ar ( A ; M ∗ ) for A ∈ Wkload (the variance of the marginal answers reconstructed from the output of M ∗ ) and pcost ( M ∗ ) , from which the privacy parameters of different flavors of differential privacy can be computed.

Theorem 4.4 says that M ∗ works by releasing M A ( x ; σ 2 A ) for each A ∈ closure ( Wkload ) for appropriately chosen values of σ 2 A . The privacy cost pcost ( M ∗ ) is the sum of the privacy costs of

the M A . Theorem 4.5 therefore shows that pcost ( M ∗ ) is a positive linear combination of the values 1 /σ 2 A for A ∈ closure ( Wkload ) and is therefore convex in the σ 2 A values. Meanwhile, Theorem 4.7 shows how to represent, for each A ∈ closure ( Wkload ) , the quantity V ar ( A ; M ∗ ) as a positive linear combination of σ 2 A ′ for A ′ ∈ closure ( A ) ⊆ closure ( Wkload ) . Therefore, the loss function L is also convex in the σ 2 A values.

Thus the optimization problems in Equations 1 and 2 can be written as minimizing a convex function of the σ 2 A subject to convex constraints. In fact, in Equation 2, the constraints are linear when the optimization variables represent the σ 2 A and in Equation 1 the constraints are linear when the optimization variables represent the 1 /σ 2 A . Furthermore, when the loss function is the weighted sum of variances of the marginal cells, the solution can be obtained in closed form (see supplementary material). Otherwise, we use CVXPY/ECOS [12, 14] for solving these convex optimization problems.

4.5 Computational Complexity

Although the universe size | Att 1 | × · · · × | Att n a | grows exponentially with the number of attributes, the following theorem shows that the time complexity of ResidualPlanner depends directly on quantities that typically grow polynomially, such as the number of desired marginals and total number of cells in those marginals.

THEOREM 4.8. Let n a be the total number of attributes. Let # cells ( A ) denote the number of cells in the marginal on attribute set A . Then:

Expressing the privacy cost of the optimal mechanism M ∗ as a linear combination of the 1 /σ 2 A values takes O ( ∑ A ∈ Wkload # cells ( A )) total time.

Expressing all of the V ar ( A ; M ∗ ) , for A ∈ Wkload , as a linear combinations of the σ 2 A values can be done in O ( ∑ A ∈ Wkload # cells ( A )) total time.

Computing all the noisy outputs of the optimal mechanism (i.e., M A ( x ; σ 2 A ) for A ∈ closure ( Wkload ) ) takes O ( n a ∑ A ∈ Wkload ∏ Att i ∈ A ( | Att i | +1) ) total time after the true answers have been precomputed (Line 1 in Algorithm 1). Note that the total number of cells on marginals in Wkload is O (∑ A ∈ Wkload ∏ Att i ∈ A | Att i | ) .

Reconstructing marginals for all A ∈ Wkload takes O ( ∑ A ∈ Wkload | A | # cells ( A ) 2 ) total time.

Computing the variance of the cells for all of the marginals for A ∈ Wkload can be done in O ( ∑ A ∈ Wkload # cells ( A )) total time.

To get a sense of these numbers, consider a dataset with 20 attributes, each having 3 possible values. If the workload consists of all 3-way marginals, there are 1,140 marginals each with 27 cells so n cells = 30 , 780 . The quantity inside the big-O for the selection step is 1 , 459 , 200 (roughly the number of scalar multiplications needed). These are all easily manageable on modern computers even without GPUs. Our experiments, under more challenging conditions, run in seconds.

5 Experiments

We next compare the accuracy and scalability of ResidualPlanner against HDMM [38], including variations of HDMM with faster reconstruction phases [41]. The hardware used was an Ubuntu 22.04.2 server with 12 Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz processors and 32GB of DDR4 RAM. We use 3 real datasets to evaluate accuracy and 1 synthetic dataset to evaluate scalability. The real datasets are (1) the Adult dataset [16] with 14 attributes, each having domain sizes 100 , 100 , 100 , 99 , 85 , 42 , 16 , 15 , 9 , 7 , 6 , 5 , 2 , 2 , respectively, resulting in a record domain size of 6 . 41 ∗ 10 17 ; (2) the CPS dataset [9] with 5 attributes, each having domain size 100 , 50 , 7 , 4 , 2 , respectively, resulting in a record domain size of 2 . 8 ∗ 10 5 ; (3) the Loans dataset [24] with 12 attributes, each having domain size 101 , 101 , 101 , 101 , 3 , 8 , 36 , 6 , 51 , 4 , 5 , 15 , respectively, resulting in a record domain size of 8 . 25 ∗ 10 15 . The synthetic dataset is called Synthn d . Here d refers to the number of attributes (we experiment from d = 2 to d = 100 ) and n is the domain size of each attribute. The running times of the algorithms only depend on n and d and not on the records in the synthetic data. For all experiments, we set the privacy cost pcost to 1, so all mechanisms being compared satisfy 0 . 5 -zCDP and 1-Gaussian DP.

5.1 Scalability of the Selection Phase

We first consider how long each method takes to perform the selection phase (i.e., determine what needs noisy answers and how much noise to use). HDMM can only optimize total variance, which is equivalent to root mean squared error. For ResidualPlanner we consider both RMSE and max variance as objectives (the latter is a harder to solve problem). Each algorithm is run 5 times and the average is taken. Table 1 shows running time results; accuracy results will be presented later.

As we can see, optimizing for max variance is more difficult than for RMSE, but ResidualPlanner does it quickly even for data settings too big for HDMM. The runtime of HDMM increases rapidly, while even for the extreme end of our experiments, ResidualPlanner needs just a few minutes.

5.2 Scalability of the Reconstruction Phase

We next evaluate the scalability of the reconstruction phase under the same settings. The reconstruction speed for ResidualPlanner does not depend on the objective of the selection phase. Here we compare against HDMM [38] and a version of HDMM with improved reconstruction scalability called HDMM+PGM [38, 41] (the PGM settings used 50 iterations of its Local-Inference estimator, as the default 1000 was too slow). Since HDMM cannot perform the selection phase after a certain point, reconstruction results also become unavailable. Table 2 shows ResidualPlanner is clearly faster.

5.3 Accuracy Comparisons

Since ResidualPlanner is optimal, the purpose of the accuracy comparisons is a sanity check. For RMSE,comparisons of the quality of ResidualPlanner to the theoretically optimal lower bound, known as the SVD bound [31], can be found in the supplementary material in Section I (ResidualPlanner

matches the lower bound). We note ResidualPlanner can provide solutions even when the SVD bound is infeasible to compute.

We also compare ResidualPlanner to HDMM when the user is interested in the maximum variance objective. This just shows that it is important to optimize for the user's objective function and that the optimal solution for RMSE (the only objective HDMM can optimize) is not a good general-purpose approximation for other objectives (as shown in Table 3). Additional comparisons are provided in the supplementary material.

6 Limitations, Conclusion, and Future Work.

In this paper, we introduced ResidualPlanner, a matrix mechanism that is scalable and optimal for marginals under Gaussian noise, for a large class of convex objective functions. While these are important improvements to the state of the art, there are limitations.

First, for some attributes, a user might not want marginals. For example, they might want range queries or queries with hierarchies (e.g., how many people drive sedans vs. vans; out of the sedans, how many are red vs. green, etc) [2, 28, 36]. In some cases, an attribute might have an infinite domain (e.g., a URL) and need to be handled differently [27, 45]. In other cases, the user may want other noise distributions, like the Laplace. These types of queries do not have the same type of symmetry as marginals that was crucial to proving the optimality of ResidualPlanner. For these situations, one of the key ideas of ResidualPlanner can be used - find a linear basis that compactly represents both the queries and 'residual' (information provided by a query that is not contained in the other queries). Such a feature would result in scalability. It is future work to determine how to extend both scalability and optimality to such situations. Another limitation is that this work considers the setting where an individual can contribute exactly one (rather than arbitrarily many) records to the dataset.

7 Acknowledgments

This work was supported by NSF grant CNS-1931686 and a gift from Facebook.

References

Nazmiye Ceren Abay, Yan Zhou, Murat Kantarcioglu, Bhavani Thuraisingham, and Latanya Sweeney. Privacy preserving synthetic data release using deep learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10-14, 2018, Proceedings, Part I 18 , pages 510-526. Springer, 2019.

John M. Abowd, Robert Ashmead, Ryan Cumings-Menon, Simson Garfinkel, Micah Heineck, Christine Heiss, Robert Johns, Daniel Kifer, Philip Leclerc, Ashwin Machanavajjhala, Brett Moran, William Sexton, Matthew Spence, and Pavel Zhuravlev. The 2020 census disclosure avoidance system topdown algorithm. Harvard Data Science Review , forthcoming. Preprint https://www.census.gov/library/working-papers/2022/ adrm/CED-WP-2022-002.html .

Hassan Jameel Asghar, Ming Ding, Thierry Rakotoarivelo, Sirine Mrabet, and Mohamed Ali Kaafar. Differentially private release of high-dimensional datasets using the gaussian copula. arXiv preprint arXiv:1902.01499 , 2019.

Sergul Aydore, William Brown, Michael Kearns, Krishnaram Kenthapadi, Luca Melis, Aaron Roth, and Ankit A Siva. Differentially private query release through adaptive projection. In International Conference on Machine Learning , pages 457-467. PMLR, 2021.

Borja Balle and Yu-Xiang Wang. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In International Conference on Machine Learning, ICML , 2018.

Boaz Barak, Kamalika Chaudhuri, Cynthia Dwork, Satyen Kale, Frank McSherry, and Kunal Talwar. Privacy, accuracy, and consistency too: a holistic solution to contingency table release. In Proceedings of the twenty-sixth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems , pages 273-282, 2007.

Mark Bun and Thomas Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Proceedings, Part I, of the 14th International Conference on Theory of Cryptography - Volume 9985 , 2016.

U.S. Census Bureau. Decennial census: 2010 summary files. https://www.census.gov/ mp/www/cat/decennial_census_2010/ .

U.S. Census Bureau. The current population survey (cps). https://www.census.gov/ programs-surveys/cps.html , 2023.

Kuntai Cai, Xiaoyu Lei, Jianxin Wei, and Xiaokui Xiao. Data synthesis via differentially private markov random fields. Proceedings of the VLDB Endowment , 14(11):2190-2202, 2021.

Rui Chen, Qian Xiao, Yu Zhang, and Jianliang Xu. Differentially private high-dimensional data publication via sampling-based inference. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pages 129-138, 2015.

Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research , 17(83):1-5, 2016.

Bolin Ding, Marianne Winslett, Jiawei Han, and Zhenhui Li. Differentially private data cubes: Optimizing noise sources and consistency. In Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data , 2011.

Alexander Domahidi, Eric Chu, and Stephen Boyd. Ecos: An socp solver for embedded systems. In 2013 European control conference (ECC) , pages 3071-3076. IEEE, 2013.

Jinshuo Dong, Aaron Roth, and Weijie J. Su. Gaussian differential privacy. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 84(1):3-37, 2022.

Dheeru Dua and Casey Graff. UCI machine learning repository, 2019.

Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In EUROCRYPT , pages 486-503, 2006.

Alexander Edmonds, Aleksandar Nikolov, and Jonathan Ullman. The power of factorization mechanisms in local and central differential privacy. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , pages 425-438, 2020.

Marco Gaboardi, Emilio Jesús Gallego Arias, Justin Hsu, Aaron Roth, and Zhiwei Steven Wu. Dual query: Practical private query release for high dimensional data. In International Conference on Machine Learning , pages 1170-1178. PMLR, 2014.

Marco Gaboardi, Hyun Lim, Ryan Rogers, and Salil Vadhan. Differentially private chi-squared hypothesis testing: Goodness of fit and independence testing. In International conference on machine learning , pages 2111-2120. PMLR, 2016.

Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023.

Moritz Hardt, Katrina Ligett, and Frank McSherry. A simple and practical algorithm for differentially private data release. Advances in neural information processing systems , 25, 2012.

James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar. Pate-gan: Generating synthetic data with differential privacy guarantees. In International conference on learning representations , 2019.

Kaggle. Loan prediction problem dataset. https://www.kaggle.com/ altruistdelhite04/loan-prediction-problem-dataset , 2021. Accessed: May 8th, 2023.

Kazuya Kakizaki, Kazuto Fukuchi, and Jun Sakuma. Differentially private chi-squared test by unit circle mechanism. In International Conference on Machine Learning , pages 1761-1770. PMLR, 2017.

Daniel Kifer and Ryan Rogers. A new class of private chi-square tests. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS , volume 17, pages 991-1000, 2016.

Aleksandra Korolova, Krishnaram Kenthapadi, Nina Mishra, and Alexandros Ntoulas. Releasing search queries and clicks privately. In Proceedings of the 18th international conference on World wide web , pages 171-180, 2009.

Yu-Hsuan Kuo, Cho-Chun Chiu, Daniel Kifer, Michael Hay, and Ashwin Machanavajjhala. Differentially private hierarchical count-of-counts histograms. arXiv preprint arXiv:1804.00370 , 2018.

Jing Lei. Differentially private m-estimators. Advances in Neural Information Processing Systems , 24, 2011.

Chao Li and Gerome Miklau. An adaptive mechanism for accurate query answering under differential privacy. Proceedings of the VLDB Endowment , 5(6), 2012.

Chao Li and Gerome Miklau. Optimal error of query sets under the differentially-private matrix mechanism. In Proceedings of the 16th International Conference on Database Theory , pages 272-283, 2013.

Chao Li, Gerome Miklau, Michael Hay, Andrew Mcgregor, and Vibhor Rastogi. The matrix mechanism: Optimizing linear counting queries under differential privacy. The VLDB Journal , 24(6):757-781, December 2015.

Haoran Li, Li Xiong, and Xiaoqian Jiang. Differentially private synthesization of multidimensional data using copula functions. In Advances in database technology: proceedings. International conference on extending database technology , volume 2014, page 475. NIH Public Access, 2014.

Terrance Liu, Giuseppe Vietri, Thomas Steinke, Jonathan Ullman, and Steven Wu. Leveraging public data for practical private query release. In International Conference on Machine Learning , pages 6968-6977. PMLR, 2021.

Terrance Liu, Giuseppe Vietri, and Steven Z Wu. Iterative methods for private synthetic data: Unifying framework and new methods. Advances in Neural Information Processing Systems , 34, 2021.

Terrance Liu and Zhiwei Steven Wu. Private synthetic data with hierarchical structure. arXiv preprint arXiv:2206.05942 , 2022.

Ryan McKenna, Gerome Miklau, Michael Hay, and Ashwin Machanavajjhala. Optimizing error of high-dimensional statistical queries under differential privacy. Proceedings of the VLDB Endowment , 11(10), 2018.

Ryan McKenna, Gerome Miklau, Michael Hay, and Ashwin Machanavajjhala. Hdmm: Optimizing error of high-dimensional statistical queries under differential privacy. arXiv preprint arXiv:2106.12118 , 2021.

Ryan McKenna, Brett Mullins, Daniel Sheldon, and Gerome Miklau. Aim: An adaptive and iterative mechanism for differentially private synthetic data. arXiv preprint arXiv:2201.12677 , 2022.

Ryan McKenna, Siddhant Pradhan, Daniel R Sheldon, and Gerome Miklau. Relaxed marginal consistency for differentially private query answering. Advances in Neural Information Processing Systems , 34:20696-20707, 2021.

Ryan McKenna, Daniel Sheldon, and Gerome Miklau. Graphical-model based estimation and inference for differential privacy. In International Conference on Machine Learning , pages 4435-4444. PMLR, 2019.

Aleksandar Nikolov. New computational aspects of discrepancy theory . PhD thesis, Rutgers University-Graduate School-New Brunswick, 2014.

Wahbeh Qardaji, Weining Yang, and Ninghui Li. Priview: practical differentially private release of marginal contingency tables. In Proceedings of the 2014 ACM SIGMOD international conference on Management of data , pages 1435-1446, 2014.

Giuseppe Vietri, Grace Tian, Mark Bun, Thomas Steinke, and Steven Wu. New oracle-efficient algorithms for private synthetic data release. In International Conference on Machine Learning , pages 9765-9774. PMLR, 2020.

Royce Wilson, Celia Yuxin Zhang, William Lam, Damien Desfontaines, Daniel SimmonsMarengo, and Bryant Gipson. Differentially private sql with bounded user contribution. In Proceedings on Privacy Enhancing Technologies Symposium , 2020.

Yingtai Xiao, Zeyu Ding, Yuxin Wang, Danfeng Zhang, and Daniel Kifer. Optimizing fitnessfor-use of differentially private linear queries. In VLDB , 2021.

Yingtai Xiao, Guanhong Wang, Danfeng Zhang, and Daniel Kifer. Answering private linear queries adaptively using the common mechanism. https://arxiv.org/abs/2212.00135 , 2022.

Chugui Xu, Ju Ren, Yaoxue Zhang, Zhan Qin, and Kui Ren. Dppro: Differentially private highdimensional data release via random projection. IEEE Transactions on Information Forensics and Security , 12(12):3081-3093, 2017.

Grigory Yaroslavtsev, Graham Cormode, Cecilia M Procopiuc, and Divesh Srivastava. Accurate and efficient private release of datacubes and contingency tables. In 2013 IEEE 29th International Conference on Data Engineering (ICDE) , pages 745-756. IEEE, 2013.

Fei Yu, Stephen E Fienberg, Aleksandra B Slavkovi´ c, and Caroline Uhler. Scalable privacypreserving data sharing methodology for genome-wide association studies. Journal of biomedical informatics , 50:133-141, 2014.

Ganzhao Yuan, Yin Yang, Zhenjie Zhang, and Zhifeng Hao. Convex optimization for linear query processing under approximate differential privacy. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2016.

Ganzhao Yuan, Zhenjie Zhang, Marianne Winslett, Xiaokui Xiao, Yin Yang, and Zhifeng Hao. Low-rank mechanism: Optimizing batch queries under differential privacy. Proc. VLDB Endow. , 5(11):1352-1363, July 2012.

Ganzhao Yuan, Zhenjie Zhang, Marianne Winslett, Xiaokui Xiao, Yin Yang, and Zhifeng Hao. Optimizing batch linear queries under exact and approximate differential privacy. ACM Transactions on Database Systems (TODS) , 40(2):1-47, 2015.

Jun Zhang, Graham Cormode, Cecilia M. Procopiuc, Divesh Srivastava, and Xiaokui Xiao. Privbayes: Private data release via bayesian networks. ACM Trans. Database Syst. , 42(4), oct 2017.

Wei Zhang, Jingwen Zhao, Fengqiong Wei, and Yunfang Chen. Differentially private highdimensional data publication via markov network. EAI Endorsed Transactions on Security and Safety , 6(19), 2019.

Zhikun Zhang, Tianhao Wang, Jean Honorio, Ninghui Li, Michael Backes, Shibo He, Jiming Chen, and Yang Zhang. Privsyn: Differentially private data synthesis. 2021.

Contents

A Table of Notation

B A Run-through of Residual Planner

In this section, we provide a complete runthrough of ResidualPlanner using a small toy dataset.

B.1 A Small Dataset and its Vectorized Representation

In our example, we have a dataset with 3 attributes, so n a = 3 . Att 1 takes values 'a' or 'b'; Att 2 takes values 'y' or 'n'; Att 3 takes values 1 or 2 or 3.

In this dataset, there are 5 people, and the tabular representation is shown in Table 5. For each attribute, we can one-hot encode its attribute values as row vectors. So, for Att 1 , the attribute value 'a' is encoded as [1 , 0] and 'b' is encoded as [0 , 1] . For Att 2 , the attribute value 'y' is encoded as [1 , 0] and 'n' is encoded as [0 , 1] . For attribute Att 3 , the attribute value '1' is encoded as [1 , 0 , 0] , the value '2' is encoded as [0 , 1 , 0] and '3' is encoded as [0 , 0 , 1] .

The kronecker product representation of a record is the kronecker product of the one-hot encoding of each attributes. So, for example, the record 'an2' is encoded as the kronecker product [1 , 0] ⊗ [0 , 1] ⊗ [0 , 1 , 0] . When this kronecker product is expanded, it has 12 components. One of the contains a 1 and the rest contain a 0. Thus the expanded kronecker product can be thought of as a one-hot encoding of the entire record.

Indeed, in the expanded kronecker product, each dimension of the resulting vector is associated with a record. In table 6, we show the kronecker product representation of each record from Table 5. The left column of Table 6 shows the record and its kronecker representation. The next 12 columns show the resulting expansion. Each record becomes as 12-dimensional vector and the column labels in Table 6 show which record is associated with which index in the 12-dimensional vector.

The sum of the kron representations of all the records is the data vector x . It is again a 12-dimensional vector. At each index i , x [ i ] is the number of people whose record is associated index i . For example, the 5th component is associated with the record 'an2' and there are 2 people with that record. For mathematical convenience, x is treated as a column vector, but for display purposes, in Table 6 it is written as a row vector.

B.2 The Marginal Workload and its Representation as a Query Matrix.

For this example, we set the marginal workload to consist of 3 marginals Wkload = {{ Att 1 } , { Att 1 , Att 2 } , { Att 2 , Att 3 }} .

The marginal on attribute set A = { Att 1 } has only two cells, which correspond to the number of people with Att 1 = a (i.e., 3) and the number with Att 1 = b (i.e., 3). This is called a one-way marginal. The other marginals are two-way marginals because they involve two attributes. For example, the marginal on A = { Att 2 , Att 3 } has 6 cells. It represents the number of people for each combination of values for Att 2 and Att 3 . For example, there are 2 people with Att 2 = y and Att 3 = 3 .

For each set A , the marginal on those attributes can be represented as a matrix Q A such that calculating the marginal is equivalent to the matrix-vector multiplication Q A x . The construction of

D

:

Dataset

r

i

:

i

th

record in

D

n

a

:

number of attributes each record has

Att

j

:

j

th

attribute.

|

Att

j

|

:

size of the domain of attribute

Att

j

.

a

(

j

)

1

, . . . , a

(

j

)

|

Att

j

|

:

possible values (domain) of

Att

j

.

d

:

Number of possible records:

d

=

n

a

∏

j

=1

|

Att

j

|

x

:

Representation of

D

as a

d

-dimensional vector of counts (e.g., histogram)

A

:

(Sub)set of attributes

Q

A

:

Matrix representation of the marginal on

A

. The counts in the marginal are the result of

matrix-vector product

Q

A

x

.

#

cells

(

A

)

:

Number of cells in the marginal on

A

. Equals

∏

Att

i

∈

A

|

Att

i

|

e

i

:

one-hot encoding vector with entry

i

being 1 and the rest 0

e

i,j

:

equal to

e

i

-

e

j

1

k

:

the

k

-dimensional vector whose entries are all

1

.

I

k

:

the

k

×

k

identity matrix

M

:

A privacy mechanism.

ω

:

Output of a mechanism.

B

:

Query matrix of a Gaussian linear query mechanism:

M

(

x

)

≡

Bx

+

N

(

0

,

Σ

)

Σ

:

Covariance matrix.

pcost

(

M

)

:

Privacy cost of a Gaussian linear mechanism

M

(

x

)

≡

Bx

+

N

(

0

,

Σ

)

.

It is defined as

the largest diagonal of

B

T

Σ

-

1

B

. Differential privacy parameters can be computed from

pcost

(

M

)

.

Wkload

:

A workload of marginals. Each element of

Wkload

is a set of attributes (representing the

marginal on those attributes).

n

cells

:

Total number of cells in the marginals in the marginal workload (i.e., the output size).

closure

(

Wkload

)

:

The set of all subsets of

Wkload

.

Formally defined as

{

A

′

:

A

′

⊆

A

for some

A

∈

Wkload

}

.

V ar

(

A

,

M

)

:

When the output of

M

is used to reconstruct answers to the marginal on

A

, then

V ar

returns the vector of variances of the marginal cells.

L

:

The loss function

†

:

The operator that gives the pseudo-inverse of a matrix

Sub

m

:

An

(

m

-

1)

×

m

subtraction matrix. The first column is filled with 1, entries of the form

(

i, i

+1)

are -1, and all other entries are 0.

R

A

:

Residual matrix. Given a set

A

⊂ {

Att

1

, . . . , Att

n

a

}

of attributes,

R

A

=

V

1

⊗···⊗

V

n

a

,

where

V

i

=

1

|

Att

i

|

if

Att

i

/

∈

A

and

V

i

=

Sub

|

Att

i

|

if

Att

i

∈

A

.

Σ

A

:

The covariance matrix used by the base mechanisms, formed as the kronecker product

⊗

Att

i

∈

A

(

Sub

|

Att

i

|

Sub

T

|

Att

i

|

)

. Also

Σ

∅

= 1

.

σ

A

:

Data-independent noise scale parameter

M

A

:

The base mechanism defined as

M

A

(

x

)

≡

R

A

x

+

N

(

0

, σ

2

A

Σ

A

)

.

It

uses a data-

independent noise parameter

σ

2

A

ω

A

:

noisy output of mechanism

M

A

the matrix Q A is straightforward. It is a kronecker product of 3 matrices. Each matrix corresponds to an attribute. If the attribute is in A then the corresponding term is the identity matrix, otherwise is is the row vector full of ones. For example, Q { Att 1 } is a kron product of 3 matrices: the first matrix corresponds to Att 1 and is the 2 × 2 identity matrix. The second matrix is actually the vector full of ones because Att 2 is not part of the marginal. This vector has 2 components because Att 2 has 2 possible values. Similarly, the third matrix is the vector full of ones with 3 components.

[1

an2:

bn3:

by3:

an2:

,

[0

0]

,

[0

⊗

1]

,

[1

,

[0

⊗

1]

,

1]

[0

⊗

0]

⊗

,

[1

⊗

1]

,

[0

,

[0

⊗

0]

,

1

[0

⊗

1]

⊗

,

[0

,

0

,

[0

,

0]

,

0

1]

,

1

,

by3:

[0

,

1]

⊗

[1

,

0]

⊗

[0

,

0

Vector of counts x :

For the marginals in Wkload , these are the the corresponding matrices:

If we multiply Q { attr 2 ,Att 3 } by the data vector x from Table 6, we get:

Comparing it to the marginals shown in Table 7 we see that it is the flattened version of the marginal. That is, we take the first column of the { Att 2 , Att 3 } marginal of Table 7, then we put the next column below it, and the third column is placed at the bottom.

,

3]

0]

3]

B.3 The Base Mechanisms

Recall that our workload, the marginals we want privacy-preserving answers to, is Wkload = {{ Att 1 } , { Att 1 , Att 2 } , { Att 2 , Att 3 }} . Its closure, denoted by closure ( Wkload ) is all of its subsets. So,

For each A ∈ closure ( Wkload ) we need to form a base mechanism M A . Each M A has a free parameter σ 2 A that we are free to choose. Each mechanism M A has the form M A ( x ; σ 2 A ) = R A x + N ( 0 , Σ A ) . That is, on input x , the mechanism multiplies it by a special 'residual' matrix R A and then adds correlated Gaussian noise, with zero mean and with covariance matrix σ 2 A Σ A . The residual and covariance matrices for each base mechanism are shown below.

Note that for any A , the residual matrix R A has a similar structure to Q A except that where Q A has an identity matrix in its kron product, R A has a subtraction matrix (e.g. [ 1 -1 ] or [ 1 -1 0 1 0 -1 ] ). Meanwhile the covariance matrix Σ A looks like R A R T A except that the vectors full of 1s have been first removed.

How do we interpret the residual matrices? Well, R ∅ is the sum query. In fact the matrix vector multiplication R ∅ x gives us the total number of people in the data.

Next, R { Att 1 } tells us the information contained in the marginal on { Att 1 } that is not contained in the sum query. If we know the total number of people in the data, then the only new information the marginal gives us is the difference between the number of people with Att 1 = a and the number of people with Att 1 = b . In other words, R { Att 1 } x is this difference. Given this difference, and the total, once can recover the marginal on attribute Att 1 .

Similarly, R { Att 2 } contains the information in the marginal on { Att 2 } that is not provided by the sum query. Finally R { Att 3 } contains the information in the marginal on { Att 3 } not provided in the sum query, which is the number of people with Att 3 = 1 minus the number with Att 3 = 2 , and also the number of people with Att 3 = 1 minus the number with Att 3 = 3 . The product R { Att 3 } x returns those two differences as a vector with two components.

Now, R { Att 1 ,Att 2 } and R { Att 2 ,Att 3 } are more complicated, but have the same idea. For example, R { Att 1 ,Att 2 } represents new information that the marginal on { Att 1 , Att 2 } provides that is not captures by the sub-marginals (the marginal on { Att 1 } and the marginal on { Att 2 } .

In general, the matrix R A represents the new information on that the marginal on A provides, which is not captured by the marginals on A ′ , for A ′ ⊂ A (strict subsets).

Now, Theorem 4.2 tells us that if we take all of the rows of all of the residual matrices, they will be linearly independent. Furthermore, given an attribute set A , the total number of rows of R A ′ for all A ′ ⊆ A is the number of rows in Q A . Furthermore, the space spanned by those rows is the same as the space spanned by the rows of Q A .

This also means that if we know R A ′ x for all A ′ ⊆ A then we can figure out Q A x (and vice versa).

Now, we want to get privacy-preserving (noisy) answers to the marginal queries in Wkload = {{ Att 1 } , { Att 1 , Att 2 } , { Att 2 , Att 3 }} that are as accurate as possible subject to privacy constraints. We quantify accuracy using a regular (Definition 4.1) loss function (e.g., sum of the variances of the answers to the marginals) and we quantify privacy by setting privacy parameters for either ( ϵ, δ ) -differential privacy, ρ -zCDP, or µ -Gaussian differential privacy.

Theorem 4.4 says that to maximize the accuracy subject to privacy constraints, we need to take the closure of the workload, closure ( Wkload ) = { ∅ , { Att 1 } , { Att 2 } , { Att 3 } , { Att 1 , Att 2 } , { Att 2 , Att 3 } } and carefully choose positive numbers σ 2 A for each A ∈ closure ( Wkload ) - so that is 6 numbers total. These numbers are chosen without looking at the data (we explain how in Section B.6). Once we have these numbers, we run the mechanisms M A ( x ; σ 2 A ) and return their outputs. In other words, we must release the outputs of:

M ∅ ( x ; σ 2 ∅ ) - produces 1 number (a vector with just one component)

M { Att 1 } ( x ; σ 2 { Att 2 } ) - produces 1 number (a vector with just one component)

M { Att 2 } ( x ; σ 2 { Att 2 } ) - produces 1 number (a vector with just one component)

M { Att 3 } ( x ; σ 2 { Att 3 } ) - produces 2 numbers (a vector with 2 components)

M { Att 1 ,Att 2 } ( x ; σ 2 { Att 1 ,Att 2 } ) - produces 1 number (a vector with 1 component)

M { Att 2 ,Att 3 } ( x ; σ 2 { Att 2 ,Att 3 } ) - produces 2 numbers (a vector with 2 components)

Which gives us 8 total (noisy) numbers. In fact, any matrix mechanism for this workload must return at least 8 noisy numbers, by Theorem 4.4.

From these outputs, one can reconstruct noisy answers to the marginals in Wkload (actually one can reconstruct noisy answers to any marginal in closure ( Wkload ) ). We show how to do this in Section B.4. Then we show how to compute the privacy cost and variances of the algorithm in Section B.5.

B.4 Reconstruction

Let ω A denote the output of M A . Thus, after running

M { Att 3 } ( x ; σ 2 { Att 3 } )

M { Att 1 ,Att 2 } ( x ; σ 2 { Att 1 ,Att 2 } )

and M { Att 2 ,Att 3 } ( x ; σ 2 { Att 2 ,Att 3 } )

we have the noisy answers

From these noisy answers we can produce noisy answers for any marginal in Wkload or even closure ( Wkload ) . To reconstruct a marginal on A , we need ω A ′ for all A ′ ⊆ A - this is not a lot as these vectors represent as many noisy numbers as there are cells in the desired histogram. So, for example, if we want to get noisy answers for the marginal on { Att 2 , Att 3 } (which has 6 cells), we need to use ω ∅ , ω { Att 2 } , ω { Att 3 } , and ω { Att 2 ,Att 3 } (together these ω vectors represent a total of 6 noisy numbers).

In order to reconstruct the marginal on A , Algorithm 2 multiplies each ω A ′ by a matrix that depends on both A and A ′ . The algorithm calls this matrix U , but to make the notation precise for this runthrough, we will call it U A ← A ′ (the U matrix that multiplies ω A ′ when reconstructing A ). It turns out that:

which means that the marginal on A could be recreated if we know the quantities R A ′ x (recall R A ′ are the matrices used to define our base mechanisms). Now, since ω A ′ is a noisy version of R A ′ x , we can get noisy marginal answers by substituting in these noisy values into the above equation.

For example, to reconstruct a noisy answer to the marginal on { Att 2 , Att 3 } , we do the following:

Note Sub † 2 and Sub † 3 are defined in Lemma 4.6.

B.5 Privacy Cost and Marginal Variances

Recall that for a marginal workload Wkload , we need to run a mechanism M A for each A ∈ closure ( Wkload ) . Theorem 4.5 shows how to compute the privacy cost pcost of each. In our running example, this means:

pcost ( M { Att 1 } ( x ; σ 2 { Att 1 } )) = 1 σ 2 { Att 1 } ∗ 1 2

pcost ( M { Att 2 } ( x ; σ 2 { Att 2 } )) = 1 σ 2 { Att 2 } ∗ 1 2

pcost ( M { Att 3 } ( x ; σ 2 { Att 3 } )) = 1 σ 2 { Att 3 } ∗ 2 3

pcost ( M { Att 1 ,Att 2 } ( x ; σ 2 { Att 1 ,Att 2 } )) = 1 σ 2 { Att 1 ,Att 2 } ∗ 1 2 ∗ 1 2

and pcost ( M { Att 2 ,Att 3 } ( x ; σ 2 { Att 2 ,Att 3 } )) = 1 σ 2 { Att 2 ,Att 3 } ∗ 1 2 ∗ 2 3

The total privacy cost is,

Thus this is a symbolic expression in terms of the (currently unknown) noise scale parameters σ 2 A . According to Definition 2.3, we can convert the privacy cost to the ρ in ρ -zCDP by dividing by 2 and we can convert it to the µ from µ -Gaussian DP by taking the square root.

For our running example, Wkload = {{ Att 1 } , { Att 1 , Att 2 } , { Att 2 , Att 3 }} and we can express the variance of these marginals (after reconstruction from the noisy ω A answers) also in terms of the noise scale parameters. We do this with the help of Theorem 4.7.

Marginal on { Att 1 } . This marginal is reconstructed from the noisy answers ω ∅ and ω { Att 1 } and so the variance of its cells depends only on σ 2 ∅ and σ 2 { Att 1 } . Applying Theorem 4.7, get that the variance in each cell of this marginal is the same and equals.

Marginal on { Att 1 , Att 2 } . This marginal is reconstructed from ω ∅ , ω { Att 1 } , ω { Att 2 } , and ω { Att 1 ,Att 2 } and hence the variance of the cells in the marginal depend on the corresponding 4 noise scale parameters. The cell variance is

Marginal on { Att 2 , Att 3 } . Similarly, this marginal also depends on 4 noise scale parameters as follows:

B.6 The Sum-of-Variances Loss Function

Now we can express the overall privacy cost symbolically in terms of the noise scale parameters. We can also express the variance of each marginal cell symbolically. We can use these symbolic expressions to set up any regular loss function and then run it through a convex optimizer to solve it.

In this section, we give an example for the weighted sum of variances, which is one of the most popular loss functions for the matrix mechanism in research settings (mostly because this loss function is easiest to work with).

Each marginal has a weight, which we set to be 1 to avoid introducing more symbols, and the objective function is computed by adding up the cell variances in a marginal, multiplying by the weight, and adding up over the workload marginals. The marginal on { Att 1 } has two cells (so we multiply the cell variance for this marginal, computed in the previous section, by 2). The marginal on { Att 1 , Att 2 } has 4 cells, and the marginal on { Att 2 , Att 3 } has 6 cells. Thus, after the dust clears, the sum of the cell variances across the workload marginals is:

Thus, we can set up the optimization problem: minimize the sum of variances subject to the privacy cost (computed in Section B.5) being less than some constant c :

If we let the coefficient of σ A be denoted by v A and the coefficient of 1 /σ 2 A be denoted by p A , then this optimization problem can be written as:

Lemma H.1 in Section H shows that the optimal solution is obtained by computing:

C Example Variance Calculations

We next illustrate how the σ 2 A parameters affect the variance of different marginals based on Theorem 4.7. We illustrate it with a toy dataset that has 5 attributes, each with 3 possible values. In this discussion, the variance of a marginal is the largest variance of its cells (all cells within the same marginal have the same variance, so the variance of a marginal is basically the variance of any cell).

EXAMPLE C.1. The objective function is to minimize the max variance among all marginals while satisfying µ -Gaussian DP with µ =1. In this case all marginals end up with the same variance of 7.594 and the σ 2 A parameters are:

σ 2 ∅ = 7 . 594

σ 2 A = 10 . 125 when A contains 1 attribute.

σ 2 A = 13 . 5 when A contains 2 attributes.

σ 2 A = 18 when A contains 3 attributes.

σ 2 A = 24 when A contains 4 attributes.

σ 2 A = 32 when A contains all of the attributes.

EXAMPLE C.2. The objective function is to minimize the weighted max variance among all marginals (i.e., minimize max m weight m ∗ var ( m ) ). We set the weight of a marginal to be 3 if it is the 5-way marginal and 1 otherwise. This objective function basically says we want more accuracy on the 5-way marginal (it has higher weight). Again the privacy constraint is 1-Gaussian DP . In the optimal solution, each cell in the 5-way marginal has variance 2.718, the 4-way marginals have variance 5.528, and the other marginals have variance 8.154. The σ 2 A parameters are:

σ 2 ∅ = 8 . 154

σ 2 A = 10 . 871 when A contains 1 attribute.

σ 2 A = 14 . 495 when A contains 2 attributes.

σ 2 A = 19 . 327 when A contains 2 attributes.

σ 2 A = 12 . 477 when A contains 2 attributes.

σ 2 A = 4 . 159 when A contains all of the attributes.

For the variance calculation, we show this for Example C.2.

We start with the variance of a one-way marginal, say the marginal on attribute 1 (i.e., A = { 1 } ). The calculation involves σ 2 { 1 } and σ 2 ∅ (because ∅ ⊆ { 1 } ) and the summation has two terms. The term involving σ 2 ∅ is multiplied by 1 (the first product in the variance expression for Theorem 4.7, since an empty product is 1) and that is multiplied by 1/9 (the second product, since each attribute has 3 possible values). Meanwhile, the term involving σ 2 { 1 } would be multiplied by 2/3 (first product) and 1 (since the second product is empty). Thus the variance for the marginal on attribute 1 is: 8 . 154 ∗ 1 ∗ 1 / 9 + 10 . 871 ∗ 2 / 3 ∗ 1 ≈ 8 . 154 .

For the variance on the 5-way marginal, it consists of 1 term for A ′ = ∅ , 5 terms for when A ′ has 1 attribute since there are 5 such A ′ (but all the terms have the same value), 10 terms for when A ′ has 2 attributes (all have the same value), 10 terms for when A ′ has 3 attributes (again these terms all are equal), 5 terms when A ′ has 4 attributes, and 1 term for when A ′ = { 1 , 2 , 3 , 4 , 5 } . First, we note that the total number of terms is 32, which is not bad because the number of cells in the 5-way marginal is 3 5 (i.e., much larger). The expression for the variance is:

So the main takeaway here is that when evaluating the variance of the 5-way marginal, the 1/9 terms in the products reduce the influence of any σ 2 A ′ for which A ′ has a small number of attributes. Hence the most important value is σ 2 { 1 , 2 , 3 , 4 , 5 } followed by the σ 2 A ′ for the 5 sets A ′ that contain 4 attributes, etc.

D Optimality Proof of ResidualPlanner

In this section, we prove the optimality of ResidualPlanner. It takes advantage of the symmetry inherent in marginals and regular loss functions.

The proof sketch is the following. Given one optimal mechanism M , we can create a variation ˜ M of that does the following. (1) ˜ M modifies each input record by applying some invertible function f i to each attribute Att i (for example, if Att i is a tertiary attribute, we can modify the value of Att i for each record using a function f i where f i (1) = 3 , f i (2) = 1 , f i (3) = 2 ). This step can be viewed as simply renaming the attribute values within an attribute. (2) Then ˜ M runs M on the resulting dataset. Note that marginals can be reconstructed from the output of ˜ M by first running the reconstruction one would do for M and then inverting the f i functions on the resulting marginals (i.e., rearranging the cells in each marginal to undo the within-attribute renaming caused by the f i ). This variation ˜ M has the same privacy properties as M and the same loss (due to the regularity condition on the loss).

Hence ˜ M is also optimal. Then we create yet another optimal privacy mechanism M ∗ that splits the privacy budget across all variations of M and returns their outputs. It turns out that the privacy cost matrix of M ∗ has eigenvectors that are equal to the rows of the residual matrices R A used by ResidualPlanner. Rewriting the privacy cost matrix of M ∗ using this eigendecomposition, we create another mechanism (the mechanism that runs the base mechanisms of ResidualPlanner) that has the same privacy cost matrix and the same value for the loss and hence is optimal.

The rest of this section explains these steps in details with formal proofs and running commentary that helps to better understand the notation and constructs in the proof.

D.1 Notation Review

We first start with a review of key notation. Recall that a dataset D = { r 1 , . . . , r n } is a collection of records. Each record r i contains attributes Att 1 , . . . , Att n a and each attribute Att j can take values a ( j ) 1 , . . . , a ( j ) | Att j | .

An attribute value a ( j ) i for attribute Att j can be represented as a vector using one-hot encoding. Specifically, let e ( j ) i be a row vector of size | Att j | with a one in component i and 0 everywhere else. In this way, e ( j ) i is a representation of a ( j ) i .

A record r with attributes Att 1 = a (1) i 1 , Att 2 = a (2) i 2 , . . . , Att n a = a ( n a ) i na can thus be represented as the kron product e (1) i 1 ⊗ e (2) i 2 ⊗···⊗ e ( n a ) i na . This vector has a 1 in exactly one position and 0s everywhere else. The position of the 1 is the index of record r .

Thus, a data vector x is a vector of integers. The value at index i is the number of times the record associated with index i appears in D .

D.2 Permutations

For each attribute Att i , let Π ( i ) be the set of permutations on the numbers 1 , . . . , | Att i | , so that each π ∈ Π ( i ) can be interpreted as a permutation (or renaming) of the attributes values of Att i . We can also view π as a function on vectors of size | Att i | that permutes their coordinates. That is, the i th coordinate of a vector y is the π ( i ) th coordinate of π ( y ) .

One can select a permutation for each attribute π (1) ∈ Π (1) , . . . , π ( n a ) ∈ Π ( n a ) and use it to define a permutation over records. This permutation maps a record represented by the kron product e (1) i 1 ⊗ e (2) i 2 ⊗···⊗ e ( n a ) i na into π (1) ( e (1) i 1 ) ⊗ π (2) ( e (2) i 2 ) ⊗···⊗ π ( n a ) ( e ( n a ) i na ) . We can think of this permutation π = ( π (1) , . . . , π ( n a ) ) as a function that independently renames each attribute value in a record. Thus this permutation can be extended to datavectors x . The value of x at the index associated with record r is the value of π ( x ) at the index associated with record π ( r ) . Another way to look at it is that π ( x ) is the histogram associated with the dataset { π ( r 1 ) , π ( r 2 ) , . . . , π ( r n ) } . This permutation can be represented as a permutation matrix W π such that W π x = π ( x ) .

We let Π = Π (1) ×···× Π ( n a ) be the set of all such permutations. We call this the space of renaming permutations since each π ∈ Π renames the values of each attribute separately.

Our first result is that permutation does not affect the privacy parameters of a mechanism.

LEMMA D.1. Let M ( x ) ≡ Bx + N ( 0 , Σ ) be a mechanism that satisfies ρ -zCDP, ( ϵ, δ ) -approximate DP, and µ -Gaussian DP. Let π be a permutation of the indices of x and W π the corresponding permutation matrix. Then M π ( x ) ≡ BW π x + N ( 0 , Σ ) satisfies ρ -zCDP, ( ϵ, δ ) -approximate DP, and µ -Gaussian DP (i.e., with the same privacy parameters).

Proof. The privacy cost pcost ( M ) of M is the largest diagonal of B T Σ -1 B . The privacy cost pcost ( M π ) of M π is the largest diagonal of W T π B T Σ -1 BW π . The effect of W π on both sides is to permute the rows and columns of B T Σ -1 B in the same way. Thus the diagonals of B T Σ -1 B and W T π B T Σ -1 BW π are the same up to permutation and hence M and M π have the same privacy cost and therefore the same privacy parameters.

The next result is that a renaming permutation preserves the accuracy of a marginal derived from the answer to a mechanism.

LEMMA D.2. Let Wkload = { A 1 , . . . , A k } be a workload on marginals. Let M ( x ) ≡ Bx + N ( 0 , Σ ) be a mechanism whose output can be used to provide unbiased estimates of those marginals. Let π ∈ Π be a renaming permutation and W π the corresponding permutation matrix. Define M π ( x ) ≡ BW π x + N ( 0 , Σ ) . Then unbiased answers to Wkload can be obtained from the output of M π and for any regular loss function L (Definition 4.1), L ( V ar ( A 1 ; M ) , . . . , V ar ( A k ; M )) = L ( V ar ( A 1 ; M π ) , . . . , V ar ( A k ; M π ))

Proof. For each set of attributes A i ∈ Wkload , let Q A i be the query matrix of the marginal (i.e., the true marginal is computed as Q A i x ). Then the best linear unbiased estimate of the marginal on A i from the output ω of M is Q A i ( B T Σ -1 B ) † B T Σ -1 ω and V ar ( A i ; M ) is the diagonal of the covariance matrix of this estimate, which is Q A i ( B T Σ -1 B ) † Q T A i . Meanwhile, the best linear unbiased estimate of the marginal on A i from the output ω ′ of M π is is Q A i ( W T π B T Σ -1 BW π ) † W T π B T Σ -1 ω ′ and V ar ( A i ; M ) is the diagonal of Q A i ( W T π B T Σ -1 BW π ) † Q T A i = Q A i W T π ( B T Σ -1 B ) † W π Q T A i .

We note that Q A i W T π is a permutation of the rows of Q A i (computing a marginal on a dataset in which attribute values within the same attribute are renamed is the same as computing the marginal on the original dataset and then renaming the marginal cells, which is permutation of the output of the marginal computation).

Therefore the diagonals of Q A i ( B T Σ -1 B ) † Q T A i and Q A i ( W T π B T Σ -1 BW π ) † Q T A i are the same up to permutation. Hence the vector V ar ( A i ; M ) is the same as the vector V ar ( A i ; M π ) up to permutation of the components, and hence does not affect a regular loss function L .

Finally, we show that there exists an optimal mechanism whose privacy cost matrix exhibits symmetries defined by the set of permutaitons Π .

LEMMA D.3. Let Wkload = { A 1 . . . , A k } be a workload of marginal queries. Let L be a regular loss function. Let U be the set of all Gaussian linear mechanisms that can provide unbiased answers to the marginals in the Wkload . Let γ be a real number. Then whenever either of the following optimization problems are feasible,

the feasible optimization problem is minimized by some mechanism of the form M ( x ) ≡ Bx + N ( 0 , Σ ) whose privacy cost matrix Γ ≡ B T Σ -1 B has the following symmetries: for all renaming permutations π ∈ Π (with W π being the associated permutation matrix), we have Γ = W T π ΓW π (in other words, permuting the rows has no effect as long as the columns are permuted in the same way).

Proof. Let M opt ( x ) ≡ B opt x + N ( 0 , Σ opt ) be an optimal mechanism to one of these problems. It may not have the required symmetries, but from it we will construct an optimal mechanism that does.

For a permutation π (and corresponding permutation matrix W π ) and a positive number λ , consider the mechanism M π,λ ( x ) ≡ B opt W π x + N ( 0 , λ Σ opt ) . By Lemma D.2, this mechanism also answers the marginals in Wkload .

Now consider the mechanism M which, on input x outputs the result of M π, | Π | for all π ∈ Π .

and the covariance matrix Σ is a block diagonal matrix with the scaled matrix | Π | Σ opt in each block. Clearly, by Lemma D.2, it also provides unbiased answers to the marginals in Wkload .

First, we claim that the pcost ( M ) ≤ pcost ( M opt ) so that the privacy parameters are at least as good. Recall pcost ( M ) is the largest diagonal entry of:

Since the privacy cost pcost ( M π, 1 ) is the largest diagonal of W T π B T opt Σ -1 opt B opt W π and equals pcost ( M opt ) , Equation 4 (and convexity of the max function) shows that the pcost ( M ) ≤ pcost ( M opt ) .

Next we consider the loss function. Let A i ∈ Wkload be a set of attributes and let Q A i be the corresponding query matrix for the marginal on A i . Then the reconstructed variances of the answers to this marginal, based on the output of M is:

For any π ∈ Π , Lemma D.2 tells us that L ( V ar ( A 1 ; M opt ) , . . . , V ar ( A k ; M opt )) = L ( V ar ( A 1 ; M π, 1 ) , . . . , V ar ( A k ; M π, 1 )) and so regularity of L (which includes convexity), means that L ( V ar ( A 1 ; M ) , . . . , V ar ( A k ; M )) ≤ L ( V ar ( A 1 ; M opt ) , . . . , V ar ( A k ; M opt )) .

Thus M is no worse in privacy or utility than M opt and hence is optimal.

Thus we consider the symmetries of the privacy cost matrix of M , which is given in Equation 4. Clearly it has the desired symmetry property that Γ = W T π ΓW π for any π ∈ Π as the permutation space Π is an algebraic group.

D.3 From permutations to interpretations

Let M opt ( x ) ≡ B opt x + N ( 0 , Σ opt ) be an optimal mechanism that has the symmetries guaranteed by Lemma D.3. Our goal is to use the symmetries in the privacy cost matrix Γ opt ≡ B T opt Σ -1 opt B opt to examine the structure of Γ opt .

If γ i,j is the ( i, j ) th entry of Γ opt and if there is a renaming permutation that maps r i (the record associated with index i ) to some r i ′ (at index i ′ ) and maps r j to some r j ′ then γ i,j = γ i ′ ,j ′ . Note that if r i and r j have the same values for attributes Att 1 and Att 2 then r i ′ and r j ′ must match on the same attributes because renaming permutations just change the names of values within each attribute. Thus we introduce notation for the set of attributes on which two records match:

DEFINITION D.4 (Common Attributes). Define ζ to be the function that takes two records and outputs the set of attributes on which they match We emphasize that ζ ( r i , r j ) is a set of attributes, not attribute values.

This discussion leads to the following result which characterizes the privacy cost matrix of an optimal mechanism.

LEMMA D.5. Under the same conditions as Lemma D.3, there exists an optimal mechanism with a privacy cost matrix Γ opt for which the following holds. In addition to the symmetry guaranteed by Lemma D.3, for every subset of attributes S ⊆ { Att 1 , . . . , Att n a } , there exists a number c S such that γ i,j , the ( i, j ) th entry of Γ opt , is equal to c ζ ( r i ,r j ) . In other words, the ( i, j ) th entry is completely determined by the set ζ ( r i , r j ) (recall r i the record value associated with index i and r j is the record value associated with index j ).

Proof. By Lemma D.3, there exists an optimal mechanism with privacy cost matrix Γ opt that is invariant under renaming permutations of its rows as long as the columns are permuted in the same way. Thus if r i is the record value corresponding to position i and r j is the record value corresponding to position j , there exists a renaming permutation that maps r i to some r i ′ and r j to some r ′ j if and only if the attributes on which r i and r j match are the same as the attributes on which r i ′ and r j ′ match each other (in symbols: ζ ( r i , r j ) = ζ ( r i ′ , r j ′ ) ). When there exists such a renaming permutation then γ i,j = γ i ′ ,j ′ . Thus the value of γ i,j is completely determined by ζ ( r i , r j ) and the result follows.

From Theorem 4.2, we know that the rows of the matrices of R A , for all A ⊆ { Att 1 , . . . , Att n a } are a linearly independent basis for R d , where d = ∏ n a i =1 | Att i | . Thus we call the rows a residual basis . DEFINITION D.6. A row vector v is a residual basis vector if it is a row in R A for some A ⊆ { Att 1 , . . . , Att n a } .

We now provide an interpretation of the residual bases. First, for an attribute Att ℓ , define the vector e ( ℓ ) i,j to be a vector of length | Att ℓ | such that the element at position i is 1, the element at position j is -1 and everywhere else is 0. In other words, e ( ℓ ) i,j = e ( ℓ ) i -e ( ℓ ) j (recall e ( ℓ ) i is 1 in position i and 0 everywhere else and is a one-hot encoding of the attribute a ( ℓ ) i ). Now, each element of the residual basis has the form v (1) ⊗···⊗ v ( n a ) where, for each ℓ , v ( ℓ ) is either the vector 1 T | Att ℓ | or a vector e ( ℓ ) 1 ,i ℓ . When the vector for attribute Att ℓ is the vector 1 T | Att ℓ | , we say that all attribute values of Att ℓ are selected . When the vector for Att ℓ is e ( ℓ ) 1 ,i ℓ , then we say attribute value a ( ℓ ) 1 is positively selected and a ( ℓ ) i ℓ is negatively selected (the other attribute values of Att ℓ are not selected at all). The attributes for which the kron term is not 1 T | Att ℓ | are called the discriminative attributes.

As an example of this notation and terminology, consider Table 8. Suppose we have three attributes: Att 1 takes values 'a' or 'b'; Att 2 takes values 'y' or 'n'; Att 3 takes values 1 or 2 or 3.

In this case, the data vector x would have 12 components. The first component corresponds to the number of appearances of record 'a,y,1' in the dataset, the second component corresponds to record 'a,y,2' and so on. The records corresponding to each index of x are listed in order as the column headings in Table 8. The first row shows the representation of record 'b,n,1' which is composed of the second value (b) for Att 1 , the second value (n) for Att 2 and the first value (1) for Att 3 . Hence its kron representation is [0 , 1] ⊗ [0 , 1] ⊗ [1 , 0 , 0] and when the kron product is evaluated, the resulting vector has a 1 in the index corresponding to 'bn1' (10th column) and 0 everywhere else.

The second and third rows show the expansions of two residual basis vectors [1 , 1] ⊗ [1 , -1] ⊗ [1 , -1 , 0] (its discriminative attributes are Att 2 and Att 3 ) and [1 , -1] ⊗ [1 , 1] ⊗ [1 , 0 , -1] (its discriminative attributes are Att 1 and Att 3 ). Consider again the kron product [1 , 1] ⊗ [1 , -1] ⊗ [1 , -1 , 0] . Note that the first part of the kron product, [1 , 1] refers to the first attribute and selects both of its values (sets them to 1). The second part of the kron product [1 , -1] refers to the Att 2 and positively selects the first attribute value 'y' (sets it to 1) and negatively selected the second attribute value 'n' (sets it to -1). The third part is [1 , -1 , 0] and it positively selects the first attribute value, negatively selects the second, but the third attribute value is not selected at all (i.e., the 3rd position is 0). These attribute selections can help us determine what the kron product looks like when it is expanded as follows. For the residual basis vector v (1) ⊗···⊗ v ( n a ) the value at the index associated with a record r is

0 if r has an attribute whose value is not selected by the residual basis vector's kron product. In this case we say the residual basis vector assigns a 0 to record r . For example, in the residual basis vector corresponding to kron product [1 , 1] ⊗ [1 , -1] ⊗ [1 , -1 , 0] , the third value of the third attribute is not selected. For any record that assigns the attribute value 3 to Att 3 , this residual basis vector assigns a 0 to such a record.

1 if for every attribute, the value assigned to it by r is selected (posititvely or negatively), and the number of negatively selected attribute values is even. In this case we say the residual basis vector assigns a 1 to record r .

-1 if the attribute value for each attribute is selected, and the number of negatively selected attribute values is odd. In this case we say the residual basis vector assigns a -1 to record r .

For example, for the residual basis vector [1 , 1] ⊗ [1 , -1] ⊗ [1 , -1 , 0] , the attribute value 3 for Att 3 is not selected. Hence the value at indices corresponding to records an3,bn3,ay3,by3 are all 0 (see Table 8). Next, consider the record an2. The value 'a' is positively selected, 'n' is negatively selected, and '2' is negatively selected. Hence all attributes are selected and an even number of attributes are negatively selected. Therefore the value at the index associated with an2 is 1. Now for the record by2. The 'b' is positively selected, 'y' is positively selected, and '2' is negatively selected. Hence there are an odd number of negative selections and so the value at the index associated with by2 is -1.

With this discussion and associated notation, we can now show that each residual basis vector is an eigenvector of the optimal privacy cost matrix, and the eigenvalue only depends on which attributes are discriminative.

̸

THEOREM D.7. Under the same conditions as Lemma D.3, there exists an optimal mechanism such that the eigenvectors of its privacy cost matrix Γ are the residual basis vectors (Definition D.6). Furthermore, if two residual basis vectors v (1) ⊗···⊗ v ( n a ) and w (1) ⊗···⊗ w ( n a ) have the same discriminative attributes (i.e., for all i , w ( i ) = 1 T | Att i | if and only v ( i ) = 1 T | Att i | ) then the two residual basis vectors have the same eigenvalues (in other words, all rows of the same residual matrix have the same eigenvalues).

̸

Proof. Recall from Definition D.4 that ζ ( r i , r j ) is the set of attributes on which r i and r j are equal.

Let Γ be the privacy cost matrix guaranteed by Lemma D.5 with the properties guaranteed by Lemma D.5, namely that for every subset of attributes S ⊆ { Att 1 , . . . , Att n a } , there exists a number c S such that γ i,j , the ( i, j ) th entry of Γ , is equal to c ζ ( r i ,r j ) - the constant associated with the set ζ ( r i , r j ) , where r i the record value associated with index i and r j is the record value associated with index j .

Let r ℓ be a record associated with index ℓ . We consider the dot product between a residual basis vector v = v (1) ⊗···⊗ v ( n a ) and the ℓ th row of Γ . Since the entries of the ℓ th row are c ζ ( r ℓ ,r 1 ) , . . . , c ζ ( r ℓ ,r d ) and the entries of v are 0,1,-1, this dot product can be expressed as:

We analyze this in three cases.

Case 1: v assigns a 0 to r ℓ . In this case, there is an attribute for which r ℓ has a value that is not selected. Without loss of generality, we may assume this is the first attribute Att 1 so that v (1) = e 1 ,i (the vector with a 1 at the first index and -1 at the i th index for some i and 0 everywhere else) and the value of Att 1 for r ℓ is therefore not a (1) 1 or a (1) i (because r ℓ got assigned 0 by v due to attribute Att 1 ). Now, if a record r appears in the left summation of Equation 5 then its value for Att 1 is either a (1) 1 or a (1) i and it does not match r ℓ on the first attribute. But this means that we can transform r into a record r ′ by replacing a (1) 1 and a (1) i with each other. This r ′ would be on the right hand side of the summation (because we are flipping the sign of the selection by v of attribute Att 1 in r ′ ). Furthermore r ′ also does not match r ℓ on Att 1 and therefore r matches r ℓ on exactly the same attributes as r ′ matches r ℓ . Thus ζ ( r ℓ , r ) = ζ ( r ℓ , r ′ ) . Thus the summation term from record r is cancelled out by r ′ in Equation 5. Using the same argument, we see that every term in the left summation is canceled out by a unique term in the right summation, and vice versa. Hence, if v assigns a 0 to record r ℓ (i.e., has a 0 in index ℓ when its kron product representation is expanded) then the dot product between v and the ℓ th row of Γ is 0.

Case 2: v assigns a 1 to r ℓ . In this case, every attribute of r ℓ has a value that is (either positively or negatively) selected by v and an even number are negatively selected. Our goal is to show that if

some other record r t is also assigned a 1 by v , then the dot product between v and ℓ th row of Γ is the same as the dot product between v and the t th row of Γ . That is, we want to show:

Let S be the set of attributes on which r ℓ and r t disagree. Now define a mapping ϕ between records such that ϕ only modifies attributes in S . For each attribute Att in S , it maps the value that record r ℓ has into the value that r t has an vice versa. (For example, suppose S = { Att 1 , Att 2 } and r ℓ has values a (1) 2 and a (2) 3 for those attributes, respectively, and suppose that r t has values a (1) 4 and a (2) 5 for those attributes. Then ϕ changes a (1) 2 in Att 1 to a (1) 4 and changes a (1) 4 into a (1) 2 ; for Att 2 it changes a (2) 3 into a (2) 5 and changes a (2) 5 into a (2) 3 . Thus ϕ ( r ℓ ) = r t and ϕ ( r t ) = r ℓ and ϕ is its own inverse. Furthermore, for any record r , ζ ( r ℓ , r ) = ζ ( ϕ ( r ℓ ) , ϕ ( r )) = ζ ( r t , ϕ ( r )) since renaming attribute values the same way in two records does not affect the set of attributes on which they match (and the last equality is because ϕ ( r ℓ ) = r t ).

We next note that since r t and r ℓ are both assigned 1 by v , then they must differ on an even number of discriminative attributes of v (if they differ on a discriminative attribute, one must have a value that is positively selected and the other must have a value that is negatively selected - there cannot be a 0 because r ℓ and r t are not assigned a 0 by v ). Therefore, due to its definition, ϕ modifies an even number of discriminative attributes and therefore for any record r , both r and ϕ ( r ) get assigned the same value by v .

Putting these facts together, we get:

and that proves Equation 6

Case 3: v assigns a -1 to r ℓ . In this case, every attribute of r ℓ has a value that is (either positively or negatively) selected by v and an odd number are negatively selected. Our goal is to show that if some other record r t is assigned a 1 by v , then the dot product between v and ℓ th row of Γ is the negative of the dot product between v and the t th row of Γ . That is, we want to show:

As in the previous case, we define ϕ in the same way and reasoning as before we see that for any record r , ζ ( r ℓ , r ) = ζ ( ϕ ( r ℓ ) , ϕ ( r )) = ζ ( r t , ϕ ( r )) and since now ϕ must change an odd number of

discriminative attributes (since r ℓ and r t are assigned -1 and 1 by v ) then for any record r , the value assigned to r by v is the negative of the value assigned to ϕ ( r ) by v . Thus we have:

and that proves Equation 7.

Thus what these 3 cases show us are that there exists some constant β such that:

If the i th position of the expansion of v is 0 (i.e., r i is assigned 0 by v ), then the i th position of Γv is also 0 (the dot product between the i th row and v is 0).

If the i th position of the expansion of v is 1 (i.e., r i is assigned 1 by v ), then the i th position of Γv is β (the dot product between the i th row and v is β ).

If the i th position of the expansion of v is -1 (i.e., r i is assigned -1 by v ), then the i th position of Γv is -β (the dot product between the i th row and v is -β ).

Thus v is an eigenvector of Γ with eigenvalue β . That proves the first part of the theorem.

The next part of the theorem is to show that if two residual basis vectors have the same discriminative attributes, then they have the same eigenvalue. So let v = v (1) ⊗···⊗ v ( n a ) and w = w (1) ⊗···⊗ w ( n a ) be two residual basis vectors that have the same discriminative attributes. Define a renaming permutation π as follows:

For an attribute Att ℓ that is not discriminative for v (and hence also not for w ), π does not rename its values (i.e., it acts as the identity for those attribute values).

For a discriminative attribute Att ℓ , let e 1 ,i ℓ be the kron component for v (i.e., v ( ℓ ) = e 1 ,i ℓ ) and let e 1 ,j ℓ be the kron component for w . Note the indices i ℓ and j ℓ are not equal to 1. In this case, we make π do the following renamings:

-a i ℓ → a j ℓ

-a j ℓ → a i ℓ

-The remaining attribute values are unchanged.

By considering which records are assigned 1,-1 and 0 by v and w , it is clear that π converts v into w (and vice versa). Let W be the matrix representation of the renaming permutation π , so that Wv = w and W T w = v (a permutation matrix is orthogonal, so its inverse is its transpose). Thus, letting β denote the eigenvalue of v with respect to Γ , we have:

= W T ΓWW T w due to the symmetry from Lemma D.3 = W T Γw , since W T is the inverse of W and so

and thus w has the same eigenvalue as v .

Thus each residual basis matrix R A has a useful property: its rows are linearly independent and are part of the same eigenspace (linear space of vectors with the same eigenvalue) of the privacy cost matrix Γ of an optimal mechanism. This allows us to prove the main result:

THEOREM 4.4. Given a marginal workload Wkload and a regular loss function L , suppose the optimization problem (either Equation 1 or 2) is feasible. Then there exist nonnegative constants σ 2 A for each A ∈ closure ( Wkload ) (the constants do not depend on the data), such that the optimal linear Gaussian mechanism M opt for loss function L releases M A ( x ; σ 2 A ) for all A ∈ closure ( Wkload ) . Furthermore, any matrix mechanism for this workload must produce at least this many noise measurements during its selection phase.

Proof of Theorem 4.4. Let ALL represent closure ( { Att 1 , . . . , Att n a } ) - all possible subsets of attributes. Theorem D.7 guarantees that there is an optimal mechanism whose privacy cost matrix Γ has eigenvectors equal to the rows of the residual matrices. Rows within the same residual matrix have the same eigenvalues. Since privacy cost matrices are symmetric positive semidefinite, this means that for every A ∈ ALL , there exists a nonnegative number β A such that:

By Theorem 3.5 of [47], if two Gaussian linear mechanisms have the same privacy cost matrix then each can be obtained by linearly processing the other. Thus they have the same privacy properties (under any postprocessing invariant privacy definition) and can be used to answer the same queries with the same exact accuracies (under any measure of accuracy). Thus we just need to construct the appropriate mechanism having privacy cost matrix Γ .

̸

For each A , let Z A be a matrix with orthonormal rows that span the row space of R A . Thus the rows of Z A are also eigenvectors of Γ (having common eigenvalue β A ) and the rows of Z A are orthogonal to the rows of Z A ′ for A = A ′ (a consequence of Theorem 4.2). Thus the set of rows of the Z A for all A ∈ ALL are a complete list of the eigenvecotrs of Γ (the are linearly independent and span R d ). Thus the (symmetric positive semidefinite) privacy cost matrix Γ can be expressed as:

and one mechanism that achieves this privacy cost matrix is the one that releases Z A x + N ( 0 , 1 β A I ) for each A ∈ ALL for which β A = 0 (i.e., we can drop the eigenvectors with eigenvalue equal to 0 as they make no difference to the privacy cost matrix).

̸

Now, since the rows of R A and Z A are independent linear bases of the same subspace, then there exists an invertible matrix Y A such that R A = Y A Z A . Furthermore, R A R T A is invertible and Z A Z T A = I by orthonormality of its rows. Therefore

Thus we have

̸

and a mechanism that achieves this privacy cost matrix is the one that releases R A x + N ( 0 , 1 β A R A R T A ) for each A for which β A = 0 .

̸

We next note that each covariance matrices we propose to use, Σ A , is proportional to R A R T A (they are equal up to positive rescaling). If we define the positive constants κ A such that R A R T A = κ A Σ A then we note that the σ 2 A in the theorem statement are equal to κ A /β A .

Next, we show that the eigenvalues β A > 0 for A ∈ closure ( Wkload ) and 0 otherwise, so that the optimal mechanism would not make use of any submechanism M A for A / ∈ closure ( Wkload ) .

̸

First, by Theorem 4.2, the rows of R A , for A ∈ closure ( Wkload ) form an independent linear basis for the space spanned by the rows of the marginals Q A for A ∈ Wkload . If a noisy R A x is not released for some A ∈ closure ( Wkload ) , then an unbiased noisy answer to at least one of the workload marginals could not be computed. Hence, they must all be part of the optimal mechanism (and thus, because of linear independence, any mechanism needs to get at least as many scalar noisy answers as this). This shows that β A > 0 for all A ∈ closure ( Wkload ) . On the other hand since the rows of R A are orthogonal to the rows of R A ′ for A = A ′ , getting answers to R A ′ x , for A ′ / ∈ closure ( Wkload ) , cannot help estimate the answers to the marginals Q A for A ∈ Wkload (by Theorem 4.2, R A ′ are orthogonal to the matrices representing these marginals when A ′ / ∈ closure ( Wkload ) ). Hence an optimal privacy mechanism cannot waste privacy budget on these irrelevant queries. This shows that β A ′ = 0 for A ′ / ∈ closure ( Wkload ) and concludes the proof.

E The other proofs about base mechanisms

̸

THEOREM 4.2. Let A be a set of attributes and let Q A be the matrix representation of the marginal on A . Then the rows of the matrices R A ′ , for all A ′ ⊆ A , form a linearly independent basis of the row space of Q A . Furthermore, if A ′ = A ′′ then R A ′ R T A ′′ = 0 (they are mutually orthogonal).

̸

Proof of Theorem 4.2. Consider two sets A ′ = A ′′ and represent there respective residual matrices as:

Since A ′ = A ′′ then one of them contains an attribute, say Att i , that the other doesn't have. Therefore either V ′ i or V ′′ i is the vector 1 T | Att i | and the other is Sub | Att i | . However, 1 T | Att i | Sub T | Att i | = 0 and Sub | Att i | 1 | Att i | = 0 and hence R A ′ R T A ′′ = 0 .

Next, for any set A ′ , it is clear that the row space of R A ′ is contained in the row space of the marginal matrix Q A ′ . It is also clear that if A ′ ⊆ A then the row space of the marginal matrix Q A ′ is contained in the row space of Q A (because Q A ′ represents a sub-marginal of Q A ). Thus the rows of the matrices R A ′ , for all A ′ ⊆ A , are contained in the rowspace of Q A . Thus we just need to show that the combined rows of R A ′ , for all A ′ ⊆ A , are linearly independent and that the number of rows is the same as the number of rows of Q A .

̸

First, each R A ′ is a kronecker product of matrices with full row rank, and so R A ′ has full row rank (therefore its rows are linearly independent). Furthermore, since R A ′ R T A ′′ = 0 whenever A ′ = A ′′ this means that the row space of R A ′ is orthogonal to the row space of R A ′′ . Hence the combined rows of the R A ′ , for all A ′ ⊆ A , are linearly independent.

̸

Next, the number of rows in R ∅ is 1 and the number of rows in R A ′ is equal to ∏ Att i ∈ A ′ ( | Att i | -1) for A ′ = ∅ and so the total number of rows in the residual matrices is 1 + ∑ A ′ ⊆ A A ′ = ∅ ∏ Att i ∈ A ′ ( | Att i | -1) .

̸

By the distributive property of multiplication, this is exactly the same as the product:

which is the number of rows in Q A and that proves that the combined rows of R A ′ , for all A ′ ⊆ A , form a linearly independent basis for the row span of Q A .

LEMMA E.1. For any i , Sub T | Att i | ( Sub | Att i | Sub T | Att i | ) -1 Sub | Att i | = I | Att i | -1 | Att i | 1 | Att i | 1 T | Att i |

Proof of Lemma E.1. For the moment, let Y denote Sub T | Att i | ( Sub | Att i | Sub T | Att i | ) -1 Sub | Att i | . Then we know:

Y is symmetric.

Y is an | Att i | × | Att i | matrix and its rank is | Att i | -1 since the rank of Sub | Att i | is | Att i | -1 .

Sub | Att i | YSub T | Att i | = Sub | Att i | Sub T | Att i | .

Now, one symmetric solution to the equation Sub | Att i | XSub T | Att i | = Sub | Att i | Sub T | Att i | is X = I | Att I | and if X 1 is another symmetric solution then Sub | Att i | ( I | Att i | -X 1 ) Sub T | Att i | = 0 .

This means that Sub | Att i | v = 0 for each eigenvector v of the symmetric matrix I | Att i | -X 1 that has a nonzero eigenvalue. Since the rank of Sub | Att i | is | Att i | -1 , the only vectors v for which Sub | Att i | v = 0 are proportional to 1 | Att i | (the null space has rank 1) and so I | Att i | -X 1 = -c 1 | Att i | 1 T | Att i | for some constant c .

This means that Y (and any other symmetric solution) has the form I | Att i | + c 1 | Att i | 1 T | Att i | . To find c , we note that Y is not full rank.

By the Sherman-Morrison-Woodbury inversion formula, if I | Att i | + c 1 | Att i | 1 T | Att i | is invertible, then its inverse is I | Att i | -c 1 | Att i | ( 1 + c 1 T | Att i | 1 | Att i | ) -1 1 T | Att i | = I | Att i | -c 1 | Att i | 1 T | Att i | 1+ c | Att i | . Thus, to prevent invertibility, we must have c = -1 / | Att i | .

THEOREM 4.5. The privacy cost of M A with noise parameter σ 2 A is 1 σ 2 A ∏ Att i ∈ A | Att i |-1 | Att i | and the evaluation of M A given in Algorithm 1 is correct - i.e., the output has the distribution N ( R A x , σ 2 A Σ A ) .

Proof of Theorem 4.5. Without loss of generality (and to simplify notation), assume A = { Att 1 , . . . , Att ℓ } consists of the first ℓ attributes.

By definition, pcost ( M A ( · ; σ 2 A )) is the largest diagonal of 1 σ 2 R T A Σ -1 A R A . Thus we can write:

Since its diagonals are | Att i |-1 | Att i | , then combined with Equation 8 it proves the result for pcost ( M A ( · , σ 2 A )) .

We next consider the correctness of Algorithm 1. First, we need to show that for the matrix H defined in Line 3 in Algorithm 1, HQ A x = R A x . Then we can write:

Next, we note that if z is distributed as N (0 , I m ) (Line 4 in Algorithm 1) then σ A Hz has the distribution N (0 , σ 2 HH T ) = Σ A and hence the algorithm is correct.

F Proofs related to the reconstruction step

LEMMA 4.6. For any Att i , let ℓ = | Att i | . The matrix Sub ℓ has the following block matrix, with dimensions ℓ × ( ℓ -1) , as its pseudo-inverse (and right inverse): Sub † ℓ = 1 ℓ [ 1 T ℓ -1 1 ℓ -1 1 T ℓ -1 -ℓ I ℓ -1 ] .

Proof of Lemma 4.6. First, if a matrix has a right inverse then that is the pseudo-inverse. Hence we just need to show that Sub ℓ Sub † ℓ = I ℓ -1 .

Note that the j th row of Sub ℓ has a 1 in position 1, -1 in position j +1 , and is 0 everywhere else.

Meanwhile, the i th column of our claimed representation of Sub † ℓ has a -( ℓ -1) /ℓ in position i +1 and 1 /ℓ everywhere else.

̸

Hence if j = i then the dot product between row j of Sub ℓ and column i of Sub † ℓ is 0 since the nonzero elements of the row from Sub ℓ are being multiplied by 1 /ℓ and 1 /ℓ .

If i = j then the corresponding first elements that are multiplied are 1 and 1 /ℓ while the elements at position i +1 being multiplied are -1 and -( ℓ -1) /ℓ . Furthermore, 1(1 /ℓ ) + ( -1)( -( ℓ -1) /ℓ ) = 1 .

LEMMA F.1. For any attribute Att i , let ℓ = | Att i | . Then Sub † ℓ ( Sub ℓ Sub T ℓ ) Sub † T ℓ = I ℓ -1 ℓ 1 ℓ 1 T ℓ

Proof of Lemma F.1. Because Sub ℓ has linearly independent rows, the pseudo-inverse of it can be expressed as,

From lemma E.1 we get,

Therefore,

THEOREM F.2. Let A be a set of attributes and let Q A be the matrix representation of the marginal on A . Given the matrices R A ′ , for all A ′ ∈ closure ( A ) , we have Q A = ∑ A ′ ∈ closure ( A ) Q A R † A ′ R A ′ .

Proof of Theorem F.2.

It is straightforward to verify that the following is a right inverse (and hence pseudo-inverse) of R A ′

Because Sub | Att i | has linearly independent rows, the pseudo-inverse of it can be expressed as,

From lemma E.1 we get,

Therefore,

Without loss of generality (and to simplify notation), assume A = { Att 1 , . . . , Att ℓ } consists of the first ℓ attributes,

Because of the distributive property of the Kronecker product,

Therefore, combining everything together,

THEOREM 4.7. Given a marginal workload Wkload and positive numbers σ 2 A for each A ∈ closure ( Wkload ) , let M be the mechanism that outputs {M A ( x ; σ 2 A ) : A ∈ closure ( Wkload ) } and let { ω A : A ∈ closure ( Wkload ) } denote the privacy-preserving noisy answers (e.g., ω A = M A ( x , σ 2 ) ). Then for any marginal on an attribute set A ∈ closure ( Wkload ) , Algorithm 2 returns the unique linear unbiased estimate of Q A x (i.e., answers to the marginal query) that can be computed from the noisy differentially private answers.

The variances V ar ( A ; M ) of all the noisy cell counts of the marginal on A is the vector whose components are all equal to ∑ A ′ ⊆ A ( σ 2 A ′ ∏ Att i ∈ A ′ | Att i |-1 | Att i | ∗ ∏ Att j ∈ ( A / A ′ ) 1 | Att j | 2 ) . The covariance between any two noisy answers of the marginal on A is ∑ A ′ ⊆ A ( σ 2 A ′ ∏ Att i ∈ A ′ -1 | Att i | ∗ ∏ Att j ∈ ( A / A ′ ) 1 | Att j | 2 ) .

Proof of Theorem 4.7. We first verify the correctness and uniqueness of the reconstruction in Algorithm 2. Uniqueness follows from the fact that the rows from all the matrices R A (for A ∈ closure ( Wkload ) ) are linearly independent.

Consider Line 3 from Algorithm 2. It uses a U matrix that depends on both the attributes A of the marginal one wants to compute and a subset A ′ of it. So, for notational dependence, we write it as U A ← A ′ . It is straightforward to verify that U A ← A ′ = Q A R † A ′ . From Theorem F.2, Q A x = ∑ A ′ ⊆ A Q A R † A ′ R A ′ x = ∑ A ′ ⊆ A U A ← A ′ R A ′ x , and so Algorithm 2 is correct because each ω A ′ is an unbiased noisy version of R A ′ x .

Having established that the q returned by Line 5 in Algorithm 2 is an unbiased estimate of the marginal query answer Q A x , the next step is to compute the covariance matrix E [ qq T ] .

Without loss of generality (and to simplify notation), assume A = { Att 1 , . . . , Att ℓ } consists of the first ℓ attributes, A ′ = { Att 1 , . . . , Att t } consists of the first t ≤ ℓ attributes, then A / A ′ = { Att t +1 , . . . , Att ℓ } .

By definition, V ar ( A ; M ) is the diagonal of E [ qq T ] = ∑ A ′ ∈ closure ( A ) σ 2 A ′ U A ← A ′ Σ A ′ U T A ← A ′ . Thus we can write:

Now, by Lemma F.1,

So the diagonals of U A ← A ′ Σ A ′ U T A ← A ′ can be computed by multiplying | Att i |-1 | Att i | for each Att i ∈ A ′ and 1 / | Att j | for each Att j ∈ A \ A ′ . Meanwhile, the off diagonals are all the same and can be computed by multiplying -1 | Att i | for each Att i ∈ A ′ and 1 | Att j | 2 for each Att j ∈ A \ A ′ .

Computing the variance and covariance of the marginal query answer is therefore the summation of these quantities for all A ′ ⊆ A and is what the theorem states.

G Computational Complexity Proofs

THEOREM 4.8. Let n a be the total number of attributes. Let # cells ( A ) denote the number of cells in the marginal on attribute set A . Then:

Expressing the privacy cost of the optimal mechanism M ∗ as a linear combination of the 1 /σ 2 A values takes O ( ∑ A ∈ Wkload # cells ( A )) total time.

Expressing all of the V ar ( A ; M ∗ ) , for A ∈ Wkload , as a linear combinations of the σ 2 A values can be done in O ( ∑ A ∈ Wkload # cells ( A )) total time.

Computing all the noisy outputs of the optimal mechanism (i.e., M A ( x ; σ 2 A ) for A ∈ closure ( Wkload ) ) takes O ( n a ∑ A ∈ Wkload ∏ Att i ∈ A ( | Att i | +1) ) total time after the true answers have been precomputed (Line 1 in Algorithm 1). Note that the total number of cells on marginals in Wkload is O (∑ A ∈ Wkload ∏ Att i ∈ A | Att i | ) .

Reconstructing marginals for all A ∈ Wkload takes O ( ∑ A ∈ Wkload | A | # cells ( A ) 2 ) total time.

Computing the variance of the cells for all of the marginals for A ∈ Wkload can be done in O ( ∑ A ∈ Wkload # cells ( A )) total time.

Proof of Theorem 4.8. First we establish that | closure ( Wkload ) | ≤ ∑ A ∈ Wkload # cells ( A ) . Given an set A ∈ Wkload , we note that it has 2 | A | subsets, so that | closure ( A ) | = 2 | A | . However, # cells ( A ) is at least 2 | A | (because each attribute has at least 2 attribute values). We also note that closure ( Wkload ) = ⋃ A ∈ Wkload closure ( A ) . Hence

To analyze the time complexity of symbolically representing the privacy cost, as a linear combi- nation of the 1 /σ 2 A values (for all A ∈ closure ( Wkload ) ) we note that the coefficient of 1 /σ 2 A is ∏ Att i ∈ A | Att i |-1 | Att i | . Thus computing the coefficient 1 /σ 2 ∅ takes O (1) time. Then, computing the coefficient of 1 /σ 2 { Att i } can be computed from the coefficient of 1 /σ 2 ∅ in O (1) additional time. Thus, we if go level by level, first computing the coefficients of 1 /σ 2 A with | A | = 1 then for | A | = 2 , etc. then computing the coefficient for each new A takes incremental O (1) time. Thus the overall time is O ( | closure ( Wkload ) | ) and therefore is O ( ∑ A ∈ Wkload # cells ( A )) .

Let n cells = ∑ A ∈ Wkload # cells ( A ) To express the variance symbolically as a linear function of the σ 2 A values via Theorem 4.7, we note from the previous part that computing ∏ ∈ ′ | Att i |-1 | Att i | for for all A ∈ closure ( Wkload ) also take total O ( n cells ) time. Once this is pre-computed, then for any A ′ ⊆ A ∈ closure ( Wkload ) , the product ∏ Att i ∈ A ′ | Att i |-1 | Att i | ∗ ∏ Att j ∈ ( A / A ′ ) 1 | Att j | 2 can be computed in O (1) time since A \ A ′ ∈ closure ( Wkload ) . Now, V ar ( A ; M ∗ ) = ∑ A ′ ⊆ A σ 2 A ′ ∏ Att i ∈ A ′ | Att i |-1 | Att i | ∗ ∏ Att j ∈ ( A / A ′ ) 1 | Att j | 2 . This is a linear combination of 2 | A | terms

Att i A all A ′ ∈ closure ( Wkload ) can be done in total O ( n cells ) time. Similarly, computing ∏ Att i ∈ A ′ 1 | Att i | 2 ′

(one term for each variable σ 2 A ′ for A ′ ⊆ A ). Each term is computed in O (1) time after the precomputation phase. Thus the symbolic representation of V ar ( A ; M ∗ ) takes O (2 | A | ) time (which is at most the number of cells in the marginal on A ) time after precomputation. Thus computing V ar ( A ; M ∗ ) for all A ∈ Wkload can be done in total O ( n cells ) time after precomputation, but precomputation also takes O ( n cells ) time. Thus the overall total time is O ( n cells ) .

We next analyze the time it takes to generate noisy answers once the true answers have been precomputed (Line 1 in Algorithm 1). This involves (1) computing the product Hv in the algorithm, (2) generating one Gaussian random variable for each column of H and (3) computing Hz . Now, the first and third steps take the same amount of time. The second step generates one Gaussian for each row of H and hence, for each M A takes time Π Att i ∈ A ( | Att i | -1) .

For the first step, the fast kronecker-product multiplication algorithm (Algorithm 1 of [38]) has the following complexity. Given a kronecker product of ℓ matrices of sizes ( m 1 -1) × m 1 , . . . , ( m ℓ -1) × m ℓ and a vector with m 1 ×···× m ℓ components, their algorithm has ℓ iterations. In iteration i , the i th matrix (with size m i -1 × m i ) is multiplied by a matrix with shape ( m i , ∏ i -1 j =1 m j ∗ ∏ ℓ j = i +1 ( m j -1)) . In our case, each m i is a subtraction matrix with two nonzero elements in each row. Thus, in each iteration, the product makes 2 ∏ i -1 j =1 m j ∗ ∏ ℓ j = i ( m j -1) scalar multiplication operations. There are ℓ iterations, so the multiplication algorithm uses O ( ℓ ∏ ℓ m i ) multiplications.

i =1

Now, to run algorithm M A , the number of kron products ℓ is | A | and each m i is | Att i | for Att i ∈ A . Hence the running time of M A is O ( | A | ∏ Att i ∈ A | Att i | ) which is at most | A | times the number of cells in the marginal on A . Note that the constant in the big-O notation is bounded across all A . Next, when adding up the complexity across all A ′ ∈ closure ( A ) , we can replace | A ′ | with | A | , and then the summation looks like the product ∏ Att i ∈ A ( | Att i | +1) when this product is expanded. Hence the time to run all Q A ′ for all A ′ ∈ closure ( A ) is O ( | A | ∏ Att i ∈ A ( | Att i | +1)) . Adding up over all

A ∈ Wkload gets the results.

Next we consider the reconstruction phase. Using the same analysis of the fast kron-product vector multiplication, we see that in each iteration of Algorithm 2, there is a kron product vector multiplication. Using similar reasoning as for the previous item, each such multiplication takes O ( | A | ∏ Att i ∈ A ) | Att i | = O ( | A | # cells ( A )) time. The number of iterations in the algorithm is 2 | A | ≤ # cells ( A ) . Thus the overall runtime is O ( ∑ A ∈ Wkload | A | # cells ( A ) 2 ) .

Finally, the variance computation is no harder than expressing the V ar ( A ; M ∗ ) as linear combinations of the optimization variables and we have shown this to be O ( n cells ) .

H Closed Form Solution to the Weighted Sum of Variances Loss

By Theorem 4.5, the privacy cost is a linear combination of the 1 /σ 2 A values. By Theorem 4.7, each reconstructed marginal's cell variances are a linear combination of the σ 2 A values. Thus, minimizing the weighted sum of reconstructed marginal variances subject to the privacy cost being ≤ c can be formulated as a problem of the following type:

where the v A are the linear coefficients of the σ 2 A and the p A are the linear coefficients of the 1 /σ 2 A in the privacy cost. The closed form solution is given by hte following lemma.

LEMMA H.1. Given the optimization problem in Equation 12 The optimal objective function value is T = (∑ A √ v A p A ) 2 /c , the optimal value of each noise scale parameter is σ 2 A = √ Tp A / ( cv A ) .

Proof. Clearly, for the optimal solution, the inequality constraint must be tight (i.e., = c ) because if it is not tight, we can lower variance while increasing privacy cost by dividing each σ 2 A by a number > 1 . Thus we just need to solve the problem subject to ∑ A p A /σ 2 A = c .

From Cauchy-Schwarz inequality,

Equality holds when v A p A σ 4 A = t for all A (for some constant t ). Since c = ∑ A p A σ 2 A = ∑ A √ v A p A /t , then we must have t = T/c . Plugging this into the definition of t , we get σ 2 A = √ Tp A / ( cv A ) .

Thus, if the loss function is the weighted sum of variances, ResidualPlanner does not need any optimization steps. The selection of the noise scales and the reconstruction phase are direct algorithms.

I Additional Experiments

In this section, we present additional experiments. Following [37], the experiments use the following type of workloads:

All k -way marginals.

All ≤ 3 -way marginals. This includes all 0 -way marginal (the total sum), all 1 -way marginals, all 2 -way marginals, and all 3 -way marginals.

Small marginals. This includes any k -way marginal that has at most 5000 cells.

We also use these metrics:

RMSE: The total variance is the sum of the variances of the reconstructed cells in each marginal in the workload. Root Mean Squared Error is obtained by taking the total variance, dividing by the total number of cells in the workload marginals, then taking the square root. The SVD Bound (SVDB for short) [31] provides a theoretical lower bound on RMSE for any matrix mechanism. For marginals, the SVDB is tight, but its computation is not scalable.

MaxVar: compute the variance of each reconstructed cell for each marginal in the workload, then take the maximum of these.

Running time (in seconds) of the different stages of the algorithms (select and reconstruct).

Unless otherwise stated, ResidualPlanner uses the open-source ECOS optimizer [14] for solving the optimization problem it generates for the select step.

For all experiments, we require all mechanisms to have privacy cost pcost ( M ) = 1 . By definition 2.3, M satisfies ρ -zCDP with ρ = 1 / 2 [46] and satisfies µ -Gaussian DP with µ = 1 [15, 46].

Each experiment is repeated 5 times, we report the mean value of these 5 results and a confidence interval consisting of ± 2 standard deviations. This is most useful for running time, as the variance loss metrics have negligible variance across all algorithms.

I.1 Scalability

In this section, we study the scalability of ResidualPlanner. This is done using the Synth -n d dataset, where d is the number of attributes and n is the domain size of each attribute. We use all ≤ 3 -way marginals as a fixed workload and vary n or d to get the computation time for HDMM and ResidualPlanner.

I.1.1 Varying Attribute Domain Size n in the Selection Step.

This experiment considers what happens when the attribute domain size n get larger. We fix the number of attributes d = 5 and vary the domain size n for each attribute, where n ranges from 2 to 1024. We evaluate the running time and accuracy of the selection step

Table 9 shows the running time for the selection step of HDMM and ResidualPlanner. The RMSE on the workload that the selection step guarantees is also measured. Both HDMM and ResidualPlanner

have no trouble here. HDMM is nearly optimal in RMSE and ResidualPlanner is optimal, as shown by agreement with the SVD Bound. ResidualPlanner is faster, but both methods are fast in this experiment setting.

Table 10 shows the running time and Max Variance comparison for the selection step. HDMM can only optimize for RMSE, not max variance, so this table shows that RMSE is not a good substitute when one needs to optimize for Max Variance.

I.1.2 Impact of varying the number of attributes in the Selection Step.

Next, we fix the domain size of each attribute to be n = 10 and vary the number of attributes d , where d ranges from 2 to 200. This experiment can test some of the limits of ResidualPlanner. While HDMMcannot perform selection when the number of attributes is 20 or larger, ResidualPlanner has no trouble optimizing RMSE even for 200 attributes. However, optimizing for Max Variance is much more difficult. ResidualPlanner can do this for d = 100 but the underlying optimization took more than 1 hour for d = 200 and we killed the process.

Table 11 shows the running time and RMSE comparison for the selection step. The running time of HDMMincreases sharply and it quickly runs out of memory. At the same point, the SVD Bound can no longer be computed. Meanwhile, ResidualPlanner continues to run efficiently.

Table 12 shows the running time and Max Variance comparison on the Selection step. Optimizing for Max Variance is much harder for ResidualPlanner compared to RMSE and we killed the process for d = 200 . Meanwhile, HDMM is not able to run at d = 20 (we emphasize again, it optimizes for RMSE even if one cares about Max Variance). There is an interesting phenomenon with HDMM that takes place for d between 8 and 15 . In this case, HDMM always produces a max variance of 1000. This maximum is always achieved for the sum query (a zero-dimensional marginal) for the following reason. For d beween 8 and 15 , HDMMdecides to add noise to all 3-way marginals and nothing else (even though the workload is all ≤ 3 marginals). The privacy loss budget is split equally among them. Thus, each of the ( d 3 ) marginals it measures gets N (0 , ( d 3 ) ) noise. The sum query gets reconstructed

as follows. For any single noisy 3-way marginal, one can estimate the sum by adding up the cells in the marginal. Since each cell has variance ( d 3 ) and there are n 3 = 1 , 000 cells, the sum estimate from a single 3-way marginal has a variance of 1000 ( d 3 ) . But one can obtain an independent estimate to the sum query from each of the ( d 3 ) noisy 3-way marginals. By averaging these noisy estimates, one can obtain an estimate of the sum query with variance 1 , 000 .

I.1.3 Scalability of the Reconstruction Step.

We conduct similar experiments, but now we measure the time in the reconstruction step. To complement the reconstruction scalability experiments from the main paper on the Synth -n d synthetic dataset, we first fix the number of attributes d = 5 and vary the domain size n for each attribute, where n ranges from 2 to 512. The reconstruction time for ResidualPlanner does not depend on the metric that the select step was optimized for. Again we compare with HDMM [38] and a version of HDMMwith improved reconstruction scalability called HDMM+PGM [38, 41] (the PGM settings used 50 iterations of its Local-Inference estimator, as the default 1000 was too slow). Table 13 shows the results. Again, at some point HDMM runs out of memory while ResidualPlanner runs efficiently. HDMMruns of out memory because of choices it had made in the selection step. When n = 128 it decided to measure a 5-way marginal, which is so large (requiring 128 5 space) that it caused HDMM and HDMM+PGM to have memory issues.

We next fix n = 3 and vary d . Table 14 shows ResidualPlanner is clearly faster. Furthermore, HDMM and HDMM+PGM are hampered by the failure of the selection step (when selection fails, there is nothing to reconstruct). It is interesting to compare HDMM+PGM behavior when n = 3 in Table

14 with n = 10 in Table 2 from the main paper. Clearly HDMM+PGM is faster for n = 10 than n = 3 . This counterintuitive result can be explained by the complex workings of HDMM as follows. When n = 3 , the selection step in HDMM returns some 4-way marginals. But when n = 10 , HDMM only returns ≤ 3 -way marginals. The 4-way marginals make the reconstruction step harder for both HDMMand HDMM + PGM.

I.2 Comparison on Real Datasets.

In this section, we compare RMSE and Max Variance on the real datasets: CPS, Adult, and Loans. The different workloads are 1-way, 2-way, 3-way, 4-way, 5-way marginals, all ≤ 3 -way marginals, and Small Marginals.

I.2.1 RMSE Comparisons

We provide an expanded comparison of RMSE on the 3 real datasets from the main paper. Here we add more workloads. Table 15, 16 and 17 show the comparison of RMSE on the CPS, Adult, and Loans datasets respectively.

We notice that ResidualPlanner matches the theoretical SVD Bound while HDMM is slightly worse, but still accurate. We conclude that when optimizing RMSE, the main advantage of ResidualPlanner is superior scalability.

Since ResidualPlanner is optimal, the purpose of the accuracy comparisons is a sanity check. For RMSE, we compare the quality of ResidualPlanner to the theoretically optimal lower bound known as the SVD bound [31] (they match, as shown in Table 18).

I.2.2 Max Variance

The next comparison is on optimization for Max Variance. We repeat that HDMM only optimizes for RMSE and this shows that optimizing for RMSE is highly suboptimal when one cares about max variance.

In contrast to RMSE, where the optimization problem generated by ResidualPlanner's selection step can be solved in closed form, for Max Variance, the optimization needs a convex solver. Hence we include comparisons between the open source ECOS [14] optimizer to the commercial Gurobi optimizer [21]. Thus, our results have columns labeled ResidualPlanner+ECOS and ResidualPlanner+Gurobi.

Tables 19, 20 and 21 show the results for the CPS, Adult, and Loans datasets, respectively. There is one item to note about numerical stability. Although Gurobi is generally faster and more numerically stable, the differences do not matter much. Situations where EOCS was worse are highlighted in red. For example, in Table 19 for the CPS dataset, the dataset has only 5 attributes, so a 5-way marginal is basically the entire dataset. The optimal mechanism for 5-way marginals simply adds N (0 , 1) noise to each cell and optimizing for RMSE is equal to optimizing Max Variance for this special case. As we see, the Max Variance for ResidualPlanner+ECOS is 1 . 008 which is 0 . 8% worse than optimal. The reason for this is the numerical precision with which ECOS can solve the optimization problem that ResidualPlanner gives it. In general, however, it looks like open source optimizers should work fairly reliably for them to be used in real applications of ResidualPlanner.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies a timely problem of improving the scalability of differentially private query releases using a matrix mechanism. The central approach is to carefully design a set of the residual basis for the representation of workload queries, such that the workload queries could be represented as Kronecker products of block matrices with small sizes. This design of basis representation allows subsequent reconstruction steps of the matrix mechanism to be computed in a closed-form, efficient manner. Consequently, this allows a significantly more efficient reconstruction step (from noisy query answers) than prior matrix mechanisms. The efficiency is proved in computation complexity and empirically shown through improved runtime compared to previous matrix mechanisms via extensive experiments. Finally, the authors show that their mechanism easily extends to different objectives (such as a weighted sum of variance and max variance) for query release. They also illustrate the improved empirical error of the proposed mechanism for the max variance objective (which is not optimized by prior matrix mechanisms).\n\n\n**Minor comments:** Some statements need to be clarified. E.g., in theorem 4.5, what does \" is correct\" mean? Does it mean the output is an unbiased estimator?\n\nSTRENGTHS:\n- Significantly improved efficiency of the matrix mechanism. The idea of designing a careful set of query representation basis is novel. The reason for efficiency improvements (via more convenient pseudo-inverse matrix computation for Kronecker product of known block matrices) is also explained in detail. \n- The proposed mechanism is general and extends to different objectives for private query release (such as max variance). By contrast, prior works mainly focus on the sum of variance objectives. However, the significance of this generalized mechanism needs to be clarified (see question Q2 for more details).\n\nWEAKNESSES:\n- The comparison with prior matrix mechanisms sometimes needs more clarity. The improvement and the necessary cost for improvement need more clarification. -- The proposed mechanism seems to sacrifice the accuracy of higher-order marginals for lower-order marginals. Is this an unavoidable cost for improved efficiency? See question Q1 for more details.\n- The difficulty and significance of extending the matrix mechanism to more general objectives, such as max variance, need to be clarified. See question Q2 for more details.\n\nQUESTIONS:\nQ1: Theorem 4.7 shows that the variance for the proposed mechanism is larger as the number of attributes for marginal increases. In other words, 3-way marginals have a larger variance than 2-way marginals. Is that correct? If so, how does this compare to the original matrix mechanism (where I suppose the variance for 3-way and 2-way marginals are the same)? Are we sacrificing the accuracy of higher-order marginals for lower-order marginals? Is this unavoidable?\n\nQ2: One of the main contributions of the paper is the investigation of other objectives (besides the sum of variance) for matrix mechanisms in a scalable manner. However, why would optimizing for other objectives be difficult for prior matrix mechanisms? Maybe the authors could clarify more regarding this to help understand the significance of this generalized mechanism for different objectives.\n\nQ3: The optimality statement Theorem 4.4 puzzles me, as it seems only about an optimal number of noisy answers. In that sense, all previous matrix mechanism-based solutions are also optimal. (As they answer the same number of noisy queries as ResidualPlanner in this paper.) In that sense, I wonder whether there is an optimality improvement. Additionally, could the authors comment on the optimality of ResidualPlanner in terms of the optimization objective, e.g., the smallest sum of variance (weighted) or max variance, among all matrix mechanisms?\n\nLIMITATIONS:\nThe paper achieves significant scalability improvement for the matrix mechanisms, and uses an interesting idea of optimizing noise added to different marginals for the overall query release objective. However, it needs to be clarified whether the algorithm inevitably generates more noisy answers to higher-order marginals. The authors also adequately discussed the limitation of marginal queries and possible future works of generalizing the mechanisms for other noise distributions in the limitation section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nINTRODUCTION:\nThis paper presents significant advancements in the matrix mechanism, a technique used for releasing differentially private linear queries or count statistics. Releasing these types of queries, for instance, “count of people living in NYC who are male” (also known as a 2-way marginal query), is a fundamental task in the differential privacy community. The proposed matrix mechanism is considerably more efficient compared to previous approaches.\n\nBASICS OF MATRIX MECHANISM:\n\nIn differential privacy, when given a collection of linear queries, a simple approach is to add independent Gaussian noise to each query response. The scale of this noise is tuned to the sensitivity of the queries, which denotes the amount of change in the output corresponding to alterations in individual data points. Informally, this implies that as the number of queries to be released increases, more noise must be added to each query answer to maintain privacy.\n\nROLE OF MATRIX MECHANISM:\n\nThe matrix mechanism is designed to reduce the amount of added noise for achieving differential privacy, aiming for better query estimates while preserving the same level of privacy. This mechanism operates by refactoring the set of queries and constructing a function with lower sensitivity, yet retaining most information from the original query set. This is effective because queries are often correlated, and this refactoring process capitalizes on such correlations. However, matrix mechanisms traditionally face computational inefficiencies in high-dimensional settings, as they require performance-intensive matrix operations.\n\nEXISTING SOLUTION AND ITS LIMITATION:\n\nThe High-Dimensional Matrix Mechanism (HDMM) is a notable prior attempt to enhance the scalability of the original matrix mechanism. Despite its significant improvements, HDMM has limitations. Specifically, HDMM struggles to approximate 3-way marginal queries on datasets comprising more than 10 attributes (each with a cardinality of 10).\n\nNEW CONTRIBUTION:\n\nThis paper breaks new ground by proposing an efficient and scalable method named 'ResidualPlanner.' This method can estimate marginal queries on datasets with 100 features (each with a cardinality of 10), outperforming the previous HDMM method. The scalability and speed of ResidualPlanner underscore a significant advancement in the application of matrix mechanisms for differential privacy. Moreover, this method handles optimizing over different loss functions. For example, prior work only optimizes over the square error over all queries, whereas this method allows for optimizing over max error.\n\nSTRENGTHS:\nThe ResidualPlanner heralds a substantial advancement in the release of linear statistical queries, an important task in the field of differential privacy. Notably, it enhances efficiency by facilitating the release of an exponentially higher number of linear queries across high-dimensional datasets, compared to its predecessor, the HDMM method. This accomplishment bears significant practical implications for the field of differential privacy, given that real-world datasets are often high dimensional.\n\nThe innovation behind ResidualPlanner's improved computational performance lies in a novel technique that produces a compact representation of the query matrix, Q. This approach deviates from the traditional matrix mechanism, which tends to be overparameterized and consequently computationally intensive. By avoiding this overparameterization, ResidualPlanner yields improved computational efficiency.\n\nWEAKNESSES:\nCertain sections of this paper have a high degree of technicality, which poses readability challenges. Thus, the paper could greatly benefit from providing more intuitive explanations of their techniques. For instance, the definition and context of the Kronecker product are noticeably absent, which detracts from the clarity of the relevant sections.\n\nAdditionally, the discourse on Algorithm 1 is notably brief. The paper lacks a clear explanation of the method's purpose and its necessity in the overall context. Providing further elaboration on these aspects would greatly improve the paper's readability and comprehension.\n\nQUESTIONS:\nIn line 69, is there a typo in the example Q_{Att_2}?\n- Line 130, Typo: “privcy”.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies scalability of matrix mechanism when answering marginal queries. The paper looks at a restricted yet large enough class which they call regular loss function, satisfying three key properties. Scalability of the matrix mechanism is often poor as it requires heavy SVD computation to compute the matrix inverses. This has been studied by several researchers since the inception of matrix mechanism in 2010, and often heuristics are proposed that provides some degree of scalability. The subject of this paper is to put it on a more firm ground.\n\nSTRENGTHS:\nAs I discussed above, the matrix mechanism has been often suggested as a mechanism that provides almost optimal accuracy guarantees (in terms of means-squared error). However, one major challenge, that several past work has not adequately addressed and becomes problematic if the problem size grows even in the medium range, is the scalability issue. This is because, without any structure in the queries, finding optimal factorization to be used in the matrix mechanism requires either solving a semidefinite program or solving a generalized regression problem, whose complexity is cubic in the size of the dimension. The scalability issue is to an extent that even considering a 2-marginal on a medium size dataset leads to out-of-memory on really high-powered clusters. \n\nTo address these scalability issues, several recent works have proposed using some heuristics, like removing the consistency constraints. This leads to a suboptimal matrix mechanism. This paper suggests the use of a new mechanism that can optimize over a large class of convex objective functions (the ones that are regular) and return solutions that are optimal under Gaussian noise addition. The matrix mechanism consists of designing two matrices: strategy matrix and reconstruction matrix. The authors propose a new strategy matrix which is in the form of a tensor product of $Sub_d$ matrix. This construction only works in full-rank structure, so they perform a Gaussian elimination to get linear independence.\n\nWEAKNESSES:\nBefore stating the weakness section, I would like to make a disclaimer that I have been unable to read the supplementary material, so if there are some questions that I raised here and are present in the supplementary material, please just state the page number and I will have a look.\n\n1. The authors mention that their result is optimal for the Gaussian mechanism. I am unsure what this statement means. Is it the case that any other mechanism will add Gaussian noise with a larger variance than what they suggest? I am a little skeptical of this statement considering that their strategy matrix is a fixed matrix and then their reconstruction matrix just depends on the inverse of the strategy matrix. This is true for a given strategy matrix, but why does this hold for an arbitrary strategy matrix?\n\n2. They design their algorithm under the assumption that the loss function is convex and regular; however, HDMM is more general. Can we simply not add another constraint in HDMM that takes into account the regularity of the loss function and achieve competitive run time? Is there a fundamental reason why that cannot be done? \n\n3. The error of the matrix mechanism is at least the trace norm of the query matrix (the SVD bound). So, to me, it feels like, for the worst case k-marginal, the mean squared error would scale like $\\Omega((d/k)^{2\\sqrt{k}})$. I might be making a mistake in my calculation, but translating it to \\ell_\\infty error, this sounds about right (comparing with the known upper bound result from ICALP 2012 paper making it tight). Is the regularity assumption the one that gets you around this bound?\n\n4. I really would like to see a concrete error bound (at least for some structured marginal queries that satisfy their requirements) to get a feel of their bound. Can the author provide me with one? \n\n\n> I am currently keeping my score to just below acceptance, but I am highly inclined to move it above the threshold and would definitely do that if the authors satisfactorily answer my questions (and other reviewer's questions)\n\nI revised my score to \"Accept\" based on the rebuttal and upon revisiting the paper once again post-rebuttal. I thank the authors for clarifying some of my misunderstandings in the first reading of the paper.\n\nQUESTIONS:\nPlease see above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a scalable and optimal, under some conditions, matrix mechanism, which can compute private marginals.  Releasing private marginals is one of the most fundamental problems in private data analytics, and as mentioned in the paper, an important primitive in synthetic data generation.  This work seems to improve on the existing work in every dimension, from scalability, generalizability, and accuracy.\n\nSTRENGTHS:\nThe paper does a great job of presenting the related work and the underlying problem they want to solve.  The results are pretty amazing when compared to HDMM.  ResidualPlanner is more scalable, more accurate, and works for more general loss functions.   The paper has a nice outline that follows the general framework of the matrix mechanism, starting with the select step, followed by measure and reconstruct steps.  One takeaway from the table of results that was not mentioned is that ResidualPlanner also tends to have less variability in its run time\n\nWEAKNESSES:\nThe paper is very notation heavy and the provided rationale behind the base mechanisms is lacking. I would have liked to have seen more discussion about why the general linear Gaussian Mechanism is necessary and why not just use an identity covariance and matrix B.  Furthermore, I would have liked to have seen more discussion on why we need more general loss functions and what applications they would be needed, in particular are other loss functions important for synthetic data?  I don’t think Table 3 is necessary in the paper, so that could be removed and more discussion could be added regarding the points above.  I like how there is some result on optimality, but I find it to be a little weak, in that it restricts optimality to linear Gaussian mechanisms.  Although this is a very general class of mechanisms, I would have liked to have seen some optimality guarantee for all DP or Gaussian DP mechanisms.  I am curious about the two problems that were presented in equations (1) and (2) where (1) follows the general approach of differential privacy (maximize utility subject to privacy), while (2) seems to follow something closer to “ex-post” privacy from Ligett, Neel, Roth, Waggoner, Wu ’17, where we want to return the best privacy subject to some accuracy constraint.\n\nQUESTIONS:\nIs there some connection between optimality for linear Gaussian mechanisms and optimality for Gaussian DP that would strengthen Theorem 4.4?\n\nDoes solving (2) provide some ex-post privacy guarantee, as in Ligett et al. ’17?\n\nIn the computational complexity section, it claims that the time complexity does not depend on the universe size, but how can that be if we are looking at all the k-way marginals with k<= 3?  If we keep the marginals the same but then increase the universe size in each dimension, I would expect the time complexity to increase.\n\nIs the time in Tables 1 and 2 in minutes or seconds?\n\nLIMITATIONS:\nThe paper points out some limitations.  Another limitation is that this assumes that each user can contribute at most one record (at least for user-level privacy).  How would bias introduced by clipping the number of records per user impact overall utility in this approach since this work is wanting to return unbiased results.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}