SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For any subset  $\mathbf{A} \subseteq \{Att_1, \ldots, Att_{n_a}\}$  of attributes, we define the residual matrix  $\mathbf{R}_{\mathbf{A}}$  as the Kronecker product  $\mathbf{R}_{\mathbf{A}} = \mathbf{V}_1 \otimes \dots \otimes \mathbf{V}_{n_a}$ , where  $\mathbf{V}_i = \mathbf{1}_{|Att_i|}^T$  if  $Att_i \notin \mathbf{A}$  and  $\mathbf{V}_i = \mathbf{Sub}_{|Att_i|}$  if  $Att_i \in \mathbf{A}$ . Continuing Example 2.1, we have  $\mathbf{R}_{\emptyset} = [11] \otimes [11]$ , and  $\mathbf{R}_{\{Att_1\}} = [1 - 1] \otimes [11]$ , and  $\mathbf{R}_{\{Att_2\}} = [11] \otimes [10] \otimes [10]$ , and  $\mathbf{R}_{\{Att_1, Att_2\}} = [1 - 1] \otimes [10] \otimes [10]$ .
Using subtraction matrices, we also define the matrix  $\Sigma_{\mathbf{A}}$  as the Kronecker product  $\bigotimes_{Att_i\in \mathbf{A}}\left(\mathbf{Sub}_{|Att_i|}\mathbf{Sub}_{|Att_i|}^T\right)$  and we note that it is proportional to  $\mathbf{R}_{\mathbf{A}}\mathbf{R}_{\mathbf{A}}^{T}$ .  $\Sigma_{\emptyset}$  is defined as 1.
Each subset  $\mathbf{A}$  of attributes can be associated with a "base" mechanism  $\mathcal{M}_{\mathbf{A}}$  that takes as input the data vector  $\mathbf{x}$  and a scalar parameter  $\sigma_{\mathbf{A}}^{2}$  for controlling how noisy the answer is.  $\mathcal{M}_{\mathbf{A}}$  is defined as:
$$
\mathcal {M} _ {\mathbf {A}} (\mathbf {x}; \sigma_ {\mathbf {A}} ^ {2}) \equiv \mathbf {R} _ {\mathbf {A}} \mathbf {x} + N \left(\mathbf {0}, \sigma_ {\mathbf {A}} ^ {2} \boldsymbol {\Sigma} _ {\mathbf {A}}\right) \tag {3}
$$
The residual matrices  $\mathbf{R}_{\mathbf{A}}$  used by base mechanisms form a linearly independent basis that compactly represent marginals, as the next result shows.
THEOREM 4.2. Let  $\mathbf{A}$  be a set of attributes and let  $\mathbf{Q}_{\mathbf{A}}$  be the matrix representation of the marginal on  $\mathbf{A}$ . Then the rows of the matrices  $\mathbf{R}_{\mathbf{A}'}$ , for all  $\mathbf{A}' \subseteq \mathbf{A}$ , form a linearly independent basis of the row space of  $\mathbf{Q}_{\mathbf{A}}$ . Furthermore, if  $\mathbf{A}' \neq \mathbf{A}''$  then  $\mathbf{R}_{\mathbf{A}'} \mathbf{R}_{\mathbf{A}''}^T = \mathbf{0}$  (they are mutually orthogonal).
REMARK 4.3. To build an intuitive understanding of residual matrices, consider again Example 2.1. Both  $\mathbf{R}_{\emptyset}$  and  $\mathbf{Q}_{\emptyset}$  are the sum query (marginal on no attributes). The rows of  $\mathbf{R}_{\{Att_1\}}$  span the subspace of  $\mathbf{Q}_{\{Att_1\}}$  that is orthogonal to  $\mathbf{Q}_{\emptyset}$  (and similarly for  $\mathbf{R}_{\{Att_2\}}$ ). The rows of  $\mathbf{R}_{\{Att_1, Att_2\}}$  span the subspace of  $\mathbf{Q}_{\{Att_1, Att_2\}}$  that is orthogonal to both  $\mathbf{Q}_{\{Att_1\}}$  and  $\mathbf{Q}_{\{Att_2\}}$ . Hence a residual matrix spans the subspace of a marginal that is orthogonal to its sub-margins.
Theorem 4.2 has several important implications. If we define the downward closure of a marginal workload  $Wkload = \{\mathbf{A}_1, \dots, \mathbf{A}_k\}$  as the collection of all subsets of the sets in  $Wkload$  (i.e., closure( $Wkload = \{\mathbf{A}' : \mathbf{A}' \subseteq \mathbf{A} \text{ for some } \mathbf{A} \in Wkload\}$ ) then the theorem implies that the combined rows from  $\{\mathbf{R}_{\mathbf{A}'}: \mathbf{A}' \in \text{closure}(Wkload)\}$  forms a linearly independent basis for the marginals in the workload. In other words, it is a linearly independent basis for the space spanned by the rows of the marginal query matrices  $\mathbf{Q}_{\mathbf{A}}$  for  $\mathbf{A} \in Wkload$ . Thus, in order to provide privacy-preserving answers to all of the marginals represented in  $Wkload$ , we need all the mechanisms  $\mathcal{M}_{\mathbf{A}'}$  for  $\mathbf{A}' \in \text{closure}(Wkload)$  – any other matrix mechanism that provides fewer noisy outputs cannot reconstruct unbiased answers to the workload marginals. This is proved in Theorem 4.4, which also states that optimality is achieved by carefully setting the  $\sigma_{\mathbf{A}}$  noise parameter for each  $\mathcal{M}_{\mathbf{A}}$ .
THEOREM 4.4. Given a marginal workload  $W$  kload and a regular loss function  $\mathcal{L}$ , suppose the optimization problem (either Equation 1 or 2) is feasible. Then there exist nonnegative constants  $\sigma_{\mathbf{A}}^2$  for each  $\mathbf{A} \in \text{closure}(W$  kload) (the constants do not depend on the data), such that the optimal linear Gaussian mechanism  $\mathcal{M}_{opt}$  for loss function  $\mathcal{L}$  releases  $\mathcal{M}_{\mathbf{A}}(\mathbf{x}; \sigma_{\mathbf{A}}^2)$  for all  $\mathbf{A} \in \text{closure}(W$  kload). Furthermore, any matrix mechanism for this workload must produce at least this many noise measurements during its selection phase.
Algorithm 1: Efficient implementation of  $\mathcal{M}_{\mathbf{A}}(\mathbf{x};\sigma_{\mathbf{A}}^{2})\equiv \mathbf{R}_{\mathbf{A}}\mathbf{x} + N(\mathbf{0},\sigma_{\mathbf{A}}^{2}\Sigma_{\mathbf{A}})$
1  $\mathbf{v}\gets \mathbf{Q}_{\mathbf{A}}\mathbf{x} / /$  Evaluate the true marginal   
2  $m\gets \prod_{Att_i\in \mathbf{A}}|Att_i|$    
3  $\mathbf{H}\leftarrow \bigotimes_{Att_i\in \mathbf{A}}\mathbf{Sub}_{|Att_i|} / /$  Use implicit representation, don't expand   
4  $\mathbf{z}\gets N(\mathbf{0},\mathcal{I}_m) / /$  independent noise   
5 return  $\mathbf{H}\mathbf{v} + \sigma_{\mathbf{A}}\mathbf{H}\mathbf{z} / /$  use kron-product/vector multiplication from [37]
$\mathcal{M}_{\mathbf{A}}$  can be evaluated efficiently, directly from the marginal of  $\mathbf{x}$  on attribute set  $\mathbf{A}$ , as shown in Algorithm 1. It uses the technique from [37] to perform fast multiplication between a Kronecker product and a vector (so that the Kronecker product does not need to be expanded). It also generates correlated noise from independent Gaussians. The privacy cost  $pcost(\mathcal{M}_{\mathbf{A}})$  of each base mechanism  $\mathcal{M}_{\mathbf{A}}$  is also easy to compute and is given by the following theorem.
THEOREM 4.5. The privacy cost of  $\mathcal{M}_{\mathbf{A}}$  with noise parameter  $\sigma_{\mathbf{A}}^{2}$  is  $\frac{1}{\sigma_{\mathbf{A}}^{2}}\prod_{Att_{i}\in \mathbf{A}}\frac{|Att_{i}| - 1}{|Att_{i}|}$  and the evaluation of  $\mathcal{M}_{\mathbf{A}}$  given in Algorithm 1 is correct - i.e., the output has the distribution  $N(\mathbf{R}_{\mathbf{A}}\mathbf{x},\sigma_{\mathbf{A}}^{2}\Sigma_{\mathbf{A}})$ .
4.3 Reconstruction
Next we explain how to reconstruct unbiased answers to marginal queries from the outputs of the base mechanisms and how to compute (co)variances of the reconstructed marginals efficiently, without any heavy matrix operations (inversion, pseudo-inverses, etc.). Then, given the closed form expressions for marginals and privacy cost (Theorem 4.5), we will be able to explain in Section 4.4 how to optimize the  $\sigma_{\mathbf{A}}^{2}$  parameters of the base mechanisms  $\mathcal{M}_{\mathbf{A}}$  to optimize regular loss functions  $\mathcal{L}$ .
Since the base mechanisms were built using a linearly independent basis, reconstruction is unique – just efficiently invert the basis. Hence, unlike PGM and its extensions [41, 40], our reconstruction algorithm does not need to solve an optimization problem and can reconstruct each marginal independently, thus allowing marginals to be reconstructed in parallel, or as needed by users. The reconstructed marginals are consistent with each other (any two reconstructed marginals agree on their sub-marginals). Just as the subtraction matrices  $\mathbf{Sub}_k$  were useful in constructing the base mechanisms  $\mathcal{M}_{\mathbf{A}}$ , their pseudo-inverses  $\mathbf{Sub}_k^\dagger$  are useful for reconstructing noisy marginals from the
noisy answers of  $\mathcal{M}_{\mathbf{A}}$ . The pseudo-inverses have a closed form. For example  $\mathbf{Sub}_4 = \begin{bmatrix} 1 & -1 & 0 & 0 \\ 1 & 0 & -1 & 0 \\ 1 & 0 & 0 & -1 \end{bmatrix}$  and  $\mathbf{Sub}_4^\dagger = \frac{1}{4} \begin{bmatrix} 1 & 1 & 1 \\ -3 & 1 & 1 \\ 1 & -3 & 1 \\ 1 & 1 & -3 \end{bmatrix}$ . More generally, they are expressed as follows:
LEMMA 4.6. For any  $Att_i$ , let  $\ell = |Att_i|$ . The matrix  $\mathbf{Sub}_{\ell}$  has the following block matrix, with dimensions  $\ell \times (\ell - 1)$ , as its pseudo-inverse (and right inverse):  $\mathbf{Sub}_{\ell}^{\dagger} = \frac{1}{\ell} \begin{bmatrix} \mathbf{1}_{\ell - 1}^{T} \\ \mathbf{1}_{\ell - 1} \mathbf{1}_{\ell - 1}^{T} - \ell \mathcal{I}_{\ell - 1} \end{bmatrix}$ .
Each mechanism  $\mathcal{M}_{\mathbf{A}}$ , for  $\mathbf{A} \in$  closure(Wkload), has a noise scale parameter  $\sigma_{\mathbf{A}}^{2}$  and a noisy output that we denote by  $\omega_{\mathbf{A}}$ . After we have obtained the noisy outputs  $\omega_{\mathbf{A}}$  for all  $\mathbf{A} \in$  closure(Wkload), we can proceed with the reconstruction phase. The reconstruction of an unbiased noisy answer for any marginal on an attribute set  $\mathbf{A} \in$  closure(Wkload) is obtained using Algorithm 2. We note that to reconstruct a marginal on attribute set  $\mathbf{A}$ , one only needs to use the noisy answers  $\omega_{\mathbf{A}'}$  for  $\mathbf{A}' \in$  closure(A). In other words, if we want to reconstruct a marginal on attribute set  $\{Att_1, Att_2\}$ , we only need the outputs of  $\mathcal{M}_{\emptyset}, \mathcal{M}_{\{Att_1\}}, \mathcal{M}_{\{Att_2\}}$ , and  $\mathcal{M}_{\{Att_1, Att_2\}}$  no matter how many other attributes are in the data and no matter what other marginals are in the Wkload. We emphasize again, the reconstruction phase does not run the base mechanisms anymore, it is purely post-processing.
Algorithm 2: Reconstruct Unbiased Answers to the Marginal on A
Input: Noise scale parameters  $\sigma_{\mathbf{A}'}^2$  and noisy answer vector  $\omega_{\mathbf{A}'}$  of mechanism  $\mathcal{M}_{\mathbf{A}'}$  for every  $\mathbf{A}' \in$  closure(A). Output: q is output as an unbiased noisy estimate of  $\mathbf{Q}_{\mathbf{A}}\mathbf{x}$ .  
1 q  $\leftarrow 0$   
2 for each  $\mathbf{A}' \in$  closure(A) do  
3  $\mathbf{U} \leftarrow \mathbf{V}_1 \otimes \dots \otimes \mathbf{V}_{n_a}$ , where  $\mathbf{V}_i = \begin{cases} \mathbf{Sub}_{|Att_i|}^\dagger & \text{if } Att_i \in \mathbf{A}' \\ \frac{1}{|Att_i|} \mathbf{1}_{|Att_i|} & \text{if } Att_i \in \mathbf{A} / \mathbf{A}' \\ [1] & \text{if } Att_i \notin \mathbf{A} \end{cases}$   
4 q  $\leftarrow \mathbf{q} + \mathbf{U} \omega_{\mathbf{A}'}$  // use kron-product/vector multiplication from [37]  
5 return q
THEOREM 4.7. Given a marginal workload  $Wkload$  and positive numbers  $\sigma_{\mathbf{A}}^{2}$  for each  $\mathbf{A} \in \text{closure}(Wkload)$ , let  $\mathcal{M}$  be the mechanism that outputs  $\{\mathcal{M}_{\mathbf{A}}(\mathbf{x}; \sigma_{\mathbf{A}}^{2}) : \mathbf{A} \in \text{closure}(Wkload)\}$  and let  $\{\omega_{\mathbf{A}} : \mathbf{A} \in \text{closure}(Wkload)\}$  denote the privacy-preserving noisy answers (e.g.,  $\omega_{\mathbf{A}} = \hat{\mathcal{M}}_{\mathbf{A}}(\mathbf{x}, \sigma^{2})$ ). Then for any marginal on an attribute set  $\mathbf{A} \in \text{closure}(Wkload)$ , Algorithm 2 returns the unique linear unbiased estimate of  $\mathbf{Q}_{\mathbf{A}}\mathbf{x}$  (i.e., answers to the marginal query) that can be computed from the noisy differentially private answers.
The variances  $Var(\mathbf{A};\mathcal{M})$  of all the noisy cell counts of the marginal on  $\mathbf{A}$  is the vector whose components are all equal to  $\sum_{\mathbf{A}^{\prime}\subseteq \mathbf{A}}\left(\sigma_{\mathbf{A}^{\prime}}^{2}\prod_{Att_{i}\in \mathbf{A}^{\prime}}\frac{|Att_{i}| - 1}{|Att_{i}|}*\prod_{Att_{j}\in (\mathbf{A} / \mathbf{A}^{\prime})}\frac{1}{|Att_{j}|^{2}}\right)$ . The covariance between any two noisy answers of the marginal on  $\mathbf{A}$  is  $\sum_{\mathbf{A}^{\prime}\subseteq \mathbf{A}}\left(\sigma_{\mathbf{A}^{\prime}}^{2}\prod_{Att_{i}\in \mathbf{A}^{\prime}}\frac{-1}{|Att_{i}|}*\prod_{Att_{j}\in (\mathbf{A} / \mathbf{A}^{\prime})}\frac{1}{|Att_{j}|^{2}}\right)$ .
To see an example of how the choices of the  $\sigma_{\mathbf{A}}^{2}$  affect the variance of different marginals, see Section C in the supplementary material.
4.4 Optimizing the Base Mechanism Selection
We now consider how to find the optimal Gaussian linear mechanism  $\mathcal{M}^*$  that solves the optimization problems in Equations 1 or 2. Given a workload on marginals  $Wkload$ , the optimization involves  $Var(\mathbf{A};\mathcal{M}^*)$  for  $\mathbf{A}\in Wkload$  (the variance of the marginal answers reconstructed from the output of  $\mathcal{M}^*$ ) and  $pcost(\mathcal{M}^*)$ , from which the privacy parameters of different flavors of differential privacy can be computed.
Theorem 4.4 says that  $\mathcal{M}^*$  works by releasing  $\mathcal{M}_{\mathbf{A}}(\mathbf{x};\sigma_{\mathbf{A}}^{2})$  for each  $\mathbf{A}\in$  closure(Wkload) for appropriately chosen values of  $\sigma_{\mathbf{A}}^{2}$ . The privacy cost  $pcost(\mathcal{M}^*)$  is the sum of the privacy costs of
the  $\mathcal{M}_{\mathbf{A}}$ . Theorem 4.5 therefore shows that  $pcost(\mathcal{M}^*)$  is a positive linear combination of the values  $1 / \sigma_{\mathbf{A}}^{2}$  for  $\mathbf{A} \in \mathrm{closure}(Wkload)$  and is therefore convex in the  $\sigma_{\mathbf{A}}^{2}$  values. Meanwhile, Theorem 4.7 shows how to represent, for each  $\mathbf{A} \in \mathrm{closure}(Wkload)$ , the quantity  $Var(\mathbf{A}; \mathcal{M}^*)$  as a positive linear combination of  $\sigma_{\mathbf{A}}^{2}$  for  $\mathbf{A}' \in \mathrm{closure}(\mathbf{A}) \subseteq \mathrm{closure}(Wkload)$ . Therefore, the loss function  $\mathcal{L}$  is also convex in the  $\sigma_{\mathbf{A}}^{2}$  values.
Thus the optimization problems in Equations 1 and 2 can be written as minimizing a convex function of the  $\sigma_{\mathbf{A}}^{2}$  subject to convex constraints. In fact, in Equation 2, the constraints are linear when the optimization variables represent the  $\sigma_{\mathbf{A}}^{2}$  and in Equation 1 the constraints are linear when the optimization variables represent the  $1 / \sigma_{\mathbf{A}}^{2}$ . Furthermore, when the loss function is the weighted sum of variances of the marginal cells, the solution can be obtained in closed form (see supplementary material). Otherwise, we use CVXPY/ECOS [12, 14] for solving these convex optimization problems.
4.5 Computational Complexity
Although the universe size  $|Att_1| \times \dots \times |Att_{n_a}|$  grows exponentially with the number of attributes, the following theorem shows that the time complexity of ResidualPlanner depends directly on quantities that typically grow polynomially, such as the number of desired marginals and total number of cells in those marginals.
THEOREM 4.8. Let  $n_a$  be the total number of attributes. Let  $\# cells(\mathbf{A})$  denote the number of cells in the marginal on attribute set  $\mathbf{A}$ . Then:
1. Expressing the privacy cost of the optimal mechanism  $\mathcal{M}^*$  as a linear combination of the  $1 / \sigma_{\mathbf{A}}^{2}$  values takes  $O(\sum_{\mathbf{A}\in Wkload}\# cells(\mathbf{A}))$  total time.
2. Expressing all of the  $Var(\mathbf{A};\mathcal{M}^*)$ , for  $\mathbf{A}\in Wkload$ , as a linear combination of the  $\sigma_{\mathbf{A}}^{2}$  values can be done in  $O(\sum_{\mathbf{A}\in Wkload}\# cells(\mathbf{A}))$  total time.
3. Computing all the noisy outputs of the optimal mechanism (i.e.,  $\mathcal{M}_{\mathbf{A}}(\mathbf{x};\sigma_{\mathbf{A}}^{2})$  for  $\mathbf{A} \in$  closure(Wkload)) takes  $O\left(n_a\sum_{\mathbf{A} \in Wkload}\prod_{Att_i \in \mathbf{A}}(|Att_i| + 1)\right)$  total time after the true answers have been precomputed (Line 1 in Algorithm 1). Note that the total number of cells on marginals in Wkload is  $O\left(\sum_{\mathbf{A} \in Wkload}\prod_{Att_i \in \mathbf{A}}|Att_i|\right)$ .
4. Reconstructing marginals for  $\underline{\underline{all}}\mathbf{A}\in Wkload$  takes  $O(\sum_{\mathbf{A}\in Wkload}|\mathbf{A}| \# cells(\mathbf{A})^2)$  total time.
5. Computing the variance of the cells for all of the marginals for  $\mathbf{A} \in W\text{kload}$  can be done in  $O(\sum_{\mathbf{A} \in W\text{kload}} \# \text{cells}(\mathbf{A}))$  total time.
To get a sense of these numbers, consider a dataset with 20 attributes, each having 3 possible values. If the workload consists of all 3-way marginals, there are 1,140 marginals each with 27 cells so  $n_{cells} = 30,780$ . The quantity inside the big-O for the selection step is 1,459,200 (roughly the number of scalar multiplications needed). These are all easily manageable on modern computers even without GPUs. Our experiments, under more challenging conditions, run in seconds.
We next compare the accuracy and scalability of ResidualPlanner against HDMM [38], including variations of HDMM with faster reconstruction phases [41]. The hardware used was an Ubuntu 22.04.2 server with 12 Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz processors and 32GB of DDR4 RAM. We use 3 real datasets to evaluate accuracy and 1 synthetic dataset to evaluate scalability. The real datasets are (1) the Adult dataset [16] with 14 attributes, each having domain sizes 100, 100, 100, 99, 85, 42, 16, 15, 9, 7, 6, 5, 2, 2, respectively, resulting in a record domain size of  $6.41 \times 10^{17}$ ; (2) the CPS dataset [9] with 5 attributes, each having domain size 100, 50, 7, 4, 2, respectively, resulting in a record domain size of  $2.8 \times 10^{5}$ ; (3) the Loans dataset [24] with 12 attributes, each having domain size 101, 101, 101, 101, 3, 8, 36, 6, 51, 4, 5, 15, respectively, resulting in a record domain size of  $8.25 \times 10^{15}$ . The synthetic dataset is called  $\mathrm{Synth - }n^d$ . Here  $d$  refers to the number of attributes (we experiment from  $d = 2$  to  $d = 100$ ) and  $n$  is the domain size of each attribute. The running times of the algorithms only depend on  $n$  and  $d$  and not on the records in the synthetic data. For all experiments, we set the privacy cost  $pcost$  to 1, so all mechanisms being compared satisfy 0.5-zCDP and 1-Gaussian DP.
5.1 Scalability of the Selection Phase
We first consider how long each method takes to perform the selection phase (i.e., determine what needs noisy answers and how much noise to use). HDMM can only optimize total variance, which is equivalent to root mean squared error. For ResidualPlanner we consider both RMSE and max variance as objectives (the latter is a harder to solve problem). Each algorithm is run 5 times and the average is taken. Table 1 shows running time results; accuracy results will be presented later.
Table 1: Time for Selection Step in seconds on  $\mathrm{Synth - }n^{d}$  dataset.  $n = 10$  and the number of attributes  $d$  varies. The workload consists of all marginals on  $\leq 3$  attributes each. Times for HDMM are reported with  $\pm 2$  standard deviations.
d	HDMMRMSE Objective	ResidualPlannerRMSE Objective	ResidualPlannerMax Variance Objective
2	0.013 ± 0.003 sec	0.001 ± 0.0008 sec	0.007 ± 0.001 sec
6	0.065 ± 0.012 sec	0.002 ± 0.0008 sec	0.009 ± 0.001 sec
10	0.639 ± 0.059 sec	0.009 ± 0.001 sec	0.018 ± 0.001 sec
12	4.702 ± 0.315 sec	0.015 ± 0.001 sec	0.028 ± 0.001 sec
14	46.054 ± 12.735 sec	0.025 ± 0.002 sec	0.041 ± 0.001 sec
15	201.485 ± 13.697 sec	0.030 ± 0.017 sec	0.050 ± 0.001 sec
20	Out of memory	0.079 ± 0.017 sec	0.123 ± 0.023 sec
30	Out of memory	0.247 ± 0.019 sec	0.461 ± 0.024 sec
50	Out of memory	1.207 ± 0.047 sec	4.011 ± 0.112 sec
100	Out of memory	9.913 ± 0.246 sec	121.224 ± 3.008 sec
As we can see, optimizing for max variance is more difficult than for RMSE, but ResidualPlanner does it quickly even for data settings too big for HDMM. The runtime of HDMM increases rapidly, while even for the extreme end of our experiments, ResidualPlanner needs just a few minutes.
5.2 Scalability of the Reconstruction Phase
We next evaluate the scalability of the reconstruction phase under the same settings. The reconstruction speed for ResidualPlanner does not depend on the objective of the selection phase. Here we compare against HDMM [38] and a version of HDMM with improved reconstruction scalability called HDMM+PGM [38, 41] (the PGM settings used 50 iterations of its Local-Inference estimator, as the default 1000 was too slow). Since HDMM cannot perform the selection phase after a certain point, reconstruction results also become unavailable. Table 2 shows ResidualPlanner is clearly faster.
Table 2: Time for Reconstruction Step in seconds on  $\mathrm{Synth} - n^d$  dataset.  $n = 10$  and the number of attributes  $d$  varies. The workload consists of all marginals on  $\leq 3$  attributes each. Times are reported with  $\pm 2$  standard deviations. Reconstruction can only be performed if the select step completed.
d	HDMM	HDMM + PGM	ResidualPlanner
2	0.003 ± 0.0006 sec	0.155 ± 0.011 sec	0.005 ± 0.003 sec
6	0.173 ± 0.011 sec	4.088 ± 0.233 sec	0.023 ± 0.004 sec
10	Out of memory in reconstruction	20.340 ± 2.264 sec	0.125 ± 0.032 sec
12	Out of memory in reconstruction	39.162 ± 1.739 sec	0.207 ± 0.004 sec
14	Out of memory in reconstruction	69.975 ± 4.037 sec	0.330 ± 0.006 sec
15	Out of memory in reconstruction	91.101 ± 7.621 sec	0.413 ± 0.006 sec
20	N/A (select step failed)	N/A (select step failed)	1.021 ± 0.011 sec
30	N/A (select step failed)	N/A (select step failed)	3.587 ± 0.053 sec
50	N/A (select step failed)	N/A (select step failed)	17.029 ± 0.212 sec
100	N/A (select step failed)	N/A (select step failed)	154.538 ± 15.045 sec
5.3 Accuracy Comparisons
Since ResidualPlanner is optimal, the purpose of the accuracy comparisons is a sanity check. For RMSE, comparisons of the quality of ResidualPlanner to the theoretically optimal lower bound, known as the SVD bound [31], can be found in the supplementary material in Section I (ResidualPlanner
matches the lower bound). We note ResidualPlanner can provide solutions even when the SVD bound is infeasible to compute.
We also compare ResidualPlanner to HDMM when the user is interested in the maximum variance objective. This just shows that it is important to optimize for the user's objective function and that the optimal solution for RMSE (the only objective HDMM can optimize) is not a good general-purpose approximation for other objectives (as shown in Table 3). Additional comparisons are provided in the supplementary material.
Table 3: Max Variance Comparisons with ResidualPlanner and HDMM (showing that being restricted to optimizing only RMSE is not a good approximation of Max Variance optimization).
Adult Dataset	CPS Dataset	Loans Dataset
Workload	ResPlan	HDMM	ResPlan	HDMM	ResPlan	HDMM
1-way Marginals	12.047	41.772	4.346	13.672	10.640	33.256
2-way Marginals	67.802	599.843	7.897	47.741	52.217	437.478
3-way Marginals	236.843	5675.238	7.706	71.549	156.638	3095.997
≤ 3-way Marginals	253.605	6677.253	13.216	415.073	180.817	4317.709
6 Limitations, Conclusion, and Future Work.
In this paper, we introduced ResidualPlanner, a matrix mechanism that is scalable and optimal for marginals under Gaussian noise, for a large class of convex objective functions. While these are important improvements to the state of the art, there are limitations.
First, for some attributes, a user might not want marginals. For example, they might want range queries or queries with hierarchies (e.g., how many people drive sedans vs. vans; out of the sedans, how many are red vs. green, etc) [2, 28, 36]. In some cases, an attribute might have an infinite domain (e.g., a URL) and need to be handled differently [27, 45]. In other cases, the user may want other noise distributions, like the Laplace. These types of queries do not have the same type of symmetry as marginals that was crucial to proving the optimality of ResidualPlanner. For these situations, one of the key ideas of ResidualPlanner can be used – find a linear basis that compactly represents both the queries and “residual” (information provided by a query that is not contained in the other queries). Such a feature would result in scalability. It is future work to determine how to extend both scalability and optimality to such situations. Another limitation is that this work considers the setting where an individual can contribute exactly one (rather than arbitrarily many) records to the dataset.
This work was supported by NSF grant CNS-1931686 and a gift from Facebook.