SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1 Introduction 1
2 Preliminaries 2
2.1 Differential Privacy 2
2.2 Matrix Mechanism 3
3 Additional Related Work 4
4 ResidualPlanner 4
4.1 Loss Functions Supported by ResidualPlanner 4
4.2 Base Mechanisms used by ResidualPlanner 5
4.3 Reconstruction 6
4.4 Optimizing the Base Mechanism Selection 7
4.5 Computational Complexity 8
5 Experiments 8
5.1 Scalability of the Selection Phase 9
5.2 Scalability of the Reconstruction Phase 9
5.3 Accuracy Comparisons 9
6 Limitations, Conclusion, and Future Work. 10
7 Acknowledgments 10
A Table of Notation 15
B A Run-through of Residual Planner 15
B.1 A Small Dataset and its Vectorized Representation 15
B.2 The Marginal Workload and its Representation as a Query Matrix. 15
B.3 The Base Mechanisms 18
B.4 Reconstruction 19
B.5 Privacy Cost and Marginal Variances 20
B.6 The Sum-of-Variances Loss Function 21
C Example Variance Calculations 22
D Optimality Proof of ResidualPlanner 23
D.1 Notation Review 24
D.2 Permutations 24
D.3 From permutations to interpretations 26
E The other proofs about base mechanisms 32
F Proofs related to the reconstruction step 34
G Computational Complexity Proofs 38
H Closed Form Solution to the Weighted Sum of Variances Loss 39
I Additional Experiments 40
I.1 Scalability 40
I.2 Comparison on Real Datasets. 43
A Table of Notation
B A Run-through of Residual Planner
In this section, we provide a complete runthrough of ResidualPlanner using a small toy dataset.
B.1 A Small Dataset and its Vectorized Representation
In our example, we have a dataset with 3 attributes, so  $n_a = 3$ .  $Att_1$  takes values 'a' or 'b';  $Att_2$  takes values 'y' or 'n';  $Att_3$  takes values 1 or 2 or 3.
In this dataset, there are 5 people, and the tabular representation is shown in Table 5. For each attribute, we can one-hot encode its attribute values as row vectors. So, for  $Att_1$ , the attribute value 'a' is encoded as [1,0] and 'b' is encoded as [0,1]. For  $Att_2$ , the attribute value 'y' is encoded as [1,0] and 'n' is encoded as [0,1]. For attribute  $Att_3$ , the attribute value '1' is encoded as [1,0,0], the value '2' is encoded as [0,1,0] and '3' is encoded as [0,0,1].
The kronecker product representation of a record is the kronecker product of the one-hot encoding of each attributes. So, for example, the record 'an2' is encoded as the kronecker product  $[1,0] \otimes [0,1] \otimes [0,1,0]$ . When this kronecker product is expanded, it has 12 components. One of the contains a 1 and the rest contain a 0. Thus the expanded kronecker product can be thought of as a one-hot encoding of the entire record.
Indeed, in the expanded kronecker product, each dimension of the resulting vector is associated with a record. In table 6, we show the kronecker product representation of each record from Table 5. The left column of Table 6 shows the record and its kronecker representation. The next 12 columns show the resulting expansion. Each record becomes as 12-dimensional vector and the column labels in Table 6 show which record is associated with which index in the 12-dimensional vector.
The sum of the kron representations of all the records is the data vector  $\mathbf{x}$ . It is again a 12-dimensional vector. At each index  $i$ ,  $\mathbf{x}[i]$  is the number of people whose record is associated index  $i$ . For example, the 5th component is associated with the record 'an2' and there are 2 people with that record. For mathematical convenience,  $\mathbf{x}$  is treated as a column vector, but for display purposes, in Table 6 it is written as a row vector.
B.2 The Marginal Workload and its Representation as a Query Matrix.
For this example, we set the marginal workload to consist of 3 marginals  $Wkload = \{\{Att_1\}, \{Att_1, Att_2\}, \{Att_2, Att_3\}\}$ .
The marginal on attribute set  $\mathbf{A} = \{Att_1\}$  has only two cells, which correspond to the number of people with  $Att_1 = a$  (i.e., 3) and the number with  $Att_1 = b$  (i.e., 3). This is called a one-way marginal. The other marginals are two-way marginals because they involve two attributes. For example, the marginal on  $\mathbf{A} = \{Att_2, Att_3\}$  has 6 cells. It represents the number of people for each combination of values for  $Att_2$  and  $Att_3$ . For example, there are 2 people with  $Att_2 = y$  and  $Att_3 = 3$ .
For each set  $\mathbf{A}$ , the marginal on those attributes can be represented as a matrix  $\mathbf{Q}_{\mathbf{A}}$  such that calculating the marginal is equivalent to the matrix-vector multiplication  $\mathbf{Q}_{\mathbf{A}}\mathbf{x}$ . The construction of
Table 4: Table of Notation
D:	Dataset
r_i:	ithrecord in D
na:	number of attributes each record has
Attj:	jthattribute.
|Attj|:	size of the domain of attribute Attj.
a1(j), ..., a|Attj|:	possible values (domain) of Attj.
d:	Number of possible records: d = ∏ni=1na |Attj|
x:	Representation of D as a d-dimensional vector of counts (e.g., histogram)
A:	(Sub)set of attributes
QA:	Matrix representation of the marginal on A. The counts in the marginal are the result of matrix-vector product QA x.
#cells(A):	Number of cells in the marginal on A. Equals ∏Atti∈A |Atti|
ei:	one-hot encoding vector with entry i being 1 and the rest 0
ei,j:	equal to ei - ej
1k:	the k-dimensional vector whose entries are all 1.
Ik:	the k × k identity matrix
M:	A privacy mechanism.
ω:	Output of a mechanism.
B:	Query matrix of a Gaussian linear query mechanism: M(x) ≡ Bx + N(0, Σ)
Σ:	Covariance matrix.
pcost(M):	Privacy cost of a Gaussian linear mechanism M(x) ≡ Bx + N(0, Σ). It is defined as the largest diagonal of BT Σ-1B. Differential privacy parameters can be computed from pcost(M).
Wkload:	A workload of marginals. Each element of Wkload is a set of attributes (representing the marginal on those attributes).
ncells:	Total number of cells in the marginals in the marginal workload (i.e., the output size).
closure(Wkload):	The set of all subsets of Wkload. Formally defined as {A' : A' ⊆ A for some A ∈ Wkload}.
Var(A, M):	When the output of M is used to reconstruct answers to the marginal on A, then Var returns the vector of variances of the marginal cells.
L:	The loss function
†:	The operator that gives the pseudo-inverse of a matrix
Subm:	An (m-1) × m subtraction matrix. The first column is filled with 1, entries of the form (i, i+1) are -1, and all other entries are 0.
RA:	Residual matrix. Given a set A ⊂{Att1, ..., Attna} of attributes, RA = V1⊗...⊗Vna, where Vi = 1|Atti| if Atti∉A and Vi = Sub|Atti| if Atti∈A.
ΣA:	The covariance matrix used by the base mechanisms, formed as the kronecker product ⊕(Sub|Atti|SubT|Atti|). Also Σ∅ = 1.
σA:	Data-independent noise scale parameter
MA:	The base mechanism defined as MA(x) ≡ RAX + N(0, σA2ΣA). It uses a data-independent noise parameter σ2A
ωA:	noisy output of mechanism MA
the matrix  $\mathbf{Q}_{\mathbf{A}}$  is straightforward. It is a kronecker product of 3 matrices. Each matrix corresponds to an attribute. If the attribute is in  $\mathbf{A}$  then the corresponding term is the identity matrix, otherwise is the row vector full of ones. For example,  $\mathbf{Q}_{\{Att_1\}}$  is a kron product of 3 matrices: the first matrix corresponds to  $Att_1$  and is the  $2\times 2$  identity matrix. The second matrix is actually the vector full of ones because  $Att_2$  is not part of the marginal. This vector has 2 components because  $Att_2$  has 2 possible values. Similarly, the third matrix is the vector full of ones with 3 components.
Att1	Att2	Att3
a	n	2
b	n	3
b	y	3
a	n	2
b	y	3
Table 5: A Toy Dataset  $\mathcal{D}$
an2: [1,0]⊗[0,1]⊗[0,1,0]
bn3: [0,1]⊗[0,1]⊗[0,0,1]
by3: [0,1]⊗[1,0]⊗[0,0,3]
an2: [1,0]⊗[0,1]⊗[0,1,0]
by3: [0,1]⊗[1,0]⊗[0,0,3]
ay1	ay2	ay3	an1	an2	an3	by1	by2	by3	bn1	bn2	bn3
0	0	0	0	1	0	0	0	0	0	0	0
0	0	0	0	0	0	0	0	0	0	0	1
0	0	0	0	0	0	0	0	1	0	0	0
0	0	0	0	1	0	0	0	0	0	0	0
0	0	0	0	0	0	0	0	1	0	0	0
Vector of counts  $\mathbf{x}$
0	0	0	0	2	0	0	0	2	0	0	1
Table 6: Kron product representations or each record and the whole dataset x. Nonzero components are shown in bold red.
For the marginals in  $Wkload$ , these are the corresponding matrices:
$$
\begin{array}{l} \mathbf {Q} _ {\{A t t _ {1} \}} = \left[ \begin{array}{c c} 1 & 0 \\ 0 & 1 \end{array} \right] \otimes \left[ \begin{array}{c c} 1 & 1 \end{array} \right] \otimes \left[ \begin{array}{c c c} 1 & 1 & 1 \end{array} \right] \\ = \left[ \begin{array}{c c c c c c c c c c c c} 1 & 1 & 1 & 1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 & 1 & 1 & 1 \end{array} \right] \\ \mathbf {Q} _ {\{A t t _ {1}, A t t _ {2} \}} = \left[ \begin{array}{c c} 1 & 0 \\ 0 & 1 \end{array} \right] \otimes \left[ \begin{array}{c c} 1 & 0 \\ 0 & 1 \end{array} \right] \otimes \left[ \begin{array}{c c c} 1 & 1 & 1 \end{array} \right] \\ = \left[ \begin{array}{c c c c c c c c c c c c} 1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 \end{array} \right] \\ \end{array}
$$
$$
\begin{array}{l} \mathbf {Q} _ {\{A t t _ {2}, A t t _ {3} \}} = \left[ \begin{array}{c c} 1 & 1 \end{array} \right] \otimes \left[ \begin{array}{c c} 1 & 0 \\ 0 & 1 \end{array} \right] \otimes \left[ \begin{array}{c c c} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{array} \right] \\ = \left[ \begin{array}{c c c c c c c c c c c c} 1 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 1 \end{array} \right] \\ \end{array}
$$
If we multiply  $\mathbf{Q}_{\{attr_2, Att_3\}}$  by the data vector  $\mathbf{x}$  from Table 6, we get:
$$
\mathbf {Q} _ {\{a t t r _ {2}, A t t _ {3} \}} \mathbf {x} = \left[ \begin{array}{l} 0 \\ 0 \\ 2 \\ 0 \\ 2 \\ 1 \end{array} \right]
$$
Comparing it to the marginals shown in Table 7 we see that it is the flattened version of the marginal. That is, we take the first column of the  $\{Att_2, Att_3\}$  marginal of Table 7, then we put the next column below it, and the third column is placed at the bottom.
$\mathbf{A} = \{Att_1\}$
$\mathbf{A} = \{Att_1, Att_2\}$
$\mathbf{A} = \{Att_2, Att_3\}$
Table 7: True answers to the marginal queries in the marginal workload  $Wkload = \{\{Att_1\}, \{Att_1, Att_2\}, \{Att_2, Att_3\}\}$ .
B.3 The Base Mechanisms
Recall that our workload, the marginals we want privacy-preserving answers to, is  $Wkload = \{\{Att_1\}, \{Att_1, Att_2\}, \{Att_2, Att_3\}\}$ . Its closure, denoted by closure(Wkload) is all of its subsets. So,
$$
\operatorname {c l o s u r e} (W k l o a d) = \{\emptyset , \{A t t _ {1} \}, \{A t t _ {2} \}, \{A t t _ {3} \}, \{A t t _ {1}, A t t _ {2} \}, \{A t t _ {2}, A t t _ {3} \} \}
$$
For each  $\mathbf{A} \in$  closure(Wkload) we need to form a base mechanism  $\mathcal{M}_{\mathbf{A}}$ . Each  $\mathcal{M}_{\mathbf{A}}$  has a free parameter  $\sigma_{\mathbf{A}}^{2}$  that we are free to choose. Each mechanism  $\mathcal{M}_{\mathbf{A}}$  has the form  $\mathcal{M}_{\mathbf{A}}(\mathbf{x}; \sigma_{\mathbf{A}}^{2}) = \mathbf{R}_{\mathbf{A}}\mathbf{x} + N(\mathbf{0}, \Sigma_{\mathbf{A}})$ . That is, on input  $\mathbf{x}$ , the mechanism multiplies it by a special "residual" matrix  $\mathbf{R}_{\mathbf{A}}$  and then adds correlated Gaussian noise, with zero mean and with covariance matrix  $\sigma_{\mathbf{A}}^{2}\Sigma_{\mathbf{A}}$ . The residual and covariance matrices for each base mechanism are shown below.
$$
\mathcal {M} _ {\emptyset}: \quad \mathbf {R} _ {\emptyset} = \left[ \begin{array}{c c} 1 & 1 \end{array} \right] \otimes \left[ \begin{array}{c c} 1 & 1 \end{array} \right] \otimes \left[ \begin{array}{c c c} 1 & 1 & 1 \end{array} \right]
$$
$$
\boldsymbol {\Sigma} _ {\emptyset} = [ 1 ]
$$
$$
\begin{array}{l} \mathcal {M} _ {\{A t t _ {1} \}}: \quad \mathbf {R} _ {\{A t t _ {1} \}} = \left[ \begin{array}{c c} 1 & - 1 \end{array} \right] \otimes \left[ \begin{array}{c c} 1 & 1 \end{array} \right] \otimes \left[ \begin{array}{c c c} 1 & 1 & 1 \end{array} \right] \\ \boldsymbol {\Sigma} _ {\{A t t _ {1} \}} = \left[ \begin{array}{c c} 1 & - 1 \end{array} \right] \left(\left[ \begin{array}{c c} 1 & - 1 \end{array} \right]\right) ^ {T} = [ 2 ] \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {M} _ {\{A t t _ {2} \}}: \quad \mathbf {R} _ {\{A t t _ {2} \}} = \left[ \begin{array}{c c} 1 & 1 \end{array} \right] \otimes \left[ \begin{array}{c c} 1 & - 1 \end{array} \right] \otimes \left[ \begin{array}{c c c} 1 & 1 & 1 \end{array} \right] \\ \Sigma_ {\{A t t _ {2} \}} = \left[ \begin{array}{c c} 1 & - 1 \end{array} \right] \left(\left[ \begin{array}{c c} 1 & - 1 \end{array} \right]\right) ^ {T} = [ 2 ] \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {M} _ {\{A t t _ {3} \}}: \quad \mathbf {R} _ {\{A t t _ {3} \}} = \left[ \begin{array}{c c} 1 & 1 \end{array} \right] \otimes \left[ \begin{array}{c c} 1 & 1 \end{array} \right] \otimes \left[ \begin{array}{c c c} 1 & - 1 & 0 \\ 1 & 0 & - 1 \end{array} \right] \\ \boldsymbol {\Sigma} _ {\{A t t _ {3} \}} = \left[ \begin{array}{c c c} 1 & - 1 & 0 \\ 1 & 0 & - 1 \end{array} \right] \left(\left[ \begin{array}{c c c} 1 & - 1 & 0 \\ 1 & 0 & - 1 \end{array} \right]\right) ^ {T} = \left[ \begin{array}{c c} 2 & 1 \\ 1 & 2 \end{array} \right] \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {M} _ {\{A t t _ {1}, A t t _ {2} \}}: \quad \mathbf {R} _ {\{A t t _ {1}, A t t _ {2} \}} = \left[ \begin{array}{c c} 1 & - 1 \end{array} \right] \otimes \left[ \begin{array}{c c} 1 & - 1 \end{array} \right] \otimes \left[ \begin{array}{c c c} 1 & 1 & 1 \end{array} \right] \\ \boldsymbol {\Sigma} _ {\left\{A t t _ {1}, A t t _ {2} \right\}} = \left(\left[ \begin{array}{l l} 1 & - 1 \end{array} \right] \otimes \left[ \begin{array}{l l} 1 & - 1 \end{array} \right]\right) \left(\left[ \begin{array}{l l} 1 & - 1 \end{array} \right] \otimes \left[ \begin{array}{l l} 1 & - 1 \end{array} \right]\right) ^ {T} = [ 4 ] \\ \end{array}
$$
$$
\begin{array}{l} \mathcal {M} _ {\{A t t _ {2}, A t t _ {3} \}}: \quad \mathbf {R} _ {\{A t t _ {2}, A t t _ {3} \}} = \left[ \begin{array}{c c} 1 & 1 \end{array} \right] \otimes \left[ \begin{array}{c c} 1 & - 1 \end{array} \right] \otimes \left[ \begin{array}{c c c} 1 & - 1 & 0 \\ 1 & 0 & - 1 \end{array} \right] \\ \boldsymbol {\Sigma} _ {\{A t t _ {2}, A t t _ {3} \}} = \left(\left[ \begin{array}{c c} 1 & - 1 \end{array} \right] \otimes \left[ \begin{array}{c c c} 1 & - 1 & 0 \\ 1 & 0 & - 1 \end{array} \right]\right) \left(\left[ \begin{array}{c c} 1 & - 1 \end{array} \right] \otimes \left[ \begin{array}{c c c} 1 & - 1 & 0 \\ 1 & 0 & - 1 \end{array} \right]\right) ^ {T} \\ = \left[ \begin{array}{c c} 4 & 2 \\ 2 & 4 \end{array} \right] \\ \end{array}
$$
Note that for any  $\mathbf{A}$ , the residual matrix  $\mathbf{R}_{\mathbf{A}}$  has a similar structure to  $\mathbf{Q}_{\mathbf{A}}$  except that where  $\mathbf{Q}_{\mathbf{A}}$  has an identity matrix in its kron product,  $\mathbf{R}_{\mathbf{A}}$  has a subtraction matrix (e.g.  $\left[ \begin{array}{cc}1 & -1 \end{array} \right]$  or  $\left[ \begin{array}{ccc}1 & -1 & 0 \\ 1 & 0 & -1 \end{array} \right]$ ). Meanwhile the covariance matrix  $\boldsymbol{\Sigma}_{\mathbf{A}}$  looks like  $\mathbf{R}_{\mathbf{A}}\mathbf{R}_{\mathbf{A}}^{T}$  except that the vectors full of 1s have been first removed.
How do we interpret the residual matrices? Well,  $\mathbf{R}_{\emptyset}$  is the sum query. In fact the matrix vector multiplication  $\mathbf{R}_{\emptyset}\mathbf{x}$  gives us the total number of people in the data.
Next,  $\mathbf{R}_{\{Att_1\}}$  tells us the information contained in the marginal on  $\{Att_1\}$  that is not contained in the sum query. If we know the total number of people in the data, then the only new information the marginal gives us is the difference between the number of people with  $Att_1 = a$  and the number of people with  $Att_1 = b$ . In other words,  $\mathbf{R}_{\{Att_1\}}\mathbf{x}$  is this difference. Given this difference, and the total, once can recover the marginal on attribute  $Att_1$ .
Similarly,  $\mathbf{R}_{\{Att_2\}}$  contains the information in the marginal on  $\{Att_2\}$  that is not provided by the sum query. Finally  $\mathbf{R}_{\{Att_3\}}$  contains the information in the marginal on  $\{Att_3\}$  not provided in the sum query, which is the number of people with  $Att_3 = 1$  minus the number with  $Att_3 = 2$ , and also the number of people with  $Att_3 = 1$  minus the number with  $Att_3 = 3$ . The product  $\mathbf{R}_{\{Att_3\}}\mathbf{x}$  returns those two differences as a vector with two components.
Now,  $\mathbf{R}_{\{Att_1, Att_2\}}$  and  $\mathbf{R}_{\{Att_2, Att_3\}}$  are more complicated, but have the same idea. For example,  $\mathbf{R}_{\{Att_1, Att_2\}}$  represents new information that the marginal on  $\{Att_1, Att_2\}$  provides that is not captures by the sub-margins (the marginal on  $\{Att_1\}$  and the marginal on  $\{Att_2\}$ .
In general, the matrix  $\mathbf{R}_{\mathbf{A}}$  represents the new information on that the marginal on  $\mathbf{A}$  provides, which is not captured by the marginals on  $\mathbf{A}'$ , for  $\mathbf{A}' \subset \mathbf{A}$  (strict subsets).
Now, Theorem 4.2 tells us that if we take all of the rows of all of the residual matrices, they will be linearly independent. Furthermore, given an attribute set  $\mathbf{A}$ , the total number of rows of  $\mathbf{R}_{\mathbf{A}'}$  for all  $\mathbf{A}' \subseteq \mathbf{A}$  is the number of rows in  $\mathbf{Q}_{\mathbf{A}}$ . Furthermore, the space spanned by those rows is the same as the space spanned by the rows of  $\mathbf{Q}_{\mathbf{A}}$ .
This also means that if we know  $\mathbf{R}_{\mathbf{A}'}\mathbf{x}$  for all  $\mathbf{A}' \subseteq \mathbf{A}$  then we can figure out  $\mathbf{Q}_{\mathbf{A}}\mathbf{x}$  (and vice versa).
Now, we want to get privacy-preserving (noisy) answers to the marginal queries in  $Wkload = \{\{Att_1\}, \{Att_1, Att_2\}, \{Att_2, Att_3\}\}$  that are as accurate as possible subject to privacy constraints. We quantify accuracy using a regular (Definition 4.1) loss function (e.g., sum of the variances of the answers to the marginals) and we quantify privacy by setting privacy parameters for either  $(\epsilon, \delta)$ -differential privacy,  $\rho$ -zCDP, or  $\mu$ -Gaussian differential privacy.
Theorem 4.4 says that to maximize the accuracy subject to privacy constraints, we need to take the closure of the workload,  $\text{closure}(\text{Wkload}) = \{\emptyset, \{Att_1\}, \{Att_2\}, \{Att_3\}, \{Att_1, Att_2\}, \{Att_2, Att_3\}\}$  and carefully choose positive numbers  $\sigma_{\mathbf{A}}^2$  for each  $\mathbf{A} \in \text{closure}(\text{Wkload})$  so that is 6 numbers total. These numbers are chosen without looking at the data (we explain how in Section B.6). Once we have these numbers, we run the mechanisms  $\mathcal{M}_{\mathbf{A}}(\mathbf{x}; \sigma_{\mathbf{A}}^2)$  and return their outputs. In other words, we must release the outputs of:
-  $\mathcal{M}_{\emptyset}(\mathbf{x};\sigma_{\emptyset}^{2})$  - produces 1 number (a vector with just one component)
-  $\mathcal{M}_{\{Att_1\}}(\mathbf{x};\sigma_{\{Att_2\}}^2)$  produces 1 number (a vector with just one component)
-  $\mathcal{M}_{\{Att_2\}}(\mathbf{x};\sigma_{\{Att_2\}}^2)$  - produces 1 number (a vector with just one component)
-  $\mathcal{M}_{\{Att_3\}}(\mathbf{x};\sigma_{\{Att_3\}}^2)$  produces 2 numbers (a vector with 2 components)
-  $\mathcal{M}_{\{Att_1, Att_2\}}(\mathbf{x}; \sigma_{\{Att_1, Att_2\}}^2)$  produces 1 number (a vector with 1 component)
-  $\mathcal{M}_{\{Att_2, Att_3\}}(\mathbf{x}; \sigma_{Att_2, Att_3}^2)$  - produces 2 numbers (a vector with 2 components)
Which gives us 8 total (noisy) numbers. In fact, any matrix mechanism for this workload must return at least 8 noisy numbers, by Theorem 4.4.
From these outputs, one can reconstruct noisy answers to the marginals in  $Wkload$  (actually one can reconstruct noisy answers to any marginal in closure( $Wkload$ )). We show how to do this in Section B.4. Then we show how to compute the privacy cost and variances of the algorithm in Section B.5.
B.4 Reconstruction
Let  $\omega_{\mathbf{A}}$  denote the output of  $\mathcal{M}_{\mathbf{A}}$ . Thus, after running
-  $\mathcal{M}_{\emptyset}(\mathbf{x};\sigma_{\emptyset}^{2})$
$\mathcal{M}_{\{Att_1\}}(\mathbf{x};\sigma_{Att_1}^2)$
$\mathcal{M}_{\{Att_2\}}(\mathbf{x};\sigma_{\{Att_2\}}^2)$
$\mathcal{M}_{\{Att_3\}}(\mathbf{x};\sigma_{Att_3}^2)$
$\mathcal{M}_{\{Att_1,Att_2\}}(\mathbf{x};\sigma_{\{Att_1,Att_2\}}^2)$
and  $\mathcal{M}_{\{Att_2, Att_3\}}(\mathbf{x}; \sigma_{\{Att_2, Att_3\}}^2)$
we have the noisy answers
$$
\omega_ {\emptyset}, \quad \omega_ {\{A t t _ {1} \}}, \quad \omega_ {\{A t t _ {2} \}}, \quad \omega_ {\{A t t _ {3} \}}, \quad \omega_ {\{A t t _ {1}, A t t _ {2} \}}, \quad \omega_ {\{A t t _ {2}, A t t _ {3} \}}
$$
From these noisy answers we can produce noisy answers for any marginal in  $Wkload$  or even closure( $Wkload$ ). To reconstruct a marginal on  $\mathbf{A}$ , we need  $\omega_{\mathbf{A}'}$  for all  $\mathbf{A}' \subseteq \mathbf{A}$  – this is not a lot as these vectors represent as many noisy numbers as there are cells in the desired histogram. So, for example, if we want to get noisy answers for the marginal on  $\{Att_2, Att_3\}$  (which has 6 cells), we need to use  $\omega_{\emptyset}, \omega_{\{Att_2\}}, \omega_{\{Att_3\}}$ , and  $\omega_{\{Att_2, Att_3\}}$  (together these  $\omega$  vectors represent a total of 6 noisy numbers).
In order to reconstruct the marginal on  $\mathbf{A}$ , Algorithm 2 multiplies each  $\omega_{\mathbf{A}'}$  by a matrix that depends on both  $\mathbf{A}$  and  $\mathbf{A}'$ . The algorithm calls this matrix  $\mathbf{U}$ , but to make the notation precise for this runthrough, we will call it  $\mathbf{U}_{\mathbf{A} \leftarrow \mathbf{A}'}$  (the  $\mathbf{U}$  matrix that multiplies  $\omega_{\mathbf{A}'}$  when reconstructing  $\mathbf{A}$ ). It turns out that:
$$
\mathbf {Q} _ {\mathbf {A}} \mathbf {x} = \sum_ {\mathbf {A} ^ {\prime} \subseteq \mathbf {A}} \mathbf {U} _ {\mathbf {A} \leftarrow \mathbf {A} ^ {\prime}} \mathbf {R} _ {\mathbf {A} ^ {\prime}} \mathbf {x}
$$
which means that the marginal on  $\mathbf{A}$  could be recreated if we know the quantities  $\mathbf{R}_{\mathbf{A}'}\mathbf{x}$  (recall  $\mathbf{R}_{\mathbf{A}'}$  are the matrices used to define our base mechanisms). Now, since  $\omega_{\mathbf{A}'}$  is a noisy version of  $\mathbf{R}_{\mathbf{A}'}\mathbf{x}$ , we can get noisy marginal answers by substituting in these noisy values into the above equation.
For example, to reconstruct a noisy answer to the marginal on  $\{Att_2, Att_3\}$ , we do the following:
Noisy Marginal on  $\{Att_2, Att_3\} = (\mathbf{U}_{\{Att_2, Att_3\} \leftarrow \emptyset})\omega_{\emptyset}$
$$
\begin{array}{l} + \left(\mathbf {U} _ {\{A t t _ {2}, A t t _ {3} \} \leftarrow \{A t t _ {2} \}}\right) \omega_ {\{A t t _ {2} \}} \\ + \left(\mathbf {U} _ {\{A t t _ {2}, A t t _ {3} \} \leftarrow \{A t t _ {3} \}}\right) \omega_ {\{A t t _ {3} \}} \\ + \left(\mathbf {U} _ {\left\{A t t _ {2}, A t t _ {3} \right\} \leftarrow \left\{A t t _ {2}, A t t _ {3} \right\}}\right) \omega_ {\left\{A t t _ {2}, A t t _ {3} \right\}} \\ \end{array}
$$
$$
\mathbf {U} _ {\{A t t _ {2}, A t t _ {3} \} \leftarrow \emptyset} = \left(\frac {1}{2} \mathbf {1} _ {2}\right) \otimes \left(\frac {1}{3} \mathbf {1} _ {3}\right) = \left[ \begin{array}{c} 1 / 2 \\ 1 / 2 \end{array} \right] \otimes \left[ \begin{array}{c} 1 / 3 \\ 1 / 3 \\ 1 / 3 \end{array} \right]
$$
$$
\mathbf {U} _ {\{A t t _ {2}, A t t _ {3} \} \leftarrow \{A t t _ {2} \}} = \left(\mathbf {S u b} _ {2} ^ {\dagger}\right) \otimes \left(\frac {1}{3} \mathbf {1} _ {3}\right) = \left[ \begin{array}{c} 1 / 2 \\ - 1 / 2 \end{array} \right] \otimes \left[ \begin{array}{c} 1 / 3 \\ 1 / 3 \\ 1 / 3 \end{array} \right]
$$
$$
\mathbf {U} _ {\{A t t _ {2}, A t t _ {3} \} \leftarrow \{A t t _ {3} \}} = \left(\frac {1}{2} \mathbf {1} _ {2}\right) \otimes \left(\mathbf {S u b} _ {3} ^ {\dagger}\right) = \left[ \begin{array}{c} 1 / 2 \\ 1 / 2 \end{array} \right] \otimes \left[ \begin{array}{c c} 1 / 3 & 1 / 3 \\ - 2 / 3 & 1 / 3 \\ 1 / 3 & - 2 / 3 \end{array} \right]
$$