SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- For a discriminative attribute  $Att_{\ell}$ , let  $e_{1,i_{\ell}}$  be the kron component for  $\mathbf{v}$  (i.e.,  $\mathbf{v}^{(\ell)} = e_{1,i_{\ell}}$ ) and let  $e_{1,j_{\ell}}$  be the kron component for  $\mathbf{w}$ . Note the indices  $i_{\ell}$  and  $j_{\ell}$  are not equal to 1. In this case, we make  $\pi$  do the following renamings:
$$
- a _ {i _ {\ell}} \rightarrow a _ {j _ {\ell}}
$$
$$
- a _ {j _ {\ell}} \rightarrow a _ {i _ {\ell}}
$$
- The remaining attribute values are unchanged.
By considering which records are assigned 1,-1 and 0 by  $\mathbf{v}$  and  $\mathbf{w}$ , it is clear that  $\pi$  converts  $\mathbf{v}$  into  $\mathbf{w}$  (and vice versa). Let  $\mathbf{W}$  be the matrix representation of the renaming permutation  $\pi$ , so that  $\mathbf{W} \mathbf{v} = \mathbf{w}$  and  $\mathbf{W}^T \mathbf{w} = \mathbf{v}$  (a permutation matrix is orthogonal, so its inverse is its transpose). Thus, letting  $\beta$  denote the eigenvalue of  $\mathbf{v}$  with respect to  $\Gamma$ , we have:
$$
\begin{array}{l} \beta \mathbf {v} = \boldsymbol {\Gamma} \mathbf {v} \\ = \boldsymbol {\Gamma} \mathbf {W} ^ {T} \mathbf {w} \\ \end{array}
$$
$$
\begin{array}{l} = \mathbf {W} ^ {T} \boldsymbol {\Gamma} \mathbf {W} \mathbf {W} ^ {T} \mathbf {w} \quad \text {d u e t o t h e s y m m e t r y f r o m L e m m a D . 3} \\ = \mathbf {W} ^ {T} \boldsymbol {\Gamma} \mathbf {w}, \\ \end{array}
$$
since  $\mathbf{W}^T$  is the inverse of  $\mathbf{W}$  and so
$$
\boldsymbol {\beta} \mathbf {w} = \boldsymbol {\beta} \mathbf {W} \mathbf {v} = \mathbf {W} \mathbf {W} ^ {T} \boldsymbol {\Gamma} \mathbf {w} = \boldsymbol {\Gamma} \mathbf {w}
$$
and thus  $\mathbf{w}$  has the same eigenvalue as  $\mathbf{v}$ .
Thus each residual basis matrix  $\mathbf{R}_{\mathbf{A}}$  has a useful property: its rows are linearly independent and are part of the same eigenspace (linear space of vectors with the same eigenvalue) of the privacy cost matrix  $\Gamma$  of an optimal mechanism. This allows us to prove the main result:
THEOREM 4.4. Given a marginal workload  $W$  kload and a regular loss function  $\mathcal{L}$ , suppose the optimization problem (either Equation 1 or 2) is feasible. Then there exist nonnegative constants  $\sigma_{\mathbf{A}}^2$  for each  $\mathbf{A} \in \text{closure}(W$  kload) (the constants do not depend on the data), such that the optimal linear Gaussian mechanism  $\mathcal{M}_{\text{opt}}$  for loss function  $\mathcal{L}$  releases  $\mathcal{M}_{\mathbf{A}}(\mathbf{x}; \sigma_{\mathbf{A}}^2)$  for all  $\mathbf{A} \in \text{closure}(W$  kload). Furthermore, any matrix mechanism for this workload must produce at least this many noise measurements during its selection phase.
Proof of Theorem 4.4. Let  $ALL$  represent closure  $(\{Att_1, \ldots, Att_{n_a}\})$  - all possible subsets of attributes. Theorem D.7 guarantees that there is an optimal mechanism whose privacy cost matrix  $\Gamma$  has eigenvectors equal to the rows of the residual matrices. Rows within the same residual matrix have the same eigenvalues. Since privacy cost matrices are symmetric positive semidefinite, this means that for every  $\mathbf{A} \in ALL$ , there exists a nonnegative number  $\beta_{\mathbf{A}}$  such that:
$$
\boldsymbol {\Gamma} \mathbf {R} _ {\mathbf {A}} ^ {T} = \beta_ {\mathbf {A}} \mathbf {R} _ {\mathbf {A}} ^ {T}
$$
By Theorem 3.5 of [47], if two Gaussian linear mechanisms have the same privacy cost matrix then each can be obtained by linearly processing the other. Thus they have the same privacy properties (under any postprocessing invariant privacy definition) and can be used to answer the same queries with the same exact accuracies (under any measure of accuracy). Thus we just need to construct the appropriate mechanism having privacy cost matrix  $\Gamma$ .
For each  $\mathbf{A}$ , let  $\mathbf{Z}_{\mathbf{A}}$  be a matrix with orthonormal rows that span the row space of  $\mathbf{R}_{\mathbf{A}}$ . Thus the rows of  $\mathbf{Z}_{\mathbf{A}}$  are also eigenvectors of  $\Gamma$  (having common eigenvalue  $\beta_{\mathbf{A}}$ ) and the rows of  $\mathbf{Z}_{\mathbf{A}}$  are orthogonal to the rows of  $\mathbf{Z}_{\mathbf{A}'}$  for  $\mathbf{A} \neq \mathbf{A}'$  (a consequence of Theorem 4.2). Thus the set of rows of the  $\mathbf{Z}_{\mathbf{A}}$  for all  $\mathbf{A} \in ALL$  is a complete list of the eigenvectors of  $\Gamma$  (the are linearly independent and span  $\mathbb{R}^d$ ). Thus the (symmetric positive semidefinite) privacy cost matrix  $\Gamma$  can be expressed as:
$$
\mathbf {\Gamma} = \sum_ {\mathbf {A} \in A L L} \beta_ {\mathbf {A}} \mathbf {Z} _ {\mathbf {A}} ^ {T} \mathbf {Z} _ {\mathbf {A}}
$$
and one mechanism that achieves this privacy cost matrix is the one that releases  $\mathbf{Z}_{\mathbf{A}}\mathbf{x} + N(\mathbf{0},\frac{1}{\beta_{\mathbf{A}}} \mathcal{I})$  for each  $\mathbf{A} \in ALL$  for which  $\beta_{\mathbf{A}} \neq 0$  (i.e., we can drop the eigenvectors with eigenvalue equal to 0 as they make no difference to the privacy cost matrix).
Now, since the rows of  $\mathbf{R}_{\mathbf{A}}$  and  $\mathbf{Z}_{\mathbf{A}}$  are independent linear bases of the same subspace, then there exists an invertible matrix  $\mathbf{Y}_{\mathbf{A}}$  such that  $\mathbf{R}_{\mathbf{A}} = \mathbf{Y}_{\mathbf{A}}\mathbf{Z}_{\mathbf{A}}$ . Furthermore,  $\mathbf{R}_{\mathbf{A}}\mathbf{R}_{\mathbf{A}}^{T}$  is invertible and  $\mathbf{Z}_{\mathbf{A}}\mathbf{Z}_{\mathbf{A}}^{T} = \mathcal{I}$  by orthonormality of its rows. Therefore
$$
\begin{array}{l} \mathbf {R} _ {\mathbf {A}} ^ {T} \left(\mathbf {R} _ {\mathbf {A}} \mathbf {R} _ {\mathbf {A}} ^ {T}\right) ^ {- 1} \mathbf {R} _ {\mathbf {A}} = \mathbf {Z} _ {\mathbf {A}} ^ {T} \mathbf {Y} _ {\mathbf {A}} ^ {T} \left(\mathbf {Y} _ {\mathbf {A}} \mathbf {Z} _ {\mathbf {A}} \mathbf {Z} _ {\mathbf {A}} ^ {T} \mathbf {Y} ^ {T}\right) ^ {- 1} \mathbf {Y} _ {\mathbf {A}} \mathbf {Z} _ {\mathbf {A}} \\ = \mathbf {Z} _ {\mathbf {A}} ^ {T} \mathbf {Y} _ {\mathbf {A}} ^ {T} \mathbf {Y} _ {\mathbf {A}} ^ {- T} \left(\mathbf {Z} _ {\mathbf {A}} \mathbf {Z} _ {\mathbf {A}} ^ {T}\right) ^ {- 1} \mathbf {Y} _ {\mathbf {A}} ^ {- 1} \mathbf {Y} _ {\mathbf {A}} \mathbf {Z} _ {\mathbf {A}} \\ = \mathbf {Z} _ {\mathbf {A}} ^ {T} (\mathbf {Z} _ {\mathbf {A}} \mathbf {Z} _ {\mathbf {A}} ^ {T}) ^ {- 1} \mathbf {Z} _ {\mathbf {A}} \\ = \mathbf {Z} _ {\mathbf {A}} ^ {T} \mathbf {Z} _ {\mathbf {A}} \quad \text {b y} \\ \end{array}
$$
$$
\boldsymbol {\Gamma} = \sum_ {\mathbf {A} \in A L L} \beta_ {\mathbf {A}} \mathbf {R} _ {\mathbf {A}} ^ {T} \left(\mathbf {R} _ {\mathbf {A}} \mathbf {R} _ {\mathbf {A}} ^ {T}\right) ^ {- 1} \mathbf {R} _ {\mathbf {A}}
$$
and a mechanism that achieves this privacy cost matrix is the one that releases  $\mathbf{R}_{\mathbf{A}}\mathbf{x} + N(\mathbf{0},\frac{1}{\beta_{\mathbf{A}}}\mathbf{R}_{\mathbf{A}}\mathbf{R}_{\mathbf{A}}^{T})$  for each  $\mathbf{A}$  for which  $\beta_{\mathbf{A}}\neq 0$
We next note that each covariance matrices we propose to use,  $\boldsymbol{\Sigma}_{\mathbf{A}}$ , is proportional to  $\mathbf{R}_{\mathbf{A}}\mathbf{R}_{\mathbf{A}}^{T}$  (they are equal up to positive rescaling). If we define the positive constants  $\kappa_{\mathbf{A}}$  such that  $\mathbf{R}_{\mathbf{A}}\mathbf{R}_{\mathbf{A}}^{T} = \kappa_{\mathbf{A}}\boldsymbol{\Sigma}_{\mathbf{A}}$  then we note that the  $\sigma_{\mathbf{A}}^{2}$  in the theorem statement are equal to  $\kappa_{\mathbf{A}} / \beta_{\mathbf{A}}$ .
Next, we show that the eigenvalues  $\beta_{\mathbf{A}} > 0$  for  $\mathbf{A} \in \mathrm{closure}(Wkload)$  and 0 otherwise, so that the optimal mechanism would not make use of any submechanism  $\mathcal{M}_{\mathbf{A}}$  for  $\mathbf{A} \notin \mathrm{closure}(Wkload)$ .
First, by Theorem 4.2, the rows of  $\mathbf{R}_{\mathbf{A}}$ , for  $\mathbf{A} \in$  closure(Wkload) form an independent linear basis for the space spanned by the rows of the marginals  $\mathbf{Q}_{\mathbf{A}}$  for  $\mathbf{A} \in$  Wkload. If a noisy  $\mathbf{R}_{\mathbf{A}}\mathbf{x}$  is not released for some  $\mathbf{A} \in$  closure(Wkload), then an unbiased noisy answer to at least one of the workload marginals could not be computed. Hence, they must all be part of the optimal mechanism (and thus, because of linear independence, any mechanism needs to get at least as many scalar noisy answers as this). This shows that  $\beta_{\mathbf{A}} > 0$  for all  $\mathbf{A} \in$  closure(Wkload). On the other hand since the rows of  $\mathbf{R}_{\mathbf{A}}$  are orthogonal to the rows of  $\mathbf{R}_{\mathbf{A}'}$  for  $\mathbf{A} \neq \mathbf{A}'$ , getting answers to  $\mathbf{R}_{\mathbf{A}'}\mathbf{x}$ , for  $\mathbf{A}' \notin$  closure(Wkload), cannot help estimate the answers to the marginals  $\mathbf{Q}_{\mathbf{A}}$  for  $\mathbf{A} \in$  Wkload (by Theorem 4.2,  $\mathbf{R}_{\mathbf{A}'}$  are orthogonal to the matrices representing these marginals when  $\mathbf{A}' \notin$  closure(Wkload)). Hence an optimal privacy mechanism cannot waste privacy budget on these irrelevant queries. This shows that  $\beta_{\mathbf{A}'} = 0$  for  $\mathbf{A}' \notin$  closure(Wkload) and concludes the proof.
E The other proofs about base mechanisms
Proof of Theorem 4.2. Consider two sets  $\mathbf{A}' \neq \mathbf{A}''$  and represent there respective residual matrices as:
$$
\mathbf {R} _ {\mathbf {A} ^ {\prime}} = \mathbf {V} _ {1} ^ {\prime} \otimes \dots \otimes \mathbf {V} _ {n _ {a}} ^ {\prime}
$$
$$
\mathbf {R} _ {\mathbf {A} ^ {\prime \prime}} = \mathbf {V} _ {1} ^ {\prime \prime} \otimes \dots \otimes \mathbf {V} _ {n _ {a}} ^ {\prime \prime}
$$
$$
\mathbf {R} _ {\mathbf {A} ^ {\prime}} \mathbf {R} _ {\mathbf {A} ^ {\prime \prime}} ^ {T} = \left(\mathbf {V} _ {1} ^ {\prime} \left(\mathbf {V} _ {1} ^ {\prime \prime}\right) ^ {T}\right) \otimes \dots \otimes \left(\mathbf {V} _ {n _ {a}} ^ {\prime} \left(\mathbf {V} _ {n _ {a}} ^ {\prime \prime}\right) ^ {T}\right)
$$
Since  $\mathbf{A}' \neq \mathbf{A}''$  then one of them contains an attribute, say  $Att_i$ , that the other doesn't have. Therefore either  $\mathbf{V}_i'$  or  $\mathbf{V}_i''$  is the vector  $\mathbf{1}_{|Att_i|}^T$  and the other is  $\mathbf{Sub}_{|Att_i|}$ . However,  $\mathbf{1}_{|Att_i|}^T\mathbf{Sub}_{|Att_i|}^T = \mathbf{0}$  and  $\mathbf{Sub}_{|Att_i|}\mathbf{1}_{|Att_i|} = \mathbf{0}$  and hence  $\mathbf{R_{A'}}\mathbf{R_{A''}}^T = \mathbf{0}$ .
Next, for any set  $\mathbf{A}'$ , it is clear that the row space of  $\mathbf{R}_{\mathbf{A}'}$  is contained in the row space of the marginal matrix  $\mathbf{Q}_{\mathbf{A}'}$ . It is also clear that if  $\mathbf{A}' \subseteq \mathbf{A}$  then the row space of the marginal matrix  $\mathbf{Q}_{\mathbf{A}'}$  is contained in the row space of  $\mathbf{Q}_{\mathbf{A}}$  (because  $\mathbf{Q}_{\mathbf{A}'}$  represents a sub-marginal of  $\mathbf{Q}_{\mathbf{A}}$ ). Thus the rows of the matrices  $\mathbf{R}_{\mathbf{A}'}$ , for all  $\mathbf{A}' \subseteq \mathbf{A}$ , are contained in the rowspace of  $\mathbf{Q}_{\mathbf{A}}$ . Thus we just need to show that the combined rows of  $\mathbf{R}_{\mathbf{A}'}$ , for all  $\mathbf{A}' \subseteq \mathbf{A}$ , are linearly independent and that the number of rows is the same as the number of rows of  $\mathbf{Q}_{\mathbf{A}}$ .
First, each  $\mathbf{R}_{\mathbf{A}'}$  is a kronecker product of matrices with full row rank, and so  $\mathbf{R}_{\mathbf{A}'}$  has full row rank (therefore its rows are linearly independent). Furthermore, since  $\mathbf{R}_{\mathbf{A}'} \mathbf{R}_{\mathbf{A}''}^T = \mathbf{0}$  whenever  $\mathbf{A}' \neq \mathbf{A}''$  this means that the row space of  $\mathbf{R}_{\mathbf{A}'}$  is orthogonal to the row space of  $\mathbf{R}_{\mathbf{A}''}$ . Hence the combined rows of the  $\mathbf{R}_{\mathbf{A}'}$ , for all  $\mathbf{A}' \subseteq \mathbf{A}$ , are linearly independent.
Next, the number of rows in  $\mathbf{R}_{\emptyset}$  is 1 and the number of rows in  $\mathbf{R}_{\mathbf{A}'}$  is equal to  $\prod_{\substack{Att_i \in \mathbf{A}' \\ \mathbf{A}' \neq \emptyset}} (|Att_i| - 1)$  for  $\mathbf{A}' \neq \emptyset$  and so the total number of rows in the residual matrices is  $1 + \sum_{\substack{\mathbf{A}' \subseteq \mathbf{A} \\ \mathbf{A}' \neq \emptyset}} \prod_{\substack{Att_i \in \mathbf{A}' \\ |Att_i| - 1}} (|Att_i| - 1)$ .
By the distributive property of multiplication, this is exactly the same as the product:
$$
\prod_ {A t t _ {i} \in \mathbf {A}} \left(\left(\left| A t t _ {i} \right| - 1\right) + 1\right) = \prod_ {A t t _ {i} \in \mathbf {A}} \left| A t t _ {i} \right|
$$
which is the number of rows in  $\mathbf{Q}_{\mathbf{A}}$  and that proves that the combined rows of  $\mathbf{R}_{\mathbf{A}'}$ , for all  $\mathbf{A}' \subseteq \mathbf{A}$ , form a linearly independent basis for the row span of  $\mathbf{Q}_{\mathbf{A}}$ .
LEMMA E.1. For any  $i$ ,  $\mathbf{Sub}_{|Att_i}^T (\mathbf{Sub}_{|Att_i} | \mathbf{Sub}_{|Att_i}^T)^{-1} \mathbf{Sub}_{|Att_i} = \mathcal{I}_{|Att_i} - \frac{1}{|Att_i|} \mathbf{1}_{|Att_i} \mathbf{1}_{|Att_i}^T$
Proof of Lemma E.1. For the moment, let  $\mathbf{Y}$  denote  $\mathbf{Sub}_{|Att_i|}^T (\mathbf{Sub}_{|Att_i|}\mathbf{Sub}_{|Att_i|}^T)^{-1}\mathbf{Sub}_{|Att_i|}$ . Then we know:
-  $\mathbf{Y}$  is symmetric.
-  $\mathbf{Y}$  is an  $|Att_i| \times |Att_i|$  matrix and its rank is  $|Att_i| - 1$  since the rank of  $\mathrm{Sub}_{|Att_i|}$  is  $|Att_i| - 1$ .
-  $\mathbf{Sub}_{|Att_i|}\mathbf{YSub}_{|Att_i|}^T = \mathbf{Sub}_{|Att_i|}\mathbf{Sub}_{|Att_i|}^T.$
Now, one symmetric solution to the equation  $\mathbf{Sub}_{|Att_i|}\mathbf{X}\mathbf{Sub}_{|Att_i|}^T = \mathbf{Sub}_{|Att_i|}\mathbf{Sub}_{|Att_i|}^T$  is  $\mathbf{X} = \mathcal{I}_{|Att_I|}$  and if  $\mathbf{X}_1$  is another symmetric solution then  $\mathbf{Sub}_{|Att_i|}(\mathcal{I}_{|Att_i|} - \mathbf{X}_1)\mathbf{Sub}_{|Att_i|}^T = \mathbf{0}$ .
This means that  $\mathbf{Sub}_{|Att_i|}\mathbf{v} = \mathbf{0}$  for each eigenvector  $\mathbf{v}$  of the symmetric matrix  $\mathcal{I}_{|Att_i|} - \mathbf{X}_1$  that has a nonzero eigenvalue. Since the rank of  $\mathbf{Sub}_{|Att_i|}$  is  $|Att_i| - 1$ , the only vectors  $\mathbf{v}$  for which  $\mathbf{Sub}_{|Att_i|}\mathbf{v} = \mathbf{0}$  are proportional to  $\mathbf{1}_{|Att_i|}$  (the null space has rank 1) and so  $\mathcal{I}_{|Att_i|} - \mathbf{X}_1 = -c\mathbf{1}_{|Att_i|}\mathbf{1}_{|Att_i|}^T$  for some constant  $c$ .
This means that  $\mathbf{Y}$  (and any other symmetric solution) has the form  $\mathcal{I}_{|Att_i|} + c\mathbf{1}_{|Att_i|}\mathbf{1}_{|Att_i|}^T$ . To find  $c$ , we note that  $\mathbf{Y}$  is not full rank.
By the Sherman-Morrison-Woodbury inversion formula, if  $\mathcal{I}_{|Att_i|} + c\mathbf{1}_{|Att_i|}\mathbf{1}_{|Att_i|}^T$  is invertible, then its inverse is  $\mathcal{I}_{|Att_i|} - c\mathbf{1}_{|Att_i|}\left(1 + c\mathbf{1}_{|Att_i|}^T\mathbf{1}_{|Att_i|}\right)^{-1}\mathbf{1}_{|Att_i|}^T = \mathcal{I}_{|Att_i|} - c\frac{\mathbf{1}_{|Att_i|}\mathbf{1}_{|Att_i|}^T}{1 + c|Att_i|}$ . Thus, to prevent invertibility, we must have  $c = -1 / |Att_i|$ .
Therefore  $\mathbf{Y} = \mathcal{I}_{|Att_i|} - \frac{1}{|Att_i|}\mathbf{1}_{|Att_i|}\mathbf{1}_{|Att_i|}^T$ .
Proof of Theorem 4.5. Without loss of generality (and to simplify notation), assume  $\mathbf{A} = \{Att_1, \ldots, Att_\ell\}$  consists of the first  $\ell$  attributes.
By definition,  $pcost(\mathcal{M}_{\mathbf{A}}(\cdot ;\sigma_{\mathbf{A}}^{2}))$  is the largest diagonal of  $\frac{1}{\sigma^2}\mathbf{R}_{\mathbf{A}}^T\Sigma_{\mathbf{A}}^{-1}\mathbf{R}_{\mathbf{A}}$ . Thus we can write:
$$
\mathbf {R} _ {\mathbf {A}} = \left(\bigotimes_ {i = 1} ^ {\ell} \mathbf {S u b} _ {| A t t _ {i} |}\right) \otimes \left(\bigotimes_ {j = \ell + 1} ^ {n _ {a}} \mathbf {1} _ {| A t t _ {j} |} ^ {T}\right)
$$
$$
\mathbf {R} _ {\mathbf {A}} ^ {T} = \left(\bigotimes_ {i = 1} ^ {\ell} \mathbf {S u b} _ {| A t t _ {i} |} ^ {T}\right) \otimes \left(\bigotimes_ {j = \ell + 1} ^ {n _ {a}} \mathbf {1} _ {| A t t _ {j} |}\right)
$$
$$
\mathbf {H} = \left(\bigotimes_ {i = 1} ^ {\ell} \mathbf {S u b} _ {| A t t _ {i} |}\right) \otimes \left(\bigotimes_ {j = \ell + 1} ^ {n _ {a}} [ 1 ]\right) \quad (\text {r i g h t m o s t k r o n s u s e} 1 \times 1 \text {m a t r i c e s})
$$
$$
\boldsymbol {\Sigma} _ {\mathbf {A}} = \mathbf {H} \mathbf {H} ^ {T} = \left(\bigotimes_ {i = 1} ^ {\ell} \left(\mathbf {S u b} _ {| A t t _ {i} |} \mathbf {S u b} _ {| A t t _ {i} |} ^ {T}\right)\right) \otimes \left(\bigotimes_ {j = \ell + 1} ^ {n _ {a}} [ 1 ]\right)
$$
$$
\begin{array}{l} \boldsymbol {\Sigma} _ {\mathbf {A}} ^ {- 1} = \left(\bigotimes_ {i = 1} ^ {\ell} \left(\mathbf {S u b} _ {| A t t _ {i} |} \mathbf {S u b} _ {| A t t _ {i} |} ^ {T}\right) ^ {- 1}\right) \otimes \left(\bigotimes_ {j = \ell + 1} ^ {n _ {a}} [ 1 ]\right) \\ \mathbf {R} _ {\mathbf {A}} ^ {T} \boldsymbol {\Sigma} _ {\mathbf {A}} ^ {- 1} \mathbf {R} _ {\mathbf {A}} = \left(\bigotimes_ {i = 1} ^ {\ell} \mathbf {S u b} _ {| A t t _ {i} |} ^ {T} \left(\mathbf {S u b} _ {| A t t _ {i} |} \mathbf {S u b} _ {| A t t _ {i} |} ^ {T}\right) ^ {- 1} \mathbf {S u b} _ {| A t t _ {i} |}\right) \otimes \left(\bigotimes_ {j = \ell + 1} ^ {n _ {a}} \mathbf {1} _ {| A t t _ {j} |} [ 1 ] \mathbf {1} _ {| A t t _ {j} |} ^ {T}\right) \tag {8} \\ \end{array}
$$
Now, by Lemma E.1,
$$
\mathbf {S u b} _ {| A t t i |} ^ {T} \left(\mathbf {S u b} _ {| A t t i |} \mathbf {S u b} _ {| A t t i |} ^ {T}\right) ^ {- 1} \mathbf {S u b} _ {| A t t i |} = \mathcal {I} _ {| A t t i |} - \frac {1}{| A t t i |} \mathbf {1} _ {| A t t i |} \mathbf {1} _ {| A t t i |} ^ {T} \tag {9}
$$
Since its diagonals are  $\frac{|Att_i| - 1}{|Att_i|}$ , then combined with Equation 8 it proves the result for  $pcost(\mathcal{M}_{\mathbf{A}}(\cdot, \sigma_{\mathbf{A}}^2))$ .
We next consider the correctness of Algorithm 1. First, we need to show that for the matrix  $\mathbf{H}$  defined in Line 3 in Algorithm 1,  $\mathbf{HQ}_{\mathbf{A}}\mathbf{x} = \mathbf{R}_{\mathbf{A}}\mathbf{x}$ . Then we can write:
$$
\mathbf {Q} _ {\mathbf {A}} = \left(\bigotimes_ {i = 1} ^ {\ell} \mathcal {I} _ {| A t t _ {i} |}\right) \otimes \left(\bigotimes_ {j = \ell + 1} ^ {n _ {a}} \mathbf {1} _ {| A t t _ {j} |} ^ {T}\right) \quad \text {r i g h t m o s t p r o d u c t i s a m a t r i x w i t h 1 r o w}
$$
$$
\mathbf {H} = \left(\bigotimes_ {i = 1} ^ {\ell} \mathbf {S u b} _ {| A t t i _ {i} |}\right) \otimes [ 1 ] \quad (\text {r i g h t m o s t t e r m i s a} 1 \times 1 \text {m a t r i x})
$$
$$
\begin{array}{l} \mathbf {H Q} _ {\mathbf {A}} = \left(\bigotimes_ {i = 1} ^ {\ell} \left(\mathbf {S u b} _ {| A t t _ {i} |} \mathcal {I} _ {| A t t _ {i} |}\right)\right) \otimes \left([ 1 ] \left(\bigotimes_ {j = \ell + 1} ^ {n _ {a}} \mathbf {1} _ {| A t t _ {j} |} ^ {T}\right)\right) \\ = \mathbf {R} _ {\mathbf {A}} \\ \end{array}
$$
Next, we note that if  $\mathbf{z}$  is distributed as  $N(0,\mathbf{I}_m)$  (Line 4 in Algorithm 1) then  $\sigma_{\mathbf{A}}\mathbf{H}\mathbf{z}$  has the distribution  $N(0,\sigma^2\mathbf{H}\mathbf{H}^T) = \pmb{\Sigma}_{\mathbf{A}}$  and hence the algorithm is correct.
F Proofs related to the reconstruction step
Proof of Lemma 4.6. First, if a matrix has a right inverse then that is the pseudo-inverse. Hence we just need to show that  $\mathbf{Sub}_{\ell}\mathbf{Sub}_{\ell}^{\dagger} = \mathcal{I}_{\ell -1}$
Note that the  $j^{\mathrm{th}}$  row of  $\mathbf{Sub}_{\ell}$  has a 1 in position 1, -1 in position  $j + 1$ , and is 0 everywhere else.
Meanwhile, the  $i^{\mathrm{th}}$  column of our claimed representation of  $\mathbf{Sub}_{\ell}^{\dagger}$  has a  $-( \ell - 1) / \ell$  in position  $i + 1$  and  $1 / \ell$  everywhere else.
Hence if  $j \neq i$  then the dot product between row  $j$  of  $\mathbf{Sub}_{\ell}$  and column  $i$  of  $\mathbf{Sub}_{\ell}^{\dagger}$  is 0 since the nonzero elements of the row from  $\mathbf{Sub}_{\ell}$  are being multiplied by  $1 / \ell$  and  $1 / \ell$ .
If  $i = j$  then the corresponding first elements that are multiplied are 1 and  $1 / \ell$  while the elements at position  $i + 1$  being multiplied are  $-1$  and  $-(\ell - 1) / \ell$ . Furthermore,  $1(1 / \ell) + (-1)(-\ell - 1) / \ell) = 1$ .
LEMMA F.1. For any attribute  $Att_i$ , let  $\ell = |Att_i|$ . Then  $\mathbf{Sub}_{\ell}^{\dagger}(\mathbf{Sub}_{\ell}\mathbf{Sub}_{\ell}^{T})\mathbf{Sub}_{\ell}^{\dagger T} = \mathcal{I}_{\ell} - \frac{1}{\ell}\mathbf{1}_{\ell}\mathbf{1}_{\ell}^{T}$
Proof of Lemma F.1. Because  $\mathbf{Sub}_{\ell}$  has linearly independent rows, the pseudo-inverse of it can be expressed as,
$$
\mathbf {S u b} _ {\ell} ^ {\dagger} = \mathbf {S u b} _ {\ell} ^ {T} \left(\mathbf {S u b} _ {\ell} \mathbf {S u b} _ {\ell} ^ {T}\right) ^ {- 1}
$$
From lemma E.1 we get,
$$
\begin{array}{l} \mathbf {S u b} _ {\ell} ^ {\dagger} \mathbf {S u b} _ {\ell} = \mathbf {S u b} _ {\ell} ^ {T} \left(\mathbf {S u b} _ {\ell} \mathbf {S u b} _ {\ell} ^ {T}\right) ^ {- 1} \mathbf {S u b} _ {\ell} \\ = \mathcal {I} _ {\ell} - \frac {1}{\ell} \mathbf {1} _ {\ell} \mathbf {1} _ {\ell} ^ {T} \\ \end{array}
$$
$$
\begin{array}{l} \operatorname {S u b} _ {\ell} ^ {\dagger} \left(\operatorname {S u b} _ {\ell} \operatorname {S u b} _ {\ell} ^ {T}\right) \operatorname {S u b} _ {\ell} ^ {\dagger T} = \left(\operatorname {S u b} _ {\ell} ^ {\dagger} \operatorname {S u b} _ {\ell}\right) \left(\operatorname {S u b} _ {\ell} ^ {\dagger} \operatorname {S u b} _ {\ell}\right) ^ {T} \\ = \left(\mathcal {I} _ {\ell} - \frac {1}{\ell} \mathbf {1} _ {\ell} \mathbf {1} _ {\ell} ^ {T}\right) \left(\mathcal {I} _ {\ell} - \frac {1}{\ell} \mathbf {1} _ {\ell} \mathbf {1} _ {\ell} ^ {T}\right) \\ = \mathcal {I} _ {\ell} - \frac {1}{\ell} \mathbf {1} _ {\ell} \mathbf {1} _ {\ell} ^ {T} - \frac {1}{\ell} \mathbf {1} _ {\ell} \mathbf {1} _ {\ell} ^ {T} + \frac {1}{\ell^ {2}} \mathbf {1} _ {\ell} (\ell) \mathbf {1} _ {\ell} ^ {T} \\ = \mathcal {I} _ {\ell} - \frac {1}{\ell} \mathbf {1} _ {\ell} \mathbf {1} _ {\ell} ^ {T} \\ \end{array}
$$
THEOREM F.2. Let  $\mathbf{A}$  be a set of attributes and let  $\mathbf{Q}_{\mathbf{A}}$  be the matrix representation of the marginal on  $\mathbf{A}$ . Given the matrices  $\mathbf{R}_{\mathbf{A}'}$ , for all  $\mathbf{A}' \in \text{closure}(\mathbf{A})$ , we have  $\mathbf{Q}_{\mathbf{A}} = \sum_{\mathbf{A}' \in \text{closure}(\mathbf{A})} \mathbf{Q}_{\mathbf{A}} \mathbf{R}_{\mathbf{A}'}^{\dagger} \mathbf{R}_{\mathbf{A}'}$ .
Proof of Theorem F.2.
$$
\mathbf {Q} _ {\mathbf {A}} = \bigotimes_ {i = 1} ^ {n _ {a}} \mathbf {K} _ {i} \quad \text {w h e r e , f o r e a c h} i, \mathbf {K} _ {i} = \left\{ \begin{array}{l l} \mathcal {I} _ {| A t t _ {i} |} & \text {i f} A t t _ {i} \in \mathbf {A} \\ \mathbf {1} _ {| A t t _ {i} |} ^ {T} & \text {i f} A t t _ {i} \notin \mathbf {A} \end{array} \right.
$$
$$
\mathbf {R} _ {\mathbf {A} ^ {\prime}} = \bigotimes_ {i = 1} ^ {n _ {a}} \mathbf {V} _ {i} \quad \text {w h e r e , f o r e a c h i ,} \mathbf {V} _ {i} = \left\{ \begin{array}{l l} \mathbf {S u b} _ {| A t t _ {i} |} & \text {i f A t t _ {i} \in \mathbf {A} ^ {\prime}} \\ \mathbf {1} _ {| A t t _ {i} |} ^ {T} & \text {i f A t t _ {i} \notin \mathbf {A} ^ {\prime}} \end{array} \right.
$$
It is straightforward to verify that the following is a right inverse (and hence pseudo-inverse) of  $\mathbf{R}_{\mathbf{A}}$
$$
\mathbf {R} _ {\mathbf {A} ^ {\prime}} ^ {\dagger} = \bigotimes_ {i = 1} ^ {n _ {a}} \mathbf {V} _ {i} ^ {\dagger} \quad \text {w h e r e , f o r e a c h i ,} \mathbf {V} _ {i} ^ {\dagger} = \left\{ \begin{array}{l l} \mathbf {S u b} _ {| A t t _ {i} |} ^ {\dagger} & \text {i f A t t _ {i} \in \mathbf {A} ^ {\prime}} \\ \frac {1}{| A t t _ {i} |} \mathbf {1} _ {| A t t _ {i} |} & \text {i f A t t _ {i} \notin \mathbf {A} ^ {\prime}} \end{array} \right.
$$