SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf{U}_{\{Att_{2},Att_{3}\} \leftarrow \{Att_{2},Att_{3}\}})\omega_{\{Att_{2},Att_{3}\}} = \left(\mathbf{Sub}_{2}^{\dagger}\right)\otimes \left(\mathbf{Sub}_{3}^{\dagger}\right) = \left[ \begin{array}{c}1 / 2\\ -1 / 2 \end{array} \right]\otimes \left[ \begin{array}{cc}1 / 3 & 1 / 3\\ -2 / 3 & 1 / 3\\ 1 / 3 & -2 / 3 \end{array} \right]
$$
Note  $\mathbf{Sub}_2^\dagger$  and  $\mathbf{Sub}_3^\dagger$  are defined in Lemma 4.6.
B.5 Privacy Cost and Marginal Variances
Recall that for a marginal workload  $Wkload$ , we need to run a mechanism  $\mathcal{M}_{\mathbf{A}}$  for each  $\mathbf{A} \in$  closure(Wkload). Theorem 4.5 shows how to compute the privacy cost  $pcost$  of each. In our running example, this means:
$$
\bullet p c o s t (\mathcal {M} _ {\emptyset} (\mathbf {x}; \sigma_ {\emptyset} ^ {2})) = \frac {1}{\sigma_ {\emptyset} ^ {2}}
$$
-  $pcost(\mathcal{M}_{\{Att_1\}}(\mathbf{x}; \sigma_{\{Att_1\}}^2)) = \frac{1}{\sigma_{\{Att_1\}}^2} * \frac{1}{2}$
-  $p$  cost  $\left( {{\mathcal{M}}_{\left\{  A{t}_{2}\right\}  }\left( {\mathbf{x};{\sigma }_{\left\{  A{t}_{2}\right\}  }}^{2}}\right) }\right)  = \frac{1}{{\sigma }_{\left\{  A{t}_{2}\right\}  }^{2}} * \frac{1}{2}$
-  ${pcost}\left( {{\mathcal{M}}_{\{ A{t}_{3}\} }\left( {\mathbf{x};{\sigma }_{\{ A{t}_{3}\} }^{2}}\right) }\right)  = \frac{1}{{\sigma }_{\{ A{t}_{3}\}}^{2}} * \frac{2}{3}$
-  $pcost(\mathcal{M}_{\{Att_1, Att_2\}}(\mathbf{x}; \sigma_{\{Att_1, Att_2\}}^2)) = \frac{1}{\sigma_{\{Att_1, Att_2\}}^2} * \frac{1}{2} * \frac{1}{2}$
and  $p\cos t(\mathcal{M}_{\{Att_2,Att_3\}}(\mathbf{x};\sigma_{\{Att_2,Att_3\}}^2)) = \frac{1}{\sigma_{\{Att_2,Att_3\}}^2}*\frac{1}{2}*\frac{2}{3}$
The total privacy cost is,
$$
\frac {1}{\sigma_ {\emptyset} ^ {2}} + \frac {1}{2} \frac {1}{\sigma_ {\{A t t _ {1} \}} ^ {2}} + \frac {1}{2} \frac {1}{\sigma_ {\{A t t _ {2} \}} ^ {2}} + \frac {2}{3} \frac {1}{\sigma_ {\{A t t _ {3} \}} ^ {2}} + \frac {1}{4} \frac {1}{\sigma_ {\{A t t _ {1} , A t t _ {2} \}} ^ {2}} + \frac {1}{3} \frac {1}{\sigma_ {\{A t t _ {2} , A t t _ {3} \}} ^ {2}}
$$
Thus this is a symbolic expression in terms of the (currently unknown) noise scale parameters  $\sigma_{\mathbf{A}}^{2}$ . According to Definition 2.3, we can convert the privacy cost to the  $\rho$  in  $\rho$ -zCDP by dividing by 2 and we can convert it to the  $\mu$  from  $\mu$ -Gaussian DP by taking the square root.
For our running example,  $Wkload = \{\{Att_1\}, \{Att_1, Att_2\}, \{Att_2, Att_3\}\}$  and we can express the variance of these marginals (after reconstruction from the noisy  $\omega_{\mathbf{A}}$  answers) also in terms of the noise scale parameters. We do this with the help of Theorem 4.7.
- Marginal on  $\{Att_1\}$ . This marginal is reconstructed from the noisy answers  $\omega_{\emptyset}$  and  $\omega_{\{Att_1\}}$  and so the variance of its cells depends only on  $\sigma_{\emptyset}^2$  and  $\sigma_{\{Att_1\}}^2$ . Applying Theorem 4.7, get that the variance in each cell of this marginal is the same and equals.
$$
\left(\sigma_ {\emptyset} ^ {2} * \frac {1}{2 ^ {2}}\right) + \left(\sigma_ {\{A t t _ {1} \}} ^ {2} * \frac {1}{2}\right)
$$
- Marginal on  $\{Att_1, Att_2\}$ . This marginal is reconstructed from  $\omega_{\emptyset}, \omega_{\{Att_1\}}, \omega_{\{Att_2\}}$ , and  $\omega_{\{Att_1, Att_2\}}$  and hence the variance of the cells in the marginal depend on the corresponding 4 noise scale parameters. The cell variance is
$$
\left(\sigma_ {\emptyset} ^ {2} * \frac {1}{2 ^ {2}} * \frac {1}{2 ^ {2}}\right) + \left(\sigma_ {\{A t t _ {1} \}} ^ {2} * \frac {1}{2} * \frac {1}{2 ^ {2}}\right) + \left(\sigma_ {\{A t t _ {2} \}} ^ {2} * \frac {1}{2} * \frac {1}{2 ^ {2}}\right) + \left(\sigma_ {\{A t t _ {1}, A t t _ {2} \}} ^ {2} * \frac {1}{2} * \frac {1}{2}\right)
$$
- Marginal on  $\{Att_2, Att_3\}$ . Similarly, this marginal also depends on 4 noise scale parameters as follows:
$$
\left(\sigma_ {\emptyset} ^ {2} * \frac {1}{2 ^ {2}} * \frac {1}{3 ^ {2}}\right) + \left(\sigma_ {\{A t t _ {2} \}} ^ {2} * \frac {1}{2} * \frac {1}{3 ^ {2}}\right) + \left(\sigma_ {\{A t t _ {3} \}} ^ {2} * \frac {2}{3} * \frac {1}{2 ^ {2}}\right) + \left(\sigma_ {\{A t t _ {2}, A t t _ {3} \}} ^ {2} * \frac {1}{2} * \frac {2}{3}\right)
$$
B.6 The Sum-of-Variances Loss Function
Now we can express the overall privacy cost symbolically in terms of the noise scale parameters. We can also express the variance of each marginal cell symbolically. We can use these symbolic expressions to set up any regular loss function and then run it through a convex optimizer to solve it.
In this section, we give an example for the weighted sum of variances, which is one of the most popular loss functions for the matrix mechanism in research settings (mostly because this loss function is easiest to work with).
Each marginal has a weight, which we set to be 1 to avoid introducing more symbols, and the objective function is computed by adding up the cell variances in a marginal, multiplying by the weight, and adding up over the workload marginals. The marginal on  $\{Att_1\}$  has two cells (so we multiply the cell variance for this marginal, computed in the previous section, by 2). The marginal on  $\{Att_1, Att_2\}$  has 4 cells, and the marginal on  $\{Att_2, Att_3\}$  has 6 cells. Thus, after the dust clears, the sum of the cell variances across the workload marginals is:
$$
= \frac {1 1}{1 2} \sigma_ {\emptyset} ^ {2} + \frac {3}{2} \sigma_ {\{A t t _ {1} \}} ^ {2} + \frac {5}{6} \sigma_ {\{A t t _ {2} \}} ^ {2} + \sigma_ {\{A t t _ {3} \}} ^ {2} + \sigma_ {\{A t t _ {1}, A t t _ {2} \}} ^ {2} + 2 \sigma_ {\{A t t _ {2}, A t t _ {3} \}} ^ {2}
$$
Thus, we can set up the optimization problem: minimize the sum of variances subject to the privacy cost (computed in Section B.5) being less than some constant  $c$ :
$\arg \min_{\sigma_{\emptyset}^2,\sigma_{\{\cdot}}^2 Att_1\}}$  11 3 11/12 6 5 6 6 6 6 6 6
$$
\sigma_ {\{A t t _ {2} \}} ^ {2}, \sigma_ {\{A t t _ {3} \}} ^ {2}
$$
$$
\sigma_ {\{A t t _ {1}, A t t _ {2} \}} ^ {2}, \sigma_ {\{A t t _ {2}, A t t _ {3} \}} ^ {2}
$$
$$
\text {s u c h} \frac {1}{\sigma_ {\emptyset} ^ {2}} + \frac {1}{2} \frac {1}{\sigma_ {\{A t t _ {1} \}} ^ {2}} + \frac {1}{2} \frac {1}{\sigma_ {\{A t t _ {2} \}} ^ {2}} + \frac {2}{3} \frac {1}{\sigma_ {\{A t t _ {3} \}} ^ {2}} + \frac {1}{4} \frac {1}{\sigma_ {\{A t t _ {1} , A t t _ {2} \}} ^ {2}} + \frac {1}{3} \frac {1}{\sigma_ {\{A t t _ {2} , A t t _ {3} \}} ^ {2}} \leq c
$$
If we let the coefficient of  $\sigma_{\mathbf{A}}$  be denoted by  $v_{\mathbf{A}}$  and the coefficient of  $1 / \sigma_{\mathbf{A}}^2$  be denoted by  $p_{\mathbf{A}}$ , then this optimization problem can be written as:
$$
\operatorname *{arg  min}_{\sigma_{\mathbf{A}}^{2}\colon \mathbf{A}\in \text{closure} (Wkload)}\sum_{\substack{\mathbf{A}\in \text{closure} (Wkload)}}v_{\mathbf{A}}\sigma_{\mathbf{A}}^{2}
$$
$$
s.t.\sum_{\mathbf{A}\in \mathrm{closure}(\text{Wkload})}\frac{p_{\mathbf{A}}}{\sigma_{\mathbf{A}}^{2}}\leq c
$$
Lemma H.1 in Section H shows that the optimal solution is obtained by computing:
$$
\begin{array}{l} T = \left(\sum_ {\mathbf {A}} \sqrt {v _ {\mathbf {A}} p _ {\mathbf {A}}}\right) ^ {2} / c = \left(\sqrt {\frac {1 1}{1 2} * 1} + \sqrt {\frac {3}{2} * \frac {1}{2}} + \sqrt {\frac {5}{6} * \frac {1}{2}} + \sqrt {2 / 3} + \sqrt {1 / 4} + \sqrt {2 / 3}\right) ^ {2} / c \\ \approx 2 1. 1 8 / c \\ \end{array}
$$
$$
\sigma_ {\mathbf {A}} ^ {2} = \sqrt {T p _ {\mathbf {A}} / (c v _ {\mathbf {A}})} \approx \sqrt {2 1 . 1 8 p _ {\mathbf {A}} / v _ {\mathbf {A}}} / c
$$
$$
\sigma_ {\varnothing} ^ {2} \approx \sqrt {2 1 . 1 8 * 1 2 / 1 1} / c \approx 4. 8 / c
$$
C Example Variance Calculations
We next illustrate how the  $\sigma_{\mathbf{A}}^{2}$  parameters affect the variance of different marginals based on Theorem 4.7. We illustrate it with a toy dataset that has 5 attributes, each with 3 possible values. In this discussion, the variance of a marginal is the largest variance of its cells (all cells within the same marginal have the same variance, so the variance of a marginal is basically the variance of any cell).
EXAMPLE C.1. The objective function is to minimize the max variance among all marginals while satisfying  $\mu$ -Gaussian  $DP$  with  $\mu = 1$ . In this case all marginals end up with the same variance of 7.594 and the  $\sigma_{\mathbf{A}}^{2}$  parameters are:
$\sigma_{\emptyset}^{2} = 7.594$
-  $\sigma_A^2 = 10.125$  when A contains  $I$  attribute.
-  $\sigma_A^2 = 13.5$  when A contains 2 attributes.
-  $\sigma_A^2 = 18$  when A contains 3 attributes.
-  $\sigma_A^2 = 24$  when  $\mathbf{A}$  contains 4 attributes.
-  $\sigma_A^2 = 32$  when  $\mathbf{A}$  contains all of the attributes.
EXAMPLE C.2. The objective function is to minimize the weighted max variance among all marginals (i.e., minimize  $\max_{m} \text{weight}_{m} * \text{var}(m)$ ). We set the weight of a marginal to be 3 if it is the 5-way marginal and 1 otherwise. This objective function basically says we want more accuracy on the 5-way marginal (it has higher weight). Again the privacy constraint is 1-Gaussian DP. In the optimal solution, each cell in the 5-way marginal has variance 2.718, the 4-way marginals have variance 5.528, and the other marginals have variance 8.154. The  $\sigma_{\mathbf{A}}^{2}$  parameters are:
$\sigma_{\emptyset}^{2} = 8.154$
-  $\sigma_A^2 = 10.871$  when A contains 1 attribute.
-  $\sigma_A^2 = 14.495$  when A contains 2 attributes.
-  $\sigma_A^2 = 19.327$  when A contains 2 attributes.
-  $\sigma_A^2 = 12.477$  when A contains 2 attributes.
-  $\sigma_A^2 = 4.159$  when  $\mathbf{A}$  contains all of the attributes.
For the variance calculation, we show this for Example C.2.
We start with the variance of a one-way marginal, say the marginal on attribute 1 (i.e.,  $\mathbf{A} = \{1\}$ ). The calculation involves  $\sigma_{\{1\}}^2$  and  $\sigma_{\emptyset}^2$  (because  $\emptyset \subseteq \{1\}$ ) and the summation has two terms. The term involving  $\sigma_{\emptyset}^2$  is multiplied by 1 (the first product in the variance expression for Theorem 4.7, since an empty product is 1) and that is multiplied by  $1/9$  (the second product, since each attribute has 3 possible values). Meanwhile, the term involving  $\sigma_{\{1\}}^2$  would be multiplied by  $2/3$  (first product) and 1 (since the second product is empty). Thus the variance for the marginal on attribute 1 is:  $8.154 * 1 * 1/9 + 10.871 * 2/3 * 1 \approx 8.154$ .
For the variance on the 5-way marginal, it consists of 1 term for  $\mathbf{A}' = \emptyset$ , 5 terms for when  $\mathbf{A}'$  has 1 attribute since there are 5 such  $\mathbf{A}'$  (but all the terms have the same value), 10 terms for when  $\mathbf{A}'$  has 2 attributes (all have the same value), 10 terms for when  $\mathbf{A}'$  has 3 attributes (again these terms all are equal), 5 terms when  $\mathbf{A}'$  has 4 attributes, and 1 term for when  $\mathbf{A}' = \{1,2,3,4,5\}$ . First, we note that the total number of terms is 32, which is not bad because the number of cells in the 5-way marginal is  $3^5$  (i.e., much larger). The expression for the variance is:
$$
1 * (8. 1 5 4 * (2 / 3) ^ {0} * (1 / 9) ^ {5})
$$
(the term involving  $\sigma_{\emptyset}^{2}$ )
$$
+ 5 * (1 0. 8 7 1 * (2 / 3) ^ {1} * (1 / 9) ^ {4})
$$
(the sum of the 5 terms having  $\mathbf{A}'$  with 1 attribute)
$$
+ 1 0 * (1 4. 4 9 5 * (2 / 3) ^ {2} * (1 / 9) ^ {3})
$$
(the sum of the 10 terms having  $\mathbf{A}'$  with 2 attributes, etc.)
$$
\begin{array}{l} + 1 0 * (1 9. 3 2 7 * (2 / 3) ^ {3} * (1 / 9) ^ {2}) \\ + 5 * (1 2. 4 7 7 * (2 / 3) ^ {4} * (1 / 9) ^ {1}) \\ + 1 * (4. 1 5 9 * (2 / 3) ^ {5} * (1 / 9) ^ {0}) \\ \approx 2. 7 1 8 \\ \end{array}
$$
So the main takeaway here is that when evaluating the variance of the 5-way marginal, the  $1/9$  terms in the products reduce the influence of any  $\sigma_{\mathbf{A}'}^2$  for which  $\mathbf{A}'$  has a small number of attributes. Hence the most important value is  $\sigma_{\{1,2,3,4,5\}}^2$  followed by the  $\sigma_{\mathbf{A}'}^2$  for the 5 sets  $\mathbf{A}'$  that contain 4 attributes, etc.
D Optimality Proof of ResidualPlanner
In this section, we prove the optimality of ResidualPlanner. It takes advantage of the symmetry inherent in marginals and regular loss functions.
The proof sketch is the following. Given one optimal mechanism  $\mathcal{M}$ , we can create a variation  $\widetilde{\mathcal{M}}$  of that does the following. (1)  $\widetilde{\mathcal{M}}$  modifies each input record by applying some invertible function  $f_{i}$  to each attribute  $Att_{i}$  (for example, if  $Att_{i}$  is a tertiary attribute, we can modify the value of  $Att_{i}$  for each record using a function  $f_{i}$  where  $f_{i}(1) = 3$ ,  $f_{i}(2) = 1$ ,  $f_{i}(3) = 2$ ). This step can be viewed as simply renaming the attribute values within an attribute. (2) Then  $\widetilde{\mathcal{M}}$  runs  $\mathcal{M}$  on the resulting dataset. Note that marginals can be reconstructed from the output of  $\widetilde{\mathcal{M}}$  by first running the reconstruction one would do for  $\mathcal{M}$  and then inverting the  $f_{i}$  functions on the resulting marginals (i.e., rearranging the cells in each marginal to undo the within-attribute renaming caused by the  $f_{i}$ ). This variation  $\widetilde{\mathcal{M}}$  has the same privacy properties as  $\mathcal{M}$  and the same loss (due to the regularity condition on the loss).
Hence  $\widetilde{\mathcal{M}}$  is also optimal. Then we create yet another optimal privacy mechanism  $\mathcal{M}^*$  that splits the privacy budget across all variations of  $\mathcal{M}$  and returns their outputs. It turns out that the privacy cost matrix of  $\mathcal{M}^*$  has eigenvectors that are equal to the rows of the residual matrices  $\mathbf{R}_{\mathbf{A}}$  used by ResidualPlanner. Rewriting the privacy cost matrix of  $\mathcal{M}^*$  using this eigendecomposition, we create another mechanism (the mechanism that runs the base mechanisms of ResidualPlanner) that has the same privacy cost matrix and the same value for the loss and hence is optimal.
The rest of this section explains these steps in details with formal proofs and running commentary that helps to better understand the notation and constructs in the proof.
D.1 Notation Review
We first start with a review of key notation. Recall that a dataset  $\mathcal{D} = \{r_1, \ldots, r_n\}$  is a collection of records. Each record  $r_i$  contains attributes  $Att_1, \ldots, Att_{n_a}$  and each attribute  $Att_j$  can take values  $a_1^{(j)}, \ldots, a_{|Att_j|}^{(j)}$ .
An attribute value  $a_{i}^{(j)}$  for attribute  $Att_{j}$  can be represented as a vector using one-hot encoding. Specifically, let  $e_{i}^{(j)}$  be a row vector of size  $|Att_{j}|$  with a one in component  $i$  and 0 everywhere else. In this way,  $e_{i}^{(j)}$  is a representation of  $a_{i}^{(j)}$ .
A record  $r$  with attributes  $Att_{1} = a_{i_{1}}^{(1)}$ ,  $Att_{2} = a_{i_{2}}^{(2)}$ , ...,  $Att_{n_{a}} = a_{i_{n_{a}}}^{(n_{a})}$  can thus be represented as the kron product  $e_{i_1}^{(1)}\otimes e_{i_2}^{(2)}\otimes \dots \otimes e_{i_{n_a}}^{(n_a)}$ . This vector has a 1 in exactly one position and 0s everywhere else. The position of the 1 is the index of record  $r$ .
Thus, a data vector  $\mathbf{x}$  is a vector of integers. The value at index  $i$  is the number of times the record associated with index  $i$  appears in  $\mathcal{D}$ .
D.2 Permutations
For each attribute  $Att_i$ , let  $\Pi^{(i)}$  be the set of permutations on the numbers  $1, \ldots, |Att_i|$ , so that each  $\pi \in \Pi^{(i)}$  can be interpreted as a permutation (or renaming) of the attributes values of  $Att_i$ . We can also view  $\pi$  as a function on vectors of size  $|Att_i|$  that permutes their coordinates. That is, the  $i^{\text{th}}$  coordinate of a vector  $\mathbf{y}$  is the  $\pi(i)^{\text{th}}$  coordinate of  $\pi(\mathbf{y})$ .
One can select a permutation for each attribute  $\pi^{(1)}\in \Pi^{(1)},\ldots ,\pi^{(n_a)}\in \Pi^{(n_a)}$  and use it to define a permutation over records. This permutation maps a record represented by the kron product  $e_{i_1}^{(1)}\otimes e_{i_2}^{(2)}\otimes \dots \otimes e_{i_{na}}^{(n_a)}$  into  $\pi^{(1)}(e_{i_1}^{(1)})\otimes \pi^{(2)}(e_{i_2}^{(2)})\otimes \dots \otimes \pi^{(n_a)}(e_{i_{na}}^{(n_a)})$ . We can think of this permutation  $\pi = (\pi^{(1)},\ldots ,\pi^{(n_a)})$  as a function that independently renames each attribute value in a record. Thus this permutation can be extended to datavectors  $\mathbf{x}$ . The value of  $\mathbf{x}$  at the index associated with record  $r$  is the value of  $\pi (\mathbf{x})$  at the index associated with record  $\pi (r)$ . Another way to look at it is that  $\pi (\mathbf{x})$  is the histogram associated with the dataset  $\{\pi (r_1),\pi (r_2),\ldots ,\pi (r_n)\}$ . This permutation can be represented as a permutation matrix  $\mathbf{W}_{\pi}$  such that  $\mathbf{W}_{\pi}\mathbf{x} = \pi (\mathbf{x})$ .
We let  $\Pi = \Pi^{(1)}\times \dots \times \Pi^{(n_a)}$  be the set of all such permutations. We call this the space of renaming permutations since each  $\pi \in \Pi$  renames the values of each attribute separately.
Our first result is that permutation does not affect the privacy parameters of a mechanism.
LEMMA D.1. Let  $\mathcal{M}(\mathbf{x})\equiv \mathbf{B}\mathbf{x} + N(\mathbf{0},\boldsymbol {\Sigma})$  be a mechanism that satisfies  $\rho -zCDP$ $(\epsilon ,\delta)$  -approximate  $DP$  and  $\mu$  -Gaussian  $DP$  .Let  $\pi$  be a permutation of the indices of  $\mathbf{x}$  and  $\mathbf{W}_{\pi}$  the corresponding permutation matrix. Then  $\mathcal{M}_{\pi}(\mathbf{x})\equiv \mathbf{BW}_{\pi}\mathbf{x} + N(\mathbf{0},\boldsymbol {\Sigma})$  satisfies  $\rho -zCDP$ $(\epsilon ,\delta)$  -approximate  $DP$  and  $\mu$  -Gaussian  $DP$  (i.e., with the same privacy parameters).
Proof. The privacy cost  $pcost(\mathcal{M})$  of  $\mathcal{M}$  is the largest diagonal of  $\mathbf{B}^T \boldsymbol{\Sigma}^{-1} \mathbf{B}$ . The privacy cost  $pcost(\mathcal{M}_{\pi})$  of  $\mathcal{M}_{\pi}$  is the largest diagonal of  $\mathbf{W}_{\pi}^{T} \mathbf{B}^{T} \boldsymbol{\Sigma}^{-1} \mathbf{B} \mathbf{W}_{\pi}$ . The effect of  $\mathbf{W}_{\pi}$  on both sides is to permute the rows and columns of  $\mathbf{B}^T \boldsymbol{\Sigma}^{-1} \mathbf{B}$  in the same way. Thus the diagonals of  $\mathbf{B}^T \boldsymbol{\Sigma}^{-1} \mathbf{B}$  and  $\mathbf{W}_{\pi}^{T} \mathbf{B}^{T} \boldsymbol{\Sigma}^{-1} \mathbf{B} \mathbf{W}_{\pi}$  are the same up to permutation and hence  $\mathcal{M}$  and  $\mathcal{M}_{\pi}$  have the same privacy cost and therefore the same privacy parameters.
The next result is that a renaming permutation preserves the accuracy of a marginal derived from the answer to a mechanism.
LEMMA D.2. Let  $W$  be a workload on marginals. Let  $\mathcal{M}(\mathbf{x}) \equiv \mathbf{B}\mathbf{x} + N(\mathbf{0},\Sigma)$  be a mechanism whose output can be used to provide unbiased estimates of those marginals. Let  $\pi \in \Pi$  be a renaming permutation and  $\mathbf{W}_{\pi}$  the corresponding permutation matrix. Define  $\mathcal{M}_{\pi}(\mathbf{x}) \equiv \mathbf{BW}_{\pi}\mathbf{x} + N(\mathbf{0},\Sigma)$ . Then unbiased answers to  $W$  kload can be obtained from the output of  $\mathcal{M}_{\pi}$  and for any regular loss function  $\mathcal{L}$  (Definition 4.1),  $\mathcal{L}(Var(\mathbf{A}_1;\mathcal{M}),\dots,Var(\mathbf{A}_k;\mathcal{M})) = \mathcal{L}(Var(\mathbf{A}_1;\mathcal{M}_\pi),\dots,Var(\mathbf{A}_k;\mathcal{M}_\pi))$
Proof. For each set of attributes  $\mathbf{A}_i \in Wkload$ , let  $\mathbf{Q}_{\mathbf{A}_i}$  be the query matrix of the marginal (i.e., the true marginal is computed as  $\mathbf{Q}_{\mathbf{A}_i} \mathbf{x}$ ). Then the best linear unbiased estimate of the marginal on  $\mathbf{A}_i$  from the output  $\omega$  of  $\mathcal{M}$  is  $\mathbf{Q}_{\mathbf{A}_i} (\mathbf{B}^T \boldsymbol{\Sigma}^{-1} \mathbf{B})^\dagger \mathbf{B}^T \boldsymbol{\Sigma}^{-1} \omega$  and  $Var(\mathbf{A}_i; \mathcal{M})$  is the diagonal of the covariance matrix of this estimate, which is  $\mathbf{Q}_{\mathbf{A}_i} (\mathbf{B}^T \boldsymbol{\Sigma}^{-1} \mathbf{B})^\dagger \mathbf{Q}_{\mathbf{A}_i}^T$ . Meanwhile, the best linear unbiased estimate of the marginal on  $\mathbf{A}_i$  from the output  $\omega'$  of  $\mathcal{M}_{\pi}$  is  $\mathbf{Q}_{\mathbf{A}_i} (\mathbf{W}_{\pi}^T \mathbf{B}^T \boldsymbol{\Sigma}^{-1} \mathbf{B} \mathbf{W}_{\pi})^\dagger \mathbf{W}_{\pi}^T \mathbf{B}^T \boldsymbol{\Sigma}^{-1} \omega'$  and  $Var(\mathbf{A}_i; \mathcal{M})$  is the diagonal of  $\mathbf{Q}_{\mathbf{A}_i} (\mathbf{W}_{\pi}^T \mathbf{B}^T \boldsymbol{\Sigma}^{-1} \mathbf{B} \mathbf{W}_{\pi})^\dagger \mathbf{Q}_{\mathbf{A}_i}^T = \mathbf{Q}_{\mathbf{A}_i} \mathbf{W}_{\pi}^T (\mathbf{B}^T \boldsymbol{\Sigma}^{-1} \mathbf{B})^\dagger \mathbf{W}_{\pi} \mathbf{Q}_{\mathbf{A}_i}^T$ .
We note that  $\mathbf{Q}_{\mathbf{A}_i}\mathbf{W}_\pi^T$  is a permutation of the rows of  $\mathbf{Q}_{\mathbf{A}_i}$  (computing a marginal on a dataset in which attribute values within the same attribute are renamed is the same as computing the marginal on the original dataset and then renaming the marginal cells, which is permutation of the output of the marginal computation).
Therefore the diagonals of  $\mathbf{Q}_{\mathbf{A}_i}(\mathbf{B}^T\pmb{\Sigma}^{-1}\mathbf{B})^\dagger \mathbf{Q}_{\mathbf{A}_i}^T$  and  $\mathbf{Q}_{\mathbf{A}_i}(\mathbf{W}_\pi^T\mathbf{B}^T\pmb{\Sigma}^{-1}\mathbf{B}\mathbf{W}_\pi)^\dagger \mathbf{Q}_{\mathbf{A}_i}^T$  are the same up to permutation. Hence the vector  $Var(\mathbf{A}_i;\mathcal{M})$  is the same as the vector  $Var(\mathbf{A}_i;\mathcal{M}_\pi)$  up to permutation of the components, and hence does not affect a regular loss function  $\mathcal{L}$ .
Finally, we show that there exists an optimal mechanism whose privacy cost matrix exhibits symmetries defined by the set of permutaitons  $\Pi$ .
LEMMA D.3. Let  $Wkload = \{\mathbf{A}_1, \dots, \mathbf{A}_k\}$  be a workload of marginal queries. Let  $\mathcal{L}$  be a regular loss function. Let  $U$  be the set of all Gaussian linear mechanisms that can provide unbiased answers to the marginals in the  $Wkload$ . Let  $\gamma$  be a real number. Then whenever either of the following optimization problems are feasible,
$$
\min  _ {\mathcal {M} \in U} p c o s t (\mathcal {M}) \quad s. t. \mathcal {L} (V a r (\mathbf {A} _ {1}; \mathcal {M}), \dots , V a r (\mathbf {A} _ {k}; \mathcal {M})) \leq \gamma
$$
$$
\min  _ {\mathcal {M} \in U} \mathcal {L} (V a r (\mathbf {A} _ {1}; \mathcal {M}), \dots , V a r (\mathbf {A} _ {k}; \mathcal {M})) \quad s. t. p c o s t (\mathcal {M}) \leq \gamma
$$
the feasible optimization problem is minimized by some mechanism of the form  $\overline{\mathcal{M}} (\mathbf{x})\equiv \overline{\mathbf{B}}\mathbf{x} + N(\mathbf{0},\overline{\mathbf{\Sigma}})$  whose privacy cost matrix  $\Gamma \equiv \overline{\mathbf{B}}^T\overline{\mathbf{\Sigma}}^{-1}\overline{\mathbf{B}}$  has the following symmetries: for all renaming permutations  $\pi \in \Pi$  (with  $\mathbf{W}_{\pi}$  being the associated permutation matrix), we have  $\Gamma = \mathbf{W}_{\pi}^{T}\Gamma \mathbf{W}_{\pi}$  (in other words, permuting the rows has no effect as long as the columns are permuted in the same way).
Proof. Let  $\mathcal{M}_{opt}(\mathbf{x})\equiv \mathbf{B}_{opt}\mathbf{x} + N(\mathbf{0},\Sigma_{opt})$  be an optimal mechanism to one of these problems. It may not have the required symmetries, but from it we will construct an optimal mechanism that does.
For a permutation  $\pi$  (and corresponding permutation matrix  $\mathbf{W}_{\pi}$ ) and a positive number  $\lambda$ , consider the mechanism  $\mathcal{M}_{\pi, \lambda}(\mathbf{x}) \equiv \mathbf{B}_{opt} \mathbf{W}_{\pi} \mathbf{x} + N(\mathbf{0}, \lambda \boldsymbol{\Sigma}_{opt})$ . By Lemma D.2, this mechanism also answers the marginals in  $Wkload$ .
Now consider the mechanism  $\overline{\mathcal{M}}$  which, on input  $\mathbf{x}$  outputs the result of  $\mathcal{M}_{\pi ,|\Pi |}$  for all  $\pi \in \Pi$