SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {Q} _ {\mathbf {A}} \mathbf {R} _ {\mathbf {A} ^ {\prime}} ^ {\dagger} \mathbf {R} _ {\mathbf {A} ^ {\prime}} = \bigotimes_ {i = 1} ^ {n _ {a}} \mathbf {K} _ {i} \mathbf {V} _ {i} ^ {\dagger} \mathbf {V} _ {i} \quad \text {w h e r e , f o r e a c h i , K} _ {i} \mathbf {V} _ {i} ^ {\dagger} \mathbf {V} _ {i} = \left\{ \begin{array}{l l} \mathbf {S u b} _ {| A t t _ {i} |} ^ {\dagger} \mathbf {S u b} _ {| A t t _ {i} |} & \text {i f A t t _ {i} \in \mathbf {A} ^ {\prime}} \\ \frac {1}{| A t t _ {i} |} \mathbf {1} _ {| A t t _ {i} |} \mathbf {1} _ {| A t t _ {i} |} ^ {T} & \text {i f A t t _ {i} \in \mathbf {A} / \mathbf {A} ^ {\prime}} \\ \mathbf {1} _ {| A t t _ {i} |} ^ {T} & \text {i f A t t _ {i} \notin \mathbf {A}} \end{array} \right.
$$
Because  $\mathbf{Sub}_{|Att_i|}$  has linearly independent rows, the pseudo-inverse of it can be expressed as,
$$
\mathbf {S u b} _ {| A t t _ {i}} ^ {\dagger} = \mathbf {S u b} _ {| A t t _ {i}} ^ {T} \left(\mathbf {S u b} _ {| A t t _ {i}} | \mathbf {S u b} _ {| A t t _ {i}} ^ {T}\right) ^ {- 1}
$$
$$
\begin{array}{l} \mathbf {S u b} _ {| A t t _ {i} |} ^ {\dagger} \mathbf {S u b} _ {| A t t _ {i} |} = \mathbf {S u b} _ {| A t t _ {i} |} ^ {T} \left(\mathbf {S u b} _ {| A t t _ {i} |} \mathbf {S u b} _ {| A t t _ {i} |} ^ {T}\right) ^ {- 1} \mathbf {S u b} _ {| A t t _ {i} |} \\ = \mathcal {I} _ {| A t t _ {i} |} - \frac {1}{| A t t _ {i} |} \mathbf {1} _ {| A t t _ {i} |} \mathbf {1} _ {| A t t _ {i} |} ^ {T} \\ \end{array}
$$
$$
\mathbf {Q} _ {\mathbf {A}} \mathbf {R} _ {\mathbf {A} ^ {\prime}} ^ {\dagger} \mathbf {R} _ {\mathbf {A} ^ {\prime}} = \bigotimes_ {i = 1} ^ {n _ {a}} \mathbf {T} _ {i} \quad \text {w h e r e , f o r e a c h i , T _ {i} =} \left\{ \begin{array}{l l} \mathcal {I} _ {| A t t _ {i} |} - \frac {1}{| A t t _ {i} |} \mathbf {1} _ {| A t t _ {i} |} \mathbf {1} _ {| A t t _ {i} |} ^ {T} & \text {i f A t t _ {i} \in \mathbf {A} ^ {\prime}} \\ \frac {1}{| A t t _ {i} |} \mathbf {1} _ {| A t t _ {i} |} \mathbf {1} _ {| A t t _ {i} |} ^ {T} & \text {i f A t t _ {i} \in \mathbf {A} / \mathbf {A} ^ {\prime}} \\ \mathbf {1} _ {| A t t _ {i} |} ^ {T} & \text {i f A t t _ {i} \notin \mathbf {A}} \end{array} \right.
$$
Without loss of generality (and to simplify notation), assume  $\mathbf{A} = \{Att_1, \ldots, Att_\ell\}$  consists of the first  $\ell$  attributes,
$$
\mathbf {Q} _ {\mathbf {A}} = \left(\bigotimes_ {i = 1} ^ {\ell} \mathcal {I} _ {| A t t _ {i} |}\right) \otimes \left(\bigotimes_ {i = \ell + 1} ^ {n _ {a}} \mathbf {1} _ {| A t t _ {i} |} ^ {T}\right)
$$
$$
\begin{array}{l} \sum_ {\mathbf {A} ^ {\prime} \in \operatorname {c l o s u r e} (\mathbf {A})} \mathbf {Q} _ {\mathbf {A}} \mathbf {R} _ {\mathbf {A} ^ {\prime}} ^ {\dagger} \mathbf {R} _ {\mathbf {A} ^ {\prime}} = \sum_ {\mathbf {A} ^ {\prime} \in \operatorname {c l o s u r e} (\mathbf {A})} \left(\bigotimes_ {i = 1} ^ {n _ {a}} \mathbf {T} _ {i}\right) \\ = \sum_ {\mathbf {A} ^ {\prime} \in \operatorname {c l o s u r e} (\mathbf {A})} \left(\left(\bigotimes_ {i = 1} ^ {\ell} \mathbf {T} _ {i}\right) \otimes \left(\bigotimes_ {i = \ell + 1} ^ {n _ {a}} \mathbf {1} _ {| A t t _ {i} |} ^ {T}\right)\right) \\ = \left(\sum_ {\mathbf {A} ^ {\prime} \in \operatorname {c l o s u r e} (\mathbf {A})} \binom {\ell} {i = 1} \mathbf {T} _ {i}\right) \otimes \left(\bigotimes_ {i = \ell + 1} ^ {n _ {a}} \mathbf {1} _ {| A t t _ {i} |} ^ {T}\right) \\ \end{array}
$$
where, for each  $i\leq \ell$ $\begin{array}{rl}\mathbf{T}_i = \left\{ \begin{array}{ll}\mathcal{I}_{|Att_i|} - \frac{1}{|Att_i|}\mathbf{1}_{|Att_i|}\mathbf{1}_{|Att_i|}^T & \mathrm{if} Att_i\in \mathbf{A}'\\ \frac{1}{|Att_i|}\mathbf{1}_{|Att_i|}\mathbf{1}_{|Att_i|}^T & \mathrm{if} Att_i\in \mathbf{A} / \mathbf{A}' \end{array} \right. \end{array}$
Because of the distributive property of the Kronecker product,
$$
\begin{array}{l} \bigotimes_ {i = 1} ^ {\ell} \mathcal {I} _ {| A t t _ {i} |} = \bigotimes_ {i = 1} ^ {\ell} \left(\left(\mathcal {I} _ {| A t t _ {i} |} - \frac {1}{| A t t _ {i} |} \mathbf {1} _ {| A t t _ {i} |} \mathbf {1} _ {| A t t _ {i} |} ^ {T}\right) + \frac {1}{| A t t _ {i} |} \mathbf {1} _ {| A t t _ {i} |} \mathbf {1} _ {| A t t _ {i} |} ^ {T}\right) \\ = \sum_ {\mathbf {A} ^ {\prime} \in \operatorname {c l o s u r e} (\mathbf {A})} \left(\bigotimes_ {i = 1} ^ {\ell} \mathbf {T} _ {i}\right) \\ \end{array}
$$
Therefore, combining everything together,
$$
\begin{array}{l} \sum_ {\mathbf {A} ^ {\prime} \in \text {c l o s u r e} (\mathbf {A})} \mathbf {Q} _ {\mathbf {A}} \mathbf {R} _ {\mathbf {A} ^ {\prime}} ^ {\dagger} \mathbf {R} _ {\mathbf {A} ^ {\prime}} = \left(\sum_ {\mathbf {A} ^ {\prime} \in \text {c l o s u r e} (\mathbf {A})} \left(\bigotimes_ {i = 1} ^ {\ell} \mathbf {T} _ {i}\right)\right) \otimes \left(\bigotimes_ {i = \ell + 1} ^ {n _ {a}} \mathbf {1} _ {| A t t _ {i}} ^ {T} \right| \\ = \left(\bigotimes_ {i = 1} ^ {\ell} \mathcal {I} _ {| A t t _ {i} |}\right) \otimes \left(\bigotimes_ {i = \ell + 1} ^ {n _ {a}} \mathbf {1} _ {| A t t _ {i} |} ^ {T}\right) \\ = \mathbf {Q} _ {\mathbf {A}} \\ \end{array}
$$
THEOREM 4.7. Given a marginal workload  $W$  kload and positive numbers  $\sigma_{\mathbf{A}}^{2}$  for each  $\mathbf{A} \in$  closure(Wkload), let  $\mathcal{M}$  be the mechanism that outputs  $\{\mathcal{M}_{\mathbf{A}}(\mathbf{x};\sigma_{\mathbf{A}}^{2}) : \mathbf{A} \in \text{closure}(W \text{kload})\}$  and let  $\{\omega_{\mathbf{A}} : \mathbf{A} \in \text{closure}(W \text{kload})\}$  denote the privacy-preserving noisy answers (e.g.,  $\omega_{\mathbf{A}} = \mathcal{M}_{\mathbf{A}}(\mathbf{x},\sigma^{2})$ ). Then for any marginal on an attribute set  $\mathbf{A} \in \text{closure}(W \text{kload})$ , Algorithm 2 returns the unique linear unbiased estimate of  $\mathbf{Q}_{\mathbf{A}}\mathbf{x}$  (i.e., answers to the marginal query) that can be computed from the noisy differentially private answers.
Proof of Theorem 4.7. We first verify the correctness and uniqueness of the reconstruction in Algorithm 2. Uniqueness follows from the fact that the rows from all the matrices  $\mathbf{R}_{\mathbf{A}}$  (for  $\mathbf{A} \in \mathrm{closure}(Wkload)$ ) are linearly independent.
Consider Line 3 from Algorithm 2. It uses a  $\mathbf{U}$  matrix that depends on both the attributes  $\mathbf{A}$  of the marginal one wants to compute and a subset  $\mathbf{A}'$  of it. So, for notational dependence, we write it as  $\mathbf{U}_{\mathbf{A} \leftarrow \mathbf{A}'}$ . It is straightforward to verify that  $\mathbf{U}_{\mathbf{A} \leftarrow \mathbf{A}'} = \mathbf{Q}_{\mathbf{A}} \mathbf{R}_{\mathbf{A}'}^{\dagger}$ . From Theorem F.2,  $\mathbf{Q}_{\mathbf{A}} \mathbf{x} = \sum_{\mathbf{A}' \subseteq \mathbf{A}} \mathbf{Q}_{\mathbf{A}} \mathbf{R}_{\mathbf{A}'}^{\dagger} \mathbf{R}_{\mathbf{A}'} \mathbf{x} = \sum_{\mathbf{A}' \subseteq \mathbf{A}} \mathbf{U}_{\mathbf{A} \leftarrow \mathbf{A}'} \mathbf{R}_{\mathbf{A}'} \mathbf{x}$ , and so Algorithm 2 is correct because each  $\omega_{\mathbf{A}'}$  is an unbiased noisy version of  $\mathbf{R}_{\mathbf{A}'} \mathbf{x}$ .
Having established that the  $\mathbf{q}$  returned by Line 5 in Algorithm 2 is an unbiased estimate of the marginal query answer  $\mathbf{Q}_{\mathbf{A}}\mathbf{x}$ , the next step is to compute the covariance matrix  $E[\mathbf{qq}^T]$ .
$$
\begin{array}{l} E [ \mathbf {q q} ^ {T} ] = E \left[ \sum_ {\mathbf {A} ^ {\prime} \subseteq \mathbf {A}} \mathbf {U} _ {\mathbf {A} \leftarrow \mathbf {A} ^ {\prime}} \left(\omega_ {\mathbf {A} ^ {\prime}} \omega_ {\mathbf {A} ^ {\prime}} ^ {T}\right) \mathbf {U} _ {\mathbf {A} \leftarrow \mathbf {A} ^ {\prime}} ^ {T} \right] \\ = \sum_ {\mathbf {A} ^ {\prime} \subseteq \mathbf {A}} \mathbf {U} _ {\mathbf {A} \leftarrow \mathbf {A} ^ {\prime}} \left(\sigma_ {\mathbf {A} ^ {\prime}} ^ {2} \boldsymbol {\Sigma} _ {\mathbf {A} ^ {\prime}}\right) \mathbf {U} _ {\mathbf {A} \leftarrow \mathbf {A} ^ {\prime}} ^ {T} \\ \end{array}
$$
Without loss of generality (and to simplify notation), assume  $\mathbf{A} = \{Att_1, \ldots, Att_\ell\}$  consists of the first  $\ell$  attributes,  $\mathbf{A}' = \{Att_1, \ldots, Att_t\}$  consists of the first  $t \leq \ell$  attributes, then  $\mathbf{A} / \mathbf{A}' = \{Att_{t+1}, \ldots, Att_\ell\}$ .
By definition,  $Var(A; \mathcal{M})$  is the diagonal of  $E[\mathbf{q}\mathbf{q}^T] = \sum_{\mathbf{A}' \in \mathrm{closure}(\mathbf{A})} \sigma_{\mathbf{A}'}^2 \mathbf{U}_{\mathbf{A} \leftarrow \mathbf{A}'} \Sigma_{\mathbf{A}'} \mathbf{U}_{\mathbf{A} \leftarrow \mathbf{A}'}^T$ . Thus we can write:
$$
\mathbf {Q} _ {\mathbf {A}} = \left(\bigotimes_ {i = 1} ^ {t} \mathcal {I} _ {| A t t _ {i} |}\right) \otimes \left(\bigotimes_ {j = t + 1} ^ {\ell} \mathcal {I} _ {| A t t _ {j} |}\right) \otimes \left(\bigotimes_ {k = \ell + 1} ^ {n _ {a}} \mathbf {1} _ {| A t t _ {k} |} ^ {T}\right)
$$
$$
\mathbf {R} _ {\mathbf {A} ^ {\prime}} = \left(\bigotimes_ {i = 1} ^ {t} \mathbf {S u b} _ {| A t t _ {i} |}\right) \otimes \left(\bigotimes_ {j = t + 1} ^ {\ell} \mathbf {1} _ {| A t t _ {j} |} ^ {T}\right) \otimes \left(\bigotimes_ {k = \ell + 1} ^ {n _ {a}} \mathbf {1} _ {| A t t _ {k} |} ^ {T}\right)
$$
$$
\mathbf {R} _ {\mathbf {A} ^ {\prime}} ^ {\dagger} = \left(\bigotimes_ {i = 1} ^ {t} \mathbf {S u b} _ {| A t t _ {i} |} ^ {\dagger}\right) \otimes \left(\bigotimes_ {j = t + 1} ^ {\ell} \frac {1}{| A t t _ {j} |} \mathbf {1} _ {| A t t _ {j} |}\right) \otimes \left(\bigotimes_ {k = \ell + 1} ^ {n _ {a}} \frac {1}{A t t _ {k}} \mathbf {1} _ {| A t t _ {k} |}\right)
$$
$$
\mathbf {U} _ {\mathbf {A} \leftarrow \mathbf {A} ^ {\prime}} = \mathbf {Q} _ {\mathbf {A}} \mathbf {R} _ {\mathbf {A} ^ {\prime}} ^ {\dagger} = \left(\bigotimes_ {i = 1} ^ {t} \mathbf {S u b} _ {| A t t _ {i} |} ^ {\dagger}\right) \otimes \left(\bigotimes_ {j = t + 1} ^ {\ell} \frac {1}{| A t t _ {j} |} \mathbf {1} _ {| A t t _ {j} |}\right) \otimes \left(\bigotimes_ {k = \ell + 1} ^ {n _ {a}} [ 1 ]\right)
$$
$$
\mathbf {U} _ {\mathbf {A} \leftarrow \mathbf {A} ^ {\prime}} ^ {T} = \left(\bigotimes_ {i = 1} ^ {t} \mathbf {S u b} _ {| A t t _ {i} |} ^ {\dagger T}\right) \otimes \left(\bigotimes_ {j = t + 1} ^ {\ell} \frac {1}{| A t t _ {j} |} \mathbf {1} _ {| A t t _ {j} |} ^ {T}\right) \otimes \left(\bigotimes_ {k = \ell + 1} ^ {n _ {a}} [ 1 ]\right)
$$
$$
\boldsymbol {\Sigma} _ {\mathbf {A} ^ {\prime}} = \left(\bigotimes_ {i = 1} ^ {t} \mathbf {S u b} _ {| A t t _ {i} |} \mathbf {S u b} _ {| A t t _ {i} |} ^ {T}\right) \otimes \left(\bigotimes_ {j = t + 1} ^ {\ell} [ 1 ]\right) \otimes \left(\bigotimes_ {k = \ell + 1} ^ {n _ {a}} [ 1 ]\right)
$$
$$
\begin{array}{l} \mathbf {U} _ {\mathbf {A} \leftarrow \mathbf {A} ^ {\prime}} \boldsymbol {\Sigma} _ {\mathbf {A} ^ {\prime}} \mathbf {U} _ {\mathbf {A} \leftarrow \mathbf {A} ^ {\prime}} ^ {T} = \left(\bigotimes_ {i = 1} ^ {t} \mathbf {S u b} _ {| A t t _ {i}} ^ {\dagger} \mathbf {S u b} _ {| A t t _ {i}} | \mathbf {S u b} _ {| A t t _ {i}} ^ {T} \mathbf {S u b} _ {| A t t _ {i}} ^ {\dagger T}\right) \\ \otimes \left(\bigotimes_ {j = t + 1} ^ {\ell} \frac {1}{| A t t _ {j} | ^ {2}} \mathbf {1} _ {| A t t _ {j} |} [ 1 ] \mathbf {1} _ {| A t t _ {j} |} ^ {T}\right) \otimes \left(\bigotimes_ {k = \ell + 1} ^ {n _ {a}} [ 1 ]\right) \tag {10} \\ \end{array}
$$
Now, by Lemma F.1,
$$
\mathbf {S u b} _ {| A t t _ {i} |} ^ {\dagger} \mathbf {S u b} _ {| A t t _ {i} |} \mathbf {S u b} _ {| A t t _ {i} |} ^ {T} \mathbf {S u b} _ {| A t t _ {i} |} ^ {\dagger T} = \mathcal {I} _ {| A t t _ {i} |} - \frac {1}{| A t t _ {i} |} \mathbf {1} _ {| A t t _ {i} |} \mathbf {1} _ {| A t t _ {i} |} ^ {T} \tag {11}
$$
So the diagonals of  $\mathbf{U}_{\mathbf{A}\leftarrow \mathbf{A}^{\prime}}\pmb{\Sigma}_{\mathbf{A}^{\prime}}\mathbf{U}_{\mathbf{A}\leftarrow \mathbf{A}^{\prime}}^{T}$  can be computed by multiplying  $\frac{|Att_i| - 1}{|Att_i|}$  for each  $Att_{i}\in$ $\mathbf{A}'$  and  $1 / |Att_j|$  for each  $Att_{j}\in \mathbf{A}\setminus \mathbf{A}^{\prime}$ . Meanwhile, the off diagonals are all the same and can be computed by multiplying  $\frac{-1}{|Att_i|}$  for each  $Att_{i}\in \mathbf{A}^{\prime}$  and  $\frac{1}{|Att_j|^2}$  for each  $Att_{j}\in \mathbf{A}\setminus \mathbf{A}^{\prime}$ .
Computing the variance and covariance of the marginal query answer is therefore the summation of these quantities for all  $\mathbf{A}' \subseteq \mathbf{A}$  and is what the theorem states.
G Computational Complexity Proofs
1. Expressing the privacy cost of the optimal mechanism  $\mathcal{M}^*$  as a linear combination of the  $1 / \sigma_{\mathbf{A}}^{2}$  values takes  $O(\sum_{\mathbf{A}\in Wkload}\# cells(\mathbf{A}))$  total time.
2. Expressing  $\underline{all}$  of the  $Var(\mathbf{A};\mathcal{M}^*)$ , for  $\mathbf{A} \in Wkload$ , as a linear combination of the  $\sigma_{\mathbf{A}}^{2}$  values can be done in  $O(\sum_{\mathbf{A} \in Wkload} \# cells(\mathbf{A}))$  total time.
3. Computing all the noisy outputs of the optimal mechanism (i.e.,  $\mathcal{M}_{\mathbf{A}}(\mathbf{x};\sigma_{\mathbf{A}}^{2})$  for  $\mathbf{A} \in$  closure(Wkload)) takes  $O\left(n_{a}\sum_{\mathbf{A} \in Wkload}\prod_{Att_{i} \in \mathbf{A}}(|Att_{i}| + 1)\right)$  total time after the true answers have been precomputed (Line 1 in Algorithm 1). Note that the total number of cells on marginals in Wkload is  $O\left(\sum_{\mathbf{A} \in Wkload}\prod_{Att_{i} \in \mathbf{A}}|Att_{i}|\right)$ .
4. Reconstructing marginals for all  $\mathbf{A} \in Wkload$  takes  $O\left(\sum_{\mathbf{A} \in Wkload} |\mathbf{A}| \# cells(\mathbf{A})^2\right)$  total time.
5. Computing the variance of the cells for all of the marginals for  $\mathbf{A} \in W\text{kload}$  can be done in  $O(\sum_{\mathbf{A} \in W\text{kload}} \# \text{cells}(\mathbf{A}))$  total time.
Proof of Theorem 4.8. First we establish that  $|\mathrm{closure}(Wkload)| \leq \sum_{\mathbf{A} \in Wkload} \# cells(\mathbf{A})$ . Given an set  $\mathbf{A} \in Wkload$ , we note that it has  $2^{|A|}$  subsets, so that  $|\mathrm{closure}(\mathbf{A})| = 2^{|A|}$ . However,  $\# cells(\mathbf{A})$  is at least  $2^{|A|}$  (because each attribute has at least 2 attribute values). We also note that closure  $(Wkload) = \bigcup_{\mathbf{A} \in Wkload} \mathrm{closure}(\mathbf{A})$ . Hence
$$
| \text {c l o s u r e} (W k l o a d) | \leq \sum_ {\mathbf {A} \in W k l o a d} | \text {c l o s u r e} (\mathbf {A}) | = \sum_ {\mathbf {A} \in W k l o a d} \# c e l l s (\mathbf {A})
$$
To analyze the time complexity of symbolically representing the privacy cost, as a linear combination of the  $1 / \sigma_{\mathbf{A}}^{2}$  values (for all  $\mathbf{A} \in$  closure(Wkload)) we note that the coefficient of  $1 / \sigma_{\mathbf{A}}^{2}$  is  $\prod_{Att_i \in \mathbf{A}} \frac{|Att_i| - 1}{|Att_i|}$ . Thus computing the coefficient  $1 / \sigma_{\emptyset}^{2}$  takes  $O(1)$  time. Then, computing the coefficient of  $1 / \sigma_{\{Att_i\}}^{2}$  can be computed from the coefficient of  $1 / \sigma_{\emptyset}^{2}$  in  $O(1)$  additional time. Thus, we if go level by level, first computing the coefficients of  $1 / \sigma_{\mathbf{A}}^{2}$  with  $|\mathbf{A}| = 1$  then for  $|\mathbf{A}| = 2$ , etc. then computing the coefficient for each new  $\mathbf{A}$  takes incremental  $O(1)$  time. Thus the overall time is  $O(|\text{closure}(Wkload)|)$  and therefore is  $O(\sum_{\mathbf{A} \in Wkload} \# \text{cells}(\mathbf{A}))$ .
Let  $n_{cells} = \sum_{\mathbf{A} \in Wkload} \# cells(\mathbf{A})$  To express the variance symbolically as a linear function of the  $\sigma_{\mathbf{A}}^{2}$  values via Theorem 4.7, we note from the previous part that computing  $\prod_{Att_i \in \mathbf{A}'} \frac{|Att_i| - 1}{|Att_i|}$  for all  $\mathbf{A}' \in \text{closure}(Wkload)$  can be done in total  $O(n_{cells})$  time. Similarly, computing  $\prod_{Att_i \in \mathbf{A}'} \frac{1}{|Att_i|^2}$  for all  $\mathbf{A}' \in \text{closure}(Wkload)$  also take total  $O(n_{cells})$  time. Once this is pre-computed, then for any  $\mathbf{A}' \subseteq \mathbf{A} \in \text{closure}(Wkload)$ , the product  $\prod_{Att_i \in \mathbf{A}'} \frac{|Att_i| - 1}{|Att_i|} * \prod_{Att_j \in (\mathbf{A} / \mathbf{A}')} \frac{1}{|Att_j|^2}$  can be computed in  $O(1)$  time since  $\mathbf{A} \setminus \mathbf{A}' \in \text{closure}(Wkload)$ . Now,  $Var(\mathbf{A}; \mathcal{M}^*) = \sum_{\mathbf{A}' \subseteq \mathbf{A}} \sigma_{\mathbf{A}'}^2 \prod_{Att_i \in \mathbf{A}'} \frac{|Att_i| - 1}{|Att_i|} * \prod_{Att_j \in (\mathbf{A} / \mathbf{A}')} \frac{1}{|Att_j|^2}$ . This is a linear combination of  $2^{|A|}$  terms (one term for each variable  $\sigma_{\mathbf{A}'}^2$  for  $A' \subseteq A$ ). Each term is computed in  $O(1)$  time after the pre-computation phase. Thus the symbolic representation of  $Var(\mathbf{A}; \mathcal{M}^*)$  takes  $O(2^{|A|})$  time (which is at most the number of cells in the marginal on  $A$ ) time after precomputation. Thus computing  $Var(\mathbf{A}; \mathcal{M}^*)$  for all  $A \in Wkload$  can be done in total  $O(n_{cells})$  time after precomputation, but precomputation also takes  $O(n_{cells})$  time. Thus the overall total time is  $O(n_{cells})$ .
We next analyze the time it takes to generate noisy answers once the true answers have been precomputed (Line 1 in Algorithm 1). This involves (1) computing the product  $\mathbf{H}\mathbf{v}$  in the algorithm, (2) generating one Gaussian random variable for each column of  $\mathbf{H}$  and (3) computing  $\mathbf{Hz}$ . Now, the first and third steps take the same amount of time. The second step generates one Gaussian for each row of  $\mathbf{H}$  and hence, for each  $\mathcal{M}_{\mathbf{A}}$  takes time  $\Pi_{Att_i\in \mathbf{A}}(|Att_i| - 1)$ .
For the first step, the fast kronecker-product multiplication algorithm (Algorithm 1 of [38]) has the following complexity. Given a kronecker product of  $\ell$  matrices of sizes  $(m_1 - 1) \times m_1, \ldots, (m_\ell - 1) \times m_\ell$  and a vector with  $m_1 \times \dots \times m_\ell$  components, their algorithm has  $\ell$  iterations. In iteration  $i$ , the  $i^{\text{th}}$  matrix (with size  $m_{i-1} \times m_i$ ) is multiplied by a matrix with shape  $(m_i, \prod_{j=1}^{i-1} m_j * \prod_{j=i+1}^\ell (m_j - 1))$ . In our case, each  $m_i$  is a subtraction matrix with two nonzero elements in each row. Thus, in each iteration, the product makes  $2 \prod_{j=1}^{i-1} m_j * \prod_{j=i}^\ell (m_j - 1)$  scalar multiplication operations. There are  $\ell$  iterations, so the multiplication algorithm uses  $O(\ell \prod_{i=1}^\ell m_i)$  multiplications.
Now, to run algorithm  $\mathcal{M}_{\mathbf{A}}$ , the number of kron products  $\ell$  is  $|\mathbf{A}|$  and each  $m_i$  is  $|Att_i|$  for  $Att_i \in \mathbf{A}$ . Hence the running time of  $\mathcal{M}_{\mathbf{A}}$  is  $O(|\mathbf{A}| \prod_{Att_i \in \mathbf{A}} |Att_i|)$  which is at most  $|\mathbf{A}|$  times the number of cells in the marginal on  $\mathbf{A}$ . Note that the constant in the big-O notation is bounded across all  $\mathbf{A}$ . Next, when adding up the complexity across all  $\mathbf{A}' \in \mathrm{closure}(\mathbf{A})$ , we can replace  $|\mathbf{A}'|$  with  $|\mathbf{A}|$ , and then the summation looks like the product  $\prod_{Att_i \in \mathbf{A}} (|Att_i| + 1)$  when this product is expanded. Hence the time to run all  $\mathbf{Q}_{\mathbf{A}'}$  for all  $\mathbf{A}' \in \mathrm{closure}(\mathbf{A})$  is  $O(|\mathbf{A}| \prod_{Att_i \in \mathbf{A}} (|Att_i| + 1))$ . Adding up over all  $\mathbf{A} \in Wkload$  gets the results.
Next we consider the reconstruction phase. Using the same analysis of the fast kron-product vector multiplication, we see that in each iteration of Algorithm 2, there is a kron product vector multiplication. Using similar reasoning as for the previous item, each such multiplication takes  $O(|\mathbf{A}| \prod_{Att_i \in \mathbf{A}} |Att_i| = O(|\mathbf{A}| \# cells(\mathbf{A}))$  time. The number of iterations in the algorithm is  $2^{|A|} \leq \# cells(\mathbf{A})$ . Thus the overall runtime is  $O(\sum_{\mathbf{A} \in Wkload} |\mathbf{A}| \# cells(\mathbf{A})^2)$ .
Finally, the variance computation is no harder than expressing the  $Var(\mathbf{A};\mathcal{M}^{*})$  as linear combinations of the optimization variables and we have shown this to be  $O(n_{cells})$ .
H Closed Form Solution to the Weighted Sum of Variances Loss
By Theorem 4.5, the privacy cost is a linear combination of the  $1 / \sigma_{\mathbf{A}}^{2}$  values. By Theorem 4.7, each reconstructed marginal's cell variances are a linear combination of the  $\sigma_{\mathbf{A}}^{2}$  values. Thus, minimizing the weighted sum of reconstructed marginal variances subject to the privacy cost being  $\leq c$  can be formulated as a problem of the following type:
$$
\sigma_ {\mathbf {A}} ^ {2}: \mathbf {A} \in \text {c l o s u r e} (W k l o a d) \sum_ {\mathbf {A} \in \text {c l o s u r e} (W k l o a d)} v _ {\mathbf {A}} \sigma_ {\mathbf {A}} ^ {2} \tag {12}
$$
$$
s. t. \sum_ {\mathbf {A} \in \text {c l o s u r e} (W k l o a d)} \frac {p _ {\mathbf {A}}}{\sigma_ {\mathbf {A}} ^ {2}} \leq c
$$
where the  $v_{\mathbf{A}}$  are the linear coefficients of the  $\sigma_{\mathbf{A}}^{2}$  and the  $p_{\mathbf{A}}$  are the linear coefficients of the  $1 / \sigma_{\mathbf{A}}^{2}$  in the privacy cost. The closed form solution is given byhte following lemma.
LEMMA H.1. Given the optimization problem in Equation 12 The optimal objective function value is  $T = \left( \sum_{\mathbf{A}} \sqrt{v_{\mathbf{A}} p_{\mathbf{A}}} \right)^2 / c$ , the optimal value of each noise scale parameter is  $\sigma_{\mathbf{A}}^2 = \sqrt{Tp_{\mathbf{A}} / (cv_{\mathbf{A}})}$ .
Proof. Clearly, for the optimal solution, the inequality constraint must be tight (i.e.,  $= c$ ) because if it is not tight, we can lower variance while increasing privacy cost by dividing each  $\sigma_{\mathbf{A}}^{2}$  by a number  $>1$ . Thus we just need to solve the problem subject to  $\sum_{\mathbf{A}} p_{\mathbf{A}} / \sigma_{\mathbf{A}}^{2} = c$ .
From Cauchy-Schwarz inequality,
$$
\sum_ {\mathbf {A}} v _ {\mathbf {A}} \sigma_ {\mathbf {A}} ^ {2} = \left(\sum_ {\mathbf {A}} v _ {\mathbf {A}} \sigma_ {\mathbf {A}} ^ {2}\right) \left(\sum_ {\mathbf {A}} \frac {p _ {\mathbf {A}}}{\sigma_ {\mathbf {A}} ^ {2}}\right) / c \geq \left(\sum_ {\mathbf {A}} \sqrt {v _ {\mathbf {A}} p _ {\mathbf {A}}}\right) ^ {2} / c = T
$$
Equality holds when  $\frac{v_{\mathbf{A}}}{p_{\mathbf{A}}} \sigma_{\mathbf{A}}^4 = t$  for all  $\mathbf{A}$  (for some constant  $t$ ). Since  $c = \sum_{\mathbf{A}} \frac{p_{\mathbf{A}}}{\sigma_{\mathbf{A}}^2} = \sum_{\mathbf{A}} \sqrt{v_{\mathbf{A}} p_{\mathbf{A}} / t}$ , then we must have  $t = T / c$ . Plugging this into the definition of  $t$ , we get  $\sigma_{\mathbf{A}}^2 = \sqrt{T p_{\mathbf{A}} / (c v_{\mathbf{A}})}$ .
Thus, if the loss function is the weighted sum of variances, ResidualPlanner does not need any optimization steps. The selection of the noise scales and the reconstruction phase are direct algorithms.
I Additional Experiments
In this section, we present additional experiments. Following [37], the experiments use the following type of workloads:
- All  $k$ -way marginals.
- All  $\leq 3$ -way marginals. This includes all 0-way marginal (the total sum), all 1-way marginals, all 2-way marginals, and all 3-way marginals.
- Small marginals. This includes any  $k$ -way marginal that has at most 5000 cells.
We also use these metrics:
- RMSE: The total variance is the sum of the variances of the reconstructed cells in each marginal in the workload. Root Mean Squared Error is obtained by taking the total variance, dividing by the total number of cells in the workload marginals, then taking the square root. The SVD Bound (SVDB for short) [31] provides a theoretical lower bound on RMSE for any matrix mechanism. For marginals, the SVDB is tight, but its computation is not scalable.
- MaxVar: compute the variance of each reconstructed cell for each marginal in the workload, then take the maximum of these.
- Running time (in seconds) of the different stages of the algorithms (select and reconstruct).
Unless otherwise stated, ResidualPlanner uses the open-source ECOS optimizer [14] for solving the optimization problem it generates for the select step.