SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where in the second step we used the Cauchy-Schwarz inequality.
Factorization vs Approximate-Factorization: Proof of Lemma 10
Proof of Lemma 10. Let  $W$  be any matrix. By definition of  $\zeta$ , there exists  $\tilde{W}$  such that  $\zeta(W; n) = \gamma_2(\tilde{W}) + \| \tilde{W} - W \|_{\infty} \cdot (\varepsilon \sqrt{n})$ . The new protocol  $\mathbb{A}'$  simply runs  $\mathbb{A}$  but on  $\tilde{W}$ . Let  $\hat{z}$  denote the output of  $\mathbb{A}'$ . Its MSE can be bounded by
$$
\begin{array}{l} \mathbb {E} \left[ \left(\hat {z} - h _ {\mathbf {x}} ^ {T} W h _ {\mathbf {x}}\right) ^ {2} \right] = \mathbb {E} \left[ \left(\left(\hat {z} - h _ {\mathbf {x}} ^ {T} \tilde {W} h _ {\mathbf {x}}\right) + \left(h _ {\mathbf {x}} ^ {T} \tilde {W} h _ {\mathbf {x}} - h _ {\mathbf {x}} ^ {T} W h _ {\mathbf {x}}\right)\right) ^ {2} \right] \\ \lesssim \mathbb {E} \left[ \left(\hat {z} - h _ {\mathbf {x}} ^ {T} \tilde {W} h _ {\mathbf {x}}\right) ^ {2} \right] + \mathbb {E} \left[ \left(h _ {\mathbf {x}} ^ {T} \tilde {W} h _ {\mathbf {x}} - h _ {\mathbf {x}} ^ {T} W h _ {\mathbf {x}}\right) ^ {2} \right] \\ \lesssim \left(c (n, \varepsilon , k) \cdot \frac {\gamma_ {2} (\tilde {W}) ^ {2}}{\varepsilon^ {2} n}\right) + \| \tilde {W} - W \| _ {\infty} ^ {2} \\ \lesssim c (n, \varepsilon , k) \cdot \frac {\zeta (W ; n) ^ {2}}{\varepsilon^ {2} n} + \frac {\zeta (W ; n) ^ {2}}{\varepsilon^ {2} n} \\ \lesssim c (n, \varepsilon , k) \cdot \frac {\zeta (W ; n) ^ {2}}{\varepsilon^ {2} n}. \\ \end{array}
$$
Projection Mechanism
As outlined in Section 1.2, we will use the projection mechanism as a subroutine for our algorithm. For  $W \in \mathbb{R}^{\ell \times k}$ , we define  $W^{\Delta}$  to denote the set  $\{Wx \mid x \in \mathbb{R}^k, \|x\|_1 \leq 1\}$ . The guarantees of the algorithm are summarized below.
Theorem 19 ([BBNS19]). For any workload  $W \in \mathbb{R}^{\ell \times k}$ , there is a non-interactive  $\varepsilon$ -local DP mechanism  $\mathcal{M}$  such that
$$
\mathbb {E} _ {\mu \sim \mathcal {M} (\mathbf {x})} [ \| \mu - W h _ {\mathbf {x}} \| _ {2} ^ {2} ] \lesssim \frac {\| W \| _ {1 \rightarrow 2} ^ {2} \sqrt {\log k}}{\varepsilon \sqrt {n}}.
$$
Moreover, the output  $\mathcal{M}(W)$  always belongs to  $W^{\Delta}$ .
We provide the proof below for completeness.
Proof of Theorem 19. Let  $C \coloneqq \| W\|_{1\to 2}$ . The algorithm works as follows:
Each user  $i\in [n]$  sends  $y_{i}\coloneqq \mathrm{VRand}_{\varepsilon ,C}(W\mathbf{1}_{x_{i}})$  to the analyst (where VRand.  $(\cdot)$  is from Theorem 9).
The analyzer computes  $Y \coloneqq \frac{1}{n} \sum_{i \in [n]} y_i$  and then outputs  $\mu \coloneqq \operatorname*{argmin}_{u \in W^{\Delta}} \| u - Y \|_2$ .
The privacy guarantee of the algorithm follows from Theorem 9.
Let  $z_{i} = y_{i} - W\mathbf{1}_{x_{i}}$ . From Theorem 9,  $z_{i}$  is zero-mean and  $\sigma$ -sub-Gaussian for  $\sigma = O(C / \varepsilon)$ . Let  $Z := \frac{1}{n}\sum_{i\in [n]}z_{i}$ ; we then have that it is zero-mean and  $\sigma'$ -sub-Gaussian for  $\sigma' = \sigma /\sqrt{n}$ . By the definition of sub-Gaussian random variable and a union bound, for every  $\beta >0$ , with probability  $1 - \beta$  the following holds:
$$
\left| \langle W _ {j}, Z \rangle \right| \lesssim \frac {C ^ {2}}{\varepsilon \sqrt {n}} \cdot \sqrt {\log (k / \beta)} \quad \forall j \in [ k ], \tag {2}
$$
where  $W_{j}$  denote the  $j$ th column of  $W$ .
It is well known that if we define  $\mu$  as we did (i.e., as projection of  $Y$  on  $W^{\Delta}$ ), then we have
$$
\langle \mu - w, \mu - Y \rangle \leq 0,
$$
for all  $w\in W^{\Delta}$
Plugging in  $w = Wh_{\mathbf{x}}$ , we have
$$
\begin{array}{l} \left\| \mu - W h _ {\mathbf {x}} \right\| _ {2} ^ {2} = \left\langle \mu - W h _ {\mathbf {x}}, \mu - Y \right\rangle + \left\langle \mu - W h _ {\mathbf {x}}, Z \right\rangle \\ \leq \langle \mu - W h _ {\mathbf {x}}, Z \rangle \\ \leq \max  _ {w ^ {\prime} \in W ^ {\Delta}} \langle w ^ {\prime}, Z \rangle \\ = \max  _ {j \in [ k ]} | \langle W _ {j}, Z \rangle |, \\ \end{array}
$$
where the equality follows from the fact  $W^{\Delta}$  is the convex hull of  $W_{1},\ldots ,W_{k}, - W_{1},\ldots , - W_{k}$ .
Putting together (2) and the above then yields
$$
\mathbb {E} [ \| \mu - W h _ {\mathbf {x}} \| _ {2} ^ {2} ] \lesssim \frac {C ^ {2} \sqrt {\log k}}{\varepsilon \sqrt {n}}.
$$
B On the Lower Bound for Linear Queries from [ENU20]
In this section, we provide details on how we can interpret the bounds of [ENU20] as stated in the form of Theorem 4. The lower bound in [ENU20] is originally for the  $\ell_{\infty}$ -error; we make the observation below that a simple modification of their proof can be made so that it applies to mMSE. Note that, in addition to getting a stronger result in terms of error metric (because a lower bound on
See e.g., [Bub15, Lemma 3.1].
mMSE implies the same lower bound on  $\ell_{\infty}$ -error), we also get a quantitatively stronger bound that does not depend on  $k$  because we avoid having to take a union bound over  $k$  queries (which was required for the proof for  $\ell_{\infty}$ -error in [ENU20]).
In [ENU20, Section 3.4], it was shown that w.l.o.g. it suffices to consider "symmetric" workload matrix  $W$  (i.e., ones that can be written as  $[W', -W']$  for some  $W'$ ). We will thus do so throughout the rest of this section. Following their notation, we also assume that  $W \in \mathbb{R}^{k \times \mathcal{X}}$ , i.e.,  $\mathcal{X}$  is the input space and there are  $k$  linear queries. We also recall the following two lemmas from their paper.
Lemma 20 ([ENU20, Lemma 11]). Let  $\varepsilon \in (0,1]$ . For any distribution  $\lambda_1,\dots ,\lambda_m,\mu_1,\dots ,\mu_m$  on  $\mathcal{X}$ , distribution  $\pi$  over  $[m]$  and  $\varepsilon$ -local DP randomizer  $\mathcal{R}$ , we have<sup>8</sup>
$$
\mathbb {E} _ {v \sim \pi} [ d _ {\mathrm {K L}} (\mathcal {R} (\lambda_ {v}) ^ {n} \parallel \mathcal {R} (\mu_ {v}) ^ {n}) ] \lesssim n \varepsilon^ {2} \cdot \| M \| _ {\ell_ {\infty} \to L _ {2} (\pi)} ^ {2},
$$
where  $M\in \mathbb{R}^{m\times \mathcal{X}}$  is a matrix such that  $M_{v,x} = \lambda_v(x) - \mu_v(x)$ .
We note that in [ENU20, Lemma 11], the LHS is not exactly the same as in our version above. However, inspecting the very first inequality from their proof shows that their bound goes through the quantity on the LHS of Lemma 20.
The next lemma, which describes the properties of the hard distributions, is arguably the main technical contribution of the lower bound in [ENU20].
Lemma 21 ([ENU20, Lemma 21]). Let  $W \in \mathbb{R}^{k \times \mathcal{X}}$  be any symmetric workload matrix and  $\alpha > 0$ . There exist  $\xi \geq 0$ , and distributions  $\tilde{\lambda}_1, \ldots, \tilde{\lambda}_k, \tilde{\mu}_1, \ldots, \tilde{\mu}_k$  on  $\mathcal{X}$  and  $\tilde{\pi}$  over  $[k]$  such that
(i)  $(W\tilde{\lambda}_v)_i = 0$  for all  $i, v \in [k]$ .
(ii) for all  $v \in \operatorname{supp}(\tilde{\pi})$ ,  $(W\tilde{\mu}_v)_v \gtrsim \frac{\xi + \alpha}{\log(\|W\|_{\infty / \alpha})}$ .<sup>10</sup>
(iii) Let  $\tilde{M}$  be defined similarly as in Lemma 20. Then  $\| \tilde{M} \|_{\ell_{\infty} \to L_2(\tilde{\pi})} \lesssim \frac{\xi}{\gamma_2(W, \alpha)}$ .
It will in fact be easier to work with the following version of the lemma, which is implicit in the proof of [ENU20, Theorem 22]:
Lemma 22. Let  $W \in \mathbb{R}^{k \times \mathcal{X}}$  be any symmetric workload matrix and  $\alpha > 0$ . There exist distributions  $\tilde{\lambda}_1, \ldots, \tilde{\lambda}_k, \tilde{\mu}_1, \ldots, \tilde{\mu}_k$  on  $\mathcal{X}$  and  $\tilde{\pi}$  over  $[k]$  such that
(i)  $(W\tilde{\lambda}_v)_i = 0$  for all  $i, v \in [k]$ .
(ii) for all  $v\in \mathrm{supp}(\tilde{\pi})$ $(W\tilde{\mu}_v)_v\gtrsim \frac{\alpha}{\log(\|W\|_{\infty} / \alpha)}.$
(iii) Let  $\tilde{M}$  be defined similarly as in Lemma 20. Then  $\| \tilde{M} \|_{\ell_{\infty} \to L_2(\tilde{\pi})} \lesssim \frac{\alpha}{\gamma_2(W, \alpha)}$ .
Proof Sketch. This is constructed by taking  $\tilde{\lambda}_1, \ldots, \tilde{\lambda}_k, \tilde{\mu}_1, \ldots, \tilde{\mu}_k$  and  $\tilde{\pi}$  from Lemma 21. Then, replace each  $\tilde{\mu}_v$  by the mixture  $(1 - \beta) \cdot \tilde{\lambda}_v + \beta \cdot \tilde{\mu}_v$  where  $\beta = \min\{1, \alpha / \xi\}$ , for all  $v \in [k]$ . Both the lower bound for  $(W\tilde{\mu}_v)_v$  and the upper bound for  $\| \tilde{M} \|_{\ell_{\infty} \to L_2(\tilde{\pi})}$  scale linearly with  $\beta$ , yielding the desired bounds.
We can now prove the following, which is a more qualitative version of the lower bound in Theorem 4.
Theorem 23. Let  $W \in \mathbb{R}^{k \times \mathcal{X}}$  be any symmetric workload matrix. Any non-interactive  $\varepsilon$ -local DP
mechanism must incur an mMSE at least  $\Omega\left(\frac{\zeta(W,n)^2}{\varepsilon^2n}\cdot\frac{1}{\log\left(\frac{\varepsilon^2n\|W\|_\infty^2}{\zeta(W,n)^2}\right)^2}\right)$ .
Here  $\| M\|_{\ell_{\infty}\to L_2(\pi)}^2 \coloneqq \max_{\| x\|_{\infty} = 1}\| Mx\|_{L_2(\pi)}$  where  $\| a\|_{L_2(\pi)} \coloneqq \sqrt{\sum_{v\in[m]}\pi(v)a_v^2}$ . Note that we will not be dealing with this quantity further than here.
<sup>9</sup>In [ENU20],  $\xi$  is related to the dual solution as  $\xi = W\bullet U - \alpha$  where  $U$  is the dual witness of the approximate factorization norm, i.e., one with  $\gamma_{2}(W) = \frac{W\bullet U - \alpha\|U\|_{1}}{\gamma_{2}^{*}(U)}$ . (See [ENU20, Section 2.3] for more details.) However, these specifics are not used in the remainder of the proof.
<sup>10</sup>Note that the  $\| W \|_{\infty}$  term does not show up in [ENU20] since they assume that  $\| W \|_{\infty} \leq 1$ .
Proof. For notational convenience, we write  $\zeta$  as a shorthand for  $\zeta(W,n)$ . Let  $\alpha = \frac{C_1\zeta}{\varepsilon\sqrt{n}}$  where  $C_1 \in (0,0.1)$  is a sufficiently small constant. Suppose for the sake of contradiction that there exists a non-interactive  $\varepsilon$ -local DP protocol (whose randomizer is  $\mathcal{R}$ ) with mMSE at most  $(\alpha')^2$  for  $\alpha' = \frac{C_2\alpha}{\log(\|W\|_{\infty} / \alpha)}$  where  $C_2 \in (0,0.1)$  is a sufficiently small constant.
Recall the definition of  $\zeta$ ; we must have  $\zeta \leq \gamma_2\left(W, \frac{\zeta}{2\varepsilon\sqrt{n}}\right) + \frac{\zeta}{2}$ . This gives
$$
\gamma_ {2} \left(W, \frac {\zeta}{2 \varepsilon \sqrt {n}}\right) \geq \frac {\zeta}{2}.
$$
Since  $\gamma_{2}(W,\cdot)$  is an increasing function, we thus have
$$
\frac {\alpha \varepsilon \sqrt {n}}{\gamma_ {2} (W , \alpha)} \leq C _ {1}. \tag {3}
$$
Let  $\tilde{\lambda}_1, \ldots, \tilde{\lambda}_k, \tilde{\mu}_1, \ldots, \tilde{\mu}_k$  and  $\tilde{\pi}$  be as in Lemma 22. By Lemma 20 and Lemma 22(iii), we have
$$
\mathbb {E} _ {v \sim \pi} \left[ d _ {\mathrm {K L}} \left(\mathcal {R} \left(\lambda_ {v}\right) ^ {n} \| \mathcal {R} \left(\mu_ {v}\right) ^ {n}\right) \right] \lesssim \left(\frac {\alpha \varepsilon \sqrt {n}}{\gamma_ {2} (W , \alpha)}\right) ^ {2}. \tag {4}
$$
On the other hand, since the protocol has mMSE at most  $(\alpha')^2$ , we may use the following algorithm to distinguish  $\mathcal{R}(\lambda_v)^n$  from  $\mathcal{R}(\mu_v)^n$ : let the analyst compute an estimate for  $Wh_{\mathbf{x}}$  (using the  $n$  samples from  $\mathcal{R}(\cdot)$  provided). If the estimate has absolute value less than  $10\alpha'$ , then output  $\lambda_v$ ; otherwise, output  $\mu_v$ . From Lemma 22(i)(ii), it is not hard to see that, for  $C_2$  that is sufficiently small, this algorithm is correct with probability at least  $2/3$ . This means that  $d_{\mathrm{tv}}(\mathcal{R}(\lambda_v)^n, \mathcal{R}(\mu_v)^n) \geq 1/3$ . Pinsker's inequality (i.e., eq. (1)) then yields  $d_{\mathrm{KL}}(\mathcal{R}(\lambda_v)^n \| \mathcal{R}(\mu_v)^n) \gtrsim 1$ . Comparing this with the above eq. (4), we get  $\frac{\alpha\varepsilon\sqrt{n}}{\gamma_2(W,\alpha)} \gtrsim 1$ , which contradicts eq. (3) when  $C_1$  is sufficiently small.
C Three-Round Algorithm
In this section, we describe and analyze our interactive algorithm. The error guarantees of the algorithm are stated below. (Note that Theorem 24 is a more precise version of Theorem 8.)
Theorem 24. For any  $W \in \mathbb{R}^{k \times k}$  and  $n \geq \tilde{\Omega}\left(\frac{\gamma_2(W)^4\log k}{\|W\|_\infty^4\varepsilon^2}\right)$ , there is a three-round  $\varepsilon$ -DP algorithm for estimating quadratic forms on  $W$  such that the MSE is  $O\left(\frac{\|W\|_{\infty}^2}{\varepsilon^2n}\right)$ .
Proof. Throughout the proof, we assume that  $n \geq Q \cdot \frac{\gamma_2(W)^4 \log k}{\|W\|_{\infty}^4 \varepsilon^2} \cdot \log \left( \frac{10 \gamma_2(W) \log k}{\|W\|_{\infty} \varepsilon} \right)$ , where  $Q$  is a sufficiently large constant. Recall from the definition of  $\gamma_2(W)$  that there must exist  $L, R$  such that  $W = L^T R$  and  $\|L\|_{1 \to 2} = \|R\|_{1 \to 2} = \sqrt{\gamma_2(W)} =: C$ .
Algorithm Description. Let  $\bar{\varepsilon} = \varepsilon /4$  and  $\tau = 4\| W\|_{\infty}$ . The algorithm works as follows:
$\triangleright$  First Round: Run the  $\bar{\varepsilon}$ -local DP protocol from Theorem 19 to get an estimate  $\mu^R$  of  $Rh_{\mathbf{x}}$ .
$\triangleright$  Second Round:
The analyzer forwards  $\mu^R$  to all users.
Each user  $i\in [n]$  sends the following to the analyst:
$\triangleright y_{i}^{L}\gets \mathrm{VRand}_{\bar{\varepsilon},C}(L\mathbf{1}_{x_{i}})$  (where VRand.  $(\cdot)$  is from Theorem 9).
$\triangleright a_{i}\gets \langle L\mathbf{1}_{x_{i}},\mu_{R}\rangle +\kappa_{i}$  where  $\kappa_{i}\sim \mathrm{Lap}(2\| W\|_{\infty} / \bar{\varepsilon})$  back to the analyst.
$\triangleright$  Third Round:
The analyst then computes  $Y^{L} \coloneqq \frac{1}{n}\sum_{i\in [n]}y_{i}^{L}$  and forwards it to the users.
Each user  $i\in [n]$  sends  $v_{i}\gets \mathrm{clip}_{\tau}(\langle Y^{L},R\mathbf{1}_{x_{i}} - \mu_{R}\rangle)$  +  $z_{i}$  where  $z_{i}\sim \mathrm{Lap}(2\tau /\bar{\varepsilon})$
Finally, the analyst outputs  $\frac{1}{n}\left(\sum_{i\in [n]}a_i\right) + \frac{1}{n}\left(\sum_{i\in [n]}v_i\right)$ .
Privacy Analysis. Each user  $i$ 's input is used four times:
To produce  $\mu^R$ . This step is  $\bar{\varepsilon}$ -DP by Theorem 19.
To produce  $y_{i}^{L}$ . This step is  $\bar{\varepsilon}$ -DP by Theorem 9.
To produce  $a_{i}$ . From the guarantee of Theorem 19,  $\mu_R$  belongs to  $R^{\Delta}$ . As a result,  $|\langle L\mathbf{1}_{x_i},\mu_R\rangle |\leq \| W\|_{\infty}$ . In other words, the sensitivity of  $\langle L\mathbf{1}_{x_i},\mu_R\rangle$  (as a function of  $x_{i}$ ) is at most  $2\| W\|_{\infty}$ . Since  $\kappa_{i}$  is sampled from  $\mathrm{Lap}(2\| W\|_{\infty} / \bar{\varepsilon})$ , this step is also  $\bar{\varepsilon}$ -DP.
To produce  $v_{i}$ . Due to clipping, the sensitivity of  $\mathrm{clip}_{\tau}(\langle Y^{L}, R\mathbf{1}_{x_{i}} - \mu_{R} \rangle)$  (as a function of  $x_{i}$ ) is at most  $2\tau$ . Thus, since we are adding a noise  $z_{i}$  drawn from  $\mathrm{Lap}(2\tau / \bar{\varepsilon})$ , this step is also  $\bar{\varepsilon}$ -DP.
Hence, by the basic composition theorem, the entire algorithm is  $\varepsilon$ -local DP as desired.
Utility Analysis. First, notice that, for a fixed  $\mu_R \in R^\Delta$ , we have
$$
\begin{array}{l} h _ {\mathbf {x}} ^ {T} W h _ {\mathbf {x}} = \left\langle L h _ {\mathbf {x}}, R h _ {\mathbf {x}} \right\rangle = \left\langle L h _ {\mathbf {x}}, \mu_ {R} \right\rangle + \left\langle L h _ {\mathbf {x}}, R h _ {\mathbf {x}} - \mu_ {R} \right\rangle \\ = \frac {1}{n} \sum_ {i \in [ n ]} \left\langle L \mathbf {1} _ {x _ {i}}, \mu_ {R} \right\rangle + \frac {1}{n} \sum_ {i \in [ n ]} \left\langle L h _ {\mathbf {x}}, R \mathbf {1} _ {\mathbf {x} _ {i}} - \mu_ {R} \right\rangle . \\ \end{array}
$$
Let  $z_{i}^{L} = y_{i}^{L} - L\mathbf{1}_{x_{i}}$ . From Theorem 9,  $z_{i}^{L}$  is zero-mean and  $\sigma$ -sub-Gaussian for  $\sigma = O(C / \varepsilon)$ . Let  $Z^{L} := \frac{1}{n}\sum_{i\in [n]}z_{i}^{L}$ ; we then have that  $Z^{L}$  is zero-mean and  $\sigma'$ -sub-Gaussian for  $\sigma' = \sigma /\sqrt{n}$ . Note that  $Y^{L} = Lh_{\mathbf{x}} + Z^{L}$ . Furthermore, let us write  $\theta_{i}$  as a shorthand for  $\mathrm{clip}_{\tau}(\langle Y^L,R\mathbf{1}_{x_i} - \mu_R\rangle) - \langle Y^L,R\mathbf{1}_{x_i} - \mu_R\rangle$ .
The MSE can be bounded by
$$
\begin{array}{l} \mathbb {E} \left[ \left(\frac {1}{n} \left(\sum_ {i \in [ n ]} a _ {i}\right) + \frac {1}{n} \left(\sum_ {i \in [ n ]} v _ {i}\right) - h _ {\mathbf {x}} ^ {T} W h _ {\mathbf {x}}\right) ^ {2} \right] \\ = \mathbb {E} \left[ \left(\frac {1}{n} \sum_ {i \in [ n ]} \kappa_ {i} + \frac {1}{n} \sum_ {i \in [ n ]} z _ {i} + \left\langle Z ^ {L}, R h _ {\mathbf {x}} - \mu_ {R} \right\rangle + \frac {1}{n} \sum_ {i \in [ n ]} \theta_ {i}\right) ^ {2} \right] \\ \lesssim \mathbb {E} \left[ \left(\frac {1}{n} \sum_ {i \in [ n ]} \kappa_ {i}\right) ^ {2} \right] + \mathbb {E} \left[ \left(\frac {1}{n} \sum_ {i \in [ n ]} z _ {i}\right) ^ {2} \right] + \mathbb {E} \left\langle Z ^ {L}, R h _ {\mathbf {x}} - \mu_ {R} \right\rangle^ {2} + \mathbb {E} \left[ \left(\frac {1}{n} \sum_ {i \in [ n ]} \theta_ {i}\right) ^ {2} \right] \\ \lesssim \frac {\| W \| _ {\infty} ^ {2}}{\varepsilon^ {2} n} + \mathbb {E} \left\langle Z ^ {L}, R h _ {\mathbf {x}} - \mu_ {R} \right\rangle^ {2} + \frac {1}{n} \sum_ {i \in [ n ]} \mathbb {E} \left[ \theta_ {i} ^ {2} \right], \tag {5} \\ \end{array}
$$
where the last step is by applying the Cauchy-Schwarz inequality to the last term.
To handle the middle term in eq. (5), note that  $Z^L$  is independent of  $\mu_R$  and, as stated earlier, is  $\sigma'$ -sub-Gaussian. Thus, we have
$$
\begin{array}{l} \mathbb {E} \left\langle Z ^ {L}, R h _ {\mathbf {x}} - \mu_ {R} \right\rangle^ {2} = \mathbb {E} _ {\mu_ {R}} \mathbb {E} _ {Z ^ {L}} \left\langle Z ^ {L}, R h _ {\mathbf {x}} - \mu_ {R} \right\rangle^ {2} \\ \lesssim \mathbb {E} _ {\mu_ {R}} (\sigma^ {\prime}) ^ {2} \| R h _ {\mathbf {x}} - \mu_ {R} \| _ {2} ^ {2} \\ \end{array}
$$
$$
\begin{array}{l} \left(\text {F r o m T h e o r e m 1 9}\right) \lesssim (\sigma^ {\prime}) ^ {2} \cdot \frac {C ^ {2} \sqrt {\log k}}{\varepsilon \sqrt {n}} \\ \lesssim \frac {C ^ {4} \sqrt {\log k}}{\varepsilon^ {3} n ^ {3 / 2}} \\ \lesssim \frac {\| W \| _ {\infty} ^ {2}}{\varepsilon^ {2} n}, \tag {6} \\ \end{array}
$$
where the last inequality is due to our assumption on  $n$ .
As for the last term in eq. (5), notice that
$$
\left\langle Y ^ {L}, R \mathbf {1} _ {x _ {i}} - \mu_ {R} \right\rangle = \left\langle Z ^ {L}, R \mathbf {1} _ {x _ {i}} - \mu_ {R} \right\rangle + \left\langle L h _ {\mathbf {x}}, R \mathbf {1} _ {x _ {i}} - \mu_ {R} \right\rangle .
$$
Since  $\mu_R$  belongs to  $R^{\Delta}$ , we have  $\frac{1}{2} (R\mathbf{1}_{x_i} - \mu_R) \in R^{\Delta}$ , which implies that  $|\langle Lh_{\mathbf{x}}, R\mathbf{1}_{x_i} - \mu_R \rangle| \leq 2\| W \|_{\infty}$  and  $\| \bar{R}\mathbf{1}_{x_i} - \mu_R \|_2 \leq 2C$ . As such, we have
$$
\begin{array}{l} \mathbb {E} \left\langle Y ^ {L}, R \mathbf {1} _ {x _ {i}} - \mu_ {R} \right\rangle^ {4} \lesssim \mathbb {E} \left\langle Z ^ {L}, R \mathbf {1} _ {x _ {i}} - \mu_ {R} \right\rangle^ {4} + \| W \| _ {\infty} ^ {4} \\ \lesssim (\sigma^ {\prime}) ^ {4} \cdot C ^ {4} + \| W \| _ {\infty} ^ {4} \\ \lesssim \frac {C ^ {8}}{\varepsilon^ {4} n ^ {2}} + \| W \| _ {\infty} ^ {4} \\ \lesssim \| W \| _ {\infty} ^ {4}, \\ \end{array}
$$
where the last inequality is due to our assumption on  $n$  and from  $\gamma_2(W) \geq \|W\|_{\infty}$ .
Meanwhile, since  $\tau = 4\| W\|_{\infty}$ , we have
$$
\begin{array}{l} \Pr \left[ \left| \left\langle Y ^ {L}, R \mathbf {1} _ {x _ {i}} - \mu_ {R} \right\rangle \right| > \tau \right] \leq \Pr \left[ \left| \left\langle Z ^ {L}, R \mathbf {1} _ {x _ {i}} - \mu_ {R} \right\rangle \right| > \tau / 2 \right] \\ \leq \exp \left(- \Omega \left(\tau / \left(\sigma^ {\prime} \cdot 2 C\right)\right) ^ {2}\right) \\ = \exp \left(- \Omega (\varepsilon \| W \| _ {\infty} \sqrt {n} / C ^ {2}) ^ {2}\right) \\ \leq \frac {1}{\varepsilon^ {4} n ^ {2}}, \\ \end{array}
$$
where the last inequality is again due to our assumption on  $n$  and from  $\gamma_2(W) \geq \|W\|_{\infty}$ .
Thus, we may apply Lemma 18 to conclude that
$$
\mathbb {E} \left[ \theta_ {i} ^ {2} \right] \lesssim \sqrt {\| W \| _ {\infty} ^ {4} \cdot \frac {1}{\varepsilon^ {4} n ^ {2}}} = \frac {\| W \| _ {\infty} ^ {2}}{\varepsilon^ {2} n}. \tag {7}
$$
Combining eqs. (5) to (7), the MSE of the estimate is at most  $O\left(\frac{\|W\|_{\infty}^{2}}{\varepsilon^{2}n}\right)$  as desired.
C.1 On Separating Non-Interactive and Interactive Local DP
We end by observing that our interactive local DP algorithm (Theorem 24) together with the non-interactive lower bound (Theorem 6 and particularly, the more quantitative version, Theorem 23) gives an asymptotic separation on the MSE achievable by  $\varepsilon$ -local DP interactive algorithms and those that are non-interactive, as long as we pick  $W$  together with  $n \in \mathbb{N}$  such that the following holds:
$$
\begin{array}{l} \triangleright n \gtrsim \frac {\gamma_ {2} (W) ^ {4} \log k}{\| W \| _ {\infty} ^ {4}} \cdot \log \left(\frac {1 0 \gamma_ {2} (W) \log k}{\| W \| _ {\infty}}\right) \text {a n d} \\ \begin{array}{l} \triangleright \frac {\zeta (W ; n)}{\| W \| _ {\infty}} \geq \omega (\log n), \end{array} \\ \end{array}
$$
where the first condition is from Theorem 24 (and letting  $\varepsilon = 1$ ) and the second condition ensures that the error from Theorem 24 is asymptotically larger than the lower bound from Theorem 23.
It is simple to verify that it suffices to pick  $W$  such that  $\| W\|_{\infty} = 1, \gamma_2(W,0.1), \gamma_2(W) = \Theta (k^c)$  for any constant  $c > 0$  and pick  $n$  to precisely satisfy the bound in the first condition. Again, it is not hard to construct such a matrix e.g., by taking  $W$  to be a random  $(k\times k)$  matrix where each entry is an i.i.d. Rademacher random variable, which yields  $\gamma_{2}(W,0.1),\gamma_{2}(W) = \Theta (\sqrt{k})$  w.h.p. [LMSS07].