SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 9 ([BBNS19]). For any  $C, \varepsilon > 0$ , there is a (non-interactive)  $\varepsilon$ -local DP algorithm  $\mathrm{VRand}_{\varepsilon, C}$  that takes in  $x \in \mathbb{R}^d$  such that  $\|x\|_2 \leq C$  and outputs  $Y \in \mathbb{R}^d$  such that  $\mathbb{E}[Y] = x$  and  $Y - x$  is  $\sigma$ -sub-Gaussian for  $\sigma = O(C / \varepsilon)$ .
Error: Factorization vs Approximate-Factorization. We show that it suffices to give errors in terms of  $\gamma_{2}(W)$  instead of  $\zeta(W; n)$ ; this will be convenient for our subsequent proofs. Due to space constraints, the proof of the following statement is deferred to Appendix A.
Lemma 10. Suppose that, for all  $W \in \mathbb{R}^{k \times k}$ , there is a non-interactive  $\varepsilon$ -local DP protocol  $\mathbb{A}$  for quadratic form on  $W$  with MSE  $O\left(c(n,\varepsilon,k) \cdot \frac{\gamma_2(W)^2}{\varepsilon^2n}\right)$  where  $c(n,\varepsilon,k) \geq \Omega(1)$ . Then there is also a non-interactive  $\varepsilon$ -local DP protocol  $\mathbb{A}'$  with MSE  $O\left(c(n,\varepsilon,k) \cdot \frac{\zeta(W;n)^2}{\varepsilon^2n}\right)$ .
3 Non-Interactive Algorithm
In this section, we prove Theorem 5. To do so, let us start by defining (approximate) rank-restricted factorization norms<sup>5</sup>, which are the same as  $\gamma_2(W), \gamma_2(W; \alpha)$  except we now restrict the number of rows of  $L, R$  to be at most  $\ell$ :
For  $W\in \mathbb{R}^{k\times k}$ $\ell \in \mathbb{N}$  , let  $\gamma_2^\ell (W)\coloneqq \min_{L^T R = W;L,R\in \mathbb{R}^{\ell \times k}}\| L\|_{1\to 2}\| R\|_{1\to 2}.$
For  $W\in \mathbb{R}^{k\times k}$ $\ell \in \mathbb{N}$  and  $\alpha \geq 0$  , let  $\gamma_2^\ell (W;\alpha)\coloneqq \min_{\| \tilde{W} -W\|_\infty \leq \alpha}\gamma_2^\ell (\tilde{W}).$
We can now use the approximate rank-restricted factorization to perform the algorithm as outlined in Section 1.2 with an error term that depends on  $\ell$  (and  $\alpha$ ):
Lemma 11. For any  $W \in \mathbb{R}^{k \times k}$ ,  $\ell \in \mathbb{N}$ , and  $\alpha \geq 0$ , there is a non-interactive  $\varepsilon$ -local DP algorithm that estimates the quadratic form on  $W$  to within an MSE of  $O\left(\alpha^2 + \gamma_2^\ell(W; \alpha)^2 \cdot \left(\frac{1}{\varepsilon^{2n}} + \frac{\ell}{\varepsilon^{4n^2}}\right)\right)$ .
Proof. By definition of  $\gamma_2^\ell (W;\alpha)$ , there exists  $\tilde{W}\in \mathbb{R}^{k\times k}$ ,  $L,R\in \mathbb{R}^{\ell \times k}$  such that  $\| \tilde{W} -W\|_{\infty}\leq \alpha$  and  $\tilde{W} = L^{T}R$  where, w.l.o.g., by rescaling,  $\| L\|_{1\to 2} = \| R\|_{1\to 2} = \sqrt{\gamma_2^\ell(W)} =: C$ .
Algorithm Description. Let  $\bar{\varepsilon} = \varepsilon /2$ . The algorithm works as follows:
Each user  $i\in [n]$  sends  $y_{i}^{L}\gets \mathrm{VRand}_{\bar{\varepsilon},C}(L\mathbf{1}_{x_{i}})$  and  $y_{i}^{R}\gets \mathrm{VRand}_{\bar{\varepsilon},C}(R\mathbf{1}_{x_{i}})$  to the analyst (where VRand.  $(\cdot)$  is from Theorem 9).
The analyst outputs  $\left\langle \frac{1}{n}\sum_{i\in [n]}y_i^L,\frac{1}{n}\sum_{i\in [n]}y_i^R\right\rangle$
As each user runs an  $(\varepsilon /2)$ -DP randomizer twice, the algorithm is  $\varepsilon$ -DP.
Utility Analysis. Let  $z_{i}^{L} = y_{i}^{L} - L\mathbf{1}_{x_{i}}$  and  $z_{i}^{R} = y_{i}^{R} - R\mathbf{1}_{x_{i}}$ . From Theorem 9,  $z_{i}^{L}, z_{i}^{R}$  are zero-mean and  $\sigma$ -sub-Gaussian for  $\sigma = O(C / \varepsilon)$ . Let  $Z^{L} := \frac{1}{n}\sum_{i\in [n]}z_{i}^{L}, Z^{R} := \frac{1}{n}\sum_{i\in [n]}z_{i}^{R}$ ; we then have that these are zero-mean and  $\sigma'$ -sub-Gaussian for  $\sigma' = \sigma / \sqrt{n}$ . The MSE of the protocol is given by
$$
\begin{array}{l} \mathbb {E} \left[ \left(\left\langle \frac {1}{n} \sum_ {i \in [ n ]} y _ {i} ^ {L}, \frac {1}{n} \sum_ {i \in [ n ]} y _ {i} ^ {R} \right\rangle - h _ {\mathbf {x}} ^ {T} W h _ {\mathbf {x}}\right) ^ {2} \right] \\ = \mathbb {E} \left[ \left(h _ {\mathbf {x}} ^ {T} (\tilde {W} - W) h _ {\mathbf {x}} + \left\langle Z ^ {L}, R h \right\rangle + \left\langle L h, Z ^ {R} \right\rangle + \left\langle Z ^ {L}, Z ^ {R} \right\rangle\right) ^ {2} \right] \\ \lesssim \mathbb {E} (h _ {\mathbf {x}} ^ {T} (\tilde {W} - W) h _ {\mathbf {x}}) ^ {2} + \mathbb {E} \left<   Z ^ {L}, R h \right> ^ {2} + \mathbb {E} \left<   L h, Z ^ {R} \right> ^ {2} + \mathbb {E} \left<   Z ^ {L}, Z ^ {R} \right> ^ {2} \\ \lesssim \| \tilde {W} - W \| _ {\infty} ^ {2} + (\sigma^ {\prime}) ^ {2} \| R h \| _ {2} ^ {2} + (\sigma^ {\prime}) ^ {2} \| L h \| _ {2} ^ {2} + \ell \cdot (\sigma^ {\prime}) ^ {4} \\ \end{array}
$$
These are not actually norms, but we use the term for consistency with other similar quantities.
$$
\begin{array}{l} \leq \alpha^ {2} + (\sigma^ {\prime}) ^ {2} \| R \| _ {1 \rightarrow 2} ^ {2} + (\sigma^ {\prime}) ^ {2} \| L \| _ {1 \rightarrow 2} ^ {2} + \ell \cdot (\sigma^ {\prime}) ^ {4} \\ \lesssim \alpha^ {2} + \left(\sigma^ {\prime}\right) ^ {2} C ^ {2} + \ell \cdot \left(\sigma^ {\prime}\right) ^ {4} \\ \lesssim \alpha^ {2} + \frac {C ^ {4}}{\varepsilon^ {2} n} + \frac {\ell \cdot C ^ {4}}{\varepsilon^ {4} n ^ {2}}. \\ \end{array}
$$
Rank-Restricted Approximate Factorization via JL. We next show that w.l.o.g. we can take  $\ell$  to be quite small in Lemma 11 above.
Lemma 12. Let  $W \in \mathbb{R}^{k \times k}$  and  $\alpha \in (0, \gamma_2(W))$ , there is  $\ell \lesssim \frac{\gamma_2(W)^2 \log k}{\alpha^2}$  with  $\gamma_2^\ell(W; \alpha) \lesssim \gamma_2(W)$ .
As mentioned earlier, this lemma is proved by applying Johnson-Lindenstrauss (JL) dimensionality reduction to each column of  $L$ ,  $R$ . We summarize a simplified version of the JL lemma below.
Lemma 13 (Johnson-Lindenstrauss Lemma (e.g., [DG03])). Let  $\beta \in (0,1)$  and  $U = \{u_{1},\dots u_{m}\} \subseteq \mathbb{R}^{d}$ . For some  $\ell \leq O(\beta^{-2}\log m)$ , there exists a matrix  $A\in \mathbb{R}^{\ell \times d}$  such that for all  $u,v\in U$ , we have  $(1 - \beta)\| u - v\| _2^2\leq \| Au - Av\| _2^2\leq (1 + \beta)\| u - v\| _2^2$ .
We are now ready to prove Lemma 12.
Proof of Lemma 12. By definition of  $\gamma_2(W)$ , there exists  $L, R \in \mathbb{R}^{d \times k}$  for some  $d \in \mathbb{N}$  such that  $W = L^T R$  where  $\|L\|_{1 \to 2} = \|R\|_{1 \to 2} = \sqrt{\gamma_2(W)} \eqqcolon C$ .
Let  $L_{1},\ldots ,L_{k}$  (resp.  $R_{1},\ldots ,R_{k})$  be the columns of  $L$  (resp.  $R$ ). Consider  $U = \{0,L_1,\dots ,L_k,R_1,\dots ,R_k, - L_1,\dots , - L_k, - R_1,\dots , - R_k\}$  and  $\beta = 0.5\alpha /C^2$ ; let  $\ell = O\left(\beta^{-2}\log k\right) = O\left(\frac{\gamma_2(W)^2\log k}{\alpha^2}\right)$  and  $A\in \mathbb{R}^{\ell \times d}$  be as guaranteed by Lemma 13.
Let  $\tilde{L} = AL, \tilde{R} = AR$ , and  $\tilde{W} = \tilde{L}^T \tilde{R}$ . For all  $i \in [k]$ , we have  $\| \tilde{L}_i \|_2^2 \leq (1 + \beta) \| L_i \|_2^2$  and  $\| \tilde{R}_i \|_2^2 \leq (1 + \beta) \| R_i \|_2^2$ . Therefore,  $\| \tilde{L} \|_{1 \to 2}, \| \tilde{R} \|_{1 \to 2} \leq O(C)$ , i.e.,  $\gamma_2^\ell (\tilde{W}) \leq O(\gamma_2(W))$ . Moreover, for each  $i, j \in [k]$ , we have
$$
\begin{array}{l} \left| \tilde {W} _ {i, j} - W _ {i, j} \right| = \left| \left\langle \tilde {L} _ {i}, \tilde {R} _ {j} \right\rangle - \left\langle L _ {i}, R _ {j} \right\rangle \right| \\ \leq \frac {1}{4} \left(\left| \| \tilde {L} _ {i} + \tilde {R} _ {j} \| _ {2} ^ {2} - \| L _ {i} + R _ {j} \| _ {2} ^ {2} \right| + \left| \| \tilde {L} _ {i} - \tilde {R} _ {j} \| _ {2} ^ {2} - \| L _ {i} - R _ {j} \| _ {2} ^ {2} \right|\right) \\ \leq \frac {\beta}{4} \left(\| L _ {i} + R _ {j} \| _ {2} ^ {2} + \| L _ {i} - R _ {j} \| _ {2} ^ {2}\right) \\ \leq 2 \beta C ^ {2} \leq \alpha . \\ \end{array}
$$
Thus,  $\| \tilde{W} - W \|_{\infty} \leq \alpha$  and therefore  $\gamma_2^\ell(W; \alpha) \leq O(\gamma_2(W))$  as claimed.
We end this section by proving Theorem 5, which is a simple combination of Lemma 11 and Lemma 12.
Proof of Theorem 5. Pick $^6$ $\alpha = \frac{\gamma_2(W)}{\varepsilon\sqrt{n}}$  and apply Lemma 12: There exists  $\ell = O\left(\frac{\gamma_2(W)^2\log k}{\alpha^2}\right) = O((\log k)\varepsilon^2 n)$  such that  $\gamma_2^\ell (W;\alpha)\leq O(\gamma_2(W))$ . Plugging this back into Lemma 11 then gives us a non-interactive  $\varepsilon$ -local DP protocol with MSE
$$
\lesssim \alpha^ {2} + \gamma_ {2} ^ {\ell} (W; \alpha) ^ {2} \cdot \left(\frac {1}{\varepsilon^ {2} n} + \frac {\ell}{\varepsilon^ {4} n ^ {2}}\right) \lesssim \gamma_ {2} (W) ^ {2} \left(\frac {1}{\varepsilon^ {2} n} + \frac {\log k}{\varepsilon^ {2} n}\right) \lesssim \frac {\gamma_ {2} (W) ^ {2} \log k}{\varepsilon^ {2} n}.
$$
Applying Lemma 10 then concludes the proof.
4 Lower Bounds for Non-Interactive Algorithms
In this section we formalize the reduction from linear queries to computing quadratic forms, as outlined in Section 1.2. The properties of the reduction are stated in the theorem below.
<sup>6</sup>We may assume w.l.o.g. that  $\alpha \leq \gamma_2(W)$ ; otherwise, the guarantee in Theorem 5 is trivial, i.e., always outputting 0 satisfies the bound.
Theorem 14. Let  $W \in \mathbb{R}^{k \times k}$  be symmetric. Suppose that there is a non-interactive  $\varepsilon$ -local DP mechanism for computing the quadratic form on  $W$  with MSE at most  $\alpha(\varepsilon, n)$ . Then there is a noninteractive  $\varepsilon$ -local DP protocol for computing the linear queries of  $W$  with mMSE  $O(\alpha(\varepsilon/2, n) + \alpha(\varepsilon/2, 2n))$ .
Theorem 14 and the lower bound in Theorem 4 immediately imply Theorem 6. In the proof below, we use subscripts  $\varepsilon, n$  to denote the privacy loss parameter and the number of users in the protocol.
Proof of Theorem 14. Let  $\mathbb{A}_{\varepsilon,n}$  be the  $\varepsilon$ -local DP protocol for computing the quadratic form on  $W$ , and let  $\mathcal{R}_{\varepsilon,n}$  denote its randomizer. We construct an algorithm  $\mathbb{A}_{\varepsilon,n}^{\prime}$  for linear queries with workload matrix  $W$ . On input  $x_{1},\ldots,x_{n}$ , proceed as follows:
$\triangleright$  Run the protocol of  $\mathbb{A}_{\varepsilon /2,n}$  on  $x_{1},\ldots ,x_{n}$  to compute an estimate  $\hat{z}$  for  $h_{\mathbf{x}}^{T}\mathbb{A}h_{\mathbf{x}}$
$\triangleright$  In the same round as above, run the randomizer  $\mathcal{R}_{\varepsilon /2,2n}$  of  $\mathbb{A}_{\varepsilon /2,2n}$  on  $x_{1},\ldots ,x_{n}$  to get the responses  $\mathcal{R}_{\varepsilon /2,2n}(x_1),\dots,\mathcal{R}_{\varepsilon /2,2n}(x_n)$ .
For each  $j\in [k]$  , do the following:
$\triangleright$  Run the randomizer  $\mathcal{R}$  of  $\mathbb{A}$  on  $y_{1} = \dots = y_{n} = j$  to get the responses  $\mathcal{R}_{\varepsilon /2,2n}(y_1),\ldots ,$ $\mathcal{R}_{\varepsilon /2,2n}(y_n)$
$\triangleright$  Compute the estimator  $z_{j}^{\prime}$  of  $\mathbb{A}$  on the  $2n$  responses  $\mathcal{R}_{\varepsilon /2,2n}(x_1),\ldots ,\mathcal{R}_{\varepsilon /2,2n}(x_n)$ $\mathcal{R}_{\varepsilon /2,2n}(y_1),\ldots ,\mathcal{R}_{\varepsilon /2,2n}(y_n)$
$\triangleright$  Set  $\hat{z}_j = 2z_j' - \frac{1}{2}\hat{z} -\frac{1}{2}\mathbf{1}_j^T W\mathbf{1}_j.$
Output  $(\hat{z}_1,\dots ,\hat{z}_j)$  as the estimates for the linear queries.
Since  $(\varepsilon /2)$ -local DP randomizers are run on each input  $x_{i}$  twice, the basic composition theorem implies that this is a  $\varepsilon$ -local DP algorithm as desired.
For each  $j\in [k]$  , we now compute the MSE of  $\hat{z}_j$  . First, observe that
$$
(W h _ {\mathbf {x}}) _ {j} = \mathbf {1} _ {j} ^ {T} W h _ {\mathbf {x}} = 2 \left(h _ {\mathbf {x} \cup \mathbf {y}} ^ {T} W h _ {\mathbf {x} \cup \mathbf {y}}\right) - \frac {1}{2} h _ {\mathbf {x}} ^ {T} W h _ {\mathbf {x}} ^ {T} - \frac {1}{2} \mathbf {1} _ {j} ^ {T} W \mathbf {1} _ {j},
$$
where  $\mathbf{y}$  denotes the dataset with  $n$  copies of  $j$ .
Therefore, we can bound the MSE of  $\hat{z}_j$  as follows:
$$
\begin{array}{l} \operatorname {M S E} \left(\hat {z} _ {j}; \left(W h _ {\mathbf {x}}\right) _ {j}\right) = \mathbb {E} \left[ \left(\hat {z} _ {j} - \left(W h _ {\mathbf {x}}\right) _ {j}\right) ^ {2} \right] \\ = \mathbb {E} \left[ \left(2 \left(z _ {j} ^ {\prime} - h _ {\mathbf {x} \cup \mathbf {y}} ^ {T} W h _ {\mathbf {x} \cup \mathbf {y}}\right) + \frac {1}{2} \left(\hat {z} - h _ {\mathbf {x}} ^ {T} W h _ {\mathbf {x}} ^ {T}\right)\right) ^ {2} \right] \\ \lesssim \mathbb {E} \left[ \left(z _ {j} ^ {\prime} - h _ {\mathbf {x} \cup \mathbf {y}} ^ {T} W h _ {\mathbf {x} \cup \mathbf {y}}\right) ^ {2} \right] + \mathbb {E} \left[ \left(\hat {z} - h _ {\mathbf {x}} ^ {T} W h _ {\mathbf {x}} ^ {T}\right) ^ {2} \right] \\ \leq \alpha (\varepsilon / 2, 2 n) + \alpha (\varepsilon / 2, n), \\ \end{array}
$$
where the last inequality follows from the guarantees of  $\mathbb{A}$ . Thus, the mMSE of  $\mathbb{A}'$  is at most  $O(\alpha(\varepsilon/2, 2n) + \alpha(\varepsilon/2, n))$ , as desired.
5 Upper Bounds for Specific Metrics
In this section, we obtain concrete upper bounds for many well-known U-statistics of degree 2. For a kernel  $f: \mathcal{X} \to \mathbb{R}$ , let  $W^{f} \in \mathbb{R}^{\mathcal{X} \times \mathcal{X}}$  denote the matrix defined by  $W_{x,x'}^{f} = f(x,x')$ . Our proof of Corollary 7 proceeds by providing an upper bound on  $\gamma_{2}(W^{f})$  for each U-statistic with kernel  $f$ ; the bounds immediately follow from Theorem 5. Similar to before, let  $k$  denote  $|\mathcal{X}|$ .
The following (non-trivial) facts about the factorization norm are useful to keep in mind:
Fact 15. The factorization norm  $\gamma_{2}$  satisfies the following properties:
1. [TJ89] For any  $A, B$ , we have  $\gamma_2(A + B) \leq \gamma_2(A) + \gamma_2(B)$ .
2. [LSS08] For any  $A, B$ ,  $\gamma_2(A \otimes B) = \gamma_2(A) \cdot \gamma_2(B)$ .
Gini's diversity index. For  $\mathcal{X} \subseteq \mathbb{R}$ , the kernel  $f(x, x') = \mathbf{1}[x \neq x']$  captures Gini's diversity index. From Fact 15(1), we have  $\gamma_2(W^f) \leq \gamma_2(\mathbf{1}_{k \times k}) + \gamma_2(\mathbf{I}_{k \times k}) \leq 1 + 1$  where the inequality  $\gamma_2(\mathbf{1}_{k \times k}) \leq 1$  is from the factorization  $L = R = \mathbf{1}_k$  and  $\gamma_2(\mathbf{I}_{k \times k}) \leq 1$  is from  $L = R = \mathbf{I}_{k \times k}$ .
Kendall's  $\tau$  coefficient and AUC. For  $\mathcal{X} = A\times B\subseteq \mathbb{R}^2$  with  $x_{i} = (y_{i},z_{i})$ , the kernel  $f((y_i,z_i),(y_j,z_j)) = \operatorname {sgn}(y_i - y_j)\cdot \operatorname {sgn}(z_i - z_j)$  yields Kendall's  $\tau$  coefficient. Let  $U_{m}\in \{-1,1\}^{m}$  denote the matrix that has  $+1$  on all entries above the main diagonal (inclusive) and  $-1$  elsewhere. It is known that  $\gamma_2(U_m) = \Theta (\log m)$ . We can bound  $\gamma_{2}(W^{f})$  for Kendall's  $\tau$  coefficient by observing that  $W^{f} = U_{A}\otimes U_{B}$ . From Fact 15(2),  $\gamma_{2}(W_{f})\leq \gamma_{2}(U_{A})\cdot \gamma_{2}(U_{B})\lesssim (\log |A|)(\log |B|)\leq (\log k)^{2}$ .
AUC for binary classification is defined in a similar manner as Kendall's tau coefficient, except that (i)  $B = \{0,1\}$  and (ii) the normalization constant being  $\frac{1}{n + n^{-}}$  instead of  $\frac{1}{\binom{n}{2}}$  where  $n^+$  (resp.,  $n^-$ ) denotes the number of 1-labeled (resp., 0-labeled) examples. The AUC result follows from the above since  $|B| = 2$  in this case. We remark that, for the AUC case, we also have to split the privacy budget and use half of it to estimate  $n^{+}$  so that we can renormalize correctly. It is not hard to see that this renormalization procedure results in at most an additive factor of  $O\left(\frac{1}{\varepsilon^2n}\right)$  in the MSE, under the balancedness assumption that  $n^{-}, n^{+} \geq \Omega(n)$ .
Lipschitz Losses. Let  $\mathcal{X} = [0,1]$  and let  $f:\mathcal{X}\to \mathbb{R}$  be any function such that  $|f(x) - f(x^{\prime})|\leq G\cdot |x - x^{\prime}|$ ; we call  $f$ $G$ -Lipschitz. This class includes U-statistics such as the Gini's mean difference, which is given by the kernel  $f(x_{i},x_{j}) = |x_{i} - x_{j}|$ , which is 1-Lipschitz.
Similar to [BBGK20], we use a discrete case where  $\mathcal{X} = [k]$  as a subroutine. Our algorithm for this is stated below. Note that Corollary 16 immediately implies the bound for the continuous case: given any function  $f:[0,1]\to \mathbb{R}$ , we may select  $k$  to be sufficiently large, e.g.,  $k = \Theta (\varepsilon n^2)$ , and discretize the function over the points  $1 / k,2 / k,\ldots ,k / k$ . Defining  $g:[k]\rightarrow \mathbb{R}$  by  $g(i) = f(i / k)$  allows us to use Corollary 16 with Lipschitz constant  $G / k$ . This leads to a MSE of  $O\left(\frac{G^2\log(\varepsilon n^2)}{\varepsilon^2n}\right)$ . The MSE resulting from the discretization error is then at most  $\frac{G^2n^2}{k^2} < \frac{G^2}{\varepsilon^2n}$ .
Corollary 16 (Discrete Lipschitz). Assuming  $\mathcal{X} = [k]$  and that  $f$  is  $G$ -Lipschitz. There is a noninteractive  $\varepsilon$ -local DP algorithm for computing pairwise statistics for  $f$  with MSE  $O\left(\frac{G^2k^2(\log k)}{\varepsilon^2n}\right)$ .
Again, the above corollary follows from Theorem 5 and the following factorization.
Lemma 17. Assuming that  $f:[k]\to \mathbb{R}$  is  $G$ -Lipschitz, then  $\gamma_{2}(W^{f})\leq O(Gk)$ .
Proof. We assume w.l.o.g. that  $k = 2^q - 1$  for some  $q \in \mathbb{N}$  and  $\| W^f \|_{\infty} \leq Lk$ ; otherwise, we may shift  $W^f$  (and the answer) without incurring any additional error. We arrange  $[k]$  into a balanced binary search tree  $\mathcal{T}$  of depth  $q - 1$  naturally (where the root is  $2^{q - 1}$  and the leaves are  $1, 3, \ldots, k$ ). Let  $P(j)$  denote the path from node  $j$  to the root (inclusive) in  $\mathcal{T}$ , and let  $\ell(j)$  denote the depth of  $j$  (where the root has depth 0). Furthermore, let  $\text{parent}(j)$  denote the parent of  $j$  in  $\mathcal{T}$ . For notational convenience, let  $\text{parent}(2^{q - 1}) = \bot$  and let  $f(i, \bot) = 0$  for all  $i \in [k]$ .
We construct  $L,R\in \mathbb{R}^{k\times k}$  as follows.
For all  $i,j\in [k]$  , let  $R_{i,j} = \left(\frac{5}{6}\right)^{\ell (i)}\mathbf{1}[i\in P(j)]$
For all  $i,j\in [k],L_{j,i} = \left(\frac{6}{5}\right)^{\ell (j)}(f(i,j) - f(i,\mathrm{parent}(j)))$
For  $i,j\in [k]$ , we have  $(L^T R)_{i,j} = \sum_{t\in P(j)}(f(i,t) - f(i,\mathrm{parent}(t))) = f(i,j)$ . Thus,  $L^T R = W$ .
Furthermore,  $\| R\|_{1\to 2}^2 = 1 + \left(\frac{5}{6}\right)^2 +\dots +\left(\frac{5}{6}\right)^{2(q - 1)}\lesssim 1$ . Meanwhile, we can bound  $\| L\|_{1\to 2}^2$  by
$$
\max _ {i \in [ k ]} \sum_ {j \in [ k ]} \left(\left(\frac {6}{5}\right) ^ {\ell (j)} (f (i, j) - f (i, \mathrm {p a r e n t} (j)))\right) ^ {2} \lesssim \sum_ {d = 0} ^ {q - 1} 2 ^ {d} \left(\left(\frac {6}{5}\right) ^ {d} \cdot \frac {G k}{2 ^ {d}}\right) ^ {2} \lesssim G ^ {2} k ^ {2},
$$
where the first inequality follows since  $f$  is  $G$ -Lipschitz. Thus  $\gamma_2(W^f) \leq O(Gk)$ .
6 Conclusion and Open Questions
In this work, we systematically study the problem of privately computing pairwise statistics. We give a non-interactive local DP algorithm and a nearly-matching lower bound for the problem. Furthermore, we show that, for some metrics, improvements can be made if interaction is allowed.
There are several immediate questions from our work. For example, is it possible to remove the  $\log k$  multiplicative factor in our non-interactive algorithm (Theorem 5)? Similarly, can the second additive term in our interactive algorithm be removed? As also suggested by [BBGK20], an intriguing research direction is to study more complicated statistics such as the "higher-degree" ones (e.g., those involving triplets instead of pairs). It would be interesting to see if techniques from linear queries and from our work can be applied to these problems.
[GMPV20] Badih Ghazi, Pasin Manurangsi, Rasmus Pagh, and Ameya Velingker. Private aggregation from fewer anonymous messages. In EUROCRYPT, pages 798–827, 2020.
[Gre16] Andy Greenberg. *Apple's "differential privacy" is about collecting your data – but not your data*. Wired, June, 13, 2016.
[HT10] Moritz Hardt and Kunal Talwar. On the geometry of differential privacy. In STOC, pages 705-714, 2010.
$\left[\mathrm{KLN}^{+}11\right]$  Shiva Prasad Kasiviswanathan, Homin K. Lee, Kobbi Nissim, Sofya Raskhodnikova, and Adam D. Smith. What can we learn privately? SIAM J. Comput., 40(3):793-826, 2011.
[KT18] Krishnaram Kenthapadi and Thanh T. L. Tran. Pripearl: A framework for privacy-preserving analytics and reporting at linkedin. In CIKM, pages 2183-2191, 2018.
$\left[\mathrm{LMH}^{+}15\right]$  Chao Li, Gerome Miklau, Michael Hay, Andrew McGregor, and Vibhor Rastogi. The matrix mechanism: optimizing linear counting queries under differential privacy. VLDB J., 24(6):757-781, 2015.
[LMSS07] Nati Linial, Shahar Mendelson, Gideon Schechtman, and Adi Shraibman. Complexity measures of sign matrices. Comb., 27(4):439-463, 2007.
[LSS08] Troy Lee, Adi Shraibman, and Robert Spalek. A direct product theorem for discrepancy. In CCC, pages 71-80, 2008.
[Nik23] Aleksandar Nikolov. Private query release via the johnson-lindenstrauss transform. In Nikhil Bansal and Viswanath Nagarajan, editors, SODA, pages 4982-5002, 2023.
[NTZ16] Aleksandar Nikolov, Kunal Talwar, and Li Zhang. The geometry of differential privacy: The small database and approximate cases. SIAM J. Comput., 45(2):575-616, 2016.
[RE19] Carey Radebaugh and Ulfar Erlingsson. Introducing TensorFlow Privacy: Learning with Differential Privacy for Training Data, March 2019. blog.tensorflow.org.
$\left[\mathrm{RSP}^{+}21\right]$  Ryan Rogers, Subbu Subramaniam, Sean Peng, David Durfee, Seunghyun Lee, Santosh Kumar Kancha, Shraddha Sahay, and Parvez Ahammad. LinkedIn's audience engagements API: A privacy preserving data analytics system at scale. J. Priv. Confiden., 11(3), 2021.
[Sha14] Stephen Shankland. How Google tricks itself to protect Chrome user privacy. CNET, October, 2014.
[TJ89] Nicole Tomczak-Jaegermann. Banach-Mazur distances and finite-dimensional operator ideals. Pure and Applied Mathematics, 38:395, 1989.
[TM20] Davide Testuggine and Ilya Mironov. PyTorch Differential Privacy Series Part 1: DP-SGD Algorithm Explained, August 2020. medium.com.
We use  $d_{\mathrm{tv}}$ ,  $d_{\mathrm{KL}}$  to denote the total variation (TV) distance and the Kullback-Leibler (KL) divergence between two distributions. Pinsker's inequality states that
$$
d _ {\mathrm {t v}} (P, Q) \leq \sqrt {\frac {1}{2} \cdot d _ {\mathrm {K L}} (P \| Q)}, \tag {1}
$$
for all distributions  $P, Q$ .
Interactive Local DP
For interactive local DP, we consider protocols that proceed in rounds. In each round, the analyst may send a message (which may depend on what the analyst has received in the previous rounds) to each user, who then replies back with a (randomized) response. The DP guarantee is then enforced on the view of the analyst.
Clipping and Resulting Error
Clipping is a standard technique in DP to achieve bounded sensitivity (cf.  $\mathrm{[ACG^+16]}$ ). In our interactive algorithm, we will use clipping on real values to bound their sensitivity. More specifically, for  $\tau \geq 0$ , let  $\mathrm{clip}_{\tau}:\mathbb{R}\to \mathbb{R}$  denote the function:
$$
\operatorname {c l i p} _ {\tau} (x) = \left\{ \begin{array}{l l} \tau & \text {i f} x > \tau , \\ x & \text {i f} \tau \geq x \geq - \tau , \\ - \tau & \text {i f} - \tau > x. \end{array} \right.
$$
There could be additional errors resulting from clipping. For the purpose of bounding such terms, we will use the following simple lemma:
Lemma 18. Let  $X$  denote any random variable over  $\mathbb{R}$  and any  $\tau \geq 0$ . Then, we have
$$
\mathbb {E} \left[ (X - \operatorname {c l i p} _ {\tau} (X)) ^ {2} \right] \leq \sqrt {\mathbb {E} [ X ^ {4} ] \cdot \Pr [ | X | > \tau ]}.
$$
Proof. We can bound the LHS term by
$$
\begin{array}{l} \mathbb {E} \left[ \left(X - \operatorname {c l i p} _ {\tau} (X)\right) ^ {2} \right] \leq \mathbb {E} \left[ X ^ {2} \cdot \mathbf {1} \left[ | X | > \tau \right] \right] \\ \leq \sqrt {\mathbb {E} [ X ^ {4} ] \cdot \mathbb {E} [ \mathbf {1} [ | X | > \tau ] ^ {2} ]} \\ = \sqrt {\mathbb {E} [ X ^ {4} ] \cdot \Pr [ | X | > \tau ]}, \\ \end{array}
$$