SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\widehat {\mathbf {x}} ^ {(\text {e s t i m a t o r})} = \bar {\beta} \Big (\frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {G} _ {i} \Big) ^ {\dagger} \Big (\frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {G} _ {i} \Big) \mathbf {x}.
$$
Note that  $\mathrm{rank}(\frac{1}{n}\sum_{i = 1}^{n}\pmb {G}_i)$  is at most  $k$ , since  $\pmb {G}_i\in \mathbb{R}^{k\times d}$ ,  $\forall i\in [k]$ . This limits the amount of information of  $\mathbf{x}$  the server can recover.
While recall that in this case, the Rand-Proj-Spatial estimator is
$$
\widehat {\mathbf {x}} ^ {(\text {R a n d - P r o j - S p a t i a l})} = \bar {\beta} \left(\sum_ {i = 1} ^ {n} \boldsymbol {G} _ {i} ^ {T} \boldsymbol {G} _ {i}\right) ^ {\dagger} \sum_ {i = 1} ^ {n} \boldsymbol {G} _ {i} ^ {T} \boldsymbol {G} _ {i} \mathbf {x} = \bar {\beta} \boldsymbol {S} ^ {\dagger} \boldsymbol {S} \mathbf {x},
$$
where  $S$  can have rank at most  $nk$ .
B.3 Why deriving the MSE of Rand-Proj-Spatial with SRHT is hard
To analyze Eq. 11, one needs to compute the distribution of eigendecomposition of  $S = \sum_{i=1}^{n} G_i^T G_i$ , i.e., the sum of the covariance of SRHT. To the best of our knowledge, there is no non-trivial closed form expression of the distribution of eigen-decomposition of even a single  $G_i^T G_i$ , when  $G_i$  is SRHT, or other commonly used random matrices, e.g., Gaussian. When  $G_i$  is SRHT, since  $G_i^T G_i = D_i H E_i^T E_i H D_i$  and the eigenvalues of  $E_i^T E_i$  are just diagonal entries, one might attempt to analyze  $H D_i$ . While the hardmard matrix  $H$ 's eigenvalues and eigenvectors are known<sup>3</sup>, the result can hardly be applied to analyze the distribution of singular values or singular vectors of  $H D_i$ .
Even if one knows the eigen-decomposition of a single  $G_i^T G_i$ , it is still hard to get the eigen-decomposition of  $S$ . The eigenvalues of a matrix  $A$  can be viewed as a non-linear function in the  $A$ , and hence it is in general hard to derive closed form expressions for the eigenvalues of  $A + B$ , given the eigenvalues of  $A$  and that of  $B$ . One exception is when  $A$  and  $B$  have the same eigenvector and the eigenvalues of  $A + B$  become a sum of the eigenvalues of  $A$  and  $B$ . Recall when  $G_i = E_i$ , Rand-Proj-Spatial recovers Rand- $k$ -Spatial. Since  $E_i^T E_i$ 's all have the same eigenvectors (i.e. same as  $I_d$ ), the eigenvalues of  $S = \sum_{i=1}^{n} E_i^T E_i$  are just the sum of diagonal entries of  $E_i^T E_i$ 's. Hence, deriving the MSE for Rand- $k$ -Spatial is not hard compared to the more general case when  $G_i^T G_i$ 's can have different eigenvectors.
Since one can also view  $S = \sum_{i=1}^{nk} \mathbf{g}_i \mathbf{g}_i^T$ , i.e. the sum of  $nk$  rank-one matrices, one might attempt to recursively analyze the eigen-decomposition of  $\sum_{i=1}^{n'} \mathbf{g}_i \mathbf{g}_i^T + \mathbf{g}_{n' + 1} \mathbf{g}_{n' + 1}^T$  for  $n' \leq n$ . One related problem is eigen-decomposition of a low-rank updated matrix in perturbation analysis: Given the eigen-decomposition of a matrix  $A$ , what is the eigen-decomposition of  $A + V V^T$ , where  $V$  is low-rank matrix (or more commonly rank-one)? To compute the eigenvalues of  $A + V V^T$  directly from that of  $A$ , the most effective and widely applied solution is to solve the so-called secular equation, e.g. [59, 60, 61]. While this can be done computationally efficiently, it is hard to get a closed form expression for the eigenvalues of  $A + V V^T$  from the secular equation.
The previous analysis of SRHT in e.g. [37, 38, 39, 45, 55] is based on asymptotic properties of SRHT, such as the limiting eigen-spectrum, or concentration bounds that bounds the singular values. To analyze the MSE of Rand-Proj-Spatial, however, we need an exact, non-asymptotic analysis of the distribution of SRHT. Concentration bounds does not apply, since computing the pseudo-inverse in Eq. 5 naturally bounds the eigenvalues, and applying concentration bounds will only lead to a loose upper bound on MSE.
3See this note https://core.ac.uk/download/pdf/81967428.pdf
B.4 More simulation results on incorporating various degrees of correlation
Figure 6: MSE comparison of estimators Rand-  $k$ , Rand-  $k$ -Spatial(Opt), Rand-Proj-Spatial, given the degree of correlation  $\mathcal{R}$ . Rand-  $k$ -Spatial(Opt) denotes the estimator that gives the lowest possible MSE from the Rand-  $k$ -Spatial family. We consider  $d = 1024$ , a smaller number of clients  $n \in \{5, 11\}$ , and  $k$  values such that  $nk < d$ . In each plot, we fix  $n, k, d$  and vary the degree of positive correlation  $\mathcal{R}$ . Note the range of  $\mathcal{R}$  is  $\mathcal{R} \in [0, n - 1]$ . We choose  $\mathcal{R}$  with equal space in this range.
C All Proof Details
C.1 Proof of Theorem 4.3
$$
\mathbb {E} \left[ \| \widehat {\mathbf {x}} ^ {(R a n d - P r o j - S p a t i a l (M a x))} - \bar {\mathbf {x}} \| _ {2} ^ {2} \right] \leq \left[ \frac {d}{(1 - \delta) n k + \delta k} - 1 \right] \| \mathbf {x} \| _ {2} ^ {2} \tag {19}
$$
Proof. All clients have the same vector  $\mathbf{x}_1 = \mathbf{x}_2 = \dots = \mathbf{x}_n = \mathbf{x} \in \mathbb{R}^d$ . Hence,  $\bar{\mathbf{x}} = \frac{1}{n}\sum_{i=1}^{n}\mathbf{x}_i = \mathbf{x}$ , and the decoding scheme is
$$
\widehat {\mathbf {x}} ^ {(\text {R a n d - P r o j - S p a t i a l} (\operatorname {M a x}))} = \bar {\beta} \left(\sum_ {i = 1} ^ {n} \boldsymbol {G} _ {i} ^ {T} \boldsymbol {G} _ {i}\right) ^ {\dagger} \sum_ {i = 1} ^ {n} \boldsymbol {G} _ {i} ^ {T} \boldsymbol {G} _ {i} \mathbf {x} = \bar {\beta} \boldsymbol {S} ^ {\dagger} \boldsymbol {S} \mathbf {x},
$$
where  $\pmb{S} = \sum_{i=1}^{n} \pmb{G}_{i}^{T} \pmb{G}_{i}$ . Let  $\pmb{S} = \pmb{U} \Lambda \pmb{U}^{T}$  be its eigendecomposition. Since  $\pmb{S}$  is a real symmetric matrix,  $\pmb{U}$  is orthogonal, i.e.,  $\pmb{U}^{T} \pmb{U} = \pmb{I}_{d} = \pmb{U} \pmb{U}^{T}$ . Also,  $\pmb{S}^{\dagger} = \pmb{U} \Lambda^{\dagger} \pmb{U}^{T}$ , where  $\Lambda^{\dagger}$  is a diagonal matrix, such that
$$
[ \Lambda^ {\dagger} ] _ {i i} = \left\{ \begin{array}{l l} 1 / [ \Lambda ] _ {i i} & \text {i f} \Lambda_ {i i} \neq 0, \\ 0 & \text {e l s e .} \end{array} \right.
$$
Let  $\delta_c$  be the probability that  $S$  has rank  $c$ , for  $c \in \{k, k + 1, \ldots, nk - 1\}$ . Note that  $\delta = \sum_{c=k}^{nk-1} \delta_c$ . For vector  $\mathbf{m} \in \mathbb{R}^d$ , we use  $\mathrm{diag}(\mathbf{m}) \in \mathbb{R}^{d \times d}$  to denote the matrix whose diagonal entries correspond to the coordinates of  $\mathbf{m}$  and the rest of the entries are zeros.
Computing  $\bar{\beta}$ . First, we compute  $\bar{\beta}$ . To ensure that our estimator  $\widehat{\mathbf{x}}^{(\mathrm{Rand - Proj - Spatial}(\mathrm{Max}))}$  is unbiased, we need  $\bar{\beta}\mathbb{E}[S^{\dagger}S\mathbf{x}] = \mathbf{x}$ . Consequently,
$$
\mathbf {x} = \bar {\beta} \mathbb {E} [ \boldsymbol {U} \boldsymbol {\Lambda} ^ {\dagger} \boldsymbol {U} ^ {T} \boldsymbol {U} \boldsymbol {\Lambda} \boldsymbol {U} ^ {T} ] \mathbf {x}
$$
$$
\begin{array}{l} = \bar {\beta} \left[ \sum_ {\boldsymbol {U} = \Phi} \Pr [ \boldsymbol {U} = \Phi ] \mathbb {E} [ \boldsymbol {U} \Lambda^ {\dagger} \Lambda \boldsymbol {U} ^ {T} \mid \boldsymbol {U} = \Phi ] \right] \mathbf {x} \\ = \bar {\beta} \left[ \sum_ {\boldsymbol {U} = \Phi} \Pr [ \boldsymbol {U} = \Phi ] \boldsymbol {U} \mathbb {E} \left[ \Lambda^ {\dagger} \Lambda \mid \boldsymbol {U} = \Phi \right] \boldsymbol {U} ^ {T} \right] \mathbf {x} \\ \stackrel {(a)} {=} \bar {\beta} \left[ \sum_ {\boldsymbol {U} = \Phi} \Pr [ \boldsymbol {U} = \Phi ] \boldsymbol {U} \mathbb {E} [ \operatorname {d i a g} (\boldsymbol {m}) \mid \boldsymbol {U} = \Phi ] \boldsymbol {U} ^ {T} \right] \mathbf {x} \\ \stackrel {(b)} {=} \bar {\beta} \sum_ {\boldsymbol {U} = \Phi} \Pr [ \boldsymbol {U} = \Phi ] \left[ \boldsymbol {U} \left((1 - \delta) \frac {n k}{d} \boldsymbol {I} _ {d} + \sum_ {c = k} ^ {n k - 1} \delta_ {c} \frac {c}{d} \boldsymbol {I} _ {d}\right) \boldsymbol {U} ^ {T} \right] \mathbf {x} \\ = \bar {\beta} \left[ (1 - \delta) \frac {n k}{d} + \sum_ {c = k} ^ {n k - 1} \delta_ {c} \frac {c}{d} \right] \mathbf {x} \\ \Rightarrow \bar {\beta} = \frac {d}{(1 - \delta) n k + \sum_ {c = k} ^ {n k - 1} \delta_ {c} c} \tag {20} \\ \end{array}
$$
where in  $(a),\mathbf{m}\in \mathbb{R}^d$  such that
$$
\mathbf {m} _ {i} = \left\{ \begin{array}{l l} 1 & \text {i f} \Lambda_ {j j} > 0 \\ 0 & \text {e l s e .} \end{array} \right.
$$
Also, by construction of  $S$ ,  $\operatorname{rank}(\operatorname{diag}(\mathbf{m})) \leq nk$ . Further,  $(b)$  follows by symmetry across the  $d$  dimensions.
Since  $\delta k\leq \sum_{c = k}^{nk - 1}\delta_{c}c\leq \delta (nk - 1)$  , there is
$$
\frac {d}{(1 - \delta) n k + \delta (n k - 1)} \leq \bar {\beta} \leq \frac {d}{(1 - \delta) n k + \delta k} \tag {21}
$$
Computing the MSE. Next, we use the value of  $\bar{\beta}$  in Eq. 20 to compute MSE.
$$
\begin{array}{l} M S E (\text {R a n d - P r o j - S p a t i a l} (\text {M a x})) = \mathbb {E} [ \| \widehat {\mathbf {x}} ^ {(\text {R a n d - P r o j - S p a t i a l} (\text {M a x}))} - \bar {\mathbf {x}} \| _ {2} ^ {2} ] = \mathbb {E} [ \| \bar {\boldsymbol {\beta}} \boldsymbol {S} ^ {\dagger} \boldsymbol {S} \mathbf {x} - \mathbf {x} \| _ {2} ^ {2} ] \\ = \bar {\beta} ^ {2} \mathbb {E} [ \| \boldsymbol {S} ^ {\dagger} \boldsymbol {S} \mathbf {x} \| _ {2} ^ {2} ] + \| \mathbf {x} \| _ {2} ^ {2} - 2 \left\langle \bar {\beta} \mathbb {E} [ \boldsymbol {S} ^ {\dagger} \boldsymbol {S} \mathbf {x} ], \mathbf {x} \right\rangle \\ = \bar {\beta} ^ {2} \mathbb {E} [ \| \boldsymbol {S} ^ {\dagger} \boldsymbol {S} \mathbf {x} \| _ {2} ^ {2} ] - \| \mathbf {x} \| _ {2} ^ {2} \quad (\text {U s i n g u n b i a s e d n e s s o f} \hat {\mathbf {x}} ^ {(\text {R a n d - P r o j - S p a t i a l (M a x)})}) \\ = \bar {\beta} ^ {2} \mathbf {x} ^ {T} \mathbb {E} \left[ \boldsymbol {S} ^ {T} \left(\boldsymbol {S} ^ {\dagger}\right) ^ {T} \boldsymbol {S} ^ {\dagger} \boldsymbol {S} \right] \mathbf {x} - \| \mathbf {x} \| _ {2} ^ {2}. \tag {22} \\ \end{array}
$$
Using  $\pmb{S}^{\dagger} = \pmb{U}\Lambda^{\dagger}\pmb{U}^{T}$
$$
\begin{array}{l} \mathbb {E} \left[ \boldsymbol {S} ^ {T} (\boldsymbol {S} ^ {\dagger}) ^ {T} \boldsymbol {S} ^ {\dagger} \boldsymbol {S} \right] = \mathbb {E} \left[ \boldsymbol {U} \Lambda \boldsymbol {U} ^ {T} \boldsymbol {U} \Lambda^ {\dagger} \boldsymbol {U} ^ {T} \boldsymbol {U} \Lambda^ {\dagger} \boldsymbol {U} ^ {T} \boldsymbol {U} \Lambda \boldsymbol {U} ^ {T} \right] \\ = \mathbb {E} [ \boldsymbol {U} \Lambda (\Lambda^ {\dagger}) ^ {2} \Lambda \boldsymbol {U} ^ {T} ] \\ = \sum_ {\boldsymbol {U} = \Phi} \boldsymbol {U} \mathbb {E} \left[ \Lambda \left(\Lambda^ {\dagger}\right) ^ {2} \Lambda \right] \boldsymbol {U} ^ {T} \cdot \Pr [ \boldsymbol {U} = \Phi ] \\ = \sum_ {\boldsymbol {U} = \Phi} \boldsymbol {U} \left[ (1 - \delta) \frac {n k}{d} \boldsymbol {I} _ {d} + \sum_ {c = k} ^ {n k - 1} \delta_ {c} \frac {c}{d} \boldsymbol {I} _ {d} \right] \boldsymbol {U} ^ {T} \cdot \Pr [ \boldsymbol {U} = \Phi ] \\ = \left[ (1 - \delta) \frac {n k}{d} + \sum_ {c = k} ^ {n k - 1} \delta_ {c} \frac {c}{d} \right] \cdot \sum_ {\boldsymbol {U} = \Phi} \boldsymbol {U} \boldsymbol {U} ^ {T} \cdot \Pr [ \boldsymbol {U} = \Phi ] \\ = \left[ (1 - \delta) \frac {n k}{d} + \sum_ {c = k} ^ {n k - 1} \delta_ {c} \frac {c}{d} \right] \boldsymbol {I} _ {d} \\ = \frac {1}{\bar {\beta}} \boldsymbol {I} _ {d} \tag {23} \\ \end{array}
$$
Substituting Eq. 23 in Eq. 22, we get
$$
\begin{array}{l} M S E (\text {R a n d - P r o j - S p a t i a l} (\text {M a x})) = \bar {\beta} ^ {2} \mathbf {x} ^ {T} \frac {1}{\bar {\beta}} \boldsymbol {I} _ {d} \mathbf {x} - \| \mathbf {x} \| _ {2} ^ {2} = (\bar {\beta} - 1) \| \mathbf {x} \| _ {2} ^ {2} \\ \leq \left[ \frac {d}{(1 - \delta) n k + \delta k} - 1 \right] \| \mathbf {x} \| _ {2} ^ {2}, \\ \end{array}
$$
where the inequality is by Eq 21.
C.2 Comparing against Rand-  $k$
Next, we compare the MSE of Rand-Proj-Spatial(Max) with the MSE of the baseline Rand- $k$  analytically in the full-correlation case. Recall that in this case,
$$
M S E (\mathrm {R a n d} - k) = \frac {1}{n} \left(\frac {d}{k} - 1\right) \| \mathbf {x} \| _ {2} ^ {2}.
$$
$$
\begin{array}{l} M S E (\text {R a n d - P r o j - S p a t i a l} (\text {M a x})) \leq M S E (\text {R a n d - k}) \\ \Leftrightarrow \frac {d}{(1 - \delta) n k + \delta k} - 1 \leq \frac {1}{n} (\frac {d}{k} - 1) \\ \Leftrightarrow \frac {d}{k} \frac {n - (1 - \delta) n - \delta}{n ((1 - \delta) n + \delta)} \leq 1 - \frac {1}{n} \\ \Leftrightarrow \frac {d}{k} \cdot \frac {\delta - \delta / n}{(1 - \delta) n + \delta} \leq \frac {n - 1}{n} \\ \Leftrightarrow d \delta (1 - \frac {1}{n}) n \leq k (n - 1) \cdot ((1 - \delta) n + \delta) \\ \Leftrightarrow d \delta \leq k \cdot ((1 - \delta) n + \delta) \\ \Leftrightarrow d \delta + k n \delta - k \delta \leq k n \\ \Leftrightarrow \delta \leq \frac {k n}{d + k n - k} \\ \Leftrightarrow \delta \leq \frac {1}{\frac {d}{k n} + 1 - \frac {1}{n}} \\ \end{array}
$$
Since  $nk \leq d$ , for  $n \geq 2$ , the above implies when
$$
\delta \leq \frac {1}{1 + \frac {1}{2}} = \frac {2}{3},
$$
the MSE of Rand-Proj-Spatial(Max) is always less than that of Rand-  $k$
C.3  $S$  has full rank with high probability
We empirically verify that  $\delta \approx 0$ . With  $d \in \{32,64,128,\dots,1024\}$  and 4 different  $nk$  value such that  $nk \leq d$  for each  $d$ , we compute  $\mathrm{rank}(S)$  for  $10^{5}$  trials for each pair of  $(nk,d)$  values, and plot the results for all trials. All results are presented in Figure 7. As one can observe from the plots,  $\mathrm{rank}(S) = nk$  with high probability, suggesting  $\delta \approx 0$ .
This implies the MSE of Rand-Proj-Spatial(Max) is
$$
M S E (\mathrm {R a n d - P r o j - S p a t i a l} (\mathrm {M a x})) \approx (\frac {d}{n k} - 1) \| \mathbf {x} \| _ {2} ^ {2},
$$
in the full correlation case.
C.4 Proof of Theorem 4.4
Theorem 4.4 (MSE under No Correlation). Consider  $n$  clients, each holding a vector  $\mathbf{x}_i \in \mathbb{R}^d$ ,  $\forall i \in [n]$ . Suppose we set  $T \equiv 1$ ,  $\bar{\beta} = \frac{d^2}{k}$  in Eq. 5, and the random linear map  $G_i$  at each client to be
Figure 7: Simulation results of  $\mathrm{rank}(S)$ , where  $S = \sum_{i=1}^{n} G_i^T G_i$ , with  $G_i$  being SRHT. With  $d \in \{32,64,128,\ldots,1024\}$  and 4 different  $nk$  values such that  $nk \leq d$  for each  $d$ , we compute  $\mathrm{rank}(S)$  for  $10^5$  trials for each pairs of  $(nk,d)$  values and plot the results for all trials. When  $d = 32$  and  $nk = 32$  in the first plot,  $\mathrm{rank}(S) = 31$  in 2100 trials, and  $\mathrm{rank}(S) = nk = 32$  in all the rest of the trials. For all other  $(nk,d)$  pairs,  $S$  always has rank  $nk$  in the  $10^5$  trials. This verifies that  $\delta = \operatorname*{Pr}[\mathrm{rank}(S) < nk] \approx 0$ .
an SRHT matrix. Then, for  $nk \leq d$ ,
$$
\mathbb {E} \left[ \| \widehat {\mathbf {x}} ^ {(R a n d - P r o j - S p a t i a l)} - \bar {\mathbf {x}} \| _ {2} ^ {2} \right] = \frac {1}{n ^ {2}} \left(\frac {d}{k} - 1\right) \sum_ {i = 1} ^ {n} \| \mathbf {x} _ {i} \| _ {2} ^ {2}.
$$
Proof. When the client vectors are all orthogonal to each other, we define the transformation function on the eigenvalue to be  $T(\lambda) = 1, \forall \lambda \geq 0$ . We show that by considering the above constant  $T$ , SRHT becomes the same as rand  $k$ . Recall  $S = \sum_{i=1}^{n} G_i^T G_i$  and let  $\bar{G}^T G = U \Lambda U^T$  be its eigendecomposition. Then,
$$
T (\boldsymbol {S}) = \boldsymbol {U} T (\Lambda) \boldsymbol {U} ^ {T} = \boldsymbol {U} \boldsymbol {I} _ {d} \boldsymbol {U} ^ {T} = \boldsymbol {I} _ {d}.
$$
Hence,  $(T(S))^{\dagger} = I_{d}$ . And the decoded vector for client  $i$  becomes
$$
\begin{array}{l} \widehat {\mathbf {x}} _ {i} = \bar {\beta} \left(T (\boldsymbol {G} ^ {T} \boldsymbol {G})\right) ^ {\dagger} \boldsymbol {G} _ {i} ^ {T} \boldsymbol {G} _ {i} \mathbf {x} _ {i} = \bar {\beta} \boldsymbol {G} _ {i} ^ {T} \boldsymbol {G} _ {i} \mathbf {x} _ {i} = \bar {\beta} \frac {1}{d} \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {H} \boldsymbol {D} _ {i} \mathbf {x} _ {i}, \\ \widehat {\mathbf {x}} = \frac {1}{n} \sum_ {i = 1} ^ {n} \widehat {\mathbf {x}} _ {i} = \frac {1}{n} \bar {\beta} \sum_ {i = 1} ^ {n} \frac {1}{d} \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {H} \boldsymbol {D} _ {i} \mathbf {x} _ {i} \tag {24} \\ \end{array}
$$
$\pmb{D}_i$  is a diagonal matrix. Also,  $\pmb{E}_i^T\pmb{E}_i \in \mathbb{R}^{d \times d}$  is a diagonal matrix, where the  $i$ -th entry is 0 or 1.
Computing  $\bar{\beta}$ . To ensure that  $\widehat{\mathbf{x}}$  is an unbiased estimator, from Eq. 24
$$
\begin{array}{l} \mathbf {x} _ {i} = \bar {\beta} \mathbb {E} \left[ \boldsymbol {G} _ {i} ^ {T} \boldsymbol {G} _ {i} \right] \mathbf {x} _ {i} \\ = \frac {\bar {\beta}}{d} \mathbb {E} \left[ \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {H} \boldsymbol {D} _ {i} \right] \mathbf {x} _ {i} \\ = \frac {\beta}{d} \mathbb {E} _ {\boldsymbol {D} _ {i}} \left[ \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \underbrace {\mathbb {E} \left[ \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \right]} _ {= (k / d) \boldsymbol {I} _ {d}} \boldsymbol {H} \boldsymbol {D} _ {i} \right] \mathbf {x} _ {i} \quad (\because \boldsymbol {E} _ {i} \text {i s i n d e p e n d e n t o f} \boldsymbol {D} _ {i}) \\ = \frac {\bar {\beta}}{d} k \mathbb {E} _ {\boldsymbol {D} _ {i}} \left[ \boldsymbol {D} _ {i} ^ {2} \right] \mathbf {x} _ {i} \quad (\because \boldsymbol {H} ^ {T} \boldsymbol {H} = d \boldsymbol {I} _ {d}) \\ = \frac {\bar {\beta} k}{d} \mathbf {x} _ {i} \quad (\because D _ {i} ^ {2} = I \text {i s n o w d e t e r m i n i s t i c .}) \\ \Rightarrow \bar {\beta} = \frac {d}{k}. \tag {25} \\ \end{array}
$$
Computing the MSE.
$$
\begin{array}{l} M S E = \mathbb {E} \left\| \hat {\mathbf {x}} - \bar {\mathbf {x}} \right\| _ {2} ^ {2} \\ = \mathbb {E} \left\| \frac {1}{n} \bar {\beta} \sum_ {i = 1} ^ {n} \frac {1}{d} \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {H} \boldsymbol {D} _ {i} \mathbf {x} _ {i} - \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} \right\| _ {2} ^ {2} \\ = \frac {1}{n ^ {2}} \left\{\mathbb {E} \left\| \bar {\beta} \sum_ {i = 1} ^ {n} \frac {1}{d} \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {H} \boldsymbol {D} _ {i} \mathbf {x} _ {i} \right\| _ {2} ^ {2} + \left\| \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} \right\| _ {2} ^ {2} \right. \\ \left. - 2 \left\langle \bar {\beta} \mathbb {E} \left[ \sum_ {i = 1} ^ {n} \frac {1}{d} \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {H} \boldsymbol {D} _ {i} \mathbf {x} _ {i} \right], \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} \right\rangle \right\} \\ = \frac {1}{n ^ {2}} \left\{\bar {\beta} ^ {2} \mathbb {E} \left\| \sum_ {i = 1} ^ {n} \frac {1}{d} \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {H} \boldsymbol {D} _ {i} \mathbf {x} _ {i} \right\| _ {2} ^ {2} - \left\| \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} \right\| _ {2} ^ {2} \right\} \quad (\because \mathbb {E} [ \widehat {\mathbf {x}} ] = \overline {{\mathbf {x}}}) \\ = \frac {1}{n ^ {2}} \left\{\sum_ {i = 1} ^ {n} \frac {\bar {\beta} ^ {2}}{d ^ {2}} \mathbb {E} \left\| \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {H} \boldsymbol {D} _ {i} \mathbf {x} _ {i} \right\| _ {2} ^ {2} - \sum_ {i = 1} ^ {n} \left\| \mathbf {x} _ {i} \right\| _ {2} ^ {2} \right. \tag {26} \\ + 2 \sum_ {i = 1} ^ {n} \sum_ {l = i + 1} ^ {n} \frac {\bar {\beta} ^ {2}}{d ^ {2}} \left\langle \mathbb {E} [ \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {H} \boldsymbol {D} _ {i} \mathbf {x} _ {i} ], \mathbb {E} [ \boldsymbol {D} _ {l} \boldsymbol {H} ^ {T} \boldsymbol {E} _ {l} ^ {T} \boldsymbol {E} _ {l} \boldsymbol {H} \boldsymbol {D} _ {l} \mathbf {x} _ {l} ] \right\rangle - 2 \sum_ {i = 1} ^ {n} \sum_ {l = i + 1} ^ {n} \left\langle \mathbf {x} _ {i}, \mathbf {x} _ {l} \right\rangle \Bigg \}. \\ \end{array}
$$
Note that in Eq. 26
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {H} \boldsymbol {D} _ {i} \mathbf {x} _ {i} \right\| _ {2} ^ {2} = \mathbb {E} [ \mathbf {x} _ {i} ^ {T} \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {H} \boldsymbol {D} _ {i} \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {H} \boldsymbol {D} _ {i} \mathbf {x} _ {i} ] \\ = d \mathbb {E} \left[ \mathbf {x} _ {i} ^ {T} \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \left(\boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i}\right) ^ {2} \boldsymbol {H} \boldsymbol {D} _ {i} \mathbf {x} _ {i} \right] \quad (\because \boldsymbol {D} _ {i} ^ {2} = \boldsymbol {I} _ {d}; \boldsymbol {H} ^ {T} \boldsymbol {H} = \boldsymbol {H} \boldsymbol {H} ^ {T} = d \boldsymbol {I} _ {d}) \\ = d \mathbf {x} _ {i} ^ {T} \mathbb {E} _ {\boldsymbol {D} _ {i}} \left[ \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \mathbb {E} \left[ \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \right] \boldsymbol {H D} _ {i} \right] \mathbf {x} _ {i} \quad (\boldsymbol {E} _ {i}, \boldsymbol {D} _ {i} \text {a r e i n d e p e n d e n t}; (\boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i}) ^ {2} = \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i}) \\ = k d \| \mathbf {x} _ {i} \| _ {2} ^ {2}, \tag {27} \\ \end{array}
$$
since  $\mathbb{E}[E_i^T E_i] = (k / d)\pmb {I}_d$ $H^{T}H = dI_{d}$  and for  $i\neq l$
$$
\left\langle \mathbb {E} [ \boldsymbol {D} _ {i} \boldsymbol {H} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {H} \boldsymbol {D} _ {i} \mathbf {x} _ {i} ], \mathbb {E} [ \boldsymbol {D} _ {l} \boldsymbol {H} ^ {T} \boldsymbol {E} _ {l} ^ {T} \boldsymbol {E} _ {l} \boldsymbol {H} \boldsymbol {D} _ {l} \mathbf {x} _ {l} ] \right\rangle = \left\langle k \mathbf {x} _ {i}, k \mathbf {x} _ {l} \right\rangle = k ^ {2} \left\langle \mathbf {x} _ {i}, \mathbf {x} _ {l} \right\rangle . \tag {28}
$$
Substituting Eq. 27, 28 in Eq. 26, we get
$$
\begin{array}{l} M S E = \frac {1}{n ^ {2}} \left\{\left(\frac {\bar {\beta} ^ {2}}{d ^ {2}} \sum_ {i = 1} ^ {n} k d \| \mathbf {x} _ {i} \| _ {2} ^ {2} + 2 \sum_ {i = 1} ^ {n} \sum_ {l = i + 1} ^ {n} \frac {\bar {\beta} ^ {2} k ^ {2}}{d ^ {2}} \left\langle \mathbf {x} _ {i}, \mathbf {x} _ {l} \right\rangle\right) - \sum_ {i = 1} ^ {n} \left\| \mathbf {x} _ {i} \right\| _ {2} ^ {2} - 2 \sum_ {i = 1} ^ {n} \sum_ {l = i + 1} ^ {n} \left\langle \mathbf {x} _ {i}, \mathbf {x} _ {l} \right\rangle \right\} \\ = \frac {1}{n ^ {2}} \left(\frac {d}{k} - 1\right) \sum_ {i = 1} ^ {n} \| \mathbf {x} _ {i} \| _ {2} ^ {2}, \\ \end{array}
$$
which is exactly the same as the MSE of rand  $k$ .
C.5 Rand-Proj-Spatial recovers Rand- $k$ -Spatial (Proof of Lemma 4.1)
Lemma 4.1 (Recovering Rand- $k$ -Spatial). Suppose client  $i$  generates a subsampling matrix  $\mathbf{E}_i = [\mathbf{e}_{i_1}, \ldots, \mathbf{e}_{i_k}]^\top$ , where  $\{\mathbf{e}_j\}_{j=1}^d$  are the canonical basis vectors, and  $\{i_1, \ldots, i_k\}$  are sampled from  $\{1, \ldots, d\}$  without replacement. The encoded vectors are given as  $\widehat{\mathbf{x}}_i = \mathbf{E}_i \mathbf{x}_i$ . Given a function  $T$ ,  $\widehat{\mathbf{x}}$  computed as in Eq. 5 recovers the Rand- $k$ -Spatial estimator.
Proof. If client  $i$  applies  $\mathbf{E}_i\in \mathbb{R}^{k\times d}$  as the random matrix to encode  $\mathbf{x}_i$  in Rand-Proj-Spatial, by Eq. 5, client  $i$ 's encoded vector is now