SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Consider  $n$  clients. Suppose client  $i$  holds a vector  $\mathbf{x}_i \in \mathbb{R}^d$ . We want to apply Rand-  $k$  or Rand-  $k$ -Spatial, while also making the encoding process more flexible than just randomly choosing  $k$  out of  $d$  coordinates. One naive way of doing this is for each client to pre-process its vector by applying an orthogonal matrix  $G \in \mathbb{R}^{d \times d}$  that is the same across all clients. Such a technique might be helpful in improving the performance of quantization because the MSE due to quantization often depends on how uniform the coordinates of  $\mathbf{x}_i$ 's are, i.e. whether the coordinates of  $\mathbf{x}_i$  have values close to each other.  $G$  is designed to be the random matrix (e.g. SRHT) that rotates  $\mathbf{x}_i$  and makes its coordinates uniform.
Each client sends the server  $\widehat{\mathbf{x}}_i = \mathbf{E}_i\mathbf{G}\mathbf{x}_i$ , where  $\mathbf{E}_i \in \mathbb{R}^{k \times d}$  is the subsampling matrix. If we use Rand- $k$ , the server can decode each client vector by first applying the decoding procedure of Rand- $k$  and then rotating it back to the original space, i.e.,  $\widehat{\mathbf{x}}_i^{(\mathrm{Naive})} = \frac{d}{k}\mathbf{G}^T\mathbf{E}_i^T\mathbf{E}_i\mathbf{G}\mathbf{x}_i$ . Note that
$$
\begin{array}{l} \mathbb {E} [ \widehat {\mathbf {x}} _ {i} ^ {(\mathrm {N a i v e})} ] = \frac {d}{k} \mathbb {E} [ \boldsymbol {G} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {G} \mathbf {x} _ {i} ] \\ = \frac {d}{k} \boldsymbol {G} ^ {T} \frac {k}{d} \boldsymbol {I} _ {d} \boldsymbol {G} \mathbf {x} _ {i} \\ = \mathbf {x} _ {i}. \\ \end{array}
$$
Hence,  $\widehat{\mathbf{x}}_i^{(\mathrm{Naive})}$  is unbiased. The MSE of  $\widehat{\mathbf{x}}^{(\mathrm{Naive})} = \frac{1}{n}\sum_{i=1}^{n}\widehat{\mathbf{x}}_i^{(\mathrm{Naive})}$  is given as
$$
\begin{array}{l} \mathbb {E} \left. \left\| \bar {\mathbf {x}} - \widehat {\mathbf {x}} ^ {(\mathrm {N a i v e})} \right\| _ {2} ^ {2} = \mathbb {E} \left\lVert \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} - \frac {1}{n} \frac {d}{k} \sum_ {i = 1} ^ {n} \pmb {G} ^ {T} \pmb {E} _ {i} ^ {T} \pmb {E} _ {i} \pmb {G} \mathbf {x} _ {i} \right\rvert \right| _ {2} ^ {2} \\ = \frac {1}{n ^ {2}} \mathbb {E} \left\| \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} - \frac {d}{k} \sum_ {i = 1} ^ {n} \boldsymbol {G} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {G} \mathbf {x} _ {i} \right\| _ {2} ^ {2} \\ = \frac {1}{n ^ {2}} \left\{\frac {d ^ {2}}{k ^ {2}} \mathbb {E} \left\| \sum_ {i = 1} ^ {n} \boldsymbol {G} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {G} \mathbf {x} _ {i} \right\| ^ {2} - \left\| \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} \right\| ^ {2} \right\} \\ = \frac {1}{n ^ {2}} \left\{\frac {d ^ {2}}{k ^ {2}} \left(\sum_ {i = 1} ^ {n} \mathbb {E} \| \boldsymbol {G} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {G} \mathbf {x} _ {i} \| _ {2} ^ {2} + \sum_ {i \neq j} \mathbb {E} \left\langle \boldsymbol {G} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {G} \mathbf {x} _ {i}, \boldsymbol {G} \boldsymbol {E} _ {l} ^ {T} \boldsymbol {E} _ {l} \boldsymbol {G} \mathbf {x} _ {l} \right\rangle\right) - \left\| \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} \right\| ^ {2} \right\}. \tag {12} \\ \end{array}
$$
Next, we bound the first term in Eq. 12.
$$
\begin{array}{l} \mathbb {E} \left\| \boldsymbol {G} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {G} \mathbf {x} _ {i} \right\| _ {2} ^ {2} = \mathbb {E} \left[ \mathbf {x} _ {i} ^ {T} \boldsymbol {G} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {G} \boldsymbol {G} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {G} \mathbf {x} _ {i} \right] = \mathbb {E} \left[ \mathbf {x} _ {i} ^ {T} \boldsymbol {G} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {G} \mathbf {x} _ {i} \right] \\ = \mathbf {x} _ {i} ^ {T} \boldsymbol {G} ^ {T} \mathbb {E} [ (\boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i}) ^ {2} ] \boldsymbol {G} \mathbf {x} _ {i} \\ = \mathbf {x} _ {i} ^ {T} \frac {k}{d} \mathbf {I} _ {d} \mathbf {x} _ {i} (\because (\mathbf {E} _ {i} ^ {T} \mathbf {E} _ {i}) ^ {2} = \mathbf {E} _ {i} ^ {T} \mathbf {E} _ {i}) \\ = \frac {k}{d} \| \mathbf {x} _ {i} \| _ {2} ^ {2} \tag {13} \\ \end{array}
$$
The second term in Eq. 12 can also be simplified as follows.
$$
\begin{array}{l} \mathbb {E} \left[ \langle \boldsymbol {G} ^ {T} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \boldsymbol {G} \mathbf {x} _ {i}, \boldsymbol {G} ^ {T} \boldsymbol {E} _ {l} ^ {T} \boldsymbol {E} _ {l} \boldsymbol {G} \mathbf {x} _ {l} \rangle \right] \\ = \left\langle \boldsymbol {G} ^ {T} \mathbb {E} \left[ \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \right] \boldsymbol {G} \mathbf {x} _ {i}, \boldsymbol {G} ^ {T} \mathbb {E} \left[ \boldsymbol {E} _ {l} ^ {T} \boldsymbol {E} _ {l} \right] \boldsymbol {G} \mathbf {x} _ {l} \right\rangle \\ = \left\langle \boldsymbol {G} ^ {T} \frac {k}{d} \boldsymbol {I} _ {d} \boldsymbol {G} \mathbf {x} _ {i}, \boldsymbol {G} ^ {T} \frac {k}{d} \boldsymbol {I} _ {d} \boldsymbol {G} \mathbf {x} _ {l} \right\rangle \\ \end{array}
$$
$$
= \frac {k ^ {2}}{d ^ {2}} \left\langle \mathbf {x} _ {i}, \mathbf {x} _ {l} \right\rangle . \tag {14}
$$
Plugging Eq. 13 and Eq. 14 into Eq. 12, we get the MSE is
$$
\begin{array}{l} \mathbb {E} \| \bar {\mathbf {x}} - \hat {\bar {\mathbf {x}}} ^ {(\text {N a i v e})} \| _ {2} ^ {2} \\ = \frac {1}{n ^ {2}} \left\{\frac {d ^ {2}}{k ^ {2}} \left(\sum_ {i = 1} ^ {n} \frac {k}{d} \| \mathbf {x} _ {i} \| _ {2} ^ {2} + 2 \sum_ {i = 1} ^ {n} \sum_ {l = i + 1} ^ {n} \frac {k ^ {2}}{d ^ {2}} \langle \mathbf {x} _ {i}, \mathbf {x} _ {l} \rangle\right) - \left\| \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} \right\| ^ {2} \right\} \\ = \frac {1}{n ^ {2}} \big (\frac {d}{k} - 1 \big) \sum_ {i = 1} ^ {n} \| \mathbf {x} _ {i} \| _ {2} ^ {2}, \\ \end{array}
$$
which has exactly the same MSE as that of Rand-  $k$ . The problem is that if each client applies the same rotational matrix  $G$ , simply rotating the vectors will not change the  $\ell_2$  norm of the decoded vector, and hence the MSE. Similarly, if one applies Rand-  $k$ -Spatial, one ends up having exactly the same MSE as that of Rand-  $k$ -Spatial as well. Hence, we need to design a new decoding procedure when the encoding procedure at the clients are more flexible.
A.2  $nk \gg d$  is not interesting
One can rewrite  $\sum_{i=1}^{n} \mathbf{G}_i^T \mathbf{G}_i$  in the Rand-Proj-Spatial estimator (Eq. 5) as  $\sum_{i=1}^{n} \mathbf{G}_i^T \mathbf{G}_i = \sum_{j=1}^{nk} \mathbf{g}_i \mathbf{g}_i^T$ , where  $\mathbf{g}_j \in \mathbb{R}^d$  and  $\mathbf{g}_{ik}, \mathbf{g}_{ik+1}, \ldots, \mathbf{g}_{(i+1)k}$  are the rows of  $\mathbf{G}_i$ . Since when  $nk \gg d$ ,  $\sum_{j=1}^{nk} \mathbf{g}_j \mathbf{g}_j^T \to \mathbb{E}[\sum_{j=1}^{n} \mathbf{g}_j \mathbf{g}_j^T]$  due to Law of Large Numbers, one way to see the limiting MSE of Rand-Proj-Spatial when  $nk$  is large is to approximate  $\sum_{i=1}^{n} \sum_{j=1}^{nk} \mathbf{g}_i \mathbf{g}_i^T$  by its expectation.
By Lemma 4.1, when  $G_{i} = E_{i}$ , Rand-Proj-Spatial recovers Rand- $k$ -Spatial. We now discuss the limiting behavior of Rand- $k$ -Spatial when  $nk \gg d$  by leveraging our proposed Rand-Proj-Spatial. In this case, each  $\mathbf{g}_j$  can be viewed as a random based vector  $\mathbf{e}_w$  for  $w$  randomly chosen in  $[d]$ .  $\sum_{i=1}^{nk} \mathbf{g}_j \mathbf{g}_j^T \rightarrow \mathbb{E}[\sum_{i=1}^{nk} \mathbf{g}_j \mathbf{g}_j^T] = \sum_{i=1}^{nk} \frac{1}{d} \mathbf{I}_d = \frac{nk}{d} \mathbf{I}_d$ . And so the scalar  $\bar{\beta}$  in Eq. 5 to ensure an unbiased estimator is computed as
$$
\begin{array}{l} \bar {\beta} \mathbb {E} [ (\frac {n k}{d} \boldsymbol {I} _ {d}) ^ {\dagger} \boldsymbol {G} _ {i} ^ {T} \boldsymbol {G} _ {i} ] = \boldsymbol {I} _ {d} \\ \bar {\beta} \frac {d}{n k} \pmb {I} _ {d} \mathbb {E} [ \pmb {G} _ {i} ^ {T} \pmb {G} _ {i} ] = \pmb {I} _ {d} \\ \bar {\beta} \frac {d}{n k} \frac {k}{d} = \boldsymbol {I} _ {d} \\ \bar {\beta} = n \\ \end{array}
$$
And the MSE is now
$$
\begin{array}{l} \mathbb {E} \left[ \| \bar {\mathbf {x}} - \hat {\mathbf {x}} \| \right] = \mathbb {E} \left[ \| \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} - \frac {1}{n} \bar {\beta} \frac {d}{n k} \boldsymbol {I} _ {d} \sum_ {i = 1} ^ {n} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \mathbf {x} _ {i} \| _ {2} ^ {2} \right] \\ = \frac {1}{n ^ {2}} \left\{\bar {\beta} ^ {2} \frac {d ^ {2}}{n ^ {2} k ^ {2}} \mathbb {E} \left[ \| \sum_ {i = 1} ^ {n} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \mathbf {x} _ {i} \| _ {2} ^ {2} \right] - \| \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} \| _ {2} ^ {2} \right\} \\ = \frac {1}{n ^ {2}} \left\{n ^ {2} \frac {d ^ {2}}{n ^ {2} k ^ {2}} \left(\sum_ {i = 1} ^ {n} \mathbb {E} \left[ \| \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \mathbf {x} _ {i} \| _ {2} ^ {2} \right] + 2 \sum_ {i = 1} ^ {n} \sum_ {l = i + 1} ^ {n} \left\langle \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \mathbf {x} _ {i}, \boldsymbol {E} _ {l} ^ {T} \boldsymbol {E} _ {l} \mathbf {x} _ {l} \right\rangle \right]\right) - \| \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} \| _ {2} ^ {2} \Bigg \} \\ = \frac {1}{n ^ {2}} \Bigl \{\frac {d ^ {2}}{k ^ {2}} \Bigl (\sum_ {i = 1} ^ {n} \mathbb {E} \Bigl [ \mathbf {x} _ {i} ^ {T} (\boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i}) ^ {2} \mathbf {x} _ {i} \Bigr ] + 2 \sum_ {i = 1} ^ {n} \sum_ {l = i + 1} ^ {n} \frac {k ^ {2}}{d ^ {2}} \langle \mathbf {x} _ {i}, \mathbf {x} _ {l} \rangle \Bigr) - \| \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} \| _ {2} ^ {2} \Bigr \} \\ = \frac {1}{n ^ {2}} \left\{\frac {d ^ {2}}{k ^ {2}} \left(\sum_ {i = 1} ^ {n} \frac {k}{d} \| \mathbf {x} _ {i} \| _ {2} ^ {2} + 2 \sum_ {i = 1} ^ {n} \sum_ {l = i + 1} ^ {n} \frac {k ^ {2}}{d ^ {2}} \langle \mathbf {x} _ {i}, \mathbf {x} _ {l} \rangle\right) - \sum_ {i = 1} ^ {n} \| \mathbf {x} _ {i} \| _ {2} ^ {2} - 2 \sum_ {i = 1} ^ {n} \sum_ {l = i + 1} ^ {n} \langle \mathbf {x} _ {i}, \mathbf {x} _ {l} \rangle \right\} \\ = \frac {1}{n ^ {2}} \left(\frac {d}{k} - 1\right) \sum_ {i = 1} ^ {n} \| \mathbf {x} _ {i} \| _ {2} ^ {2} \\ \end{array}
$$
which is exactly the same MSE as Rand-  $k$ . This implies when  $nk$  is large, the MSE of Rand-  $k$ -Spatial does not get improved compared to Rand-  $k$  with correlation information. Intuitively, this implies when  $nk \gg d$ , the server gets enough amount of information from the client, and does not need correlation to improve its estimator. Hence, we focus on the more interesting case when  $nk < d$  — that is, when the server does not have enough information from the clients, and thus wants to use additional information, i.e. cross-client correlation, to improve its estimator.
B Additional Details on the Rand-Proj-Spatial Family Estimator
B.1  $\bar{\beta}$  is a scalar
From Eq. 20 in the proof of Theorem 4.3 and Eq. 25 in the proof of Theorem 4.4, it is evident that the unbiasedness of the mean estimator  $\widehat{\mathbf{x}}^{\mathrm{Rand - Proj - Spatio}}$  is ensured collectively by
- The random sampling matrices  $\{\pmb{E}_i\}$ .
- The orthogonality of scaled Hadamard matrices  $\pmb{H}^T\pmb{H} = d\pmb{I}_d = \pmb{H}\pmb{H}^T$
- The rademacher diagonal matrices, with the property  $(D_{i})^{2} = I_{d}$ .
B.2 Alternative motivating regression problems
Alternative motivating regression problem 1.
Let  $G_{i} \in \mathbb{R}^{k \times d}$  and  $W_{i} \in \mathbb{R}^{d \times k}$  be the encoding and decoding matrix for client  $i$ . One possible alternative estimator that translates the intuition that the decoded vector should be close to the client's original vector, for all clients, is by solving the following regression problem,
$$
\hat {\mathbf {x}} = \underset {\boldsymbol {W}} {\arg \min } f (\boldsymbol {W}) = \mathbb {E} [ \| \bar {\mathbf {x}} - \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {W} _ {i} \boldsymbol {G} _ {i} \mathbf {x} _ {i} \| _ {2} ^ {2} ]
$$
$$
\text {s u b j e c t} \bar {\mathbf {x}} = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \boldsymbol {W} _ {i} \boldsymbol {G} _ {i} \mathbf {x} _ {i} \right] \tag {15}
$$
where  $\mathbf{W} = (\mathbf{W}_1, \mathbf{W}_2, \dots, \mathbf{W}_n)$  and the constraint enforces unbiasedness of the estimator. The estimator is then the solution of the above problem. However, we note that optimizing a decoding matrix  $\mathbf{W}_i$  for each client leads to performing individual decoding of each client's compressed vector instead of a joint decoding process that considers all clients' compressed vectors. Only a joint decoding process can achieve the goal of leveraging cross-client information to reduce the estimation error. Indeed, we show as follows that solving the above optimization problem in Eq. 15 recovers the MSE of our baseline Rand- $k$ . Note
$$
\begin{array}{l} f (\boldsymbol {W}) = \mathbb {E} [ \| \frac {1}{n} \sum_ {i = 1} ^ {n} (\mathbf {x} _ {i} - \boldsymbol {W} _ {i} \boldsymbol {G} _ {i} \mathbf {x} _ {i}) \| _ {2} ^ {2} ] = \mathbb {E} [ \| \frac {1}{n} \sum_ {i = 1} ^ {n} (\boldsymbol {I} _ {d} - \boldsymbol {W} _ {i} \boldsymbol {G} _ {i}) \mathbf {x} _ {i} \| _ {2} ^ {2} ] \\ = \mathbb {E} \Big [ \frac {1}{n ^ {2}} \Big (\sum_ {i = 1} ^ {n} \| (\pmb {I} _ {d} - \pmb {W} _ {i} \pmb {G} _ {i} \mathbf {x} _ {i}) \| _ {2} ^ {2} + \sum_ {i \neq j} \Big \langle (\pmb {I} _ {d} - \pmb {W} _ {i} \pmb {G} _ {i}) \mathbf {x} _ {i}, (\pmb {I} _ {d} - \pmb {W} _ {j} \pmb {G} _ {j}) \mathbf {x} _ {j} \Big \rangle \Big) \Big ] \\ = \frac {1}{n ^ {2}} \left(\sum_ {i = 1} ^ {n} \mathbb {E} \left[ \| \left(\boldsymbol {I} _ {d} - \boldsymbol {W} _ {i} \boldsymbol {G} _ {i}\right) \mathbf {x} _ {i} \| _ {2} ^ {2} \right] + \sum_ {i \neq j} \mathbb {E} \left[ \left\langle \left(\boldsymbol {I} _ {d} - \boldsymbol {W} _ {i} \boldsymbol {G} _ {i}\right) \mathbf {x} _ {i}, \left(\boldsymbol {I} _ {d} - \boldsymbol {W} _ {j} \boldsymbol {G} _ {j}\right) \mathbf {x} _ {j} \right\rangle \right]\right). \tag {16} \\ \end{array}
$$
By the constraint of unbiasedness, i.e.,  $\bar{\mathbf{x}} = \frac{1}{n}\sum_{i = 1}^{n}\mathbf{x}_i = \frac{1}{n}\sum_{i = 1}^{n}\mathbb{E}[W_iG_i\mathbf{x}_i]$ , there is
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {x} _ {i} - \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} [ \boldsymbol {W} _ {i} \boldsymbol {G} _ {i} \mathbf {x} _ {i} ] = 0 \Leftrightarrow \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} [ (\boldsymbol {I} _ {d} - \boldsymbol {W} _ {i} \boldsymbol {G} _ {i}) \mathbf {x} _ {i} ] = 0.
$$
We now show that a sufficient and necessary condition to satisfy the above unbiasedness constraint is that for all  $i \in [n]$ ,  $\mathbb{E}[\pmb{W}_i\pmb{G}_i] = \pmb{I}_d$ .
Sufficiency. It is obvious that if for all  $i \in [n]$ ,  $\mathbb{E}[W_i G_i] = I_d$ , then we have  $\frac{1}{n} \mathbb{E}[(I_d - W_i G_i) \mathbf{x}_i] = 0$ . Necessity. Consider the special case that for some  $i \in [n]$  and  $\lambda \in [d]$ ,  $\mathbf{x}_i = n \mathbf{e}_{\lambda}$ , where  $\mathbf{e}_{\lambda}$  is the  $\lambda$ -th canonical basis vector, and  $\mathbf{x}_j = 0$ , and for all  $j \in [n] \setminus \{i\}$ . Then,
$$
\mathbf {e} _ {\lambda} = \bar {\mathbf {x}} = \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbb {E} [ \boldsymbol {W} _ {i} \boldsymbol {G} _ {i} \mathbf {x} _ {i} ] = \frac {1}{n} \mathbb {E} [ \boldsymbol {W} _ {i} \boldsymbol {G} _ {i} ] \mathbf {e} _ {\lambda} = [ \mathbb {E} [ \boldsymbol {W} _ {i} \boldsymbol {G} _ {i} ] ] _ {\lambda},
$$
where  $[\cdot ]_{\lambda}$  denotes the  $\lambda$  -th column of matrix  $\mathbb{E}[W_iG_i]$
Since our approach is agnostic to the choice of vectors, we need this choice of decoder matrices, by varying  $\lambda$  over  $[d]$ , we see that we need  $\mathbb{E}[\pmb{W}_i\pmb{G}_i] = \pmb{I}_d$ . And by varying  $i$  over  $[n]$ , we see that we need  $\mathbb{E}[\pmb{W}_j\pmb{G}_j] = \pmb{I}_d$  for all  $j \in [n]$ .
Therefore,  $\bar{\mathbf{x}} = \frac{1}{n}\sum_{i = 1}^{n}\mathbb{E}\big[\pmb {W}_i\pmb {G}_i\mathbf{x}_i\big]\Leftrightarrow \forall i\in [n],\mathbb{E}\big[\pmb {W}_i\pmb {G}_i\big] = \pmb {I}_d.$
This implies the second term of  $f(\mathbf{W})$  in Eq. 16 is 0, that is,
$$
\sum_ {i \neq j} \mathbb {E} \left[ \right.\left\langle\left(\boldsymbol {I} _ {d} - \boldsymbol {W} _ {i} \boldsymbol {G} _ {i}\right) \mathbf {x} _ {i}, \left(\boldsymbol {I} _ {d} - \boldsymbol {W} _ {j} \boldsymbol {G} _ {j}\right) \mathbf {x} _ {j} \right\rangle = 0.
$$
Hence, we only need to solve
$$
\hat {\mathbf {x}} = \underset {\boldsymbol {W}} {\arg \min } f _ {2} (\boldsymbol {W}) = \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \| \left(\boldsymbol {I} _ {d} - \boldsymbol {W} _ {i} \boldsymbol {G} _ {i}\right) \mathbf {x} _ {i} \| _ {2} ^ {2} \right] \tag {17}
$$
Since each  $\mathbf{W}_i$  appears in  $f_2(\mathbf{W})$  separately, each  $\mathbf{W}_i$  can be optimized separately, via solving
$$
\min  _ {\boldsymbol {W} _ {i}} \mathbb {E} \left[ \left\| (\boldsymbol {I} _ {d} - \boldsymbol {W} _ {i} \boldsymbol {G} _ {i}) \mathbf {x} _ {i} \right\| _ {2} ^ {2} \right] \quad \text {s u b j e c t t o} \mathbb {E} [ \boldsymbol {W} _ {i} \boldsymbol {G} _ {i} ] = \boldsymbol {I} _ {d}.
$$
One natural solution is to take  $\pmb{W}_i = \frac{d}{k}\pmb{G}_i^\dagger$ ,  $\forall i \in [n]$ . For  $i \in [n]$ , let  $\pmb{G}_i = \pmb{V}_i\Lambda_i\pmb{U}_i^T$  be its SVD, where  $\pmb{V}_i \in \mathbb{R}^{k\times d}$  and  $\pmb{U}_i \in \mathbb{R}^{d\times d}$  are orthogonal matrices. Then,
$$
\pmb {W} _ {i} \pmb {G} _ {i} = \frac {d}{k} \pmb {U} _ {i} \boldsymbol {\Lambda} _ {i} ^ {\dagger} \pmb {V} _ {i} ^ {T} \pmb {V} _ {i} \boldsymbol {\Lambda} \pmb {U} ^ {T} = \frac {d}{k} \pmb {U} _ {i} \boldsymbol {\Lambda} _ {i} ^ {\dagger} \boldsymbol {\Lambda} \pmb {U} ^ {T} = \frac {d}{k} \pmb {U} _ {i} \boldsymbol {\Sigma} \pmb {U} _ {i} ^ {T},
$$
where  $\Sigma$  is a diagonal matrix with 0s and 1s on the diagonal.
For simplicity, we assume the random matrix  $\mathbf{U}_i$  follows a continuous distribution.  $\mathbf{U}_i$  being discrete follows a similar analysis. Let  $\mu(\mathbf{U}_i)$  be the measure of  $\mathbf{U}_i$ .
$$
\begin{array}{l} \mathbb {E} [ \boldsymbol {W} _ {i} \boldsymbol {G} _ {i} ] = \frac {d}{k} \mathbb {E} [ \boldsymbol {U} _ {i} \boldsymbol {\Sigma} \boldsymbol {U} _ {i} ^ {T} ] = \frac {d}{k} \int_ {\boldsymbol {U} _ {i}} \mathbb {E} [ \boldsymbol {U} _ {i} \boldsymbol {\Sigma} _ {i} \boldsymbol {U} _ {i} ^ {T} | \boldsymbol {U} _ {i} ] \cdot d \mu (\boldsymbol {U} _ {i}) \\ = \frac {d}{k} \int_ {\boldsymbol {U} _ {i}} \boldsymbol {U} _ {i} \mathbb {E} [ \Sigma_ {i} \mid \boldsymbol {U} _ {i} ] \boldsymbol {U} _ {i} ^ {T} \cdot \mu (\boldsymbol {U} _ {i}) \\ = \frac {d}{k} \int_ {\boldsymbol {U} _ {i}} \boldsymbol {U} _ {i} \frac {k}{d} \boldsymbol {I} _ {d} \boldsymbol {U} _ {i} ^ {T} \cdot d \mu (\boldsymbol {U} _ {i}) \\ = \frac {d}{k} \frac {k}{d} \boldsymbol {I} _ {d} = \boldsymbol {I} _ {d}, \\ \end{array}
$$
which means the estimator  $\frac{1}{n}\sum_{i = 1}^{n}\frac{k}{d}\pmb{G}_i^\dagger \pmb{G}_i$  satisfies unbiasedness. The MSE is now
$$
\begin{array}{l} M S E = \mathbb {E} \left[ \| \bar {\mathbf {x}} - \frac {1}{n} \sum_ {i = 1} ^ {n} W _ {i} G _ {i} \mathbf {x} _ {i} \| _ {2} ^ {2} \right] = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \| \left(\boldsymbol {I} _ {d} - \boldsymbol {W} _ {i} G _ {i}\right) \mathbf {x} _ {i} \| _ {2} ^ {2} \right] \\ = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \left(\| \mathbf {x} _ {i} \| _ {2} ^ {2} + \mathbb {E} [ \| \boldsymbol {W} _ {i} \boldsymbol {G} _ {i} \mathbf {x} _ {i} \| _ {2} ^ {2} ] - 2 \langle \mathbf {x} _ {i}, \mathbb {E} [ \boldsymbol {W} _ {i} \boldsymbol {G} _ {i} ] \mathbf {x} _ {i} \rangle\right) \\ = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \left(\| \mathbf {x} _ {i} \| _ {2} ^ {2} + \mathbb {E} [ \| \boldsymbol {W} _ {i} \boldsymbol {G} _ {i} \mathbf {x} _ {i} \| _ {2} ^ {2} ] - 2 \langle \mathbf {x} _ {i}, \mathbf {x} _ {i} \rangle\right) \\ \end{array}
$$
$$
\begin{array}{l} = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \left(\mathbb {E} \left[ \| \boldsymbol {W} _ {i} \boldsymbol {G} _ {i} \mathbf {x} _ {i} \| _ {2} ^ {2} - \| \mathbf {x} _ {i} \| _ {2} ^ {2} \right]\right) \\ = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \left(\mathbf {x} _ {i} \mathbb {E} \left[ \left(\boldsymbol {W} _ {i} \boldsymbol {G} _ {i}\right) ^ {T} \left(\boldsymbol {W} _ {i} \boldsymbol {G} _ {i}\right) \right] \mathbf {x} _ {i} - \| \mathbf {x} _ {i} \| _ {2} ^ {2}\right). \\ \end{array}
$$
Again, let  $\pmb{G}_i = \pmb{V}_i \Lambda_i \pmb{U}_i^T$  be its SVD and consider  $\pmb{W}_i \pmb{G}_i = \frac{d}{k} \pmb{U}_i \Sigma_i \pmb{U}_i^T$ , where  $\Sigma_i$  is a diagonal matrix with 0s and 1s. Then,
$$
\begin{array}{l} M S E = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \sum_ {i = 1} ^ {n} \left(\mathbf {x} _ {i} ^ {T} \frac {d ^ {2}}{k ^ {2}} \mathbb {E} [ \boldsymbol {U} _ {i} \Sigma_ {i} \boldsymbol {U} _ {i} ^ {T} \boldsymbol {U} _ {i} \Sigma_ {i} \boldsymbol {U} _ {i} ^ {T} ] \mathbf {x} _ {i} - \| \mathbf {x} _ {i} \| _ {2} ^ {2}\right) \\ = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \left(\frac {d ^ {2}}{k ^ {2}} \mathbf {x} _ {i} ^ {T} \mathbb {E} [ U _ {i} \Sigma^ {2} U _ {i} ^ {T} ] \mathbf {x} _ {i} - \| \mathbf {x} _ {i} \| _ {2} ^ {2}\right). \\ \end{array}
$$
Since  $\pmb{G}_i$  has rank  $k$ ,  $\Sigma_i$  is a diagonal matrix with  $k$  out of  $d$  entries being 1 and the rest being 0. Let  $\mu(\pmb{U}_i)$  be the measure of  $\pmb{U}_i$ . Hence, for  $i \in [n]$ ,
$$
\begin{array}{l} \mathbb {E} \left[ \boldsymbol {U} _ {i} \Sigma_ {i} ^ {2} \boldsymbol {U} _ {i} ^ {T} \right] = \int_ {\boldsymbol {U} _ {i}} \mathbb {E} \left[ \boldsymbol {U} _ {i} \Sigma_ {i} ^ {2} \boldsymbol {U} _ {i} ^ {T} \mid \boldsymbol {U} _ {i} \right] d \mu (\boldsymbol {U} _ {i}) \\ = \int_ {\boldsymbol {U} _ {i}} \boldsymbol {U} _ {i} \mathbb {E} \left[ \Sigma_ {i} ^ {2} \mid \boldsymbol {U} _ {i} \right] \boldsymbol {U} _ {i} ^ {T} d \mu (\boldsymbol {U} _ {i}) \\ = \int_ {\boldsymbol {U} _ {i}} \frac {k}{d} \boldsymbol {U} _ {i} \boldsymbol {I} _ {d} \boldsymbol {U} _ {i} ^ {T} d \mu (\boldsymbol {U} _ {i}) \\ = \frac {k}{d} \int_ {\boldsymbol {U} _ {i}} \boldsymbol {I} _ {d} d \mu (\boldsymbol {U} _ {i}) \\ = \frac {k}{d} I _ {d}. \\ \end{array}
$$
Therefore, the MSE of the estimator, which is the solution of the optimization problem in Eq. 15, is
$$
M S E = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \left(\frac {d ^ {2}}{k ^ {2}} \mathbf {x} _ {i} ^ {T} \frac {k}{d} \mathbf {I} _ {d} \mathbf {x} _ {i} - \| \mathbf {x} _ {i} \| _ {2} ^ {2}\right) = \frac {1}{n ^ {2}} (\frac {d}{k} - 1) \sum_ {i = 1} ^ {n} \| \mathbf {x} _ {i} \| _ {2} ^ {2},
$$
which is the same MSE as that of Rand-  $k$
Alternative motivating regression problem 2.
Another motivating regression problem based on which we can design our estimator is
$$
\widehat {\mathbf {x}} = \arg \min  _ {\mathbf {x}} \| \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {G} _ {i} \mathbf {x} - \frac {1}{n} \sum_ {i = 1} ^ {n} \mathbf {G} _ {i} \mathbf {x} _ {i} \| _ {2} ^ {2} \tag {18}
$$
Note that  $\pmb{G}_i \in \mathbb{R}^{k \times d}, \forall i \in [n]$ , and so the solution to the above problem is
$$
\widehat {\mathbf {x}} ^ {(\mathrm {s o l u t i o n})} = \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {G} _ {i}\right) ^ {\dagger} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {G} _ {i} \mathbf {x} _ {i}\right),
$$
and to ensure unbiasedness of the estimator, we can set  $\bar{\beta} \in \mathbb{R}$  and have the estimator as
$$
\widehat {\mathbf {x}} ^ {(\text {e s t i m a t o r})} = \bar {\beta} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} G _ {i}\right) ^ {\dagger} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} G _ {i} \mathbf {x} _ {i}\right).
$$
It is not hard to see this estimator does not lead to an MSE as low as Rand-Proj-Spatial does. Consider the full correlation case, i.e.,  $\mathbf{x}_i = \mathbf{x},\forall i\in [n]$ , for example, the estimator is now