SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\hat {\mathbf {x}} _ {i} ^ {(\text {R a n d - P r o j - S p a t i a l})} = \bar {\beta} \left(T \left(\sum_ {i = 1} ^ {n} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i}\right)\right) ^ {\dagger} \boldsymbol {E} _ {i} ^ {T} \boldsymbol {E} _ {i} \mathbf {x} _ {i} \tag {29}
$$
Notice  $\mathbf{E}_i^T\mathbf{E}_i$  is a diagonal matrix, where the  $j$ -th diagonal entry is 1 if coordinate  $j$  of  $\mathbf{x}_i$  is chosen. Hence,  $\mathbf{E}_i^T\mathbf{E}_i\mathbf{x}_i$  can be viewed as choosing  $k$  coordinates of  $\mathbf{x}_i$  without replacement, which is exactly the same as Rand-  $k$ -Spatial's (and Rand-  $k$ 's) encoding procedure.
Notice  $\sum_{i=1}^{n} \boldsymbol{E}_{i}^{T} \boldsymbol{E}_{i}$  is also a diagonal matrix, where the  $j$ -th diagonal entry is exactly  $M_{j}$ , i.e. the number of clients who selects the  $j$ -th coordinate as in Rand- $k$ -Spatial [12]. Furthermore, notice  $\left(T \left(\sum_{i=1}^{n} \boldsymbol{E}_{i}^{T} \boldsymbol{E}_{i}\right)\right)^{\dagger}$  is also a diagonal matrix, where the  $j$ -th diagonal entry is  $\frac{1}{T(M_{j})}$ , which recovers the scaling factor used in Rand- $k$ -Spatial's decoding procedure.
Rand-Proj-Spatial computes  $\bar{\beta}$  as  $\bar{\beta} \mathbb{E}\left[\left(T\left(\sum_{i=1}^{n} \boldsymbol{E}_{i}^{T} \boldsymbol{E}_{i}\right)\right)^{\dagger} \boldsymbol{E}_{i}^{T} \boldsymbol{E}_{i} \mathbf{x}_{i}\right] = \mathbf{x}_{i}$ . Since  $\left(T\left(\sum_{i=1}^{n} \boldsymbol{E}_{i}^{T} \boldsymbol{E}_{i}\right)\right)^{\dagger}$  and  $\boldsymbol{E}_{i}^{T} \boldsymbol{E}_{i} \mathbf{x}_{i}$  recover the scaling factor and the encoding procedure of Rand- $k$ -Spatial, and  $\bar{\beta}$  is computed in exactly the same way as Rand- $k$ -Spatial does,  $\bar{\beta}$  will be exactly the same as in Rand- $k$ -Spatial.
Therefore,  $\hat{\mathbf{x}}_i^{(\mathrm{Rand - Proj - Spatial})}$  in Eq. 29 with  $E_{i}$  as the random matrix at client  $i$  recovers  $\hat{\mathbf{x}}_i^{(\mathrm{Rand - k - Spatial})}$ . This implies Rand-Proj-Spatial recovers Rand-  $k$ -Spatial in this case.
D Additional Experiment Details and Results
Implementation. All experiments are conducted in a cluster of 20 machines, each of which has 40 cores. The implementation is in Python, mainly based on numpy and scipy. All code used for the experiments can be found at https://github.com/11hifish/Rand-Proj-Spatial.
Data Split. For the non-IID dataset split across the clients, we follow [62] to split Fashion-MNIST, which is used in distributed power iteration and distributed  $k$ -means. Specifically, the data is first sorted by labels and then divided into  $2n$  shards with each shard corresponding to the data of a particular label. Each client is then assigned 2 shards (i.e., data from 2 classes). However, this approach only works for datasets with discrete labels (i.e. datasets used in classification tasks). For the other dataset UJIndoor, which is used in distributed linear regression, we first sort the dataset by the ground truth prediction and then divides the sorted dataset across the clients.
D.1 Additional experimental results
For each one of the three tasks, distributed power iteration, distributed  $k$ -means, and distributed linear regression, we provide additional results when the data split is IID across the clients for smaller  $n, k$  values in Section D.1.1, and when the data split is Non-IID across the clients in Section D.1.2. For the Non-IID case, we use the same settings (i.e.  $n, k, d$  values) as in the IID case.
Discussion. For smaller  $n, k$  values compared to the data dimension  $d$ , there is less information or less correlation from the client vectors. Hence, both Rand- $k$ -Spatial and Rand-Proj-Spatial perform better as  $nk$  increases. When  $n, k$  is small, one might notice Rand-Proj-Spatial performs worse than Rand- $k$ -Wangni in some settings. However, Rand- $k$ -Wangni is an adaptive estimator, which optimizes the sampling weights for choosing the client vector coordinates through an iterative process. That means Rand- $k$ -Wangni requires more computation from the clients, while in practice, the clients often have limited computational power. In contrast, our Rand-Proj-Spatial estimator is non-adaptive and the server does more computation instead of the clients. This is more practical since the central server usually has more computational power than the clients in applications like FL. See the introduction for more discussion.
In most settings, we observe the proposed Rand-Proj-Spatial has a better performance compared to Rand- $k$ -Spatial. Furthermore, as one would expect, both Rand- $k$ -Spatial and Rand-Proj-Spatial perform better when the data split is IID across the clients since there is more correlation among the client vectors in the IID case than in the Non-IID case.
D.1.1 More results in the IID case
Distributed Power Iteration and Distributed  $K$ -Means. We use the Fashion-MNIST dataset for both distributed power iteration and distributed  $k$ -means, which has a dimension of  $d = 1024$ . We consider more settings for distributed power iteration and distributed  $k$ -means here:  $n = 10, k \in \{5, 25, 51\}$ , and  $n = 50, k \in \{5, 10\}$ .
Figure 8: More results of distributed power iteration on Fashion-MNIST (IID data split) with  $d = 1024$  when  $n = 10, k \in \{5, 25, 51\}$  and when  $n = 50, k \in \{5, 10\}$ .
Figure 9: More results on distributed  $k$ -means on Fashion-MNIST (IID data split) with  $d = 1024$  when  $n = 10, k \in \{5, 25, 51\}$  and when  $n = 50, 28 \in \{10, 51\}$ .
Distributed Linear Regression. We use the UJIndoor dataset distributed linear regression, which has a dimension of  $d = 512$ . We consider more settings here:  $n = 10, k \in \{5, 25\}$  and  $n = 50, k \in \{1, 5\}$ .
Figure 10: More results of distributed linear regression on UJIndoor (IID data split) with  $d = 512$ , when  $n = 10, k \in \{5, 25\}$  and when  $n = 50, k \in \{1, 5\}$ . Note when  $k = 1$ , the Induced estimator is the same as Rand- $k$ .
D.1.2 Additional results in the Non-IID case
In this section, we report results when the dataset split across the clients are Non-IID, using the same datasets as in the IID case. We choose exactly the same set of  $n, k$  values as in the IID case.
Distributed Power Iteration and Distributed  $K$ -Means. Again, both distributed power iteration and distributed  $k$ -means use the Fashion-MNIST dataset, with a dimension  $d = 1024$ . We consider the following settings for both tasks:  $n = 10$ ,  $k \in \{5, 25, 51, 102\}$  and  $n = 50$ ,  $k \in \{5, 10, 20\}$ .
Figure 11: Results of distributed power iteration when the data split is Non-IID.  $n = 10, k \in \{5, 25, 51, 102\}$  and  $n = 50, k \in \{5, 10, 20\}$ .
Distributed Linear Regression. Again, we use the UJIndoor dataset for distributed linear regression, which has a dimension  $d = 512$ . We consider the following settings:  $n = 10, k \in \{5, 25, 50\}$  and  $n = 50, k \in \{1, 5, 50\}$ .
Distributed K Means (Non-IID)
Distributed K Means (Non-ILD)
Figure 12: Results of distributed  $k$ -means when the data split is Non-IID.  $n = 10, k \in \{5, 25, 51, 102\}$  and  $n = 50, k \in \{5, 10, 20\}$ .
Figure 13: Results of distributed linear regression when the data split is Non-IID.  $n = 10, k \in \{5,25,50\}$  and  $n = 50, k \in \{1,5,50\}$ .