SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\widehat {\mathbf {x}} ^ {(\text {R a n d - P r o j - S p a t i a l})} = \bar {\beta} \left(\sum_ {i = 1} ^ {n} \boldsymbol {G} _ {i} ^ {T} \boldsymbol {G} _ {i}\right) ^ {\dagger} \sum_ {i = 1} ^ {n} \boldsymbol {G} _ {i} ^ {T} \boldsymbol {G} _ {i} \mathbf {x} = \bar {\beta} \boldsymbol {S} ^ {\dagger} \boldsymbol {S} \mathbf {x}, \tag {8}
$$
where  $S \coloneqq \sum_{i=1}^{n} G_i^T G_i$ . By construction,  $\operatorname{rank}(S) \leq nk$ , and we focus on the case  $nk \leq d$ .
Limitation of Subsampling matrices. As mentioned above, with  $G_{i} = E_{i}, \forall i \in [n]$ , we recover the Rand- $k$ -Spatial family of estimators. In this case,  $S$  is a diagonal matrix, where each diagonal entry  $S_{jj} = M_j$ ,  $j \in [d]$ .  $M_j$  is the number of clients which sent their  $j$ -th coordinate to the server. To ensure  $\text{rank}(S) = nk$ , we need  $S_{jj} \leq 1, \forall j$ , i.e., each of the  $d$  coordinates is sent by at most one client. If all the clients sample their matrices  $\{E_i\}_{i=1}^n$  independently, this happens with probability  $\frac{\binom{d}{nk}}{\binom{d}{k}^n}$ . As an example, for  $k = 1$ ,  $\text{Prob}(\text{rank}(S) = n) = \frac{\binom{d}{n}}{d^n} \leq \frac{1}{n!}$  (because  $\frac{d^n}{n^n} \leq \binom{d}{n} \leq \frac{d^n}{n!}$ ). Therefore, to guarantee that  $S$  is full-rank, each client would need the subsampling information of all the other clients. This not only requires additional communication but also has serious privacy implications. Essentially, the limitation with subsampling matrices  $E_{i}$  is that the eigenvectors of  $S$  are restricted to be canonical basis vectors  $\{\mathbf{e}_j\}_{j=1}^d$ . Generalizing  $G_{i}$ 's to general rank  $k$  matrices relaxes this constraint and hence we can ensure that  $S$  is full-rank with high probability. In the next result, we show the benefit of choosing  $G_{i}$  as SRHT matrices. We call the resulting estimator Rand-Proj-Spatial(Max).
Theorem 4.3 (MSE under Full Correlation). Consider  $n$  clients, each holding the same vector  $\mathbf{x} \in \mathbb{R}^d$ . Suppose we set  $T(\lambda) = \lambda$ ,  $\bar{\beta} = \frac{d}{k}$  in Eq. 5, and the random linear map  $G_i$  at each client to be an SRHT matrix. Let  $\delta$  be the probability that  $S = \sum_{i=1}^{n} G_i^T G_i$  does not have full rank. Then, for  $nk \leq d$ ,
$$
\mathbb {E} \left[ \| \widehat {\mathbf {x}} ^ {(R a n d - P r o j - S p a t i a l (M a x))} - \bar {\mathbf {x}} \| _ {2} ^ {2} \right] \leq \left[ \frac {d}{(1 - \delta) n k + \delta k} - 1 \right] \| \mathbf {x} \| _ {2} ^ {2} \tag {9}
$$
The proof details are in Appendix C.1. To compare the performance of Rand-Proj-Spatial(Max) against Rand-  $k$ , we show in Appendix C.2 that for  $n \geq 2$ , as long as  $\delta \leq \frac{2}{3}$ , the MSE of Rand-Proj-Spatial(Max) is less than that of Rand-  $k$ . Furthermore, in Appendix C.3 we empirically demonstrate that with  $d \in \{32, 64, 128, \dots, 1024\}$  and different values of  $nk \leq d$ , the rank of  $S$  is full with high probability, i.e.,  $\delta \approx 0$ . This implies  $\mathbb{E}[\| \widehat{\mathbf{x}}^{(\text{Rand-Proj-Spatial(Max)})} - \bar{\mathbf{x}} \|_2^2] \approx (\frac{d}{nk} - 1) \| \mathbf{x} \|_2^2$ .
Furthermore, since setting  $G_{i}$  as SRHT significantly increases the probability of recovering  $nk$  coordinates of  $\mathbf{x}$ , the MSE of Rand-Proj-Spatial with SRHT (Eq. 4.3) is strictly less than that of Rand-  $k$ -Spatial (Eq. 3). We also compare the MSEs of the three estimators in Figure 2 in the following setting:  $\| \mathbf{x}\| _2 = 1$ ,  $d = 1024$ ,  $n\in \{10,20,50,100\}$  and small  $k$  values such that  $nk < d$ .
Figure 2: MSE comparison of Rand-  $k$ , Rand-  $k$ -Spatial(Max) and Rand-Proj-Spatial(Max) estimators, when all clients have identical vectors (maximum inter-client correlation).
4.2 Case II: Orthogonal Client Vectors  $(\mathcal{R} = 0)$
When all the client vectors are orthogonal to each other, [12] showed that  $\operatorname{Rand} - k$  has the lowest MSE among the Rand- $k$ -Spatial family of decoders. We show in the next result that if we set the
random linear maps  $G_{i}$  at client  $i$  to be SRHT, and choose the fixed transformation  $T \equiv 1$  as in [12], Rand-Proj-Spatial achieves the same MSE as that of Rand- $k$ .
Theorem 4.4 (MSE under No Correlation). Consider  $n$  clients, each holding a vector  $\mathbf{x}_i \in \mathbb{R}^d$ ,  $\forall i \in [n]$ . Suppose we set  $T \equiv 1$ ,  $\bar{\beta} = \frac{d^2}{k}$  in Eq. 5, and the random linear map  $G_i$  at each client to be an SRHT matrix. Then, for  $nk \leq d$ ,
$$
\mathbb {E} \left[ \| \widehat {\mathbf {x}} ^ {(R a n d - P r o j - S p a t i a l)} - \bar {\mathbf {x}} \| _ {2} ^ {2} \right] = \frac {1}{n ^ {2}} \left(\frac {d}{k} - 1\right) \sum_ {i = 1} ^ {n} \| \mathbf {x} _ {i} \| _ {2} ^ {2}. \tag {10}
$$
The proof details are in Appendix C.4. Theorem 4.4 above shows that with zero correlation among client vectors, Rand-Proj-Spatial achieves the same MSE as that of Rand-  $k$ .
4.3 Incorporating Varying Degrees of Correlation
In practice, it unlikely that all the client vectors are either identical or orthogonal to each other. In general, there is some "imperfect" correlation among the client vectors, i.e.,  $\mathcal{R} \in (0, n - 1)$ . Given correlation level  $\mathcal{R}$ , [12] shows that the estimator from the Rand- $k$ -Spatial family that minimizes the MSE is given by the following transformation.
$$
T (m) = 1 + \frac {\mathcal {R}}{n - 1} (m - 1) \tag {11}
$$
Recall from Section 4.1 (Section 4.2) that setting  $T(m) = 1$  ( $T(m) = m$ ) leads to the estimator among the Rand- $k$ -Spatial family that minimizes MSE when there is zero (maximum) correlation among the client vectors. We observe the function  $T$  defined in Eq. 11 essentially interpolates between the two extreme cases, using the normalized degree of correlation  $\frac{\mathcal{R}}{n-1} \in [-\frac{1}{n-1}, 1]$  as the weight. This motivates us to apply the same function  $T$  defined in Eq. 11 on the eigenvalues of  $S = \sum_{i=1}^{n} G_i^T G_i$  in Rand-Proj-Spatial. As we shall see in our results, the resulting Rand-Proj-Spatial family estimator improves over the MSE of both Rand- $k$  and Rand- $k$ -Spatial family estimator.
We note that deriving a closed-form expression of MSE for Rand-Proj-Spatial with SRHT in the general case with the transformation function  $T$  (Eq. 11) is hard (we elaborate on this in Appendix B.3), as this requires a closed form expression for the non-asymptotic distributions of eigenvalues and eigenvectors of the random matrix  $S$ . To the best of our knowledge, previous analyses of SRHT, for example in [37, 38, 39, 45, 55], rely on the asymptotic properties of SRHT, such as the limiting eigen spectrum, or concentration bounds on the singular values, to derive asymptotic or approximate guarantees. However, to analyze the MSE of Rand-Proj-Spatial, we need an exact, non-asymptotic analysis of the eigenvalues and eigenvectors distribution of SRHT. Given the apparent intractability of the theoretical analysis, we compare the MSE of Rand-Proj-Spatial, Rand- $k$ -Spatial, and Rand- $k$  via simulations.
Simulations. In each experiment, we first simulate  $\bar{\beta}$  in Eq. 5, which ensures our estimator is unbiased, based on 1000 random runs. Given the degree of correlation  $\mathcal{R}$ , we then compute the squared error, i.e.  $\| \widehat{\mathbf{x}}^{(\mathrm{Rand - Proj - Spatial})} - \overline{\mathbf{x}}\| _2^2$ , where Rand-Proj-Spatial has  $G_{i}$  as SRHT matrix (Eq. 6) and  $T$  as in Eq. 11. We plot the average over 1000 random runs as an approximation to MSE. Each client holds a  $d$ -dimensional base vector  $\mathbf{e}_j$  for some  $j\in [d]$ , and so two clients either hold the same or orthogonal vectors. We control the degree of correlation  $\mathcal{R}$  by changing the number of clients which hold the same vector. We consider  $d = 1024$ ,  $n\in \{21,51\}$ . We consider positive correlation values, where  $\mathcal{R}$  is chosen to be linearly spaced within  $[0,n - 1]$ . Hence, for  $n = 21$ , we use  $\mathcal{R}\in \{4,8,12,16\}$  and for  $n = 51$ , we use  $\mathcal{R}\in \{10,20,30,40\}$ . All results are presented in Figure 3. As expected, given  $\mathcal{R}$ , Rand-Proj-Spatial consistently achieves a lower MSE than the lowest possible MSE from the Rand- $k$ -Spatial family decoder. Additional results with different values of  $n,d,k$ , including the setting  $nk\ll d$ , can be found in Appendix B.4.
A Practical Configuration. In reality, it is hard to know the correlation information  $\mathcal{R}$  among the client vectors. [12] uses the transformation function which interpolates to the middle point between the full correlation and no correlation cases, such that  $T(m) = 1 + \frac{n}{2} \frac{m - 1}{n - 1}$ . Rand- $k$ -Spatial with such  $T$  is called Rand- $k$ -Spatial(Avg). Following this approach, we evaluate Rand-Proj-Spatial with SRHT using this  $T$ , and call it Rand-Proj-Spatial(Avg) in practical settings (see Figure 4).
Figure 3: MSE comparison of estimators Rand-  $k$ , Rand-  $k$ -Spatial(Opt), Rand-Proj-Spatial, given the degree of correlation  $\mathcal{R}$ . Rand-  $k$ -Spatial(Opt) denotes the estimator that gives the lowest possible MSE from the Rand-  $k$ -Spatial family. We consider  $d = 1024$ , number of clients  $n \in \{21, 51\}$ , and  $k$  values such that  $nk < d$ . In each plot, we fix  $n, k, d$  and vary the degree of positive correlation  $\mathcal{R}$ . The y-axis represents MSE. Notice since each client has a fixed  $\| \mathbf{x}_i \|_2 = 1$ , and Rand-  $k$  does not leverage cross-client correlation, the MSE of Rand-  $k$  in each plot remains the same for different  $\mathcal{R}$ .
We consider three practical distributed optimization tasks for evaluation: distributed power iteration, distributed  $k$ -means and distributed linear regression. We compare Rand-Proj-Spatial(Avg) against Rand- $k$ , Rand- $k$ -Spatial(Avg), and two more sophisticated but widely used sparsification schemes: non-uniform coordinate-wise gradient sparsification [15] (we call it Rand- $k$ (Wangni)) and the Induced compressor with Rand- $k + \operatorname{Top}-k$  [14]. The results are presented in Figure 4.
Dataset. For both distributed power iteration and distributed  $k$ -means, we use the test set of the Fashion-MNIST dataset [56] consisting of 10000 samples. The original images from Fashion-MNIST are  $28 \times 28$  in size. We preprocess and resize each image to be  $32 \times 32$ . Resizing images to have their dimension as a power of 2 is a common technique used in computer vision to accelerate the convolution operation. We use the UJIndoor dataset for distributed linear regression. We subsample 10000 data points, and use the first 512 out of the total 520 features on signals of phone calls. The task is to predict the longitude of the location of a phone call. In all the experiments in Figure 4, the datasets are split IID across the clients via random shuffling. In Appendix D.1, we have additional results for non-IID data split across the clients.
Setup and Metric. Recall that  $n$  denotes the number of clients,  $k$  the per-client communication budget, and  $d$  the vector dimension. For Rand-Proj-Spatial, we use the first 50 iterations to estimate  $\bar{\beta}$  (see Eq. 5). Note that  $\bar{\beta}$  only depends on  $n, k, d$ , and  $T$  (the transformation function in Eq. 5), but is independent of the dataset. We repeat the experiments across 10 independent runs, and report the mean MSE (solid lines) and one standard deviation (shaded regions) for each estimator. For each task, we plot the squared error of the mean estimator  $\hat{\mathbf{x}}$ , i.e.,  $\| \hat{\mathbf{x}} - \bar{\mathbf{x}} \|_2^2$ , and the values of the task-specific loss function, detailed below.
Tasks and Settings:
1. Distributed power iteration. We estimate the principle eigenvector of the covariance matrix, with the dataset (Fashion-MNIST) distributed across the  $n$  clients. In each iteration, each client computes a local principle eigenvector estimate based on a single power iteration and sends an encoded version to the server. The server then computes a global estimate and sends it back to the clients. The task-specific loss here is  $\| \mathbf{v}_t - \mathbf{v}_{top} \|_2$ , where  $\mathbf{v}_t$  is the global estimate of the principal eigenvector at iteration  $t$ , and  $\mathbf{v}_{top}$  is the true principle eigenvector.
$^{2}$ https://archive.ics.uci.edu/ml/datasets/ujiindoorloc
2. Distributed  $k$ -means. We perform  $k$ -means clustering [57] with the data distributed across  $n$  clients (Fashion-MNIST, 10 classes) using Lloyd's algorithm. At each iteration, each client performs a single iteration of  $k$ -means to find its local centroids and sends the encoded version to the server. The server then computes an estimate of the global centroids and sends them back to the clients. We report the average squared mean estimation error across 10 clusters, and the  $k$ -means loss, i.e., the sum of the squared distances of the data points to the centroids.
For both distributed power iterations and distributed  $k$ -means, we run the experiments for 30 iterations and consider two different settings:  $n = 10$ ,  $k = 102$  and  $n = 50$ ,  $k = 20$ .
Distributed Power Iteration
Distributed K Means
Figure 4: Experiment results on three distributed optimization tasks: distributed power iteration, distributed  $k$ -means, and distributed linear regression. The first two use the Fashion-MNIST dataset with the images resized to  $32 \times 32$ , hence  $d = 1024$ . Distributed linear regression uses UJIndoor dataset with  $d = 512$ . All the experiments are repeated for 10 random runs, and we report the mean as the solid lines, and one standard deviation using the shaded region. The violet line in the plots represents our proposed Rand-Proj-Spatial(Avg) estimator.
Distributed Linear Regression
3. Distributed linear regression. We perform linear regression on the UJIndoor dataset distributed across  $n$  clients using SGD. At each iteration, each client computes a local gradient and sends an encoded version to the server. The server computes a global estimate of the gradient, performs an SGD step, and sends the updated parameter to the clients. We run the experiments for 50 iterations with learning rate 0.001. The task-specific loss is the linear regression loss, i.e. empirical mean squared error. To have a proper scale that better showcases the difference in performance of different estimators, we plot the results starting from the 10th iteration.
Results. It is evident from Figure 4 that Rand-Proj-Spatial(Avg), our estimator with the practical configuration  $T$  (see Section 4.3) that does not require the knowledge of the actual degree of correlation among clients, consistently outperforms the other estimators in all three tasks. Additional experiments for the three tasks are included in Appendix D.1. Furthermore, we present the wall-clock time to encode and decode client vectors using different sparsification schemes in Figure 5. Though Rand-Proj-Spatial(Avg) has the longest decoding time, the encoding time of Rand-Proj-Spatial(Avg) is less than that of the adaptive Rand-  $k$  (Wangni) sparsifier. In practice, the server has more compu
Figure 5: The corresponding wall-clock time to encode and decode client vectors (in seconds) using different sparsification schemes, across the three tasks.
tational power than the clients and hence can afford a longer decoding time. Therefore, it is more important to have efficient encoding procedures.
We note two practical limitations of the proposed Rand-Proj-Spatial.
1) Computation Time of Rand-Proj-Spatial. The encoding time of Rand-Proj-Spatial is  $O(kd)$ , while the decoding time is  $O(d^2 \cdot nk)$ . The computation bottleneck in decoding is computing the eigendecomposition of the  $d \times d$  matrix  $\sum_{i=1}^{n} G_i^T G_i$  of rank at most  $nk$ . Improving the computation time for both the encoding and decoding schemes is an important direction for future work.
2) Perfect Shared Randomness. It is common to assume perfect shared randomness between the server and the clients in distributed settings [58]. However, to perfectly simulate randomness using Pseudo Random Number Generator (PRNG), at least  $\log_2d$  bits of the seed need to be exchanged in practice. We acknowledge this gap between theory and practice.
In this paper, we propose the Rand-Proj-Spatial estimator, a novel encoding-decoding scheme, for communication-efficient distributed mean estimation. The proposed client-side encoding generalizes and improves the commonly used Rand-  $k$  sparsification, by utilizing projections onto general  $k$ -dimensional subspaces. On the server side, cross-client correlation is leveraged to improve the approximation error. Compared to existing methods, the proposed scheme consistently achieves better mean estimation error across a variety of tasks. Potential future directions include improving the computation time of Rand-Proj-Spatial and exploring whether the proposed Rand-Proj-Spatial achieves the optimal estimation error among the class of non-adaptive estimators, given correlation information. Furthermore, combining sparsification and quantization techniques and deriving such algorithms with the optimal communication cost-estimation error trade-offs would be interesting.
We would like to thank the anonymous reviewer for providing valuable feedback on the title of this work, interesting open problems, alternative motivating regression problems and practical limitations of shared randomness. This work was supported in part by NSF grants CCF 2045694, CCF 2107085, CNS-2112471, and ONR N00014-23-1-2149.
[1] Jakub Konečny, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527, 2016.
[2] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273–1282. PMLR, 2017.
[3] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1-2):1-210, 2021.
[4] Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. arXiv preprint arXiv:2107.06917, 2021.
[5] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[7] John A Gubner. Distributed estimation and quantization. IEEE Transactions on Information Theory, 39(4):1456-1459, 1993.
[8] Peter Davies, Vijaykrishna Gurunathan, Niusha Moshrefi, Saleh Ashkboos, and Dan Alistarh. New bounds for distributed mean estimation and variance reduction, 2021.
[9] Shay Vargaftik, Ran Ben Basat, Amit Portnoy, Gal Mendelson, Yaniv Ben-Itzhak, and Michael Mitzenmacher. Eden: Communication-efficient and robust distributed mean estimation for federated learning, 2022.
[10] Ananda Theertha Suresh, X Yu Felix, Sanjiv Kumar, and H Brendan McMahan. Distributed mean estimation with limited communication. In International conference on machine learning, pages 3329-3337. PMLR, 2017.
[11] Jakub Konečný and Peter Richtárik. Randomized distributed mean estimation: Accuracy vs. communication. Frontiers in Applied Mathematics and Statistics, 4:62, 2018.
[12] Divyansh J Hunjhunwala, Ankur Mallick, Advait Harshal Gadhikar, Swanand Kadhe, and Gauri Joshi. Leveraging spatial and temporal correlations in sparsified mean estimation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[13] Ananda Theertha Suresh, Ziteng Sun, Jae Ro, and Felix Yu. Correlated quantization for distributed mean estimation and optimization. In International Conference on Machine Learning, pages 20856-20876. PMLR, 2022.
[14] Samuel Horváth and Peter Richtarik. A better alternative to error feedback for communication-efficient distributed learning. In International Conference on Learning Representations, 2021.
[48] Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. Fetchsgd: Communication-efficient federated learning with sketching. In International Conference on Machine Learning, pages 8253-8265. PMLR, 2020.
[49] Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
[50] Kilian Q Weinberger, Fei Sha, and Lawrence K Saul. Learning a kernel matrix for nonlinear dimensionality reduction. In Proceedings of the twenty-first international conference on Machine learning, page 106, 2004.
[51] Rohit Tripathy, Ilias Billionis, and Marcial Gonzalez. Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation. Journal of Computational Physics, 321:191-223, 2016.
[52] Gregory T. Minton and Eric Price. Improved concentration bounds for count-sketch, 2013.
[53] R. Gallager. Low-density parity-check codes. IRE Transactions on Information Theory, 8(1):21-28, 1962.
[54] Amin Shokrollahi. Fountain codes. Iee Proceedings-communications - IEE PROC-COMMUN, 152, 01 2005.
[55] Zijian Lei and Liang Lan. Improved subsampled randomized hadamard transform for linear svm. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4519-4526, 2020.
[56] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
[57] Maria-Florina F Balcan, Steven Ehrlich, and Yingyu Liang. Distributed  $k$ -means and  $k$ -median clustering on general topologies. Advances in neural information processing systems, 26, 2013.
[58] Mingxun Zhou, Tianhao Wang, T-H. Hubert Chan, Giulia Fanti, and Elaine Shi. Locally differentially private sparse vector aggregation. In 2022 IEEE Symposium on Security and Privacy (SP), pages 422-439, 2022.
[59] Gene H. Golub. Some modified matrix eigenvalue problems. SIAM Review, 15(2):318-334, 1973.
[60] Ming Gu and Stanley C. Eisenstat. A stable and efficient algorithm for the rank-one modification of the symmetric eigenproblem. SIAM Journal on Matrix Analysis and Applications, 15(4):1266-1276, 1994.
[61] Peter Arbenz, Walter Gander, and Gene H. Golub. Restricted rank modification of the symmetric eigenvalue problem: Theoretical considerations. Linear Algebra and its Applications, 104:75-95, 1988.
[62] H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficient learning of deep networks from decentralized data, 2023.
A Additional Details on Motivation in Introduction
A.1 Preprocessing all client vectors by the same random matrix does not improve performance