SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 1. Let  $u \in S^{d-1}$ , be drawn uniformly at random. Then  $||Bu||^2 \sim \mathrm{Beta}(\frac{c}{2}, \frac{d-c}{2}, 0, \frac{d}{c})$ , the four parameter Beta distribution with pdf  $f(x) = \frac{(2x/c)^{c/2-1}(1-2x/c)^{(d-c)/2-1}}{(d/c)\beta(c/2, (d-c)/2)}$  and  $\mathbb{E}||Bu||^2 = 1$ .
We defer proof of Proposition 1 to the Appendix D. Since the folded magnitude  $||Bu||^2$  is unit in expectation, the distortion term  $\alpha^2 /\lambda^2$  in Theorem 2 will often be close to 1, greatly tightening the angle distortion bounds.
4.2 PGHash algorithm
Below, we detail our protocol for deploying PGHash in a centralized distributed setting (presented algorithmically in Algorithm 1). Over a network of  $N$  devices, the central host identifies a target layer  $P$  whose weight  $W_{t}^{P}\in \mathbb{R}^{d\times n}$  (at
iteration  $t$ ) is too expensive for memory-constrained devices to fully train or host in local memory. Neurons (columns of  $W_{t}^{P}$ ) are pruned by devices according to their estimated cosine similarity to the output  $x^{P - 1}$  of the previous layer.
The central host begins each round of distributed training  $t$  by sending each device (1) all weights  $\{W_t^\ell\}_{\ell=1}^{P-1}$  required to generate the input  $x^{P-1}$  and (2) the compressed target layer  $BW_t^P$ . Using these weights, each device conducts PGHash analysis (via Algorithm 2) using its current batch of local data to determine its activated neurons. The central host sends each device  $i$  their set of activated neurons  $W_t^P(\Theta_i)$ , and each device performs a standard gradient update on their new model  $\{W_t^\ell\}_{\ell=1}^{P-1} \bigcup W_t^P(\Theta_i) \bigcup \{W_t^\ell\}_{\ell=P+1}^L$ . Finally, the central host receives updated models from each device and averages only the weights that are activated during training.
The on-device PGHash analysis (Algorithm 2) consists of first running a forward pass up to the target layer to generate the input  $x^{P - 1}$ . Devices generate personal hash tables by performing left-multiplication of  $BW_{t}^{P}$  and  $x^{P - 1}$  by a random Gaussian matrix  $S\sim \mathcal{N}(0,I_c)$ , as described in Section 3. The number of tables  $\tau$  is specified by the user. A neuron  $j$  is marked as active if the input
Algorithm 1 Distributed PGHash
Require: weights  $\{W_0^\ell\}_{\ell=1}^L$ , objective  $\mathcal{F}$ ,  $N$  devices, target layer  $P$ ,  $T$  iterations, folding matrix  $B$ ,  $\tau$  hash tables, hash length  $k$ , sketch dim  $c$ , comp. ratio  $CR$ .
1: Server Executes:
2: for  $t = 1,2,\ldots ,T$  do
3: Compile pre-target weights  $W_{A} = \{W_{t}^{\ell}\}_{\ell = 1}^{P - 1}$
4: Compile post-target weights  $W_{B} = \{W_{t}^{\ell}\}_{\ell = P + 1}^{L}$
5: for each device  $i$  in parallel do
6:  $\Theta_{i}\gets \mathbf{DeviceLSH}(\mathrm{i},W_{A},BW_{t}^{P},CR,\tau ,k,c)$
7:  $W_{t + 1,i}\gets \mathbf{DeviceUpdate}(\mathrm{i},W_t^P (\Theta_i)\bigcup W_B)$
8: end for
9:  $W_{t + 1}\gets$  Average the active weights  $W_{t + 1,i}$  across all devices
0: end for
11: return  $\{W_T^\ell\}_{\ell=1}^L$
12: DeviceLSH(i,  $W_{A}$ ,  $BW$ ,  $CR$ ,  $\tau$ ,  $k$ ,  $c$ ):
13: Sample input  $\xi$  from local data distribution  $\mathcal{D}_i$
14:  $\xi^{P - 1} = \mathcal{F}(W_A,\xi)W_A$  is stored locally
15:  $\Theta_{i}\gets \mathsf{PGHash}(\xi^{P - 1},BW,CR,\tau ,k,c)$
16: return  $\Theta_{i}$
17: DeviceUpdate(i,  $W_B$ ):
18:  $W\gets W_A\bigcup W_B$
19: return  $W - \eta \nabla_{W} \mathcal{F}(W; \xi)$
(a) SimHash (10 tables).
(c) SimHash (100 tables).
Figure 2: Correlation between angle and Hamming distance. We plot the average Hamming distance (x-axis) between a PG/SimHash hashed fixed vector  $x$  and collection of vectors  $\mathcal{V}$  versus their true angles (y-axis). Vectors are unit, length 100, and hashed down to dimension  $k = 25$  binary vectors according to  $\tau = 10$  or 100 hash tables. PGHash has a sketch dimension of  $c = 25$ . Both PGHash and SimHash show strong correlation between Hamming distance and angular similarity.
(b) PGHash (10 tables).
(d) PGHash (100 tables).
hash code  $\mathrm{sgn}(Sx^{P - 1})$  is identical to  $j$ -th weight's hash code  $\mathrm{sgn}(SB[W_t^P ]_{:,j})$ . In Appendix A.2, we detail how Hamming distance can also be used for neuron selection.
Computational Complexity Remark. Through the use of dynamic pruning, PGHash significantly reduces both the forward and backward training computational complexities. PGHash activates at most  $CR \cdot n$  neurons per sample as opposed to  $n$  for full training. In practice, PGHash activates only a fraction of the  $CR \cdot n$  neurons (as shown in Figure 6a). Therefore, the number of floating point operations within forward and backward training is dramatically reduced.
Communication Complexity Remark. By reducing the size of the model needed in local memory and subsequently requesting a pruned version of the architecture we improve communication efficiency. For a fixed number of rounds  $T$  and target weight size  $dn$ , the total communication complexity, with respect to this data structure, is  $\mathcal{O}(T \cdot CR \cdot dn)$ , which is significantly fewer bits than the vanilla  $\mathcal{O}(Tdn)$  communication cost of vanilla federated training. In Section 5, we show that PGHash achieves near
state-of-the-art results with only  $CR = 0.1$  (10% of a massive weight matrix).
Algorithm 2 PGHash
Require: batched input  $X\in \mathbb{R}^{d\times M}$ , projected weight  $BW\in \mathbb{R}^{c\times n}$ , compression rate  $CR$ ,  $\tau$  hash tables, hash length  $k$ , sketch dim  $c$ .
1: Set  $\Theta = [ ]$
2: for  $i = 1,2,\dots ,\tau$  do
3: Draw random Gaussian  $S \in \mathbb{R}^{k \times c}$
4: for each sample  $x$  in  $X$  do
5: for  $j = 1,2,\ldots ,n$  do
6: if  $\operatorname{sgn}(SBx) = [\operatorname{sgn}(SBW)]_{:,j}$  then
7:  $\Theta .\mathrm{add}(j)$
8: end if
11: if  $|\Theta| > \lfloor CR \cdot n \rfloor$  then
12: Randomly remove selected neurons from table-  $i$  until  $|\Theta | = \lfloor CR\cdot n\rfloor$
13: return  $\Theta$
16: return  $\Theta$
(a)  $90\%$  Compression.
(b)  $75\%$  Compression.
Figure 3: Compressed PGHash. We record model accuracy of a large recommendation system on an extreme classification task (Delicious-200K) using PGHash for varying compression rates  $(CR)$ . Compressed PGHash, even at  $90\%$  compression, is competitive with full training (without even including effects of sparsity-induced neuronal drop out). Hyperparameters are in Appendix A.1.
(c) No Compression.
In this section, we (1) gauge the sensitivity of PGHash and (2) analyze the performance of PGHash and our own DWTA variant (PGHash-D) in training large-scale recommender networks. PGHash and PGHash-D require only  $6.25\%$  ( $c = 8$ ) of the final layer sent by the server to perform on-device LSH in our experiments. In PGHash, devices receive the compressed matrix  $BW \in \mathbb{R}^{c \times n}$  via the procedure outlined in Section 4. In PGHash-D, devices receive  $c$  out of  $d$  randomly selected coordinates for all  $n$  neurons in the final layer weight. Using  $k$  of the  $c$  coordinates (ensuring privacy since the server is unaware of the coordinates used for LSH), PGHash-D selects neurons that, within the  $k$  coordinates, share the same index of highest-magnitude entry between the input and weight. We employ PGHash for Delicious-200K and PGHash-D for Amazon-670K and WikiLSHTC-325K.
PGHash Sensitivity Analysis. Our first experiment measures the ability of  $\mathcal{H}^{PG}(c,d)$  to estimate cosine similarity. We produce a fixed unit vector  $x\in \mathbb{R}^{100}$  and set of 180 vectors  $\{v_{i}\}_{i = 1}^{180}$  of the same dimension. Both the Gaussian vector  $x$  and collection of vectors  $V$  are fed through varying numbers of SimHash and PGHash tables. We produce a scatter plot measuring the correlation between the angle and average Hamming distance. PGHash, as seen in Figure 2, is an effective estimator of cosine similarity. We observe that PGHash, like SimHash, successfully produces low average Hamming distances for vectors that are indeed close in angle. This provides evidence that selecting neurons with exact hash code matches (vanilla sampling) is effective for choosing neurons that are close in angle to the input vector. Finally, we find increasing the number of hash tables helps reduce variance.
Large-Scale Recommender Network Training. Our second experiment tests how well PGHash(-D) can train large-scale recommender networks. We train these networks efficiently by utilizing dynamic neuronal dropout as done in [8]. We use three extreme multi-label datasets for training recommender networks: Delicious-200K, Amazon-670K, and WikiLSHTC-325K. These datasets come from the Extreme Classification Repository [4]. The dimensionality of these datasets is large: 782,585/205,443 (Delicious-200K), 135,909/670,091 (Amazon-670K), and 1,617,899/325,056 (WikiLSHTC-325K) features/terms. Due to space, Wiki results are found in Appendix A.3.
The feature and label sets of these datasets are extremely sparse. Akin to [8, 7, 34], we train a recommender network using a fully-connected neural network with a single hidden layer of size 128. Therefore, for Amazon-670K, our two dense layers have weight matrices of size  $(135,909\times 128)$  and  $(128\times 670,091)$ . The final layer weights output logits for label prediction, and we use PGHash(-D) to prune its size to improve computational efficiency during training.
Unlike [8, 7, 34], PGHash(-D) can be deployed in a federated setting. Within our experiments, we show the efficacy of PGHash for both single- and multi-device settings. Training in the federated setting (following the protocols of Algorithm 1) allows each device to rapidly train portions of the entire neural network in tandem. We partition data evenly (in an IID manner) amongst devices. Finally, we train our neural network using TensorFlow. We use the Adam [21] optimizer with an initial learning rate of 1e-4. A detailed list of the hyper-parameters we use in our experiments can be found in Appendix A.1. Accuracy in our figures refers to the  $P@1$  metric, which measures whether the predicted label with the highest probability is within the true list of labels. These experiments are run on a cloud cluster using Intel Xeon Silver 4216 processors with 128GB of total memory.
(a) 1 Device Results.
(b) 4 Device Results.
Figure 4: Federated Delicious-200K PGHash. We record model accuracy of a large recommendation system on an extreme classification task (Delicious-200K) trained in a federated setting. PGHash achieves competitive accuracies compared with Federated SLIDE and FedAvg. In fact, PGHash converges quicker to near-optimal accuracy. Hyperparameters are in Appendix A.1.
(c) 8 Device Results.
Sampling Strategy. One important aspect of training is how we select activated neurons for each sample through LSH. Like [8], we utilize vanilla sampling. In our vanilla sampling protocol, a total of  $CR \cdot n$  neurons are selected across the entire sampled batch of data. As detailed in Section 4 and Algorithm 2, a neuron is selected when its hash code exactly matches the hash code of the input. We retrieve neurons until either  $CR \cdot n$  are selected or all  $\tau$  tables have been looked up.
Compression Efficacy. We begin by analyzing how PGHash performs when varying the compression rate  $CR$ . Figure 3 showcases how PGHash performs for compression rates of  $75\%$  and  $90\%$  as well as no compression. Interestingly, PGHash reaches near-optimal accuracy even when compressed. This shows the effectiveness of PGHash at accurately selecting fruitful active neurons given a batch of data. The difference between the convergence of PGHash for varying compression rates lies within the volatility of training. As expected, PGHash experiences more volatile training (Figures 3a and 3b) when undergoing compression as compared to non-compressed training (Figure 3c).
Distributed Efficacy. In Figures 4 and 5, we analyze how well PGHash(-D) performs in a federated setting. We compare PGHash(-D) to a federated version of SLIDE [8] that we implemented (using respectively, a full SimHash or DWTA), as well as fully-dense Federated Averaging (FedAvg) for Delicious-200K. One can immediately see in Figures 4 and 5 that PGHash(-D) performs identically to, or better than, Federated SLIDE. In fact, for Delicious-200K, PGHash and Federated SLIDE outperform the dense baseline (FedAvg). In Appendix A.3 we detail the difficulties of PGHash-D and SLIDE in matching the dense baseline as well as the failure of SimHash to achieve performance akin to DWTA for Amazon-670K.
(a) 1 Device Results.
Figure 5: Federated Amazon-670K PGHash-D. We record model accuracy of a large recommendation system on Amazon-670K trained in a federated setting. PGHash-D matches the convergence of Federated SLIDE without requiring LSH to be performed by the central server.
(b) 4 Device Results.
(a) Average Activated Neurons.
Figure 6: PGHash(-D) Computational Efficiency. In Figure 6a, we showcase that PGHash(-D) activates only a fraction of the total final layer neurons even without compression. Through this induced sparsity, PGHash(-D) greatly reduces the computational complexity of forward and backward training compared to full training. In Figures 6b and 6c, we compare PGHash(-D) with the Sampled Softmax heuristic [17] (randomly sampling  $10\%$  of the total neurons) for efficiently training recommender systems. PGHash(-D) outperforms the Sampled Softmax baseline, as it selects a better set of activated neurons via LSH to more efficiently train the recommender system.
(b) Sampled Softmax (Delicious).
(c) Sampled Softmax (Amazon).
PGHash(-D) and Federated SLIDE smartly train portions of the network related to each batch of local device data, via LSH, in order to make up for the lack of a full output layer. However, unlike Federated SLIDE, PGHash(-D) can perform on-device LSH using as little as  $6.25\%$  of the full weight  $W(c = 8)$  for both Delicious-200K and Amazon-670K experiments. Furthermore, for Delicious-200K, PGHash generates a dense Gaussian that is only  $6.25\%$  ( $c = 8$ ) the size of that for Federated SLIDE. In summary, PGHash(-D) attains similar performance to Federated SLIDE while storing less than a tenth of the parameters.
Induced Sparsity. PGHash(-D) induces a large amount of sparsity through its LSH process. This is especially prevalent in large-scale recommender networks, where the number of labels for each data point is a minuscule fraction of the total output layer size (e.g. Delicious-200K has on average only 75.54 labels per point). PGHash(-D) performs well at identifying this small subset of neurons as training progresses. As one can see in Figure 6a, even when PGHash is allowed to select all possible neurons (i.e., no compression  $CR = 1$ ), it still manages to select fewer than  $1\%$  of the total neurons after only 50 iterations of training over Delicious-200K. For Amazon-670K, PGHash-D requires less than  $30\%$  of the total neurons for the majority of training. Therefore, PGHash(-D) greatly increases sparsity within the NN, improving the computational efficiency of the algorithm by reducing the number of floating point operations required in the forward and backward training.
In this work, we present a new hashing family, PGHash, which enables the generation of multiple LSH hash tables using a single base projection of a massive target weight. These hash tables can be used to dynamically select neurons that are similar to the layer input. This alleviates memory, communication, and privacy costs associated with conventional LSH-training approaches. As a proof of concept, we demonstrate that (i) the PGHash family is effective at mimicking SimHash and (ii) our framework is competitive against other, memory-inefficient, LSH-based federated training baselines of large-scale recommender networks. For future work, we intend to explore how multi-layer PGHash pruning affects model performance and incorporate learnable hashes as in the Mongoose [7] pipeline.
**Limitations.** Our theory indicates that PGHash is useful for detecting high angular similarity, but could prove unreliable for differentiating between intermediately dissimilar vectors. Additionally, LSH-based pruning has only shown success on large classification layers or attention layers in transformers [22]. When considering broader impacts, large-scale recommender networks, and any subsequent improvements to their design, can be used for strategically negative advertising purposes.
Rabbani, Bornstein, and Huang are supported by the National Science Foundation NSF-IIS-FAI program, DOD-ONR-Office of Naval Research, DOD Air Force Office of Scientific Research, DOD-DARPA-Defense Advanced Research Projects Agency Guaranteeing AI Robustness against Deception (GARD), Adobe, Capital One, and JP Morgan faculty fellowships. Rabbani was supported in part by NSF award DGE-1632976.
[1] Zareen Alamgir, Farwa K Khan, and Saira Karim. Federated recommenders: methods, challenges and future. Cluster Computing, 25(6):4075-4096, 2022.
[2] Alexandr Andoni and Piotr Indyk. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. Communications of the ACM, 51(1):117-122, 2008.
[3] Alexandr Andoni, Piotr Indyk, Thijs Laarhoven, Ilya Razenshteyn, and Ludwig Schmidt. Practical and optimal lsh for angular distance. Advances in neural information processing systems, 28, 2015.
[4] Kush Bhatia, Kunal Dahiya, Himanshu Jain, Anshul Mittal, Yashoteja Prabhu, and Manik Varma. The extreme classification repository: Multi-label datasets and code. URL http://manikvarma.org/downloads/XC/XMLRepository.html, 2016.
[5] Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pages 21-29. IEEE, 1997.
[6] Moses S Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the thirty-fourth annual ACM symposium on Theory of computing, pages 380-388, 2002.
[7] Beidi Chen, Zichang Liu, Binghui Peng, Zhaozhuo Xu, Jonathan Lingjie Li, Tri Dao, Zhao Song, Anshumali Shrivastava, and Christopher Re. Mongoose: A learnable lsh framework for efficient neural network training. In International Conference on Learning Representations, 2020.
[8] Beidi Chen, Tharun Medini, James Farwell, Charlie Tai, Anshumali Shrivastava, et al. Slide: In defense of smart algorithms over hardware acceleration for large-scale deep learning systems. Proceedings of Machine Learning and Systems, 2:291-306, 2020.
[9] Beidi Chen and Anshumali Shrivastava. Revisiting winner take all (wta) hashing for sparse datasets. arXiv preprint arXiv:1612.01834, 2016.
[10] Beidi Chen and Anshumali Shrivastava. Densified winner take all (wta) hashing for sparse datasets. In Uncertainty in artificial intelligence, 2018.
[11] Lars Engebretsen, Piotr Indyk, and Ryan O'Donnell. Derandomized dimensionality reduction with applications. 2002.
[12] Michel X Goemans and David P Williamson. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM (JACM), 42(6):1115-1145, 1995.
[13] Arjun K Gupta and Saralees Nadarajah. Handbook of beta distribution and its applications. CRC press, 2004.
[14] Piotr Indyk. Stable distributions, pseudorandom generators, embeddings, and data stream computation. Journal of the ACM (JACM), 53(3):307-323, 2006.
[15] Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pages 604-613, 1998.
[34] Ian En-Hsu Yen, Satyen Kale, Felix Yu, Daniel Holtmann-Rice, Sanjiv Kumar, and Pradeep Ravikumar. Loss decomposition for fast learning in large output spaces. In International Conference on Machine Learning, pages 5640-5649. PMLR, 2018.
[35] Felix Yu, Sanjiv Kumar, Yunchao Gong, and Shih-Fu Chang. Circulant binary embedding. In International conference on machine learning, pages 946-954. PMLR, 2014.
[36] Shijie Zhang, Hongzhi Yin, Tong Chen, Zi Huang, Quoc Viet Hung Nguyen, and Lizhen Cui. Pipattack: Poisoning federated recommender systems for manipulating item promotion. arXiv preprint arXiv:2110.10926, 2021.
[37] Andrew Ziegler, Eric Christiansen, David Kriegman, and Serge Belongie. Locally uniform comparison image descriptor. Advances in Neural Information Processing Systems, 25, 2012.
Large-Scale Distributed Learning via Private On-Device LSH Appendix
A Experiment details
In this section, we provide deeper background into how our experiments were run as well as some additional results and observations. We first detail the hyper-parameters we used in order to reproduce our results. Then, we provide additional comments and details into our sampling approach. Finally, we describe some of the interesting observations we encountered while solving the Amazon-670K and Wiki-325K recommender system problems.
A.1 Experiment hyper-parameters
Below, we detail the hyper-parameters we used when running our federated experiments.
Table 1: Hyper-parameters for Federated Experiments (PGHash and Federated SLIDE).
Dataset	Algorithm	Hash Type	LR	Batch Size	Steps per LSH	k	c	Tables	CR
Delicious-200K	PGHash	PGHash	1e-4	128	1	8	8	50	1
Delicious-200K	SLIDE	SimHash	1e-4	128	1	8	N/A	50	1
Amazon-670K	PGHash	PGHash-D	1e-4	256	50	8	8	50	1
Amazon-670K	SLIDE	DWTA	1e-4	256	50	8	N/A	50	1
Wiki-325K	PGHash	PGHash-D	1e-4	256	50	5	16	50	1
Wiki-325K	SLIDE	DWTA	1e-4	256	50	5	N/A	50	1
What one can immediately see from Table 1, is that we use a Densified Winner Take All (DWTA) variant of PGHash for the larger output datasets Amazon-670K and Wiki-325K. As experienced in [8, 7, 24], SimHash fails to perform well on these larger datasets. We surmise that SimHash fails due in part to its inability to select a large enough number of neurons per sample (we observed this dearth of activated neurons empirically). Reducing the hash length  $k$  does increase the number of neurons selected, however this decreases the accuracy. Therefore, DWTA is used because it utilizes more neurons per sample on these larger problems and also still achieves good accuracy.
Table 2: Hyper-parameters for Compression Experiments (PGHash).
Dataset	Algorithm	Hash Type	LR	Batch Size	Steps per LSH	k	c	Tables	CR
Delicious-200K	PGHash	PGHash	1e-4	128	1	8	8	50	0.1/0.25/1
As a quick note, we record test accuracy every so often (around 100 iterations for Delicious-200K and Amazon-670K). Similar to [8], to reduce the test accuracy computations (as the test sets are very large) we compute the test accuracy of 30 randomly sampled large batches of test data.
A.2 Neuron sampling
Speed of Neuron Sampling. In Table 3 we display the time it takes to perform LSH for PGHash given a set number of tables. These times were collected locally during training. The entries in Table 3 denote the time it takes to compute hashing of the final layer weights  $w_{i}$  and each sample  $x$  in batch  $M$  as well as vanilla-style matching (neuron selection) for each sample.
Table 3: Average LSH time for PGHash over a range of tables. We compute the average  $\mu$  time (and standard deviation  $\sigma$ ) it takes for PGHash to perform vanilla sampling (exact matches) between the hash codes of sample  $x$  and each weight  $w_{i}$  in the final dense layer. Times are sampled for PGHash on Delicious-200k for batch size  $M = 128$ ,  $k = 9$ , and  $c = 8$  for one device.
Method	1 table (seconds)	50 tables (seconds)	100 tables (seconds)
PGHash	μ = 0.0807, σ = 0.0076	μ = 3.1113, σ = 0.0555	μ = 6.2091, σ = 0.1642
SLIDE	μ = 0.0825, σ = 0.0099	μ = 3.2443, σ = 0.1671	μ = 6.2944, σ = 0.0689
We find in Table 3 that PGHash achieves near sub-linear speed with respect to the number of tables  $\tau$  and slightly outperforms SLIDE. PGHash edges out SLIDE due to the smaller matrix multiplication cost, as PGHash utilizes a smaller random Gaussian matrix (size  $c \times c$ ). The speed-up over SLIDE will