SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(b) 4 Device Results.
Figure 7: Wiki-325K PGHash-D. We record model accuracy of a large recommendation network on Wiki-325K trained in a federated setting. PGHash-D matches or outperforms the convergence of SLIDE without requiring LSH to be performed by the central server.
become more significant when the input layer is larger (as  $d = 128$  in our experiments). Therefore, PGHash obtains superior sampling performance to SLIDE.
Hamming Distance Sampling. An alternative method to vanilla sampling is to instead select final layer weights (neurons)  $w_{i}$  which have a small Hamming distance relative to a given sample  $x$ . As a refresher, the Hamming distance simply computes the number of non-matching entries between two binary codes (strings). If two binary codes match exactly, then the Hamming distance is zero. In this sampling routine, either (i) the top-k weights  $w_{i}$  with the smallest Hamming distance to sample  $x$  are selected to be activated or (ii) all weights  $w_{i}$  with a Hamming distance of  $\beta$  or smaller to sample  $x$  are selected to be activated. Interestingly, the vanilla-sampling approach we use in our work is equivalent to using  $\beta = 0$  in (ii).
In either of the scenarios listed above, hash codes for  $w_{i}$  and  $x$  are computed as done in PGHash(-D). From there, however, the hash code for  $x$  is compared to the hash codes for all final layer weights in order to compute the Hamming distance for each  $w_{i}$ . The process of computing  $n$  Hamming distances for each sample  $x$  is very expensive (much harder than just finding exact matches). That is why our work, as well as [8, 7], use vanilla sampling instead of other methods.
A.3 Amazon-670K and Wiki-325K experiment analysis
Sub-par SimHash Performance. SimHash is known to perform worse than DWTA on Amazon-670K and Wiki-325K. Utilizing SimHash for these experiments is unfair as it is shown by [8, 7], for example, that DWTA achieves much higher performance on Amazon-670K. For this reason, DWTA is the chosen hash function in [8] for Amazon-670K experiments. To verify this observation, we performed experiments on Amazon-670K with PGHash (not PGHash-D) and SLIDE (with a SimHash hash function). Table 4 displays the SimHash approach for Amazon-670K.
Table 4: PGHash and SLIDE performance on Amazon-670K using SimHash. Accuracy across the first 5,000 iterations for a single device. Batch size  $M = {1024},k = 8$  ,and  $c = 8$  .
Iteration	SLIDE	PGHash
1,000	10.82%	10.04%
2,000	18.27%	15.99%
3,000	21.83%	19.51%
4,000	23.72%	21.65%
5,000	25.08%	23.38%
As shown in Table 4, even with a much larger batch size, SLIDE and PGHash are unable to crack  $30\%$  on Amazon-670K. We would like to note that using a smaller batch size (like the  $M = 256$  value we use in our Amazon-670K experiments) resulted in an even further drop in accuracy. These empirical results back-up the notion that SimHash is ill-fit for Amazon-670K.
Wiki-325K Performance. In Figure 7, we showcase how PGHash-D performs on Wiki-325K. Quite similar to the Amazon-670K results (shown in Figure 5), PGHash-D almost exactly matches up with SLIDE. In order to map how well our training progresses, we periodically check test accuracies. However, since the test set is very large, determining test accuracies over the entire test set is infeasible due to time constraints on the cluster. Therefore, we determine test accuracies over 30 batches of test data as a substitute as is done in [8, 7]. For Delicious-200K and Amazon-670K the entire test set accuracies matched the randomly sampled batches, however the randomly sampled batches underestimate the true test accuracies for Wiki-325K. For Wiki-325K, the true test accuracy ran about  $5\%$  greater than the sampled test accuracy values.
Matching Full-Training Performance. Along with the failure for SimHash to perform well on Amazon-670K and Wiki-325K, SLIDE and PGHash(-D) are unable to match the performance of full-training on these data-sets. This is observed empirically for Amazon-670K by GResearch in the following article https://www.gresearch.co.uk/blog/article/implementing-slide/. We surmise that the failure of SLIDE and PGHash(-D) to match full-training performance on Amazon-670K and Wiki-325K arises due to the small average labels per point in these two data-sets (5.45 and 3.19 respectively). Early on in training, SLIDE and PGHash(-D) do not utilize enough activated neurons. This is detrimental to performance when there are only a few labels per sample, as the neurons corresponding to the true label are rarely selected at the beginning of training (and these final layer weights are tuned much slower). In full-training, the true neurons are always selected and therefore the final layer weights are better adjusted from the beginning. We also note that [33] requires a hidden layer size of 1024 for a distributed version of SLIDE to achieve improved test accuracies for Amazon-670K. Thus, increasing the hidden layer size may have improved our performance (we kept it as 128 to match the original SLIDE paper [8]).
B PHash: angle versus Hamming distance
In this section, we visually explore the degree to which PGHash is a consistent estimator of angular similarity. Specifically, let  $x, y \in \mathbb{R}^d$ : then we know by Theorem 1 that  $\mathcal{H}^{PG}(c,d)$  is an LSH for  $\cos(x_c, y_c)$ . We demonstrate that in the unit vector regime,  $\theta_c = \arccos(\cos(x_c, Y_c))$  is an acceptable surrogate for  $\theta = \arccos(x, Y)$ , where  $Y = \{y^i\}_{i=1}^N$  and  $Y_c = \{y_c^i\}_{i=1}^N$ .
(a)  $c = 5$
(d)  $c = 5$
(b)  $c = 10$
(e)  $c = 10$
(c)  $c = 25$
(f)  $c = 25$
Figure 8: Angle/Hamming Distance as a function of sketch dimension. The average Hamming distance between a PGHashed fixed unit vector  $x \in \mathbb{R}^{100}$  and a collection of vectors  $y_{i} \in \mathbb{R}^{100}$  which form different angles with  $x$ . Increasing sketch dimension  $c$  smooths and reduces the variance of the scatter towards linear correlation. Furthermore, the Hamming scales linearly with  $c$ , improving discernibility. (a)-(c) & (d)-(f) are independent series.
(a) 5 tables
(b) 10 tables
(c) 50 tables
Figure 9: Angle/Hamming Distance as a function of tables The average Hamming distance between a PGhaled fixed unit vector  $x\in \mathbb{R}^1 00$  and a collection of vectors  $y_{i}\in \mathbb{R}^{100}$  which form different angles with  $x$  and fixed sketch dimension  $c = 10$ . Increasing the number of tables reduces variance.
C Additional proofs
Fact 3. Let  $x, y, e_1, e_2$  be  $d$ -dimensional unit vectors such that the  $e_i$  lie on the unit circle contained with the plane spanned by  $x$  and  $y$  (denoted as  $S_{x,y}$ ) and  $e_1 \perp e_2$ . Consider the point  $v$  on  $S_{x,y}$  such that the line through it bisects the angle of the lines passing through  $x$  and  $y$ . Let  $\eta = \arccos(\cos(v, e_1))$ . Denote  $\theta = \frac{1}{2} \arccos(\cos(x, y))$ . Then we may write  $x = \cos(\eta + \theta)e_1 + \sin(\eta + \theta)e_2$  and  $y = \cos(\eta - \theta)e_1 + \sin(\eta - \theta)e_2$ .
Proof. Let  $\theta = \frac{1}{2}\arccos (\cos (x,y))$  where  $x,y\in \mathbb{S}^{d - 1}$  and  $A = B^{\top}B$ . The cosine similarity between  $x_{c} = Bx$  and  $y_{c} = By$  (for  $B$  correspondent to a  $(d,c)$ -folding), is expressible as
$$
\cos \left(x _ {c}, y _ {c}\right) = \frac {x ^ {\top} B ^ {\top} B y}{\left(x ^ {\top} B ^ {\top} B x\right) \left(y ^ {\top} B ^ {\top} B y\right)} = \frac {x ^ {\top} A y}{\sqrt {\left(x ^ {\top} A x\right) \left(y ^ {\top} A y\right)}}. \tag {1}
$$
Consider the SVD  $B = U D V^{\top}$  where  $U$  and  $V$  are orthogonal and  $D$  is  $c \times d$  rectangular diagonal matrix. We have then that  $A = B^{\top}B = V\hat{D}^{2}V^{\top}$ . (Here  $\hat{D}$  is now a square diagonal matrix containing squared  $D_{ii}$  along the diagonal and 0 everywhere else.) Notice that choice of  $U$  nor the ordering of columns  $v_{i}$  of  $V$  affects the angle calculation in Equation 1. First, we re-order the columns of  $V$  so as to order the diagonal entries  $d_{i}$  of  $D$  (i.e., the squared singular values) in decreasing order, and as an abuse of notation set  $B = \frac{1}{d_1} DV^{\top}$ . Denoting  $\hat{\lambda}_i = d_i / d_1$  for  $1 \leq i \leq n$ , we have that  $Bv_{i} = \hat{\lambda}_{i}e_{i}$ . (By construction of  $B$  we have that  $d_{i} \in \{\frac{d}{c}, 0\}$ , therefore,  $\hat{\lambda}_i \in \{1, 0\}$ )
Consider  $B$  acting on  $S^{d - 1}$ : it scales each dimension by  $\hat{\lambda}_i$ , thus (as with any linear transformation of a sphere), transforms it into an ellipsoid, with  $c$  principal axes determined by the  $v_{i}$ . The greatest possible distance from the origin to the ellipsoid  $BS^{d - 1}$  is 1 while the shortest possible distance is 0. Now consider the unit circle  $S_{x,y} = \{v\in \operatorname {span}(x,y):\| v\| = 1\}$ . We have that  $BS_{x,y}\subset BS^{d - 1}\cap BU$  is an ellipse (since the intersection of an ellipsoid and plane is always an ellipse).
Choose unit  $w_{1}$  and  $w_{2}$  belonging to  $S_{x,y}$  such that  $w_{1} \perp w_{2}$ . By Fact 3, we may parameterize our vectors as  $x = \cos(\eta - \theta)w_{1} + \sin(\eta - \theta)w_{2}$  and  $y = \sin(\eta + \theta)w_{1} + \sin(\eta + \theta)w_{2}$ , where  $\eta$  is the angle made with  $w_{1}$  with the bisector of  $x$  and  $y$ . By assumption,  $||Bw|| \geq \alpha$  (the minimal shrinking factor of  $B$  on  $S_{x,y}$ ) for some positive  $\alpha$ . Denoting  $\lambda = \frac{d}{c}$  (the maximal stretching factor of  $B$  on  $S_{x,y}$ ), we have that the angle between  $Bx$  and  $By$  is upper-bounded by
$$
f (\eta) = \arctan \left(\frac {\alpha}{\lambda} \tan (\eta + \theta)\right) - \arctan \left(\frac {\alpha}{\lambda} \tan (\eta - \theta)\right) \tag {2}
$$
The numerator of  $\frac{df}{d\eta}$  is  $\beta (1 - \beta)(1 + \beta)\sin (2\theta)\sin (2\eta)$  where  $\beta = \alpha /\lambda$ . The derivative is trivially 0 if (1)  $\beta = 0$ , (2)  $\beta = 1$ , or (3)  $\theta = 0$ . (1) will not occur as we assume that  $S_{x,y}$  does not contain a 0-eigenvector of  $A = B^{\top}B$ . (2) can only occur if  $A$  is a multiple of the identity matrix (which it is not by construction), and (3) implies that  $x$  and  $y$  are parallel, in which case their angle will not be
distorted. Aside from these pathological cases, the critical points occur at  $\eta = 0, \pi / 2$ . We have then that  $\cos(Bx, By)$  lives between  $\cos(f(0)) = \frac{1 - \beta^2 \tan^2 \theta}{1 + \beta^2 \tan^2 \theta}$  and  $\cos(f(\pi / 2)) = -\frac{\tan^2 \theta^2 - \beta^2}{\tan^2 \theta^2 + \beta^2}$ .
Remark. The constant  $\beta$  has an enormous influence on the bounds in Theorem 2. The smaller the  $\alpha$  (i.e., shrinking of  $||w||$ ), the greater the bounds on distortion. Although we have imposed constraints on  $x, y$ , if we treat them as any possible pair of random unit vectors, then the  $w$  in  $S_{x,y}$  effectively becomes a random unit vector as well. We can exactly characterize the distribution of  $||BX||$  where  $X$  denotes a random variable which selects a  $d$ -dimensional unit vector uniformly at random.
Proof. We can sample a  $d$ -dimensional vector uniformly at random from the unit sphere  $S^{d-1}$  by drawing a  $d$ -dimensional Gaussian vector with iid entries and normalizing. Let us represent this as the random variable  $X = Z' / ||Z'||$  where  $Z' \sim \mathcal{N}(0, I_d)$ . Consider a  $(c, d)$ -folding matrix  $B$ , i.e., a  $d/c$  horizontal stack of  $c \times c$  identity matrices (let us assume  $c|d$ ). We are interested in determining the distribution of  $||BX||^2$ . For ease of notation, consider the permutation  $Z$  of  $Z'$  where  $Z_i = Z'_{(\lfloor \frac{d/c}{i} \rfloor - 1) * (d/c) + i \pmod{d/c}}$ . Since this permutation is representable as an orthogonal matrix  $P$  (and multi-variate Gaussians are invariant in distribution under orthogonal transformations), we may instead consider  $X := P(Z' / ||Z'||)^2 = Z / ||Z||^2$ . We may write the norm-squared as
$$
\left\| B X \right\| ^ {2} = \frac {\left(Z _ {1} + \cdots + Z _ {d / c}\right) ^ {2}}{\left\| Z \right\| ^ {2}} + \frac {\left(Z _ {d / c + 1} + \cdots + Z _ {2 d / c}\right) ^ {2}}{\left\| Z \right\| ^ {2}} + \dots + \frac {\left(Z _ {(c - 1) (d / c) + 1} + \cdots + Z _ {d}\right) ^ {2}}{\left\| Z \right\| ^ {2}}. \tag {3}
$$
Consider the first term  $\frac{(Z_1 + \cdots + Z_{d/c})^2}{||Z||^2}$ . First note that for any unit vector  $u$ , the distribution of  $\frac{(u^\top Z)^2}{||Z||^2}$  does not depend on choice of  $u$ . Consider the unit vector  $u'$  then which contains  $\sqrt{d/c}$  in the first  $d/c$  entries and 0 otherwise. Then  $\frac{(u'^{\top}Z)^2}{||Z||^2}$  is equivalent to  $d/c$  times our first term. Of course, since  $\frac{(e_1^\top Z)^2}{||Z||^2}$  has the same distribution as  $\frac{(u'^{\top}Z)^2}{||Z||^2}$ , we have by transitivity that  $\frac{Z_1^2}{||Z||^2} \stackrel{d}{=} (n/q)\frac{(Z_1 + \cdots + Z_{d/c})^2}{||Z||^2}$ .
By extending the discussion above to the other terms, and by their independence with respect to rotation of  $Z$  (since their numerators contain squared sums of mutually disjoint  $Z$  coordinates), we have that
$$
\left\| B X \right\| ^ {2} \stackrel {d} {=} \frac {d}{c} \cdot \frac {Z _ {1} ^ {2} + Z _ {d / c} ^ {2} + Z _ {2 d / c} ^ {2} + \cdots + Z _ {d} ^ {2}}{\left\| Z \right\| ^ {2}}. \tag {4}
$$
The distribution of  $\frac{Z_1^2 + Z_{d/c}^2 + Z_{2d/c}^2 + \cdots + Z_d^2}{||Z||^2}$  is well-known to follow a  $\mathrm{Beta}\left(\frac{c}{2}, \frac{d-c}{2}\right)$  distribution [13]. In totality,  $||BX||^2 \stackrel{d}{=} \frac{d}{c} \mathrm{Beta}\left(\frac{c}{2}, \frac{d-c}{2}\right)$ . However, we will move to the four parameter description of this scaled Beta distribution which is  $\mathrm{Beta}\left(\frac{c}{2}, \frac{d-c}{2}, 0, \frac{d}{c}\right)$ . The pdf and expected value follows by the usual statistical descriptions of this distribution, which can also be found in [13].
Figure 10 depicts how  $(d, c)$ -foldings affect the norms of unit vectors.
D Additional theory
In this section, we provide additional theory relevant to SimHash.
We present several well-known results regarding SimHash.
Proposition 2 (SimHash estimation). Let  $x, y \in \mathbb{S}$ , i.e., unit  $d$ -dimensional vectors. Denote  $\theta = \arccos(|\cos(x, y)|)$ . Let  $v \in S^d$  be a unit vector drawn uniformly at random (according to the Haar measure, for example). Then,
$$
P r \left[ s g n \left(v ^ {\top} x\right) \neq s g n \left(v ^ {\top} y\right) \right] = \frac {\theta}{\pi}. \tag {5}
$$
Figure 10: Distribution of folded norms. 100k randomly drawn unit vectors  $(d = 128)$  are folded down to length 16 by are usual  $(d, c)$ -folding procedure. Depicted is a binned histogram of the norms. As predicted by the statistical description of  $||BX||^2$ , where  $X$  is a randomly drawn unit vector, the mass is centered at 1, i.e., most norms are preserved. Empirically we observe that folded rarely exceed  $\sqrt{128}16$ , although the theoretical support is  $[0, 8]$ : this concurs with the pdf.
Proof. We reproduce the argument of [12]. We have by symmetry that  $Pr[\operatorname{sgn}(v^{\top}x) \neq \operatorname{sgn}(v^{\top}y)] = 2Pr[v^{\top}x > 0, v^{\top}y < 0]$ . The set  $\mathcal{U} = \{v \in S^{d} : v^{\top}x > 0, v^{\top}y \leq 0\}$  corresponds to the intersection of two half-spaces whose dihedral angle (i.e., angle between the normals of both spaces) is exactly  $\theta$ . Intersecting with the  $d$ -dimensional unit sphere produces a subspace of measure  $\frac{\theta}{2\pi}$ , therefore,  $2Pr[v^{\top}x > 0, v^{\top}y < 0] = \frac{\theta}{\pi}$ , completing the argument.
Corollary 2. Let  $v$  instead be a  $d$ -dimensional random Gaussian vector with iid entries  $\sim \mathcal{N}(0,1)$ . Then for  $x, y \in \mathbb{R}^d$ ,
$$
P r \left[ \operatorname {s g n} \left(v ^ {\top} x\right) \neq \operatorname {s g n} \left(v ^ {\top} y\right) \right] = \frac {\theta}{\pi} \tag {6}
$$
Proof. Randomly drawn, normalized Gaussian vectors are well-known to be uniformly distributed on the unit sphere.
In the setup as above, let the  $X$  be a random variable which returns 1 if  $x$  and  $y$  have differing signs when taking the standard inner product with a randomly drawn Gaussian  $v$ . Let  $X_{1}, X_{2}, \ldots, X_{n}$  represent a sequence of independent  $X$  events. Then,
Proposition 3.  $\mathbb{E}\left[\frac{1}{n}\sum_{i = 1}^{n}X_{i}\right] = 1 - \frac{\theta}{\pi}$  and  $\mathbb{V}[X] = \frac{1}{N}\frac{\theta}{\pi} (1 - \frac{\theta}{\pi})$
Given that PGHash is equivalent to a SimHash over  $(d, c)$ -foldings of  $R^d$ , the variance reduction we observe by using multiple tables (Figure 9 is explainable by Proposition 3.