Title: Large-Scale Distributed Learning via Private On-Device LSH
Venue: NeurIPS 2023
Accepted: yes
Keywords: distributed learning

locality-sensitive hashing

recommender systems

compression
Abstract: Locality-sensitive hashing (LSH) based frameworks have been used efficiently to select weight vectors in a dense hidden layer with high cosine similarity to an input, enabling dynamic pruning. 
 While this type of scheme has been shown to improve computational training efficiency, existing algorithms require repeated randomized projection of the full layer weight, which is impractical for computational- and memory-constrained devices. 
 In a distributed setting, deferring LSH analysis to a centralized host is (i) slow if the device cluster is large and (ii) requires access to input data which is forbidden in a federated context. 
 Using a new family of hash functions, we develop the first private, personalized, and memory-efficient on-device LSH framework.
Our framework enables privacy and personalization by allowing each device to generate hash tables, without the help of a central host, using device-specific hashing hyper-parameters (e.g., number of hash tables or hash length).
Hash tables are generated with a compressed set of the full weights, and can be serially generated and discarded if the process is memory-intensive.
This allows devices to avoid maintaining (i) the fully-sized model and (ii) large amounts of hash tables in local memory for LSH analysis. We prove several statistical and sensitivity properties of our hash functions, and experimentally demonstrate that our framework is competitive in training large scale recommender networks compared to other LSH frameworks which assume unrestricted on-device capacity.

Large-Scale Distributed Learning via Private On-Device Locality-Sensitive Hashing

Abstract

Locality-sensitive hashing (LSH) based frameworks have been used efficiently to select weight vectors in a dense hidden layer with high cosine similarity to an input, enabling dynamic pruning. While this type of scheme has been shown to improve computational training efficiency, existing algorithms require repeated randomized projection of the full layer weight, which is impractical for computational- and memory-constrained devices. In a distributed setting, deferring LSH analysis to a centralized host is (i) slow if the device cluster is large and (ii) requires access to input data which is forbidden in a federated context. Using a new family of hash functions, we develop one of the first private, personalized, and memory-efficient on-device LSH frameworks. Our framework enables privacy and personalization by allowing each device to generate hash tables, without the help of a central host, using device-specific hashing hyper-parameters ( e.g. number of hash tables or hash length). Hash tables are generated with a compressed set of the full weights, and can be serially generated and discarded if the process is memory-intensive. This allows devices to avoid maintaining (i) the fully-sized model and (ii) large amounts of hash tables in local memory for LSH analysis. We prove several statistical and sensitivity properties of our hash functions and experimentally demonstrate that our framework is competitive in training large-scale recommender networks compared to other LSH frameworks which assume unrestricted on-device capacity.

1 Introduction

Locality-sensitive hashing (LSH) has proven to be a remarkably effective tool for memory- and computationally-efficient data clustering and nearest neighbor search [6, 15, 2]. LSH algorithms such as SimHash [6] can be used to search for vectors in collection W ⊂ R d of massive cardinality which will form a large inner product with a reference vector x ∈ R d . This procedure, known as maximum inner product search (MIPS) [27], has been applied to neural network (NN) training. In NN training, the weights of a dense layer that are estimated to produce a large inner product with the input (thereby, a large softmax, for example) are activated while the remainder is dropped out.

While LSH-based pruning greatly reduces training costs associated with large-scale models, popular frameworks such as SLIDE [8] and Mongoose [7] cannot be deployed in distributed settings over

memory-constrained devices such as GPUs or mobile phones for the following reasons: (a) required maintenance of a large target layer in memory and (b) access to the input is needed to conduct LSH.

With many modern NN architectures reaching billions of parameters in size, requiring resource- constrained devices to conduct LSH analysis over even part of such a large model is infeasible as it requires many linear projections of massive weights. The hope of offloading this memoryand computationally-intensive task to a central host in the distributed setting is equally fruitless. LSH-based pruning cannot be conducted by a central host as it requires access to either local client data or hashed mappings of such data. Both of these violate the fundamental host-client privacy contract, especially in a federated setting [20]. Therefore, in order to maintain privacy, devices are forced to conduct LSH themselves, returning us back to our original drawback in a vicious circle. We raise the following question then:

Can a resource-constrained device conduct LSH-like pruning of a large dense layer without ever needing to see the entirety of its underlying weight?

This work makes the following contributions to positively resolve this question:

(1) Introduce a novel family of hash functions, PGHash, for the detection of high cosine similarity amongst vectors. PGHash improves upon the efficiency of SimHash by comparing binarized random projections of folded vectors. We prove several statistical properties about PGHash, including angle/norm distortion bounds, and that it is an LSH family.

(2) Present an algorithmic LSH framework, leveraging our hash functions, which allows for private, personalized, and memory-efficient distributed/federated training of large-scale recommender networks via dynamic pruning.

(3) Showcase experimentally that our PGHash-based framework is able to efficiently train large-scale recommender networks. Our approach is competitive against a distributed implementation of SLIDE [8] using full-scale LSH. Furthermore, where entry-magnitude similarity is desired over angular similarity (training over Amazon-670K, for example), we empirically demonstrate that using our DWTA [9] variant of PGHash, PGHash-D, matches the performance of using full-scale DWTA.

2 Related work

LSH Families. Locality-sensitive hashing families have been used to efficiently solve the approximate nearest neighbors problem [16, 15, 2]. SimHash [6], based on randomized hyperplane projections, is used to estimate cosine similarity. Each SimHash function requires a significant number of random bits if the dimensionality of each target point is large. However, bit reduction using Nisan's pseudorandom generator [23] is often suggested [11, 14]. MinHash [5], a competitor to SimHash [28], measures Jaccard similarity between binary vectors and has been used for document classification. The Winner Take All (WTA) hash [32] compares the ordering of entries by magnitude (corresponding to Kendall-Tau similarity); such comparative reasoning has proven popular in vision applications [25]. However, it was observed that WTA was ineffective at differentiating highly sparse vectors leading to the development of Densified WTA (DWTA) [9]. Since MinHash, WTA, and DWTA are better suited for binary vector comparison, and we require comparison over real-valued vectors, PGHash is founded on SimHash.

Hash-based Pruning. One of the earlier proposals of pruning based on input-neuron angular similarity via LSH tables is in [29], where a scheme for asynchronous gradient updates amongst multiple threads training over a batch along with hashed backpropagation are also outlined. These principles are executed to great effect in both the SLIDE [8] and Mongoose [7] frameworks for training extreme-scale recommender networks. Mongoose improves SLIDE by using an adaptive scheduler to determine when to re-run LSH over a layer weight, and by utilizing learnable hash functions. Both works demonstrated that a CPU using LSH-based dynamic dropout could achieve competitive training complexity against a GPU conducting fully-dense training. Reformer [22] uses LSH to reduce the memory complexity of self-attention layers.

Distributed Recommender Networks. Several works which prune according to input-neuron angular similarity estimations via LSH utilize multiple workers on a single machine [29, 24, 7, 8]. Federated training of recommender networks is an emerging topic of interest, with particular interest in personalized training [18, 26] malicious clients [30, 36], and wireless unreliability [1]. D-SLIDE [33], which is the federated version of SLIDE, eases local on-device memory and computational requirements by sharding the network across clients. However, in the presence of low client numbers,

S

1 0 0

0 1 0

0 1 0

1 0 0

(a) SimHash

B

S

1 1 1

1 1 0

0 1 0

1 0 0

(b) PGHash

the proportion of the model owned per device can still be taxing, whereas our compression is independent of the number of federated agents. In [31], clients query the server for weights based on the results of LSH conducted using server-provided hash functions. We regard this complete server control over the hashing family, and therefore access to hash-encoding of local client data, as non-private and potentially open to honest-but-curious attacks.

3 Preliminaries

Let W = { w 1 , w 2 , . . . , w n } ⊂ R d be the weights of a dense hidden layer and x ∈ R d be the input. For brevity, we refer to W ∈ R d × n as the weight of the layer. In particular, w i ∈ R d corresponds to the i th neuron of the layer. We assume that the layer contains dn parameters. Within our work, we perform MIPS, as we select weights that produce large inner products with x . Mathematically, we can begin to define this by first letting p = max w ⊤ i x for 1 ≤ i ≤ n . For 0 < ϵ < 1 we are interested in selecting S ⊂ W , such that for ∀ w i ∈ S , w ⊤ i x > ϵp . The weights of S will pass through activation while the rest are dropped out, reducing the computational complexity of the forward and backward pass through this layer. As detailed in Section 4 and illustrated in Figure 1, we will determine S by estimating angles with a projection BW = { Bw i } n i =1 , with B ∈ R c × d such that c << d .

Locality-sensitive Hashing (LSH). LSH [2] is an efficient framework for solving the ϵ -approximate nearest neighbor search (NNS) problem:

Definition 1 ( ϵ -NNS) . Given a set of points P = { p 1 , p 2 , . . . , p n } in a metric space M and similarity funcion Sim : M×M→ R over this space, find a point p ∈ P , such that for a query point q ∈ X and all p ′ ∈ P , we have that Sim ( p, q ) ≤ (1 + ϵ ) Sim ( p ′ , q ) .

It is important to note that the similarity function Sim ( · ) need not be a distance metric, but rather any general comparison mapping. Popular choices include Euclidean distance and cosine similarity, the latter of which is the primary focus of this paper. The cosine similarity for x, y ∈ R d is defined as cos( x, y ) ≜ x ⊤ y/ ( || x || 2 || y || 2 ) . We can frame the MIPS problem described previously as an ϵ -NNS one if we assume that the weights w i ∈ W are of unit length. Thus, we are searching for ϵ -nearest neighbors in W of the query x according to cosine similarity.

Consider a family H containing hash functions of the form h : M→S , where S is a co-domain with significantly lower feature dimensionality than M . We say that H is locality-sensitive if the hashes of a pair of points x, y in M , computed by an h ∈ H (selected uniformly at random), have a higher collision (matching) probability in S the more similar x and y are according to Sim . We now formally define this notion following [15].

Definition 2 (Locality-sensitive Hashing) . A family H is called ( S 0 , ϵS 0 , p 1 , p 2 ) -sensitive if for any two points x, y ∈ R d and h chosen uniformly at random from H satisfies the following, 1. if Sim ( x, y ) ≥ S 0 ⇒ Pr ( h ( x ) = h ( y )) ≥ p 1 , 2. if Sim ( x, y ) ≤ ϵS 0 ⇒ Pr ( h ( x ) = h ( y )) ≤ p 2 .

For an effective LSH, p 1 < p 2 and ϵ < 1 is required. An LSH family allows us to conduct a similarity search over a collection of vectors through a comparison of their hashed mappings. Of course, locality loss is inevitable if Sim is a dimension-lowering projection. Through a mixture of increased precision

(raising the output dimension of H ) and repeated trials (running several trials over independently chosen h ), we may tighten the correspondence between Sim and matches over H , following the spirit of the Johnson-Lindenstrauss Lemma [19].

SimHash. A popular LSH algorithm for estimating cosine similarity is SimHash, which uses signed random projections [6] as its hash functions. Specifically, for a collection of vectors W ⊂ R d , the SimHash family H Sim consists of hash functions h v , each indexed by a random Gaussian vector v ∼ N (0 , I n ) , i.e., an n -dimensional vector with iid entries drawn from N (0 , 1) . For x ∈ R n , we define the hash mapping h v ( x ) := sgn ( v ⊤ x ) . Here, we modify sgn ( x ) to return 1 if x > 0 , else it returns 0. For Gaussian v chosen uniformly at random and fixed x, y ∈ R d , we have

Pr ( h v ( x ) = h v ( y )) = 1 -arccos( x ⊤ y || x ||·|| y || ) π . This hashing scheme was popularized in [12] as part of a randomized approximation algorithm for solving MAX-CUT. Notice that the probability of a hashed pair matching is monotonically increasing with respect to the cosine similarity of x and y , satisfying Definition 2. More precisely, if we set S 0 = cos( x, y ) then H Sim is ( S 0 , ϵS 0 , ( 1 -arccos( S 0 π ) ) , ( 1 -arccos( ϵS 0 π ) )) -sensitive [6, 28]. The above discussion considers the sign of a single random projection, but in practice, we will perform multiple projections.

Definition 3 (Hash table) . Let X = { x 1 , . . . , x n } ⊂ R d and V = { v ⊤ 1 , . . . , v ⊤ k } ⊂ N (0 , I d ) , where k is the hash length . Define h V : R d → R k by [ h V ( x )] i = h v i ( x ) where h v i ∈ H Sim for 1 ≤ i ≤ k . For fixed V , the hash table h V ( X ) ∈ R k × n is a binary matrix with columns h V ( x j ) for 1 ≤ j ≤ n .

Following the notation above, we may estimate the similarity between an input q ∈ R and a collection of vectors X ⊂ R d by measuring the Hamming distances (or exact sign matches) between h V ( q ) and columns of h V ( X ) . SimHash is now more discriminatory, as H Sim can separate R d into 2 k buckets corresponding to all possible length k binary vectors (which we refer to as hash codes ). Finally, counting the frequency of exact matches or computing the average Hamming distance over several independently generated hash tables further improves our estimation of closeness. Implementations of the well-known SLIDE framework [8, 24], which utilize SimHash for LSH-based weight pruning, require upwards of 50 tables.

DWTA. Another popular similarity metric is to measure how often high-magnitude entries between two vectors occur at the exact same positions. The densified winner-take-all (DWTA) LSH family [10] estimates this similarity by uniformly drawing k random coordinates over W and recording the position of the highest-magnitude entry. Similar to SimHash, this process is repeated several times, and vectors with the highest frequency of matches are expected to have similar magnitude ordering. This type of comparative reasoning is useful for computer vision applications [37].

4 PGHash

In this section, we develop a family of hash functions H PG which allow for memory-efficient serial generation of hash tables using a single dimensionally-reduced sketch of W . This is in contrast to traditional LSH frameworks, which produce hash tables via randomized projections over the entirety of W . We first present an overview of the Periodic Gaussian Hash (PGHash) followed by its algorithm for distributed settings, an exploration of several statistical properties regarding the local sensitivity of H PG .

PGHash Motivation. As detailed in Section 3, our goal is to efficiently estimate cosine similarity between an input to a layer x ∈ R d and the columns of a large weight matrix W ∈ R d × n . SimHash performs hash table generation by first multiplying a matrix of uniformly drawn Gaussian hyperplanes T V ∈ R c × d with W . The full hash table is computed as h V ( W ) = sgn ( T V W ) . Then, the j th neuron is activated if sgn ( T V x ) = h V ( w j ) for a layer input x .

One can immediately notice that generation of a new hash table h ˜ V ( W ) requires both (i) computation of T ˜ V W which requires access to the fully-sized weights and (ii) the storage of T ˜ V to compute sgn ( T ˜ V x ′ ) for further inputs x ′ . This is problematic for a memory-constrained device, as it would need to maintain both W and T ˜ V to generate further tables and perform dynamic pruning. To solve this issue, we introduce a family of hash functions generated from a single projection BW of W .

4.1 PGHash theory

Definition 4 (Periodic Gaussian Hash) . Assume sketch dimension c << d divides d for simplicity. Let B = [ I c | I c | · · · | I c ] ∈ R c × d , where I c is the c × c identity matrix and | denotes d c concatenations. Let S ∈ R k × c be a random Gaussian matrix with iid entries drawn from N (0 , 1) . We may define a Periodic Gaussian Hash (PGHash) function h PG S : R d → R k by [ h PG S ( x )] i = [ sgn ( SBx )] i for 1 ≤ i ≤ k . We denote the family of all such hash functions as H PG ( c, d ) .

We use the term 'periodic' to describe the hash functions described in Definition 4, since unlike SimHash which projects a point via a fully random Gaussian vector as in SimHash, our projection is accomplished using a repeating concatenation of a length c Gaussian vector. Furthermore, for h PG S ⊂ H PG ( c, d ) , the matrix representation SB is a tiling of a single Gaussian matrix. Notice that we may easily extend the notion of a PGHash of one vector to an entire hash table over multiple vectors following Definition 3. In this manner, we may generate a sequence of hash tables { h PG S i ( W ) } τ i =1 over a weight matrix W simply by drawing random Gaussian matrices S i ∈ R k × c for 1 ≤ i ≤ τ (where k is the hash length) and computing sgn ( S i BW ) .

Extension to DWTA (PGHash-D) Remark. When DWTA (described in Section 3) is preferred over SimHash for similarity estimation, we may modify Definition 4 as follows: B = D ✶ P where P is a d × d random permutation matrix, and D ✶ is a c × d rectangular diagonal matrix with D ii = 1 for 1 ≤ i ≤ c . We denote our hash functions as h PG -D S : R d → R k by h PG -D S ( x ) = max i [ SBx ] i for 1 ≤ i ≤ k , with k ≤ c , where S is now a rectangular permutation matrix which selects k rows of Bx at random. We refer to this scheme as PGHash-D, whereas PGHash refers to Definition 4.

Local Memory Complexity Remark. When generating a new table using PGHash, a device maintains S and needs access to just BW , which costs O ( kc ) and O ( cn ) space complexity respectively. This is much smaller than the O ( kd ) and O ( dn ) local memory requirements of SimHash.

Sensitivity of H PG . In this section, we will explore the sensitivity of H PG ( c, d ) .

Definition 5. Let x ∈ R d and c ∈ R such that c | d . Define the ( d, c ) -folding x c ∈ R c of x as [ x c ] i = ∑ d c j =1 [ x ] i + j · d c . Equivalently, x c = Bx , with B as specified in Definition 4.

Theorem 1. Let x, y ∈ R d . Define the following similarity function Sim d c ( x, y ) ≜ cos( x c , y c ) , where x c , y c are ( d, c ) -foldings of x, y . H PG ( c, d ) is an LSH family with respect to Sim d c .

Proof. Let h v ∈ H PG . This means that for a randomly chosen v ′ ∼ N (0 , I d c ) , v is a c -times concatenation of v . We see that sgn ( v ⊤ x ) = sgn ( v ⊤ x || v ||·|| x || ) = sgn (√ c d || x c || || x || v ′⊤ x c || v ′ |||| x c || ) . Since sgn is unaffected by the positive multiplicative factors, we conclude that sgn ( v ⊤ x ) = sgn ( v ′ ⊤ x c ) . Through symmetric argument, we find sgn ( v ⊤ y ) = sgn ( v ′⊤ y c ) . Since v ′ ∼ N (0 , I c ) , comparing the sign of v ⊤ x to v ⊤ y is equivalent to a standard SimHash over x c and y c , i.e., estimation of cos( x c , y c ) .

Proof. This follows directly from the well-known sensitivity of SimHash [6].

We see that H PG is LSH with respect to the angle between ( d, c ) -foldings of vectors. The use of periodic Gaussian vectors restricts the degrees of freedom (from d to d/c ) of our projections. However, the usage of pseudo-random and/or non-iid hash tables has been observed to perform well in certain regimes[3, 35]. Although H PG ( c, d ) is LSH, is cos( x c , y c ) necessarily an acceptable proxy for cos( x, y ) , in particular, for high angular similarity? Heuristically, yes, for highly-cosine similar vectors: assuming x and y are both unit (since scaling does not affect angle) then we have that || x -y || 2 = 2 -2 cos( x, y ) . If ℓ 2 similarity between x and y is already high, then the ℓ 2 similarity

of their (normalized) ( d, c ) -foldings will also be high, and thus their cosine similarity as well. We now provide a characterization of the angle distortion of a ( d, c ) -folding.

Theorem 2. Let x, y ∈ S d -1 . Assume that neither x nor y vanish under multiplication by B and define the set S x,y := { v ∈ span( x, y ) : || v || = 1 } . We denote the following quantities: λ = d/c , θ := 1 2 arccos(cos( x, y )) , α = inf { c > 0 | ||| Bv || ≥ c, ∀ v ∈ S x,y } , and β = α/λ . Assume α > 0 . Then cos( x c , y c ) lives between 1 -β 2 tan 2 θ 1+ β 2 tan 2 θ and -tan 2 θ 2 -β 2 tan 2 θ 2 + β 2 .

Proof sketch. Consider the unit circle S x,y contained in span ( x, y ) (Let us assume x and y are unit, WLOG). The linear distortion BS x,y is an ellipse containing x c = Bx and y c = By . The length of the axes of this ellipse is determined by the eigenvalues of B ⊤ B . The bounds follow from further trigonometric arguments, by considering when the axes of BS x,y are maximally stretched and shrunk respectively. These distortions are strongly related to λ and β .

We can see that as cos( x, y ) → 0 we have cos( x c , y c ) → 0 . It is natural to consider the distribution of α in Theorem 2 as how extreme shrinking by B (the folding matrix) can greatly distort the angle. We can characterize this statistical distribution exactly.

Proposition 1. Let u ∈ S d -1 , be drawn uniformly at random. Then || Bu || 2 ∼ Beta( c 2 , d -c 2 , 0 , d c ) , the four parameter Beta distribution with pdf f ( x ) = (2 x/c ) c/ 2 -1 (1 -2 x/c ) ( d -c ) / 2 -1 ( d/c ) β ( c/ 2 , ( d -c ) / 2 and E || Bu || 2 = 1 .

We defer proof of Proposition 1 to the Appendix D. Since the folded magnitude || Bu || 2 is unit in expectation, the distortion term α 2 /λ 2 in Theorem 2 will often be close to 1, greatly tightening the angle distortion bounds.

4.2 PGHash algorithm

Below, we detail our protocol for deploying PGHash in a centralized distributed setting (presented algorithmically in Algorithm 1). Over a network of N devices, the central host identifies a target layer P whose weight W P t ∈ R d × n (at

Algorithm 1 Distributed PGHash

Require: weights { W ℓ 0 } L ℓ =1 , objective F , N devices, target layer P , T iterations, folding matrix B , τ hash tables, hash length k , sketch dim c , comp. ratio CR .

1: Server Executes:

2: for t = 1 , 2 , . . . , T do

3: Compile pre-target weights W A = { W ℓ t } P -1 ℓ =1

5: for each device i in parallel do

4: Compile post-target weights W B = { W ℓ t } L ℓ = P +1

6: Θ i ← DeviceLSH (i, W A , BW P t , CR , τ, k, c )

7: W t +1 ,i ← DeviceUpdate (i, W P t (Θ i ) ⋃ W B )

8: end for

9: W t +1 ← Average the active weights W t +1 ,i across all devices

10: end for

11: return { W ℓ T } L ℓ =1

12: DeviceLSH ( i, W A , BW, CR, τ, k, c ):

13: Sample input ξ from local data distribution D i

14: ξ P -1 = F ( W A , ξ ) W A is stored locally

15: Θ i ← PGHash ( ξ P -1 , BW, CR, τ, k, c )

16: return Θ i

17: DeviceUpdate ( i, W B ):

18: W ← W A ⋃ W B

19: return W -η ∇ W F ( W ; ξ )

iteration t ) is too expensive for memory-constrained devices to fully train or host in local memory. Neurons (columns of W P t ) are pruned by devices according to their estimated cosine similarity to the output x P -1 of the previous layer.

The central host begins each round of distributed training t by sending each device (1) all weights { W ℓ t } P -1 ℓ =1 required to generate the input x P -1 and (2) the compressed target layer BW P t . Using these weights, each device conducts PGHash analysis (via Algorithm 2) using its current batch of local data to determine its activated neurons. The central host sends each device i their set of activated neurons W P t (Θ i ) , and each device performs a standard gradient update on their new model { W ℓ t } P -1 ℓ =1 ⋃ W P t (Θ i ) ⋃ { W ℓ t } L ℓ = P +1 . Finally, the central host receives updated models from each device and averages only the weights that are activated during training.

The on-device PGHash analysis (Algorithm 2) consists of first running a forward pass up to the target layer to generate the input x P -1 . Devices generate personal hash tables by performing leftmultiplication of BW P t and x P -1 by a random Gaussian matrix S ∼ N (0 , I c ) , as described in Section 3. The number of tables τ is specified by the user. A neuron j is marked as active if the input

(a) SimHash (10 tables).

(b) PGHash (10 tables).

(c) SimHash (100 tables).

(d) PGHash (100 tables).

175

150

：(degrees)

125

100

Angle

75

50

25

0

0

5

10

15

20

25

Avg Hamming Distance

175

150

：(degrees)

125

100

Angle (

75

50

25

0

0

5

10

15

20

25

Avg Hamming Distance

175

150

Angle(Degrees)

125

100

75

50

25

0

0

5

10

15

20

25

Avg Hamming Distance

175

150

Angle (Degrees)

125

100

75

50

25

0

0

5

10

15

20

25

Avg. Hamming Distance

hash code sgn ( Sx P -1 ) is identical to j -th weight's hash code sgn ( SB [ W P t ] : ,j ) . In Appendix A.2, we detail how Hamming distance can also be used for neuron selection.

Computational Complexity Remark. Through the use of dynamic pruning, PGHash significantly reduces both the forward and backward training computational complexities. PGHash activates at most CR · n neurons per sample as opposed to n for full training. In practice, PGHash activates only a fraction of the CR · n neurons (as shown in Figure 6a). Therefore, the number of floating point operations within forward and backward training is dramatically reduced.

Communication Complexity Remark. By reducing the size of the model needed in local memory and subsequently requesting a pruned version of the architecture we improve communication efficiency. For a fixed number of rounds T and target weight size dn , the total communication complexity, with respect to this data structure, is O ( T · CR · dn ) , which is significantly fewer bits than the vanilla O ( Tdn ) communication cost of vanilla federated training. In Section 5, we show that PGHash achieves near

state-of-the-art results with only CR = 0 . 1 (10% of a massive weight matrix).

Algorithm 2 PGHash

10 2

10 3

Iterations

0.25

0.30

0.35

0.40

0.45

Test Accuracy

PGHash: 0.1CR

Dense Baseline

(a) 90% Compression.

10 2

10 3

Iterations

0.25

0.30

0.35

0.40

0.45

Test Accuracy

PGHash: 0.25CR

Dense Baseline

(b) 75% Compression.

10 2

10 3

Iterations

0.25

0.30

0.35

0.40

0.45

Test Accuracy

Full PGHash

Dense Baseline

(c) No Compression.

5 Experiments

In this section, we (1) gauge the sensitivity of PGHash and (2) analyze the performance of PGHash and our own DWTA variant (PGHash-D) in training large-scale recommender networks. PGHash and PGHash-D require only 6.25% ( c = 8 ) of the final layer sent by the server to perform on-device LSH in our experiments. In PGHash, devices receive the compressed matrix BW ∈ R c × n via the procedure outlined in Section 4. In PGHash-D, devices receive c out of d randomly selected coordinates for all n neurons in the final layer weight. Using k of the c coordinates (ensuring privacy since the server is unaware of the coordinates used for LSH), PGHash-D selects neurons that, within the k coordinates, share the same index of highest-magnitude entry between the input and weight. We employ PGHash for Delicious-200K and PGHash-D for Amazon-670K and WikiLSHTC-325K.

PGHash Sensitivity Analysis. Our first experiment measures the ability of H PG ( c, d ) to estimate cosine similarity. We produce a fixed unit vector x ∈ R 100 and set of 180 vectors { v i } 180 i =1 of the same dimension. Both the Gaussian vector x and collection of vectors V are fed through varying numbers of SimHash and PGHash tables. We produce a scatter plot measuring the correlation between the angle and average Hamming distance. PGHash, as seen in Figure 2, is an effective estimator of cosine similarity. We observe that PGHash, like SimHash, successfully produces low average Hamming distances for vectors that are indeed close in angle. This provides evidence that selecting neurons with exact hash code matches (vanilla sampling) is effective for choosing neurons that are close in angle to the input vector. Finally, we find increasing the number of hash tables helps reduce variance.

Large-Scale Recommender Network Training. Our second experiment tests how well PGHash(D) can train large-scale recommender networks. We train these networks efficiently by utilizing dynamic neuronal dropout as done in [8]. We use three extreme multi-label datasets for training recommender networks: Delicious-200K, Amazon-670K, and WikiLSHTC-325K. These datasets come from the Extreme Classification Repository [4]. The dimensionality of these datasets is large: 782,585/205,443 (Delicious-200K), 135,909/670,091 (Amazon-670K), and 1,617,899/325,056 (WikiLSHTC-325K) features/labels. Due to space, Wiki results are found in Appendix A.3.

The feature and label sets of these datasets are extremely sparse. Akin to [8, 7, 34], we train a recommender network using a fully-connected neural network with a single hidden layer of size 128. Therefore, for Amazon-670K, our two dense layers have weight matrices of size (135 , 909 × 128) and (128 × 670 , 091) . The final layer weights output logits for label prediction, and we use PGHash(-D) to prune its size to improve computational efficiency during training.

Unlike [8, 7, 34], PGHash(-D) can be deployed in a federated setting. Within our experiments, we show the efficacy of PGHash for both single- and multi-device settings. Training in the federated setting (following the protocols of Algorithm 1) allows each device to rapidly train portions of the entire neural network in tandem. We partition data evenly (in an IID manner) amongst devices. Finally, we train our neural network using TensorFlow. We use the Adam [21] optimizer with an initial learning rate of 1e-4. A detailed list of the hyper-parameters we use in our experiments can be found in Appendix A.1. Accuracy in our figures refers to the P @1 metric, which measures whether the predicted label with the highest probability is within the true list of labels. These experiments are run on a cloud cluster using Intel Xeon Silver 4216 processors with 128GB of total memory.

10 2

10 3

Iterations

0.25

0.30

0.35

0.40

0.45

Test Accuracy

Single Device PGHash

Single Device SLIDE

Single Device FedAvg

(a) 1 Device Results.

10 2

10 3

Iterations

0.25

0.30

0.35

0.40

0.45

Test Accuracy

4 Device PGHash

4 Device SLIDE

4 Device FedAvg

(b) 4 Device Results.

10 2

10 3

Iterations

0.25

0.30

0.35

0.40

0.45

Test Accuracy

8 Device PGHash

8 Device SLIDE

8 Device FedAvg

(c) 8 Device Results.

Sampling Strategy. One important aspect of training is how we select activated neurons for each sample through LSH. Like [8], we utilize vanilla sampling. In our vanilla sampling protocol, a total of CR · n neurons are selected across the entire sampled batch of data. As detailed in Section 4 and Algorithm 2, a neuron is selected when its hash code exactly matches the hash code of the input. We retrieve neurons until either CR · n are selected or all τ tables have been looked up.

Compression Efficacy. We begin by analyzing how PGHash performs when varying the compression rate CR . Figure 3 showcases how PGHash performs for compression rates of 75% and 90% as well as no compression. Interestingly, PGHash reaches near-optimal accuracy even when compressed. This shows the effectiveness of PGHash at accurately selecting fruitful active neurons given a batch of data. The difference between the convergence of PGHash for varying compression rates lies within the volatility of training. As expected, PGHash experiences more volatile training (Figures 3a and 3b) when undergoing compression as compared to non-compressed training (Figure 3c).

Distributed Efficacy. In Figures 4 and 5, we analyze how well PGHash(-D) performs in a federated setting. We compare PGHash(-D) to a federated version of SLIDE [8] that we implemented (using respectively, a full SimHash or DWTA), as well as fully-dense Federated Averaging (FedAvg) for Delicious-200K. One can immediately see in Figures 4 and 5 that PGHash(-D) performs identically to, or better than, Federated SLIDE. In fact, for Delicious-200K, PGHash and Federated SLIDE outperform the dense baseline (FedAvg). In Appendix A.3 we detail the difficulties of PGHash-D and SLIDE in matching the dense baseline as well as the failure of SimHash to achieve performance akin to DWTA for Amazon-670K.

10 2

10 3

10 4

Iterations

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Test Accuracy

Single Device PGHash-D

Single Device SLIDE

(a) 1 Device Results.

10 2

10 3

10 4

Iterations

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Test Accuracy

4 Device PGHash-D

4 Device SLIDE

(b) 4 Device Results.

10 0

10 1

10 2

10 3

10 4

Iterations

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Average Activated Neurons per Sample (%)

PGHash Delicious-200K

PGHash-D Amazon-670K

10 2

10 3

Iterations

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

Test Accuracy

PGHash

Sampled Softmax

10 2

10 3

10 4

Iterations

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Test Accuracy

Sampled Softmax

PGHash-D

(a) Average Activated Neurons.

(b) Sampled Softmax (Delicious).

(c) Sampled Softmax (Amazon).

PGHash(-D) and Federated SLIDE smartly train portions of the network related to each batch of local device data, via LSH, in order to make up for the lack of a full output layer. However, unlike Federated SLIDE, PGHash(-D) can perform on-device LSH using as little as 6.25% of the full weight W ( c = 8 ) for both Delicious-200K and Amazon-670K experiments. Furthermore, for Delicious-200K, PGHash generates a dense Gaussian that is only 6.25% ( c = 8 ) the size of that for Federated SLIDE. In summary, PGHash(-D) attains similar performance to Federated SLIDE while storing less than a tenth of the parameters.

Induced Sparsity. PGHash(-D) induces a large amount of sparsity through its LSH process. This is especially prevalent in large-scale recommender networks, where the number of labels for each data point is a minuscule fraction of the total output layer size ( e.g. Delicious-200K has on average only 75.54 labels per point). PGHash(-D) performs well at identifying this small subset of neurons as training progresses. As one can see in Figure 6a, even when PGHash is allowed to select all possible neurons ( i.e., no compression CR = 1 ), it still manages to select fewer than 1% of the total neurons after only 50 iterations of training over Delicious-200K . For Amazon-670K, PGHash-D requires less than 30% of the total neurons for the majority of training. Therefore, PGHash(-D) greatly increases sparsity within the NN, improving the computational efficiency of the algorithm by reducing the number of floating point operations required in the forward and backward training.

6 Conclusion

In this work, we present a new hashing family, PGHash, which enables the generation of multiple LSH hash tables using a single base projection of a massive target weight. These hash tables can be used to dynamically select neurons that are similar to the layer input. This alleviates memory, communication, and privacy costs associated with conventional LSH-training approaches. As a proof of concept, we demonstrate that (i) the PGHash family is effective at mimicking SimHash and (ii) our framework is competitive against other, memory-inefficient, LSH-based federated training baselines of large-scale recommender networks. For future work, we intend to explore how multi-layer PGHash pruning affects model performance and incorporate learnable hashes as in the Mongoose [7] pipeline.

Limitations. Our theory indicates that PGHash is useful for detecting high angular similarity, but could prove unreliable for differentiating between intermediately dissimilar vectors. Additionally, LSH-based pruning has only shown success on large classification layers or attention layers in transformers [22]. When considering broader impacts, large-scale recommender networks, and any subsequent improvements to their design, can be used for strategically negative advertising purposes.

References

Zareen Alamgir, Farwa K Khan, and Saira Karim. Federated recommenders: methods, challenges and future. Cluster Computing , 25(6):4075-4096, 2022.

Alexandr Andoni and Piotr Indyk. Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions. Communications of the ACM , 51(1):117-122, 2008.

Alexandr Andoni, Piotr Indyk, Thijs Laarhoven, Ilya Razenshteyn, and Ludwig Schmidt. Practical and optimal lsh for angular distance. Advances in neural information processing systems , 28, 2015.

Kush Bhatia, Kunal Dahiya, Himanshu Jain, Anshul Mittal, Yashoteja Prabhu, and Manik Varma. The extreme classification repository: Multi-label datasets and code. URL http://manikvarma. org/downloads/XC/XMLRepository. html , 2016.

Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171) , pages 21-29. IEEE, 1997.

Moses S Charikar. Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing , pages 380-388, 2002.

Beidi Chen, Zichang Liu, Binghui Peng, Zhaozhuo Xu, Jonathan Lingjie Li, Tri Dao, Zhao Song, Anshumali Shrivastava, and Christopher Re. Mongoose: A learnable lsh framework for efficient neural network training. In International Conference on Learning Representations , 2020.

Beidi Chen, Tharun Medini, James Farwell, Charlie Tai, Anshumali Shrivastava, et al. Slide: In defense of smart algorithms over hardware acceleration for large-scale deep learning systems. Proceedings of Machine Learning and Systems , 2:291-306, 2020.

Beidi Chen and Anshumali Shrivastava. Revisiting winner take all (wta) hashing for sparse datasets. arXiv preprint arXiv:1612.01834 , 2016.

Beidi Chen and Anshumali Shrivastava. Densified winner take all (wta) hashing for sparse datasets. In Uncertainty in artificial intelligence , 2018.

Lars Engebretsen, Piotr Indyk, and Ryan O'Donnell. Derandomized dimensionality reduction with applications. 2002.

Michel X Goemans and David P Williamson. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM (JACM) , 42(6):1115-1145, 1995.

Arjun K Gupta and Saralees Nadarajah. Handbook of beta distribution and its applications . CRC press, 2004.

Piotr Indyk. Stable distributions, pseudorandom generators, embeddings, and data stream computation. Journal of the ACM (JACM) , 53(3):307-323, 2006.

Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the thirtieth annual ACM symposium on Theory of computing , pages 604-613, 1998.

Piotr Indyk, Rajeev Motwani, Prabhakar Raghavan, and Santosh Vempala. Locality-preserving hashing in multidimensional spaces. In Proceedings of the twenty-ninth annual ACM symposium on Theory of computing , pages 618-625, 1997.

Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. arXiv preprint arXiv:1412.2007 , 2014.

Junjie Jia and Zhipeng Lei. Personalized recommendation algorithm for mobile based on federated matrix factorization. In Journal of Physics: Conference Series , volume 1802, page 032021. IOP Publishing, 2021.

William B Johnson. Extensions of lipschitz mappings into a hilbert space. Contemp. Math. , 26:189-206, 1984.

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning , 14(1-2):1-210, 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020.

Noam Nisan. Pseudorandom generators for space-bounded computations. In Proceedings of the twenty-second annual ACM symposium on Theory of computing , pages 204-212, 1990.

Zaifeng Pan, Feng Zhang, Hourun Li, Chenyang Zhang, Xiaoyong Du, and Dong Deng. G-slide: A gpu-based sub-linear deep learning engine via lsh sparsification. IEEE Transactions on Parallel and Distributed Systems , 33(11):3015-3027, 2021.

Devi Parikh and Kristen Grauman. Relative attributes. In 2011 International Conference on Computer Vision , pages 503-510. IEEE, 2011.

Chengshuai Shi, Cong Shen, and Jing Yang. Federated multi-armed bandits with personalization. In International Conference on Artificial Intelligence and Statistics , pages 2917-2925. PMLR, 2021.

Anshumali Shrivastava and Ping Li. Asymmetric lsh (alsh) for sublinear time maximum inner product search (mips). Advances in neural information processing systems , 27, 2014.

Anshumali Shrivastava and Ping Li. In defense of minhash over simhash. In Artificial Intelligence and Statistics , pages 886-894. PMLR, 2014.

Ryan Spring and Anshumali Shrivastava. Scalable and sustainable deep learning via randomized hashing. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 445-454, 2017.

Chuhan Wu, Fangzhao Wu, Tao Qi, Yongfeng Huang, and Xing Xie. Fedattack: Effective and covert poisoning attack on federated recommendation via hard sampling. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 4164-4172, 2022.

Zhaozhuo Xu, Luyang Liu, Zheng Xu, and Anshumali Shrivastava. Adaptive sparse federated learning in large output spaces via hashing. In Workshop on Federated Learning: Recent Advances and New Challenges (in Conjunction with NeurIPS 2022) .

Jay Yagnik, Dennis Strelow, David A Ross, and Ruei-sung Lin. The power of comparative reasoning. In 2011 International Conference on Computer Vision , pages 2431-2438. IEEE, 2011.

Minghao Yan, Nicholas Meisburger, Tharun Medini, and Anshumali Shrivastava. Distributed slide: Enabling training large neural networks on low bandwidth and simple cpu-clusters via model parallelism and sparsity. arXiv preprint arXiv:2201.12667 , 2022.

Ian En-Hsu Yen, Satyen Kale, Felix Yu, Daniel Holtmann-Rice, Sanjiv Kumar, and Pradeep Ravikumar. Loss decomposition for fast learning in large output spaces. In International Conference on Machine Learning , pages 5640-5649. PMLR, 2018.

Felix Yu, Sanjiv Kumar, Yunchao Gong, and Shih-Fu Chang. Circulant binary embedding. In International conference on machine learning , pages 946-954. PMLR, 2014.

Shijie Zhang, Hongzhi Yin, Tong Chen, Zi Huang, Quoc Viet Hung Nguyen, and Lizhen Cui. Pipattack: Poisoning federated recommender systems formanipulating item promotion. arXiv preprint arXiv:2110.10926 , 2021.

Andrew Ziegler, Eric Christiansen, David Kriegman, and Serge Belongie. Locally uniform comparison image descriptor. Advances in Neural Information Processing Systems , 25, 2012.

Large-Scale Distributed Learning via Private On-Device LSH Appendix

A Experiment details

In this section, we provide deeper background into how our experiments were run as well as some additional results and observations. We first detail the hyper-parameters we used in order to reproduce our results. Then, we provide additional comments and details into our sampling approach. Finally, we describe some of the interesting observations we encountered while solving the Amazon-670K and Wiki-325K recommender system problems.

A.1 Experiment hyper-parameters

Below, we detail the hyper-parameters we used when running our federated experiments.

What one can immediately see from Table 1, is that we use a Densified Winner Take All (DWTA) variant of PGHash for the larger output datasets Amazon-670K and Wiki-325K. As experienced in [8, 7, 24], SimHash fails to perform well on these larger datasets. We surmise that SimHash fails due in part to its inability to select a large enough number of neurons per sample (we observed this dearth of activated neurons empirically). Reducing the hash length k does increase the number of neurons selected, however this decreases the accuracy. Therefore, DWTA is used because it utilizes more neurons per sample on these larger problems and also still achieves good accuracy.

As a quick note, we record test accuracy every so often (around 100 iterations for Delicious-200K and Amazon-670K). Similar to [8], to reduce the test accuracy computations (as the test sets are very large) we compute the test accuracy of 30 randomly sampled large batches of test data.

A.2 Neuron sampling

Speed of Neuron Sampling. In Table 3 we display the time it takes to perform LSH for PGHash given a set number of tables. These times were collected locally during training. The entries in Table 3 denote the time it takes to compute hashing of the final layer weights w i and each sample x in batch M as well as vanilla-style matching (neuron selection) for each sample.

We find in Table 3 that PGHash achieves near sub-linear speed with respect to the number of tables τ and slightly outperforms SLIDE. PGHash edges out SLIDE due to the smaller matrix multiplication cost, as PGHash utilizes a smaller random Gaussian matrix (size c × c ). The speed-up over SLIDE will

10 2

10 3

10 4

Iterations

0.14

0.16

0.18

0.20

0.22

0.24

Test Accuracy

Single Device PGHash-D

Single Device SLIDE

(a) 1 Device Results.

10 2

10 3

10 4

Iterations

0.14

0.16

0.18

0.20

0.22

0.24

Test Accuracy

4 Device PGHash-D

4 Device SLIDE

(b) 4 Device Results.

become more significant when the input layer is larger (as d = 128 in our experiments). Therefore, PGHash obtains superior sampling performance to SLIDE.

Hamming Distance Sampling. An alternative method to vanilla sampling is to instead select final layer weights (neurons) w i which have a small Hamming distance relative to a given sample x . As a refresher, the Hamming distance simply computes the number of non-matching entries between two binary codes (strings). If two binary codes match exactly, then the Hamming distance is zero. In this sampling routine, either (i) the top-k weights w i with the smallest Hamming distance to sample x are selected to be activated or (ii) all weights w i with a Hamming distance of β or smaller to sample x are selected to be activated. Interestingly, the vanilla-sampling approach we use in our work is equivalent to using β = 0 in (ii) .

In either of the scenarios listed above, hash codes for w i and x are computed as done in PGHash(-D). From there, however, the hash code for x is compared to the hash codes for all final layer weights in order to compute the Hamming distance for each w i . The process of computing n Hamming distances for each sample x is very expensive (much harder than just finding exact matches). That is why our work, as well as [8, 7], use vanilla sampling instead of other methods.

A.3 Amazon-670K and Wiki-325K experiment analysis

Sub-par SimHash Performance. SimHash is known to perform worse than DWTA on Amazon670K and Wiki-325K. Utilizing SimHash for these experiments is unfair as it is shown by [8, 7], for example, that DWTA achieves much higher performance on Amazon-670K. For this reason, DWTA is the chosen hash function in [8] for Amazon-670K experiments. To verify this observation, we performed experiments on Amazon-670K with PGHash (not PGHash-D) and SLIDE (with a SimHash hash function). Table 4 displays the SimHash approach for Amazon-670K.

As shown in Table 4, even with a much larger batch size, SLIDE and PGHash are unable to crack 30% on Amazon-670K. We would like to note that using a smaller batch size (like the M = 256 value we use in our Amazon-670K experiments) resulted in an even further drop in accuracy. These empirical results back-up the notion that SimHash is ill-fit for Amazon-670K.

Wiki-325K Performance. In Figure 7, we showcase how PGHash-D performs on Wiki-325K. Quite similar to the Amazon-670K results (shown in Figure 5), PGHash-D almost exactly matches up with SLIDE. In order to map how well our training progresses, we periodically check test accuracies. However, since the test set is very large, determining test accuracies over the entire test set is infeasible due to time constraints on the cluster. Therefore, we determine test accuracies over 30 batches of test data as a substitute as is done in [8, 7]. For Delicious-200K and Amazon-670K the entire test set accuracies matched the randomly sampled batches, however the randomly sampled batches underestimate the true test accuracies for Wiki-325K. For Wiki-325K, the true test accuracy ran about 5% greater than the sampled test accuracy values.

Matching Full-Training Performance. Along with the failure for SimHash to perform well on Amazon-670K and Wiki-325K, SLIDE and PGHash(-D) are unable to match the performance of full-training on these data-sets. This is observed empirically for Amazon-670K by GResearch in the following article https://www.gresearch.co.uk/blog/article/implementing-slide/ . We surmise that the failure of SLIDE and PGHash(-D) to match full-training performance on Amazon670K and Wiki-325K arises due to the small average labels per point in these two data-sets (5.45 and 3.19 respectively). Early on in training, SLIDE and PGHash(-D) do not utilize enough activated neurons. This is detrimental to performance when there are only a few labels per sample, as the neurons corresponding to the true label are rarely selected at the beginning of training (and these final layer weights are tuned much slower). In full-training, the true neurons are always selected and therefore the final layer weights are better adjusted from the beginning. We also note that [33] requires a hidden layer size of 1024 for a distributed version of SLIDE to achieve improved test accuracies for Amazon-670K. Thus, increasing the hidden layer size may have improved our performance (we kept it as 128 to match the original SLIDE paper [8]).

B PGHash: angle versus Hamming distance

In this section, we visually explore the degree to which PGHash is a consistent estimator of angular similarity. Specifically, let x, y ∈ R d : then we know by Theorem 1 that H PG ( c, d ) is an LSH for cos( x c , y c ) . We demonstrate that in the unit vector regime, θ c = arccos(cos( x c , Y c )) is an acceptable surrogate for θ = arccos( x, Y ) , where Y = { y i } N i =1 and Y c = { y i c } N i =1 .

(a)

c

=5

(b)

c

=10

(c)

c

=25

(d)

c

=5

(e)

c

=10

(f)

c

=25

175

175

175

150

150

150

Angle (Degrees)

125

Angle (Degrees)

125

Angle (Degrees)

125

100

100

100

75

75

50

50

50

25

SZ

25

0

0

0

0

1

2

3

0

1

2

3

4

0

5

10

15

Z

25

Avg Hamming Distance

Avg Hamming Distance

Avg Hamming Distance

175

175

175

150

150

150

Angle (Degrees)

125

Angle (Degrees)

125

(saaa) a

125

100

100

100

万

75

50

50

50

25

25

25

0

0

0

0

1

2

3

4

5

0

2

4

6

8

10

0

10

15

20

25

Avg Hamming Distance

Avg Hamming Distance

Avg Hamming Distance

(a) 5 tables

(b) 10 tables

(c) 50 tables

175

175

175

150

150

150

(aa)

125

(a

125

(aaa)

125

100

100

100

Angle

75

Angle

75

Angle

75

50

50

50

25

25

25

0

0

0

0

2

4

8

10

0

2

4

6

8

10

0

2

4

6

8

10

Avg Hamming Distance

Avg Hamming Distance

Avg Hamming Distance

C Additional proofs

Fact 3. Let x, y, e 1 , e 2 be d -dimensional unit vectors such that that the e i lie on the unit circle contained with the plane spanned by x and y (denoted as S x,y ) and e 1 ⊥ e 2 . Consider the point v on S x,y such that the line through it bisects the angle of the lines passing through x and y . Let η = arccos(cos( v, e 1 )) . Denote θ = 1 2 arccos(cos( x, y )) . Then we may write x = cos( η + θ ) e 1 + sin( η + θ ) e 2 and y = cos( η -θ ) e 1 +sin( η -θ ) e 2 .

C.1 Proof of Theorem 2

Proof. Let θ = 1 2 arccos(cos( x, y )) where x, y ∈ S d -1 and A = B ⊤ B . The cosine similarity between x c = Bx and y c = By (for B correspondent to a ( d, c ) -folding), is expressible as

Consider the SVD B = UDV ⊤ where U and V are orthogonal and D is c × d rectangular diagonal matrix. We have then that A = B ⊤ B = V ˆ D 2 V ⊤ . (Here ˆ D is now a square diagonal matrix containing squared D ii along the diagonal and 0 everywhere else.) Notice that choice of U nor the ordering of columns v i of V affects the angle calculation in Equation 1. First, we re-order the columns of V so as to order the diagonal entries d i of D (i.e., the squared singular values) in decreasing order, and as an abuse of notation set B = 1 d 1 DV ⊤ . Denoting ˆ λ i = d i /d 1 for 1 ≤ i ≤ n , we have that Bv i = ˆ λ i e i . (By construction of B we have that d i ∈ { d c , 0 } , therefore, ˆ λ i ∈ { 1 , 0 } )

Consider B acting on S d -1 : it scales each dimension by ˆ λ i , thus (as with any linear transformation of a sphere), transforms it into an ellipsoid, with c principal axes determined by the v i . The greatest possible distance from the origin to the ellipsoid BS d -1 is 1 while the shortest possible distance is 0 . Now consider the unit circle S x,y = { v ∈ span( x, y ) : || v || = 1 } . We have that BS x,y ⊂ BS d -1 ∩ BU is an ellipse (since the intersection of an ellipsoid and plane is always an ellipse).

Choose unit w 1 and w 2 belonging to S x,y such that w 1 ⊥ w 2 . by By Fact 3, we may parameterize our vectors as x = cos( η -θ ) w 1 +sin( η -θ ) w 2 and y = sin( η + θ ) w 1 +sin( η + θ ) w 2 , where η is the angle made with w 1 with the bisector of x and y . By assumption, || Bw || ≥ α (the minimal shrinking factor of B on S x,y ) for some positive α . Denoting λ = d c (the maximal stretching factor of B on S x,y ), we have that the angle between Bx and By is upper-bounded by

.

The numerator of df dη is β (1 -β )(1 + β ) sin(2 θ ) sin(2 η ) where β = α/λ . The derivative is trivially 0 if (1) β = 0 , (2) β = 1 , or (3) θ = 0 . (1) will not occur as we assume that S x,y does not contain a 0-eigenvector of A = B ⊤ B . (2) can only occur if A is a multiple of the identity matrix (which it is not by construction), and (3) implies that x and y are parallel, in which case their angle will not be

distorted. Aside from these pathological cases, the critical points occur at η = 0 , π/ 2 . We have then that cos( Bx,By ) lives between cos( f (0)) = 1 -β 2 tan 2 θ 1+ β 2 tan 2 θ and cos( f ( π/ 2)) = -tan 2 θ 2 -β 2 tan 2 θ 2 + β 2 .

Remark. The constant β has an enormous influence on the bounds in Theorem 2. The smaller the α (i.e., shrinking of || w || ), the greater the bounds on distortion. Although we have imposed constraints on x , y , if we treat them as any possible pair of random unit vectors, then the w in S x,y effectively becomes a random unit vector as well. We can exactly characterize the distribution of || BX || where X denotes a random variable which selects a d -dimensional unit vector uniformly at random.

C.2 Proof of Proposition 1

Proof. We can sample a d -dimensional vector uniformly at random from the unit sphere S d -1 by drawing a d -dimensional Gaussian vector with iid entries and normalizing. Let us represent this as the random variable X = Z ′ / || Z ′ || where Z ′ ∼ N (0 , I d ) . Consider a ( c, d ) -folding matrix B , i.e., a d/c horizontal stack of c × c identity matrices (let us assume c | d ). We are interested in determining the distribution of || BX || 2 . For ease of notation, consider the permutation Z of Z ′ where Z i = Z ′ ( ⌊ d/c ⌋-1) ∗ ( d/c )+ i (mod d/c ) . Since this permutation is representable as an orthogonal i matrix P (and multi-variate Gaussians are invariant in distribution under orthogonal transformations), we may instead consider X := P ( Z ′ / || Z ′ || ) 2 = Z/ || Z || 2 . We may write the norm-squared as

Consider the first term ( Z 1 + ··· + Z d/c ) 2 || Z || 2 . First note that for any unit vector u , the distribution of ( u ⊤ Z ) 2 || Z || 2 does not depend on choice of u . Consider the unit vector u ′ then which contains √ d/c in the first d/c entries and 0 otherwise. Then ( u ′⊤ Z ) 2 || Z || 2 is equivalent to d/c times our first term. Of course, since ( e ⊤ 1 Z ) 2 || Z || 2 has the same distribution as ( u ′⊤ Z ) 2 || Z || 2 , we have by transitivity that Z 2 1 || Z || 2 d = ( n/q ) ( Z 1 + ··· + Z d/c ) 2 || Z || 2 .

By extending the discussion above to the other terms, and by their independence with respect to rotation of Z (since their numerators contain squared sums of mutually disjoint Z coordinates), we have that

The distribution of Z 2 1 + Z 2 d/c + Z 2 2 d/c + ··· + Z 2 d || Z || 2 is well-known to follow a Beta( c 2 , d -c 2 ) distribution [13].

In totality, || BX || 2 d = d c Beta( c 2 , d -c 2 ) . However, we will move to the four parameter description of this scaled Beta distribution which is Beta( c 2 , d -c 2 , 0 , d c ) . The pdf and expected value follows by the usual statistical descriptions of this distribution, which can also be found in [13].

Figure 10 depicts how ( d, c ) -foldings affect the norms of unit vectors.

D Additional theory

In this section, we provide additional theory relevant to SimHash.

We present several well-known results regarding SimHash.

Proposition 2 (SimHash estimation) . Let x, y ∈ S , i.e., unit d -dimensional vectors. Denote θ = arccos( | cos( x, y ) | ) . Let v ∈ S d be a unit vector drawn uniformly at random (according to the Haar measure, for example). Then,

̸

Distributionoffoldednorms

2000

Frequency

1500

1000

500

0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Bvl~ 2

̸

Proof. We reproduce the argument of [12]. We have by symmetry that Pr [ sgn ( v ⊤ x ) = sgn ( v ⊤ y )] = 2 Pr [ v ⊤ x > 0 , v ⊤ y < 0] . The set U = { v ∈ S d : v ⊤ x > 0 , v ⊤ y ≤ 0 } corresponds to the intersection of two half-spaces whose dihedral angle (i.e., angle between the normals of both spaces) is exactly θ . Intersecting with the d -dimensional unit sphere produces gives a subspace of measure θ 2 π , therefore, 2 Pr [ v ⊤ x > 0 , v ⊤ y < 0] = θ π , completing the argument.

Corollary 2. Let v instead be a d -dimensional random Gaussian vector with iid entries ∼ N (0 , 1) . Then for x, y ∈ R d ,

̸

Proof. Randomly drawn, normalized Gaussian vectors are well-known to be uniformly distributed on the unit sphere.

In the setup as above, let the X be a random variable which returns 1 if x and y have differing signs when taking the standard inner product with a randomly drawn Gaussian v . Let X 1 , X 2 , . . . , X n represent a sequence of independent X events. Then,

Given that PGHash is equivalent to a SimHash over ( d, c ) -foldings of R d , the variance reduction we observe by using multiple tables (Figure 9 is explainable by Proposition 3.

Peer reviews:
comments: SUMMARY:
This paper presents a novel framework for on-device locality-sensitive hashing (LSH) that addresses the limitations of existing algorithms in terms of computational efficiency, memory constraints, and privacy concerns. The authors introduce a new family of hash functions that enables each device to generate hash tables independently, without relying on a central host. This approach allows for personalized and private LSH analysis while conserving memory and computational resources.

STRENGTHS:
1. The proposed PGhash family is novel. It addresses computational and memory challenges introduced by repeated randomized projection of full-layer weights in distributed settings.

2. The authors claim to have proven several statistical and sensitivity properties of their PGhash functions, ensuring the reliability and robustness of the proposed framework. They also present experimental results that demonstrate the competitiveness of their approach in training large-scale recommender networks compared to existing LSH frameworks that assume unrestricted on-device capacity.

3. The paper is well-written. The theoretical formulation is well-organized.

WEAKNESSES:
NA

QUESTIONS:
Is there any intuition on the choosing between PGhash version of DWTA and SimHash in practice?

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

comments: SUMMARY:
The paper proposes a family of hashing functions that reduces the memory cost. Let W be a weight matrix. It is observed that in many applications, one needs to draw a Gaussian matrix T and compute T * W when constructing hash tables. Also, given a query x, one needs to compute T * x which requires the storage of the entire matrix T. These two steps will be less practical on memory-limited devices. The main contribution of the work is a more efficient hashing scheme. The key idea is to perform a low-dimensional projection B of x on to dimension c, and use a smaller Gaussian matrix S. It is shown that the cosine distance on Bx preserves LSH property.

STRENGTHS:
+ The problem and results are clearly presented. 

+ The saving of memory is important to many applications.

+ The techniques are easy to follow.

+ A set of experiments were included to support the theory.

WEAKNESSES:
- My primary concern is the novelty of the approach. It turns out that Theorem 1 is the main result of the paper, which shows that projection with the matrix B = [I_c | I_c | ... | I_c] would preserve cosine distance in the sense of LSH. I am however uncertain whether the design of B is novel, or has been considered before in a similar context. (I guess it appeared somewhere but I cannot recount.) I defer to authors' clarification and other reviewers' comments.

- The parameter 'c' controls the new dimension, which is supposed to be lower bounded by some quantity relevant to d or the training size, or will change the performance of the similarity function Sim_c, but there was no such discussion or condition.

- Algorithm 1 looks superficial and lacks guarantee.

QUESTIONS:
see weakness

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Main Idea** : If you are using LSH based sparsity, then why not project the weight matrices down using random / even structured projections and then use the LSH. \n\n**Potential applications**: device-based training / inference (resource-constrained) / federated learning. \n\nThe authors provide theoretical analyse of proposed PGHash and provide some experiments in single and multi-device settings. \n\n**[UPDATE AFTER REBUTTAL]** \nSorry for the late response. Due to unavoidable reasons, i could not get to this earlier. \n\n1. As agreed by authors on most questions on theoretical results, there were issues in the provided theory. While authors provide quick fixes to the theorems. In order to evaluate these fixes, one would have to reconsider the entire theory again. \n\n2. I do not agree with authors argument in favor of structured matrix B. I still believe it has issues and detailed proof must be provided to support it. I am afraid structured matrix B will kill any best bounds you can give. For example, in the example I gave in the review, how can \none ever cope up with the error caused there. For such as example, the bounds have to be trivial [-1,1] for cosine similarity. \n\n3. I am skeptical with the statement that \"The block identity structure solves the following problem: given a fixed $c \\times d$ matrix $B$ with $c<<d$, produce a sequence of $c \\times c$ matrices $S_i$ for $1\\leq i \\leq n$ such that the $S_iB$ resemble random $c \\times d$ Gaussian matrices\".  \"Resemble\" is not what we are looking for in the theory. **Would request ACs to take a serious look at the theory to confirm its correctness.**\n\nAbout experiments, I would request the authors to show results after the peak as well to get the full picture. \n\n\nI will be maintaining my score for the submission in its current form.\n\nSTRENGTHS:\nThe idea of using projections before LSH might be useful in practice depending on the sensitivity of the model to the compression.\n\nWEAKNESSES:\nSee question section.\n\nQUESTIONS:\n**Theoretical soundness of approach and provided theorems.**\n\n1. Theorem 1 does not provide any information on the approximation. \n\n     The statement of theorem 1 is obvious as the similarity is defined as cosine similarity over folded vectors. After folding, PGhash is just Simhash so it is not surprising that it is a LSH on folded vectors\n\n2. Theorem 2 does provide some information. But has issues.\n\n   a). The $\\alpha$ is infimum of a set and can be equal to $0$ even if the set itself does not contain $0$. (on a related note, i believe the definition of $\\alpha$ can be improved in terms of how the set is written). Is it that $\\alpha = inf \\\\{ ||Bv|| , v \\in S_{x,y} \\\\}$ ? I assume this is what the authors meant. \n\n   b) If $\\alpha$ is 0, then the range of distortion is trivially -1,1\n\n   c) Proposition 1 is used to comment on the value of $\\alpha$ but it's not exactly clear how they are related, the set used in proposition 1 is that of unit sphere in R^d and that in theorem 2 is linear span of $x$ and $y$ where coefficients are from a unit circle . Clearly the two sets are very different. Also, E||Bu|| = 1 cannot be directly extended to understand the range of $\\alpha$ as $\\alpha$ is infimum of the $ \\\\{ ||Bv|| , v \\in S_{x,y}\\\\}$ \n\n   d) It is possible that writing issues in this section might have mislead me a bit. It would be good to get clarification from authors. \n\n   e) The structured form of matrix B is actually troublesome for theoretical guarantees. Consider a simple example , $d=20$, $c=5$, the vectors be,  $S = \\\\{ [x_1, 0,0,0,0, x_2, 0,0,0,0, x_3, 0,0,0,0, x_4, 0,0,0,0] | x_1+x_2+x_3+x_4 = 1 \\\\}$.  Note that two vectors from S can have cosine similarities that vary from $-1$ to $1$ but under PGHash , they will always have similarities of $1$. \n\n\n\n**Experiments:**\n\n1. Both from Figure 5 and Figure 4, it seems like higher number of devices causes PGHash to perform worse. Is there any reason why this occurs?\n2. I do expect that using projections to reduce size of weight matrices is going to lose accuracy. However, the hope for this paper is that the trade-off only significantly kicks in after large amounts of compression.\n\n    a. The plots provided for Amazon-670K and Delicious-200K are not shown to convergence. So it is impossible to know what the eventual loss of accuracy is. \n\n    b. With multi-device settings, the plots already show that PGhash is losing accuracy as compared to baseline. Given that multi-device setting is the proposed application area of PGhash, it is unclear if the overall proposal has value.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-08-01"}