SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tahseen Rabbani*
trabbani@umd.edu
Marco Bornstein*
marcob@umd.edu
Furong Huang
furongh@umd.edu
Locality-sensitive hashing (LSH) based frameworks have been used efficiently to select weight vectors in a dense hidden layer with high cosine similarity to an input, enabling dynamic pruning. While this type of scheme has been shown to improve computational training efficiency, existing algorithms require repeated randomized projection of the full layer weight, which is impractical for computational- and memory-constrained devices. In a distributed setting, deferring LSH analysis to a centralized host is (i) slow if the device cluster is large and (ii) requires access to input data which is forbidden in a federated context. Using a new family of hash functions, we develop one of the first private, personalized, and memory-efficient on-device LSH frameworks. Our framework enables privacy and personalization by allowing each device to generate hash tables, without the help of a central host, using device-specific hashing hyper-parameters (e.g. number of hash tables or hash length). Hash tables are generated with a compressed set of the full weights, and can be serially generated and discarded if the process is memory-intensive. This allows devices to avoid maintaining (i) the fully-sized model and (ii) large amounts of hash tables in local memory for LSH analysis. We prove several statistical and sensitivity properties of our hash functions and experimentally demonstrate that our framework is competitive in training large-scale recommender networks compared to other LSH frameworks which assume unrestricted on-device capacity.
Locality-sensitive hashing (LSH) has proven to be a remarkably effective tool for memory- and computationally-efficient data clustering and nearest neighbor search [6, 15, 2]. LSH algorithms such as SimHash [6] can be used to search for vectors in collection  $W \subset \mathbb{R}^d$  of massive cardinality which will form a large inner product with a reference vector  $x \in \mathbb{R}^d$ . This procedure, known as maximum inner product search (MIPS) [27], has been applied to neural network (NN) training. In NN training, the weights of a dense layer that are estimated to produce a large inner product with the input (thereby, a large softmax, for example) are activated while the remainder is dropped out.
While LSH-based pruning greatly reduces training costs associated with large-scale models, popular frameworks such as SLIDE [8] and Mongoose [7] cannot be deployed in distributed settings over
memory-constrained devices such as GPUs or mobile phones for the following reasons: (a) required maintenance of a large target layer in memory and (b) access to the input is needed to conduct LSH.
With many modern NN architectures reaching billions of parameters in size, requiring resource-constrained devices to conduct LSH analysis over even part of such a large model is infeasible as it requires many linear projections of massive weights. The hope of offloading this memory and computationally-intensive task to a central host in the distributed setting is equally fruitless. LSH-based pruning cannot be conducted by a central host as it requires access to either local client data or hashed mappings of such data. Both of these violate the fundamental host-client privacy contract, especially in a federated setting [20]. Therefore, in order to maintain privacy, devices are forced to conduct LSH themselves, returning us back to our original drawback in a vicious circle. We raise the following question then:
Can a resource-constrained device conduct LSH-like pruning of a large dense layer without ever needing to see the entirety of its underlying weight?
This work makes the following contributions to positively resolve this question:
(1) Introduce a novel family of hash functions, PGHash, for the detection of high cosine similarity amongst vectors. PGHash improves upon the efficiency of SimHash by comparing binarized random projections of folded vectors. We prove several statistical properties about PGHash, including angle/norm distortion bounds, and that it is an LSH family.
(2) Present an algorithmic LSH framework, leveraging our hash functions, which allows for private, personalized, and memory-efficient distributed/federated training of large-scale recommender networks via dynamic pruning.
(3) Showcase experimentally that our PGHash-based framework is able to efficiently train large-scale recommender networks. Our approach is competitive against a distributed implementation of SLIDE [8] using full-scale LSH. Furthermore, where entry-magnitude similarity is desired over angular similarity (training over Amazon-670K, for example), we empirically demonstrate that using our DWTA [9] variant of PGHash, PGHash-D, matches the performance of using full-scale DWTA.
LSH Families. Locality-sensitive hashing families have been used to efficiently solve the approximate nearest neighbors problem [16, 15, 2]. SimHash [6], based on randomized hyperplane projections, is used to estimate cosine similarity. Each SimHash function requires a significant number of random bits if the dimensionality of each target point is large. However, bit reduction using Nisan's pseudorandom generator [23] is often suggested [11, 14]. MinHash [5], a competitor to SimHash [28], measures Jaccard similarity between binary vectors and has been used for document classification. The Winner Take All (WTA) hash [32] compares the ordering of entries by magnitude (corresponding to Kendall-Tau similarity); such comparative reasoning has proven popular in vision applications [25]. However, it was observed that WTA was ineffective at differentiating highly sparse vectors leading to the development of Densified WTA (DWTA) [9]. Since MinHash, WTA, and DWTA are better suited for binary vector comparison, and we require comparison over real-valued vectors, PGHash is founded on SimHash.
Hash-based Pruning. One of the earlier proposals of pruning based on input-neuron angular similarity via LSH tables is in [29], where a scheme for asynchronous gradient updates amongst multiple threads training over a batch along with hashed backpropagation are also outlined. These principles are executed to great effect in both the SLIDE [8] and Mongoose [7] frameworks for training extreme-scale recommender networks. MongoDB improves SLIDE by using an adaptive scheduler to determine when to re-run LSH over a layer weight, and by utilizing learnable hash functions. Both works demonstrated that a CPU using LSH-based dynamic dropout could achieve competitive training complexity against a GPU conducting fully-dense training. Reformer [22] uses LSH to reduce the memory complexity of self-attention layers.
Distributed Recommender Networks. Several works which prune according to input-neuron angular similarity estimations via LSH utilize multiple workers on a single machine [29, 24, 7, 8]. Federated training of recommender networks is an emerging topic of interest, with particular interest in personalized training [18, 26] malicious clients [30, 36], and wireless unreliability [1]. D-SLIDE [33], which is the federated version of SLIDE, eases local on-device memory and computational requirements by sharding the network across clients. However, in the presence of low client numbers,
(a) SimHash
(b) PGHash
Figure 1: Hash-based dropout. Two procedures for conducting cosine similarity estimation between full weight matrix  $W \in \mathbb{R}^{d \times n}$  (column  $w_{i}$  denotes the weight of neurons  $i$ ) and an input  $x \in \mathbb{R}^d$ . (a) SimHash generates a hash table via left-multiplication by a randomized rectangular Gaussian matrix  $S \sim \mathcal{N}(0, I_d)$  onto the fully-sized  $W$ . (b) PGHash generates a hash table via left-multiplication by a randomized square Gaussian matrix  $S \sim \mathcal{N}(0, I_c)$  onto a base projection  $BW \in \mathbb{R}^{c \times n}$  of  $W$ . In both procedures, weight  $w_{i}$  is selected for activation if its signed hash matches the signed hash of  $x$ .
the proportion of the model owned per device can still be taxing, whereas our compression is independent of the number of federated agents. In [31], clients query the server for weights based on the results of LSH conducted using server-provided hash functions. We regard this complete server control over the hashing family, and therefore access to hash-encoding of local client data, as non-private and potentially open to honest-but-curious attacks.
Let  $W = \{w_{1}, w_{2}, \ldots, w_{n}\} \subset \mathbb{R}^{d}$  be the weights of a dense hidden layer and  $x \in \mathbb{R}^{d}$  be the input. For brevity, we refer to  $W \in \mathbb{R}^{d \times n}$  as the weight of the layer. In particular,  $w_{i} \in \mathbb{R}^{d}$  corresponds to the  $i^{\text{th}}$  neuron of the layer. We assume that the layer contains  $dn$  parameters. Within our work, we perform MIPS, as we select weights that produce large inner products with  $x$ . Mathematically, we can begin to define this by first letting  $p = \max w_{i}^{\top} x$  for  $1 \leq i \leq n$ . For  $0 < \epsilon < 1$  we are interested in selecting  $S \subset W$ , such that for  $\forall w_{i} \in S, w_{i}^{\top} x > \epsilon p$ . The weights of  $S$  will pass through activation while the rest are dropped out, reducing the computational complexity of the forward and backward pass through this layer. As detailed in Section 4 and illustrated in Figure 1, we will determine  $S$  by estimating angles with a projection  $BW = \{Bw_{i}\}_{i=1}^{n}$ , with  $B \in \mathbb{R}^{c \times d}$  such that  $c << d$ .
Locality-sensitive Hashing (LSH). LSH [2] is an efficient framework for solving the  $\epsilon$ -approximate nearest neighbor search (NNS) problem:
Definition 1 (ε-NNS). Given a set of points  $P = \{p_1, p_2, \ldots, p_n\}$  in a metric space  $\mathcal{M}$  and similarity function  $Sim: \mathcal{M} \times \mathcal{M} \to \mathbb{R}$  over this space, find a point  $p \in P$ , such that for a query point  $q \in X$  and all  $p' \in P$ , we have that  $Sim(p, q) \leq (1 + \epsilon) Sim(p', q)$ .
It is important to note that the similarity function  $Sim(\cdot)$  need not be a distance metric, but rather any general comparison mapping. Popular choices include Euclidean distance and cosine similarity, the latter of which is the primary focus of this paper. The cosine similarity for  $x, y \in \mathbb{R}^d$  is defined as  $\cos(x, y) \triangleq x^\top y / (||x||_2||y||_2)$ . We can frame the MIPS problem described previously as an  $\epsilon$ -NNS one if we assume that the weights  $w_i \in W$  are of unit length. Thus, we are searching for  $\epsilon$ -nearest neighbors in  $W$  of the query  $x$  according to cosine similarity.
Consider a family  $\mathcal{H}$  containing hash functions of the form  $h: \mathcal{M} \to \mathcal{S}$ , where  $\mathcal{S}$  is a co-domain with significantly lower feature dimensionality than  $\mathcal{M}$ . We say that  $\mathcal{H}$  is locality-sensitive if the hashes of a pair of points  $x, y$  in  $\mathcal{M}$ , computed by an  $h \in \mathcal{H}$  (selected uniformly at random), have a higher collision (matching) probability in  $S$  the more similar  $x$  and  $y$  are according to Sim. We now formally define this notion following [15].
Definition 2 (Locality-sensitive Hashing). A family  $\mathcal{H}$  is called  $(S_0, \epsilon S_0, p_1, p_2)$ -sensitive if for any two points  $x, y \in \mathbb{R}^d$  and  $h$  chosen uniformly at random from  $\mathcal{H}$  satisfies the following,
$$
\begin{array}{l} 1. \quad i f \quad S i m (x, y) \quad \geq \quad S _ {0} \quad \Rightarrow \quad P r (h (x) \quad = \quad h (y)) \quad \geq \quad p _ {1}, \quad 2. \quad i f \\ S i m (x, y) \leq \epsilon S _ {0} \Rightarrow P r (h (x) = h (y)) \leq p _ {2}. \end{array}
$$
For an effective LSH,  $p_1 < p_2$  and  $\epsilon < 1$  is required. An LSH family allows us to conduct a similarity search over a collection of vectors through a comparison of their hashed mappings. Of course, locality loss is inevitable if Sim is a dimension-lowering projection. Through a mixture of increased precision
(raising the output dimension of  $\mathcal{H}$ ) and repeated trials (running several trials over independently chosen  $h$ ), we may tighten the correspondence between Sim and matches over  $\mathcal{H}$ , following the spirit of the Johnson-Lindenstrauss Lemma [19].
SimHash. A popular LSH algorithm for estimating cosine similarity is SimHash, which uses signed random projections [6] as its hash functions. Specifically, for a collection of vectors  $W \subset R^{d}$ , the SimHash family  $\mathcal{H}^{Sim}$  consists of hash functions  $h_v$ , each indexed by a random Gaussian vector  $v \sim \mathcal{N}(0, I_n)$ , i.e., an  $n$ -dimensional vector with iid entries drawn from  $\mathcal{N}(0, 1)$ . For  $x \in R^n$ , we define the hash mapping  $h_v(x) := \operatorname{sgn}(v^\top x)$ . Here, we modify  $\operatorname{sgn}(x)$  to return 1 if  $x > 0$ , else it returns 0. For Gaussian  $v$  chosen uniformly at random and fixed  $x, y \in \mathbb{R}^d$ , we have  $Pr(h_v(x) = h_v(y)) = 1 - \frac{\arccos\left(\frac{x^\top y}{||x|| \cdot ||y||}\right)}{\pi}$ . This hashing scheme was popularized in [12] as part of a randomized approximation algorithm for solving MAX-CUT. Notice that the probability of a hashed pair matching is monotonically increasing with respect to the cosine similarity of  $x$  and  $y$ , satisfying Definition 2. More precisely, if we set  $S_0 = \cos(x, y)$  then  $\mathcal{H}^{Sim}$  is  $(S_0, \epsilon S_0, (1 - \arccos(S_0/\pi)), (1 - \arccos(\frac{\epsilon S_0}{\pi}))^-)$ -sensitive [6, 28]. The above discussion considers the sign of a single random projection, but in practice, we will perform multiple projections.
Definition 3 (Hash table). Let  $X = \{x_{1},\ldots ,x_{n}\} \subset \mathbb{R}^{d}$  and  $V = \{v_{1}^{\top},\dots,v_{k}^{\top}\} \subset \mathcal{N}(0,I_{d})$ , where  $k$  is the hash length. Define  $h_V:\mathbb{R}^d\to \mathbb{R}^k$  by  $[h_V(x)]_i = h_{v_i}(x)$  where  $h_{v_i}\in \mathcal{H}^{Sim}$  for  $1\leq i\leq k$ . For fixed  $V$ , the hash table  $h_V(X)\in \mathbb{R}^{k\times n}$  is a binary matrix with columns  $h_V(x_j)$  for  $1\leq j\leq n$ .
Following the notation above, we may estimate the similarity between an input  $q \in \mathbb{R}$  and a collection of vectors  $X \subset \mathbb{R}^d$  by measuring the Hamming distances (or exact sign matches) between  $h_V(q)$  and columns of  $h_V(X)$ . SimHash is now more discriminatory, as  $\mathcal{H}^{Sim}$  can separate  $R^d$  into  $2^k$  buckets corresponding to all possible length  $k$  binary vectors (which we refer to as hash codes). Finally, counting the frequency of exact matches or computing the average Hamming distance over several independently generated hash tables further improves our estimation of closeness. Implementations of the well-known SLIDE framework [8, 24], which utilize SimHash for LSH-based weight pruning, require upwards of 50 tables.
DWTA. Another popular similarity metric is to measure how often high-magnitude entries between two vectors occur at the exact same positions. The densified winner-take-all (DWTA) LSH family [10] estimates this similarity by uniformly drawing  $k$  random coordinates over  $W$  and recording the position of the highest-magnitude entry. Similar to SimHash, this process is repeated several times, and vectors with the highest frequency of matches are expected to have similar magnitude ordering. This type of comparative reasoning is useful for computer vision applications [37].
4 PGHash
In this section, we develop a family of hash functions  $\mathcal{H}^{PG}$  which allow for memory-efficient serial generation of hash tables using a single dimensionally-reduced sketch of  $W$ . This is in contrast to traditional LSH frameworks, which produce hash tables via randomized projections over the entirety of  $W$ . We first present an overview of the Periodic Gaussian Hash (PGHash) followed by its algorithm for distributed settings, an exploration of several statistical properties regarding the local sensitivity of  $\mathcal{H}^{PG}$ .
PGHash Motivation. As detailed in Section 3, our goal is to efficiently estimate cosine similarity between an input to a layer  $x \in \mathbb{R}^d$  and the columns of a large weight matrix  $W \in \mathbb{R}^{d \times n}$ . SimHash performs hash table generation by first multiplying a matrix of uniformly drawn Gaussian hyperplanes  $T_V \in \mathbb{R}^{c \times d}$  with  $W$ . The full hash table is computed as  $h_V(W) = \operatorname{sgn}(T_VW)$ . Then, the  $j^{\text{th}}$  neuron is activated if  $\operatorname{sgn}(T_Vx) = h_V(w_j)$  for a layer input  $x$ .
One can immediately notice that generation of a new hash table  $h_{\tilde{V}}(W)$  requires both (i) computation of  $T_{\tilde{V}}W$  which requires access to the fully-sized weights and (ii) the storage of  $T_{\tilde{V}}$  to compute  $\operatorname{sgn}(T_{\tilde{V}}x')$  for further inputs  $x'$ . This is problematic for a memory-constrained device, as it would need to maintain both  $W$  and  $T_{\tilde{V}}$  to generate further tables and perform dynamic pruning. To solve this issue, we introduce a family of hash functions generated from a single projection  $BW$  of  $W$ .
4.1 PGHash theory
Definition 4 (Periodic Gaussian Hash). Assume sketch dimension  $c < d$  divides  $d$  for simplicity. Let  $B = [I_c|I_c|\dots |I_c] \in \mathbb{R}^{c \times d}$ , where  $I_c$  is the  $c \times c$  identity matrix and  $| \cdot|$  denotes  $\frac{d}{c}$  concatenations. Let  $S \in \mathbb{R}^{k \times c}$  be a random Gaussian matrix with iid entries drawn from  $\mathcal{N}(0,1)$ . We may define a Periodic Gaussian Hash (PGHash) function  $h_S^{PG}: \mathbb{R}^d \to \mathbb{R}^k$  by  $[h_S^{PG}(x)]_i = [sgn(SBx)]_i$  for  $1 \leq i \leq k$ . We denote the family of all such hash functions as  $\mathcal{H}^{PG}(c,d)$ .
We use the term "periodic" to describe the hash functions described in Definition 4, since unlike SimHash which projects a point via a fully random Gaussian vector as in SimHash, our projection is accomplished using a repeating concatenation of a length  $c$  Gaussian vector. Furthermore, for  $h_S^{PG} \subset H^{PG}(c,d)$ , the matrix representation  $SB$  is a tiling of a single Gaussian matrix. Notice that we may easily extend the notion of a PGHash of one vector to an entire hash table over multiple vectors following Definition 3. In this manner, we may generate a sequence of hash tables  $\{h_{S_i}^{PG}(W)\}_{i=1}^{\tau}$  over a weight matrix  $W$  simply by drawing random Gaussian matrices  $S_i \in \mathbb{R}^{k \times c}$  for  $1 \leq i \leq \tau$  (where  $k$  is the hash length) and computing  $\operatorname{sgn}(S_i BW)$ .
Extension to DWTA (PGHash-D) Remark. When DWTA (described in Section 3) is preferred over SimHash for similarity estimation, we may modify Definition 4 as follows:  $B = D_{\mathbb{1}}P$  where  $P$  is a  $d \times d$  random permutation matrix, and  $D_{\mathbb{1}}$  is a  $c \times d$  rectangular diagonal matrix with  $D_{ii} = 1$  for  $1 \leq i \leq c$ . We denote our hash functions as  $h_S^{PG - D}: \mathbb{R}^d \to \mathbb{R}^k$  by  $h_S^{PG - D}(x) = \max_i [SBx]_i$  for  $1 \leq i \leq k$ , with  $k \leq c$ , where  $S$  is now a rectangular permutation matrix which selects  $k$  rows of  $Bx$  at random. We refer to this scheme as PGHash-D, whereas PGHash refers to Definition 4.
Local Memory Complexity Remark. When generating a new table using PGHash, a device maintains  $S$  and needs access to just  $BW$ , which costs  $\mathcal{O}(kc)$  and  $\mathcal{O}(cn)$  space complexity respectively. This is much smaller than the  $\mathcal{O}(kd)$  and  $\mathcal{O}(dn)$  local memory requirements of SimHash.
Sensitivity of  $\mathcal{H}^{PG}$ . In this section, we will explore the sensitivity of  $\mathcal{H}^{PG}(c,d)$ .
Definition 5. Let  $x \in \mathbb{R}^d$  and  $c \in \mathbb{R}$  such that  $c|d$ . Define the  $(d, c)$ -folding  $x_c \in \mathbb{R}^c$  of  $x$  as  $[x_c]_i = \sum_{j=1}^{d/c} [x]_{i+j \cdot \frac{d}{c}}$ . Equivalently,  $x_c = Bx$ , with  $B$  as specified in Definition 4.
Theorem 1. Let  $x, y \in R^d$ . Define the following similarity function  $\operatorname{Sim}_c^d(x, y) \triangleq \cos(x_c, y_c)$ , where  $x_c, y_c$  are  $(d, c)$ -foldings of  $x, y$ .  $\mathcal{H}^{PG}(c, d)$  is an LSH family with respect to  $\operatorname{Sim}_c^d$ .
Proof. Let  $h_v \in \mathcal{H}^{PG}$ . This means that for a randomly chosen  $v' \sim \mathcal{N}(0, I_{\frac{d}{c}})$ ,  $v$  is a  $c$ -times concatenation of  $v$ . We see that  $\operatorname{sgn}(v^\top x) = \operatorname{sgn}\left(\frac{v^\top x}{||v|| \cdot ||x||}\right) = \operatorname{sgn}\left(\sqrt{\frac{c}{d}} \frac{||x_c||}{||x||} \frac{v'^\top x_c}{||v'||||x_c||}\right)$ . Since  $\operatorname{sgn}$  is unaffected by the positive multiplicative factors, we conclude that  $\operatorname{sgn}(v^\top x) = \operatorname{sgn}(v'^\top x_c)$ . Through symmetric argument, we find  $\operatorname{sgn}(v^\top y) = \operatorname{sgn}(v'^\top y_c)$ . Since  $v' \sim \mathcal{N}(0, I_c)$ , comparing the sign of  $v^\top x$  to  $v^\top y$  is equivalent to a standard SimHash over  $x_c$  and  $y_c$ , i.e., estimation of  $\cos(x_c, y_c)$ .
Corollary 1. Let  $x, y \in \mathbb{R}^d$ , then  $\mathcal{H}^{PG}(c,d)$  is  $\left(S_c, \epsilon S_c, \left(1 - \arccos \left(\frac{S_c}{\pi}\right)\right), \left(1 - \arccos \left(\frac{\epsilon S_c}{\pi}\right)\right)\right)$ -sensitive where  $S_c = \cos(x_c, y_c)$ .
Proof. This follows directly from the well-known sensitivity of SimHash [6].
We see that  $\mathcal{H}^{PG}$  is LSH with respect to the angle between  $(d, c)$ -foldings of vectors. The use of periodic Gaussian vectors restricts the degrees of freedom (from  $d$  to  $d / c$ ) of our projections. However, the usage of pseudo-random and/or non-ideal hash tables has been observed to perform well in certain regimes[3, 35]. Although  $\mathcal{H}^{PG}(c, d)$  is LSH, it is  $\cos(x_c, y_c)$  necessarily an acceptable proxy for  $\cos(x, y)$ , in particular, for high angular similarity? Heuristically, yes, for highly-cosine similar vectors: assuming  $x$  and  $y$  are both unit (since scaling does not affect angle) then we have that  $\|x - y\|^2 = 2 - 2\cos(x, y)$ . If  $\ell_2$  similarity between  $x$  and  $y$  is already high, then the  $\ell_2$  similarity
of their (normalized)  $(d, c)$ -foldings will also be high, and thus their cosine similarity as well. We now provide a characterization of the angle distortion of a  $(d, c)$ -folding.
Theorem 2. Let  $x, y \in \mathbb{S}^{d-1}$ . Assume that neither  $x$  nor  $y$  vanish under multiplication by  $B$  and define the set  $S_{x,y} := \{v \in \operatorname{span}(x,y) : ||v|| = 1\}$ . We denote the following quantities:  $\lambda = d/c$ ,  $\theta := \frac{1}{2} \arccos(\cos(x,y))$ ,  $\alpha = \inf \{c > 0 \mid |||Bv||| \geq c, \forall v \in S_{x,y}\}$ , and  $\beta = \alpha/\lambda$ . Assume  $\alpha > 0$ . Then  $\cos(x_c, y_c)$  lives between  $\frac{1 - \beta^2 \tan^2 \theta}{1 + \beta^2 \tan^2 \theta}$  and  $-\frac{\tan^2 \theta^2 - \beta^2}{\tan^2 \theta^2 + \beta^2}$ .
Proof sketch. Consider the unit circle  $S_{x,y}$  contained in  $\operatorname{span}(x, y)$  (Let us assume  $x$  and  $y$  are unit, WLOG). The linear distortion  $BS_{x,y}$  is an ellipse containing  $x_c = Bx$  and  $y_c = By$ . The length of the axes of this ellipse is determined by the eigenvalues of  $B^\top B$ . The bounds follow from further trigonometric arguments, by considering when the axes of  $BS_{x,y}$  are maximally stretched and shrunk respectively. These distortions are strongly related to  $\lambda$  and  $\beta$ .
We can see that as  $\cos (x,y)\to 0$  we have  $\cos (x_c,y_c)\rightarrow 0$ . It is natural to consider the distribution of  $\alpha$  in Theorem 2 as how extreme shrinking by  $B$  (the folding matrix) can greatly distort the angle. We can characterize this statistical distribution exactly.