SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sin (q, x) = \int_ {- \infty} ^ {\infty} \sum_ {k \in [ K ]} \boldsymbol {S} _ {q} (\iota \omega_ {k}) ^ {\top} \boldsymbol {S} _ {x} (\iota \omega_ {k}) d \omega_ {k} = \int_ {\boldsymbol {\omega} \in \mathbb {R} ^ {K}} \boldsymbol {S} _ {q} (\iota \boldsymbol {\omega}) ^ {\top} \boldsymbol {S} _ {x} (\iota \boldsymbol {\omega}) d \boldsymbol {\omega} \tag {8}
$$
Here,  $S_{\bullet}(\iota \omega) = [S_{\bullet}(\iota \omega_1),\dots,S_{\bullet}(\iota \omega_K)],\omega = [\omega_1,\dots,\omega_K]$ . Note that, the expression of  $S_{q}(\iota \omega_{k})$  is different from  $S_{x}(\iota \omega_{k})$  in Eq. (7). This maintains the asymmetry in the final dot product in Eq. (8).
Inspired by the seminal work of Rahimi and Recht [40], several works [50, 37] have exploited Fourier transformations to approximate various functions using inner product between the feature maps. However, the functions that Rahimi and Recht [40] considered are shift invariant positive definite kernels. This allowed them to leverage Bochner's theorem [44] which establishes that the Fourier
transformation of these kernels are probability distributions. However, in Eq. (8), there is no such readily available probability distribution. In response, we attempt to find out a probability distribution  $p(\omega)$  which allows us to draw samples using an importance sampling like procedure, as follows:
$$
\operatorname {s i m} (q, x) = \mathbb {E} _ {\omega \sim p (\omega)} \left[ \boldsymbol {F} _ {q} (\iota \omega) ^ {\top} \boldsymbol {F} _ {x} (\iota \omega) \right], \text {w h e r e ,} \boldsymbol {F} _ {q} (\iota \omega) = \frac {\boldsymbol {S} _ {q} (\iota \omega)}{\sqrt {p (\omega)}}, \boldsymbol {F} _ {x} (\iota \omega) = \frac {\boldsymbol {S} _ {x} (\iota \omega)}{\sqrt {p (\omega)}}, \tag {9}
$$
Let  $\{\omega^j\}_{j = 1}^M\sim p(\omega)$  be  $M$  i.i.d random samples. We compute the Monte Carlo estimate as follows:
$$
\operatorname {s i m} (q, x) \approx \frac {1}{M} \sum_ {j \in [ M ]} \boldsymbol {F} _ {q} \left(\iota \boldsymbol {\omega} ^ {j}\right) ^ {\top} \boldsymbol {F} _ {x} \left(\iota \boldsymbol {\omega} ^ {j}\right) \propto \cos \left(\boldsymbol {F} _ {q} \left(\iota \boldsymbol {\omega} ^ {1.. M}\right), \boldsymbol {F} _ {x} \left(\iota \boldsymbol {\omega} ^ {1.. M}\right)\right) \tag {10}
$$
Here,  $F_{\bullet}(\iota \omega^{1\dots M}) = [F_{\bullet}(\iota \omega^{1}),..,F_{\bullet}(\iota \omega^{M})]$ . Note that, as suggested by Eqs. (7) and (9),  $||F_q(\iota \omega^{1\dots M})||_2 = ||F_x(\iota \omega^{1\dots M})||_2 = \sum_{j=1}^M \sum_{k=1}^K \frac{|\mathrm{Re}(S(\iota\omega_k^j))| + |\mathrm{Im}(S(\iota\omega_k^j))|}{p(\omega_k^j)}$ . Thus, the value is independent of the query or corpus, which leads to the proportionality relation. We choose the probability distribution  $p(\omega)$  guided the proportionality constant  $||F_{\bullet}(\iota \omega^{1\dots M})||$  and set  $p(\omega) = \prod_{k \in [K]} p(\omega)$ , where  $p(\omega) \propto |\mathrm{Re}(S(\iota \omega))| + |\mathrm{Im}(S(\iota \omega))|$ . However, the integral of these terms may not be bounded. Therefore, we set the support of  $p(\omega)$  between  $[-\omega_{\max}, \omega_{\max}]$ , thus eliminating the higher frequency terms. The effect on the overall score is small. Attenuation of the higher frequency signals can be seen as a multiplication with a low pass filter in the frequency domain, which affects a convolution in the time domain. Its impact on the similarity score is proportional to  $1 / \omega_{\max}$ . This still allows us for ALSH despite frequency truncation.
Theorem 3.2. (Proven in Appendix D) Let  $\pmb{q}$ ,  $\pmb{x} \in \mathbb{R}^K$ ,  $\cos^{-1}$  be Lipschitz with Lipschitz constant  $L_{\mathrm{cos}}$ ; the hyperparameter  $T$  in Eq. (4) be chosen such that  $T > ||\pmb{q} - \pmb{x}||_{\infty}$ ; the frequency sampling distribution  $p(\omega_k^j) \propto [|\mathrm{Re}(S(\omega_k^j))| + |\mathrm{Im}(S(\omega_k^j))|]$  with the support set  $\omega_k^j \in [-\omega_{\max}, \omega_{\max}]$  and the proportionality constant  $I(\omega_{\max}) = \int_{-\omega_{\max}}^{\omega_{\max}} [|\mathrm{Re}(S(\omega)| + |\mathrm{Im}(S(\omega))|] d\omega$ . Then, the mapping  $g(q)[i] = \mathrm{sign}(\mathbf{w}_i^\top \mathbf{F}_q(\omega^{1\dots M}))$  and  $h(x)[i] = \mathrm{sign}(\mathbf{w}_i^\top \mathbf{F}_x(\omega^{1\dots M}))$  where  $\mathbf{w}_i \sim N(0,\mathbb{I})$ , constitutes a  $(S_0, cS_0, p_1, p_2)$ -ALSH for some  $p_1$  and  $p_2$  if we choose the support set  $[-\omega_{\max}, \omega_{\max}]$  and the number of samples  $M$  as follows:
$$
\omega_ {\max } > \frac {4 K L _ {\cos}}{\pi (1 - c) S _ {0}} \left(6 + \frac {2 T}{T - \max  _ {\boldsymbol {x} , \boldsymbol {q}} \left\| \boldsymbol {x} - \boldsymbol {q} \right\| _ {\infty}}\right) \text {a n d} M > \left[ \frac {4 L _ {\cos}}{(1 - c) S _ {0}} \right] ^ {2} K I \left(\omega_ {\max }\right). \tag {11}
$$
Based on the outlined assumptions, the above Theorem guarantees that FOURIERHASHNET is an  $(S_0,cS_0,p_1,p_2)$ -ALSH for the asymmetric dominance similarity score, subject to appropriate choices of  $\omega_{\mathrm{max}}$  to bound the effect of frequency truncation and  $M$  to bound the variance of the Monte Carlo sample estimate.
3.3 Trainable hashing network
Random hyperplane LSH Eq. (10) provides an asymmetric transformation on the input querycorpus pair, which maps it into the cosine similarity space, thus allowing for Random Hyperplanes hashing. We sample  $H$  spherically symmetrically distributed normal vectors  $\{\mathbf{w}_i\}_{i = 1}^H$ , i.e.,  $\mathbf{w}_i\sim \mathcal{N}(0,\mathbb{I})$ , each perpendicular to a random hyperplane passing though the origin. For each query  $q$  and the corpus  $x$ , we can generate  $H$ -bit hashcodes  $g(q)$ ,  $h(x)\in \{\pm 1\} ^H$  from the Fourier features (10) as follows:  $g(q)[i] = \mathrm{sign}(\mathbf{w}_i^\top \mathbf{F}_q(\iota \omega^{1\dots M}))$  and  $h(x)[i] = \mathrm{sign}(\mathbf{w}_i^\top \mathbf{F}_x(\iota \omega^{1\dots M}))$ . Consequently, we can index the given corpus with  $N$  items, into a hash table with  $2^{H}$  buckets. For each query  $q$ , we restrict our search within bucket  $b = g(q)$ . If the corpus items are uniformly distributed across all buckets, then it enables sub-quadratic time retrieval with  $N / 2^{H}$  comparisons (per trial).
Data driven hashcode generation The above random hyperplane LSH approach suffers from two distinct limitations: (1) the quality of Monte Carlo approximation obtained in Eq. (10), depends on the suitability of  $p(\omega)$ , and (2) the hyperplanes are data oblivious. Data oblivious hyperplanes provide the best efficiency if the corpus embeddings are uniformly spread over the  $K$  dimensional sphere, which allows the random hyperplanes to evenly allocate the corpus items across different hashcodes. However, in practice, the spatial distribution of the embeddings is not uniform. This results in a skewed distribution of the corpus items across the hash buckets.
To tackle the first problem, we improve the quality of the Fourier features through a trainable nonlinear transformation. Here, we use two networks  $\phi_q$  and  $\phi_x$  which takes the Fourier features for the query and corpus, i.e.,  $F_q(\iota \omega^{1..M})$  and  $F_x(\iota \omega^{1..M})$  as input and outputs corresponding
transformed Fourier representations  $\mathbf{z}_q = \phi_q(\mathbf{F}_q(\iota \omega^{1\ldots M}))$  and  $\mathbf{z}_x = \phi_x(\mathbf{F}_x(\iota \omega^{1\ldots M}))$ . We train  $\phi_q$  and  $\phi_x$  by minimizing a BCE loss on  $\{\cos (\mathbf{z}_q,\mathbf{z}_x),\mathrm{rel}(q,x)\}$  pairs for  $q\in Q$  and  $x\in X$  as follows:
$$
\min  _ {\phi_ {q}, \phi_ {x}} \sum_ {q \in Q, x \in X} - [ \operatorname {r e l} (q, x) \log (1 + \cos \left(\boldsymbol {z} _ {q}, \boldsymbol {z} _ {x}\right)) + (1 - \operatorname {r e l} (q, x) \log (1 - \cos \left(\boldsymbol {z} _ {q}, \boldsymbol {z} _ {x}\right)) ] \tag {12}
$$
Next, we train the random hyperplanes  $\mathbf{W} = [\mathbf{w}_1, \mathbf{w}_2, \dots]$  using the transformed Fourier features  $\{\mathbf{z}_q\}$  and  $\{\mathbf{z}_x\}$ . The final hashcodes  $g(q)$  and  $h(x)$  are obtained as  $g(q) = \mathrm{sign}(\widehat{\mathbf{W}} \mathbf{z}_q)$ ,  $h(x) = \mathrm{sign}(\widehat{\mathbf{W}} \mathbf{z}_x)$ , where  $\widehat{\mathbf{W}}$  are the final trained random hyperplanes. For training purposes, we use  $\tanh(\mathbf{W} \bullet)$  as a smooth surrogate of  $\mathrm{sign}(\mathbf{W} \bullet)$ . The loss function  $\mathrm{loss}(Q, X | \mathbf{W})$  used to train  $\mathbf{W}$  consists of three components.
(1) Collision minimizer For any query  $q$ , our goal is to ensure that assigned bucket contains only positive items. Assuming corpus items are uniformly distributed across buckets, we ensure that for any query  $q$ , the  $N / 2^H$  most relevant items  $X_{q\checkmark}$  measured in terms of  $d(q,x)$  will have higher amount of bit overlap than rest of the items  $X_{q\checkmark}$ . Here,  $X_{q\checkmark}$  and  $X_{q\checkmark}$  indicate positive and negative silver instances (not gold instances) indicating top  $N / 2^H$  items in terms of the (possibly
trained) hinge distance  $d(q, x)$ . We encode this by minimizing the following ranking loss.
$$
\Delta_ {1} = \sum_ {q \in Q} \sum_ {x \in X _ {q \checkmark}, x ^ {\prime} \in X _ {q \times}} \left[ 1 + \tanh  \left(\boldsymbol {W} \boldsymbol {z} _ {q}\right) ^ {\top} \tanh  \left(\boldsymbol {W} \boldsymbol {z} _ {x ^ {\prime}}\right) - \tanh  \left(\boldsymbol {W} \boldsymbol {z} _ {q}\right) ^ {\top} \tanh  \left(\boldsymbol {W} \boldsymbol {z} _ {x}\right) \right] _ {+} \tag {13}
$$
Algorithm 1 FOURIERHASHNET
1: function  $\operatorname{Train}(X, Q, \{\operatorname{rel}(q, x)\}_{q \in Q, x \in X})$
2: Draw  $\omega^{1\dots M}\sim p(\omega)$
3: Compute  $\pmb{F}_{q}(\iota \omega^{1\ldots M}), \pmb{F}_{x}(\iota \omega^{1\ldots M})$  (Eq. (9))
4: Train  $\phi_q, \phi_x$  from  $\operatorname{rel}(q, x)$ ,  $F_{\bullet}(\iota \omega^{1..M})$  (Eq. (12))
5: Train  $\vec{W}$  by minimizing the loss (14)
6: Return  $\widehat{\phi_x},\widehat{\phi_q},\widehat{W}$
1: function Index  $(\{\pmb{F}_x(\iota \omega^{1..M})\}_{x\in X})$
2: Require: Trained networks  $\widehat{\phi_x}$ ,  $\widehat{W}$
3:  $h(x)\gets \mathrm{sign}(\widehat{\pmb{W}\phi_x} (\pmb {F}_x(\iota \omega^{1..M})))\forall x\in X$
4: for  $x\in X$  do
5: hash  $x$  to bucket  $b = h(x)$
6: Return the bucket sets  $B$
1: function Retrieve  $(q^{\prime})$
2: Require: Trained networks  $\widehat{\phi_q}$ ,  $\widehat{W}$
3: Compute  $\pmb{F}_q(\iota \omega^{1\ldots M})$  based on  $\pmb{q}'$
4:  $g(q') \gets \mathrm{sign}(\widehat{\pmb{W}}\widehat{\phi_q}(\pmb{F}_q(\iota\omega^{1\dots M})))$
5: Rank all  $x$  in the bucket  $b = g(q')$  based on the distance  $d(q', x)$  to obtain the list  $\operatorname{List}_{q'}$ .
6: Return  $\mathrm{List}_{q'}$
minimizing the following ranking loss.
This loss encourages that  $\tanh (\boldsymbol {W}\boldsymbol{z}_q)^\top \tanh (\boldsymbol {W}\boldsymbol{z}_x) > \tanh (\boldsymbol {W}\boldsymbol{z}_q)^\top \tanh (\boldsymbol {W}\boldsymbol{z}_{x'}) + 1$ , i.e., the number of common bits between  $q$  and  $x\in X_{q\checkmark}$  is at least one more than the same between  $q$  and  $x^{\prime}$ .
(2) Fence Sitting We set fence sitting loss as  $\Delta_2 = \sum_{x\in X}||\tanh (\pmb {W}\pmb {z}_x)| - 1||_1$  . This prevents the optimizer from arriving at a trivial solution by setting all hashcodes to zero.
(3) Bit Balance We set the bit balance loss as  $\Delta_3 = \sum_{i\in [H]}\left|\sum_{x\in X}\tanh (Wz_x)[i]\right|$ . This enforces that each position should have an equal number of  $+1$  and  $-1$ , thus ensuring that each random hyperplane evenly splits the set of points. Finally, we estimate  $\mathbf{W}$  by minimizing the loss, with  $\lambda_{\bullet}$  as hyperparameters such that  $\sum_{i}\lambda_{i} = 1$ , which is given as follows:
$$
\left. \operatorname {l o s s} (Q, X \mid \boldsymbol {W}) = \lambda_ {1} \Delta_ {1} + \lambda_ {2} \Delta_ {2} + \lambda_ {3} \Delta_ {3}, \right. \tag {14}
$$
Algorithm 1 summarizes the overall procedure.
Difference from existing trainable LSH LSH training has been extensively studied [54, 15, 43], with Fence Sitting and Bit Balance losses being well known. However, the Collision Minimizer loss differs significantly from existing approaches. Current techniques seek to ensure load balance across hash buckets for all corpus items, including the ones that may not be relevant to most queries. This is unnecessary for query workloads which touch upon only a small subset of the corpus to generate the best responses. In contrast, our Collision Minimizer loss ensures that only the top-most bucket for any given query allows relevant items and explicitly denies irrelevant items. Thus, it is informed by the query workload, rather than assuming load balance for all items in the corpus. Such an approach may result in balanced bucket loads, but not necessarily.
In this section, we provide a comprehensive evaluation of our method against several baselines and ablations on four datasets. Appendix F describes additional experiments. Our code is in https://github.com/structlearning/fhashnet.
Datasets We experiment on datasets sampled from anonymized real-world Web log data, viz, MSWEB and MSNBC. MSWEB [5] is generated using logs from www.microsoft.com, containing
(a) MSWEB-1
(b) MSWEB-2
Figure 2: Effect of different similarity measures on LSH, measured in terms of variation of MAP vs. average query time (in sec) for all methods. Here, the final score used for ranking the relevant items is that similarity score for which the LSH is designed for.
(c) MSNBC-1
(d) MSNBC-2
records of the areas of the website visited by the users. MSNBC [8] is a collection of logs of user page requests from msnbc.com. In both cases, a record (either  $q$  or  $x$ ), is a passage that is regarded as a bag of words. Given a collection  $V$  of such word bags,  $(|V| = 11234$  for MsWEB and  $|V| = 111290$  for MsNBC), we sample  $|Q| = 500$  bags from  $V$ , designating them as queries, and designate the rest as corpus items  $X = V \backslash Q$ . Consistent with typical information retrieval application scenarios [51], we generate gold relevance labels based on (multi)set containment for MsWEB (MsNBC). (Additional methods for evaluation are explored in Appendix F.) We build the query set  $Q$ , such that the number of relevant items  $N_{q\oplus} = |\{x \in X : \mathrm{rel}(q,x) = +1\}| \in [5,500]$  for each query  $q$ . We create four datasets by changing average relevance counts per query,  $\overline{N}_{q\oplus}$ . They are: (1) MsWEB-1 where  $\overline{N}_{q\oplus} = 35.624$ . (2) MsWEB-2 where  $\overline{N}_{q\oplus} = 20.392$ . (3) MsNBC-1 where  $\overline{N}_{q\oplus} = 24.09$ . (4) MsNBC-2 where  $\overline{N}_{q\oplus} = 19.78$ . The set of queries  $Q$  is partitioned into  $20\%$  training set  $Q_{\mathrm{tr}}$ ,  $20\%$  validation set  $Q_{\mathrm{dev}}$  and  $60\%$  test set  $Q_{\mathrm{test}}$ .
Design of query and corpus embeddings  $q, x$  We begin with a pre-trained sentence transformer model [41] to obtain 768 dimensional dense contextual representations  $\mathbf{feature}_q$  and  $\mathbf{feature}_x$  for the each word in bags  $q$  and  $x$ . Embeddings of words belonging to a bag are fed into a deep set [60] network to obtain a bag representation  $q, x \in \mathbb{R}^K$ , with  $K = 294$  (chosen via hyperparameter sweep). To train the parameters inside the deep set network, we use  $q, x$  to compute the proposed asymmetric hinge distance  $d(q, x)$  (1), feed it into a trainable sigmoid layer  $\sigma$  and minimize
$$
\sum_ {q, x} \operatorname {B C E} (\operatorname {r e l} (q, x), \sigma (- d (q, x))) \tag {15}
$$
which uses a BCE loss on the gold relevance labels. Once we obtain  $\mathbf{q}$  and  $\mathbf{x}$ , we use Algorithm 1 to obtain trained  $\widehat{\phi_q}$ ,  $\widehat{\phi_x}$  and  $\widehat{\mathbf{W}}$  (Train(·)), which are then used for indexing (Index(·)).
Evaluation Given a test query  $q \in Q_{\mathrm{test}}$  and a set of  $N_q'$  candidate corpus items, we rank them in increasing order of their hinge distances  $d(q, x)$ . Then we evaluate the average precision (AP) for the query and average over queries to report mean average precision (MAP) — see Appendix E.6.
4.2 Effect of different similarity measures on LSH
Setup Here, we compare FOURIERHASHNET against the three LSH baselines, viz., Random hyperlane (RH) [9], Dot product LSH (DP-RH) [35] and Weighted MinHash (WMH) [12], that are tailored towards cosine similarity, dot product similarity and Weighted Jaccard Similarity, respectively. For each LSH method, we train the embeddings  $\mathbf{q}$ ,  $\mathbf{x}$  and the final hashcodes  $g(q)$  and  $h(x)$  using the same networks, as in our method. Furthermore, we set the final relevance measure for ranking to be the similarity score for which the LSH is designed.
Results We vary the mixing hyperparameters  $\lambda_{1}$  and  $\lambda_{2}$  in our loss (14) and the number of buckets  $B$  to explore the tradeoff between accuracy (MAP) and average query time. In Figure 2, we summarize the results. We observe that: (1) FOURIERHASHNET outperforms all the baselines by providing significantly better time-vs.-MAP trade-off across all datasets. In MSWEB datasets, all the baselines except DP-RH show poor performance. All baselines perform poorly for the MSNBC dataset. We remark that cosine similarity, dot product or weighted Jaccard similarity are not suited for vector dominance search. Therefore, the maximum possible MAP obtained by them are severely constrained. (2) In MSWEB datasets, DP-RH performs moderately, by achieving a MAP value around 0.4-0.42 within 0.03 seconds (average query time). This is because dot product can be computed significantly faster than all the other distance/similarity measures. In particular, it is  $\sim 7.5\times$  faster than our hinge distance (1),  $\sim 10.3\times$  faster than cosine, and  $\sim 5.1\times$  faster than Jaccard similarity.
(b) MSWEB-2
Figure 3: Trade-off between query time and accuracy (MAP) for MSWEB and MSNBC datasets where there is  $\geq 10\mathrm{X}$  speedup compared to exhaustive search. We apply different LSH methods on hinge distance guided embeddings, viz, RH+Hinge, DP-RH+Hinge, WMH+Hinge, FLORA and FOURIERHASHNET; and, then use the hinge distance to finally rank the retrieved items.
4.3 Comparison against other efficient indexing techniques
In Section 4.2, we used the similarity score corresponding to each LSH method for final candidate ranking. The baselines performed poorly, which may result from a poor choice of final similarity score or the indexing method. Here, we evaluate FOURIERHASHNET against baseline indexing methods applied on hinge distance guided embeddings. We consider LSH and Inverted File Indexing (IVF) based indexing variants in this section, and discuss graph-based indexing methods in Appendix F.
Comparison with LSH based indexing. Shrivastava and Li [47] showed that an LSH not tailored to the final scoring function may still provide an effective filter. Accordingly, we set the final similarity function to be dominance similarity, and compare against four possible LSH baselines.
Given the embeddings  $\pmb{q}, \pmb{x}$  trained (15) using hinge distance, we feed them into the four baselines, each of which trains a hashing network in a different way. (1) RH+Hinge: We train a set of random hyperplanes represented by  $\pmb{W}$  and compute the hashcodes as  $h(q) = \mathrm{sign}(\pmb{W}\pmb{q})$  and  $h(x) = \mathrm{sign}(\pmb{W}\pmb{x})$ . (2) DP-RH+Hinge: We train random hyperplanes  $\pmb{W}$  for these embeddings to compute the hashcodes as  $g(q) = \mathrm{sign}(\pmb{W}[0,\pmb{q} / ||\pmb{q}||])$  and  $h(x) = \mathrm{sign}(\pmb{W}[\sqrt{T^2 - ||\pmb{x}||^2},\pmb{x}])$ . (3) WMH+Hinge: We use the best performing WMH implementation from DrHash toolkit [58] to obtain the hashcodes. (4) FLORA[15]: We train asymmetric hash networks  $(\mathrm{net}_1, \mathrm{net}_2)$  using an end-to-end data-driven approach, which minimizes bit balance and decorrelation loss, along with a consistency loss which predicts the final similarity score using  $\cos(\mathrm{net}_1(\pmb{q}), \mathrm{net}_2(\pmb{x}))$ .
Figure 3 compares the performance of FOURIERHASHNET, RH+Hinge, DP-RH+Hinge, WMH+Hinge and FLORA in terms of MAP for MSWEB and MSNBC datasets. Here we analyze the section of the trade-off curve which provides  $\geq 10\mathrm{X}$  speedup compared to exhaustive search. The complete tradeoff curve is provided in Appendix F. (1) The newly designed baselines are now seen to perform significantly better than those used in the previous experiments with Figure 2. However FOURIERHASHNET still outperforms all the baselines. (2) RH+Hinge, despite achieving the second highest scores in many cases, is seen to suffer from a large variance in performance within any given time budget. This would make it difficult to tune the hyperparameters to achieve the requisite performance v/s retrieval speed trade-off. (3) DP-RH+Hinge is seen to have a significantly worse performance than FOURIERHASHNET everywhere. This indicates that DP-RH is ill-suited to asymmetric hinge distance based retrieval.(4) We observe that for the same amount of query time invested, FLORA's MAP can lag ours by over  $10\%$  , particularly when faster average query times are required. FLORA's hyperparameter tuning is also more delicate, with there being unsuccessful settings (where MAP grows very slowly with query time) very close to relatively successful ones.
Comparison with IVF indexing We use the widely used FAISS-IVF [21] library, which supports IVF indexing based on L2 distance (IVF-L2) and Inner Product similarity (IVF-IP). Additionally, we propose an alternative Fourier+IVF+IP, where we apply Fourier transformation on the input embeddings, before using IVF-IP. We provide embeddings  $\mathbf{q}$ ,  $\mathbf{x}$ , trained using hinge distance to all the methods, and use hinge distance to rank retrieved items.
Figure 4 compares the performances in terms of MAP, across all datasets. We observe that: (1) FOURIERHASHNET outperforms both IVF-L2 and IVF-IP across all datasets. FAISS-IVF retrieval suffers because its quantizers, that assign vectors to the Voronoi cells, rely on a metric like L2 or IP, which are unsuitable for asymmetric hinge distance. (2) Fourier transformation provides a significant boost in performance across all datasets, as seen while comparing Fourier+IVF+IP against IVF+IP. However, FOURIERHASHNET still outperforms Fourier+IVF+IP, most noticeably in MSWEB-2.
(b) MSWEB-2
Data driven vs data oblivious LSH To perform ablation study on our proposed hashcode training method, we propose an alternative FHash (UN-TRAINED). This applies our Fourier features followed by a data oblivious random hyperplane LSH, without any data driven hashcode training.
In Figure 5, we compare the complete design of our method, i.e., FOURIERHASHNET and FHash (UNTRAINED) against the untrained versions of
Figure 4: Trade-off between number of corpus items being evaluated and accuracy (MAP) for MSWEB and MSNBC datasets where there is  $\geq 10X$  speedup compared to exhaustive search. We apply FOURIERHASHNET and different IVF methods, viz, IVF+L2, IVF+IP, and Fourier+IVF+IP, on hinge distance guided embeddings; and then use the hinge distance to finally rank the retrieved items.
(a) MSWEB-1
Figure 5: Effect of untrained RH
RH+Hinge and DP-RH+Hinge. We make the following observations: (1) Benefit of Fourier transformation: The MAP vs time trade-off curve of FHash (UNTRAINED), consistently dominates all the baselines across both datasets. (2) Benefit of hashcode training: Compared to FHash (UNTRAINED), we observe that FOURIERHASHNET allows for significantly more choices of trade-off points, where higher MAP is required.
Ablation on collision minimizer Here, we replace the collision minimizer in  $\mathrm{loss}(Q,X\mid W)$  (14) with decorrelation loss which encourages hashcodes to be dissimilar:  $\Delta_1 = \sum_{x\neq y}|\tanh (\pmb {W}\pmb {z}_x)^\top \tanh (\pmb {W}\pmb {z}_y)|$  , a commonly used loss in prior work [54, 15].