SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 6 compares the performance of the two variations of the losses in terms of MAP, for MSWEB datasets. We observe that: (1) Our loss containing the collision minimizer term performs better than its variant which uses the decorrelation loss. In MSWEB-2, latter provides a MAP of 0.4 in 0.014 secs, which our loss achieves in  $50\%$  of the time. (2) Our method allows for greater freedom in navigating the performance vs average query time trade-off, as seen in MSWEB-2, as it is more spread out across the time axis.
(a) MSWEB-1
Figure 6: Collision minimizer vs. decorrelation.
We have presented FOURIERHASHNET, an ALSH for asymmetric hinge distance, strongly motivated by text, image and graph retrieval applications. By converting hinge distance to a proposed dominance similarity and applying a suitable Fourier transform to the dominance similarity, we can estimate the distance as an inner product over an importance-sampled spectrum, which further enables the use of a trainable LSH in the frequency domain. Experiments show that FOURIERHASHNET dramatically speeds up queries while preserving or improving retrieval accuracy. Our approach can be extended to any shift invariant functions including Chamfer distance, box embeddings, etc. Box embeddings are known to model more complex set operations like set overlap and set difference [42, 13], making them an interesting avenue for future research. One limitation of FOURIERHASHNET compared to simple symmetric LSHs is the increase in computational cost to compute the Fourier transform. One can explore other types of transformations to mitigate this cost.
[52] Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language. arXiv preprint arXiv:1511.06361, 2015. URL https://arxiv.org/pdf/1511.06361.
[53] Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. Feature hashing for large scale multitask learning. In ICML, pages 1113-1120, 2009. URL https://arxiv.org/pdf/0902.2206.pdf.
[54] Yair Weiss, Antonio Torralba, and Rob Fergus. Spectral hashing. Advances in neural information processing systems, 21, 2008.
[55] Wei Wu, Bin Li, Ling Chen, and Chengqi Zhang. Canonical consistent weighted sampling for real-value weighted min-cache. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pages 1287-1292. IEEE, 2016.
[56] Wei Wu, Bin Li, Ling Chen, and Chengqi Zhang. Consistent weighted sampling made more practical. In Proceedings of the 26th International Conference on World Wide Web, pages 1035-1043, 2017.
[57] Wei Wu, Bin Li, Ling Chen, Chengqi Zhang, and S Yu Philip. Improved consistent weighted sampling revisited. IEEE Transactions on Knowledge and Data Engineering, 31(12):2332-2345, 2018.
[58] Wei Wu, Bin Li, Ling Chen, Junbin Gao, and Chengqi Zhang. A review for weighted minhash algorithms. IEEE Transactions on Knowledge and Data Engineering, 34(6):2553-2573, 2022.
[59] Xiao Yan, Jinfeng Li, Xinyan Dai, Hongzhi Chen, and James Cheng. Norm-ranging lsh for maximum inner product search. 31, 2018.
[60] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Locality Sensitive Hashing in Fourier Frequency Domain For Soft Set Containment Search (Appendix)
A Limitations of our work
(1) In Eq. (9), the probability distribution  $p(\omega)$  is determined based on the proportionality constant  $||F_{\bullet}(\iota \omega^{1\dots M})||$ , and we set  $p(\omega) = \prod_{k\in [K]}p(\omega)$ , where  $p(\omega)\propto |\mathrm{Re}(S(\iota \omega))| + |\mathrm{Im}(S(\iota \omega))|$ . We note that this choice of distribution is not informed by the data distribution. Doing so may further improve FOURIERHASHNET.
(2) In our approach, the dominance similarity function is represented as an expectation of inner products of functions in the frequency domain, as described in Eq. (10). The accuracy of this representation relies on the quality of Monte Carlo approximations, which is influenced by the number of  $\omega$  samples used. Our experimental findings, presented in Figure 13, suggest that it may be necessary to generate up to 100 samples per dimension to reduce the approximation error to acceptable levels. A better choice of  $p(\omega)$  may reduce the number of samples needed.
(3) During our experimental investigations, we discovered that the computation of our proposed dominance similarity score is approximately 7.5X slower, compared to the dot product similarity on our datasets. This aligns with earlier observations where matrix subtraction operations have been known to be significantly slower than dot product. This disparity in computation speed represents another potential limitation of our approach, which could be addressed by exploring alternative designs for the dominance similarity function.
B Example applications of soft set containment
Natural Language Inference (NLI) In (NLI) [4],  $q$  and  $x$  are sentences, regarded as sequences of words as items. A transformer network [14, 41] converts each sentence to an embedding vector. We infer  $x \Rightarrow q$  if  $q \leq x$  [26]. Consider now a claim verification application that, given a claim as a query and a Web-scale corpus, needs to quickly retrieve passages that best support the given claim. This application exactly motivates FOURIERHASHNET.
Market basket Given a basket of supermarket items, we may query purchase logs for frequent superset to make recommendations. The corpus contains itemsets purchased in the past, each is a 'document'  $x$ . The query  $q$  is the current basket. Hard set containment tests for  $q \subset x$ . However, we would like to 'soften' items in  $q$  (say, from one toothpaste brand to another. Each item has a short textual description, which is passed into BERT [41] and the [CLS] embedding read out as the item representation. An itemset is then embedded using a (suitably fine-tuned) set encoder (such as Deep Set), giving us  $q$  and  $x$ . To the extent  $q \leq x$ , we regard the query basket as "soft-contained" in the document basket.
Knowledge graph (KG) completion Vendrov et al. [52] embedded types and entities in a KG to vectors such that if two types are related via  $t_1$  is-subtype-of  $t_2$  then  $\vec{t_1} \leq \vec{t_2}$  was encouraged, and if entity  $e$  is-instance-of  $t$ , then  $\vec{e} \leq \vec{t}$  was encouraged in suitable loss functions. Later, these order embeddings were generalized to box embeddings where is-subtype-of and is-instance-of were modeled as boxes in high dimensions contained in other boxes [10]. These models naturally motivate fast retrieval using dominance similarity.
Subgraph isomorphism search Here we expect a corpus graph  $x$  will be relevant if query graph  $q$  is a subgraph of  $x$ , i.e., that  $x$  has a subgraph that is isomorphic to  $q$ . In reality, we want to score highly corpus graphs that have subgraphs almost isomorphic to the query graph. A graph neural network (GNN) [25, 18] can build suitable contextual embeddings  $q$  and  $x$  for the entire graphs, which can be used to test for approximate subgraph isomorphism, i.e.,  $q \leq x$ . There are several applications of subgraph isomorphism search. In material and drug design, there are large molecule databases. A researcher wishes to predict properties of a new query molecule by retrieving similar molecules in the database. Each molecule is modeled as a modest-sized graph with nodes (atom, DNA bases) and edges (valence, etc.). In image search [22], the query  $q$  may be a graph fragment, e.g.,  $\langle \text{person, feeding, pet} \rangle$ , and the goal is to find corpus graphs  $x$  where  $q$  is approximately a subgraph [31], e.g.,  $x$  can contain  $\langle \text{man, feeding, dog} \rangle$ .
C Further discussion on related work
In this section, we discuss existing work related to each of the three major components of our work—set embeddings, use of frequency domain for computing representations, and locality sensitive hashing.
Neural set embeddings Motivated by various machine learning questions that are better formalized by using a set of items as primitive, there has also been a recent line work on set embeddings. Zaheer et al. [60] consider models that act on sets and characterize the structure of such permutation-invariant models, but do not consider asymmetric query based measures e.g., containment. Skianis et al. [48] casts the similarity measurement between sets as a combinatorial flow problem, which in turn is approximated by a linear program. Lee et al. [27] propose the Set Transformer, a model that uses self-attention to model interactions among the input set elements. Our model is different from the existing line of work in focusing on asymmetric metrics that can measure containment, as well as in using the frequency domain representation of the metric to build a scalable LSH.
Application of frequency domain transformation in machine learning One of the most celebrated uses of the frequency domain representation was by [40], who proposed using random Fourier features for shift-invariant kernels. Since dot-product kernels are not shift-invariant, Bochner's theorem, a key tool in creating random Fourier features, does not apply. Hence, alternative random feature-based approximations have been proposed, primarily focusing on polynomial kernels [23, 38, 3, 49, 30]. All of the above work is on symmetric kernels. For our asymmetric dominance similarity function, we design a sampling distribution by taking into account the individual frequency-level coefficients of the Fourier representation. Raginsky and Lazebnik [39] provided a random Fourier based approach for LSH. However they only focused on similarity of the Mercer Kernel class, which is symmetric in nature.
Locality sensitive hashing The third main pillar of our work is locality sensitive hashing (LSH) which enables efficient retrieval. Answering queries using sketches or hashes in order to measure the similarity or containment of documents has a long history, pioneered by Broder [6]. In more recent years, semantic search or vector search, sometimes called "dense passage retrieval" [24] employing scalable near-neighbor search engines, has emerged as a credible, often more powerful, alternative [34, 29] to standard information retrieval schemes, as the vector embeddings can capture more nuanced contextualization and semantics. While there are a number of variants of LSH, including multi-probe [32], our presentation and experimentation focus on a single-probe setting in which the hashing hyperplanes are learned from the data. In particular, we build upon the asymmetric hash constructions in Shrivastava and Li [46] and Neyshabur and Srebro [35].
D Proofs of the technical results
D.1 Proof of Proposition 3.1
Proof. Consider two functions  $\mathrm{BOX}_{a,b}$  (for arbitrary positive constants  $a, b$ ) and  $\mathrm{RELUT}$  defined as follows:
$$
\operatorname {B O X} _ {a, b} (t) = \left\{ \begin{array}{l} a \text {i f} - b \leq t \leq b \\ 0 \text {e l s e} \end{array} \right. \tag {16}
$$
$$
\operatorname {R E L U} _ {T} (t) = \left\{ \begin{array}{l} t \text {i f} 0 \leq t \leq T \\ 0 \text {e l s e} \end{array} \right. \tag {17}
$$
Observe that  $s$  can be written in terms of these new functions as follows:  $s(t) = \mathrm{BOX}_{T,T}(t) - \mathrm{RELU}_T(t)$ . By linearity of Fourier transform,
$$
S (\iota \omega) = \mathcal {F} _ {\mathrm {B O X} _ {T, T}} (\iota \omega) - \mathcal {F} _ {\mathrm {R E L U} _ {T}} (\iota \omega) \tag {18}
$$
where,  $\mathcal{F}_f(\iota \omega)$  denotes Fourier transform of  $f(t)$  for any function  $f$ . Now let us compute  $\mathcal{F}_{\mathrm{RELUT}}(\iota \omega)$ .
$$
\begin{array}{l} \mathcal {F} _ {\mathrm {R E L U} _ {T}} (\iota \omega) = \frac {1}{2 \pi} \int_ {0} ^ {T} t e ^ {- \iota \omega t} d t (19) \\ = - \frac {1}{2 \pi \omega^ {2}} + \frac {e ^ {- \iota \omega T}}{2 \pi \omega^ {2}} + \frac {\iota T e ^ {- \iota \omega T}}{2 \pi \omega} (20) \\ = - \frac {1}{2 \pi \omega^ {2}} + \frac {\cos (\omega T) - \iota \sin (\omega T)}{2 \pi \omega^ {2}} + \frac {T (\iota \cos (\omega T) + \sin (\omega T))}{2 \pi \omega} (21) \\ = \frac {- 2 \sin^ {2} (\omega T / 2)}{2 \pi \omega^ {2}} + \frac {T \sin (\omega T)}{2 \pi \omega} - \iota \frac {\sin (\omega T)}{2 \pi \omega^ {2}} + \iota \frac {T \cos (\omega T)}{2 \pi \omega} (22) \\ \end{array}
$$
Since  $\mathrm{Box}_{T,T}(t)$  is a rectangular pulse, its Fourier transform is
$$
\mathcal {F} _ {\mathrm {B O X} _ {T, T}} (\iota \omega) = 2 T \frac {\sin (\omega T)}{2 \pi \omega} \tag {23}
$$
Substituting the above Eqs. (22) and (23) into Eq. (18), we get  $S(\iota \omega)$  as follows.
$$
\begin{array}{l} S (\iota \omega) = \underbrace {2 T \frac {\sin (\omega T)}{2 \pi \omega}} _ {\operatorname {B o x} _ {T, T} (\iota \omega)} - \underbrace {\left[ \frac {- 2 \sin^ {2} (\omega T / 2)}{2 \pi \omega^ {2}} + \frac {T \sin (\omega T)}{2 \pi \omega} - \iota \frac {\sin (\omega T)}{2 \pi \omega^ {2}} + \iota \frac {T \cos (\omega T)}{2 \pi \omega} \right]} _ {G (\iota \omega)} (24) \\ = \overbrace {T \frac {\sin (\omega T)}{2 \pi \omega} + 2 \frac {\sin^ {2} \left(\frac {\omega T}{2}\right)}{2 \pi \omega^ {2}}} ^ {\operatorname {R e} (S (\iota \omega))} + \iota \underbrace {\left[ \frac {\sin (\omega T)}{2 \pi \omega^ {2}} - \frac {T \cos (\omega T)}{2 \pi \omega} \right]} _ {\operatorname {I m} (S (\iota \omega))} (25) \\ \end{array}
$$
D.2 Proof of Theorem 3.2
Theorem 3.2. Let  $\pmb{q}, \pmb{x} \in \mathbb{R}^{K}$ ,  $\cos^{-1}$  be Lipschitz with Lipschitz constant  $L_{\mathrm{cos}}$ ; the hyperparameter  $T$  in Eq. (4) be chosen such that  $T > ||\pmb{q} - \pmb{x}||_{\infty}$ ; the frequency sampling distribution  $p(\omega_k^j) \propto [|\operatorname{Re}(S(\omega_k^j))| + |\operatorname{Im}(S(\omega_k^j))|]$  with the support set  $\omega_k^j \in [-\omega_{\max}, \omega_{\max}]$ . Furthermore, we define  $\Delta_{\mathrm{sim}} := \frac{T}{T - \max_{\pmb{x}, \pmb{q}} ||\pmb{x} - \pmb{q}||_{\infty}}$  and  $I(\omega_{\max}) = \int_{-\omega_{\max}}^{\omega_{\max}} [|\operatorname{Re}(S(\omega)| + |\operatorname{Im}(S(\omega))|] d\omega$ . Then, the mapping  $g(q)[i] = \mathrm{sign}(\mathbf{w}_i^\top \mathbf{F}_q(\omega^{1\dots M}))$  and  $h(x)[i] = \mathrm{sign}(\mathbf{w}_i^\top \mathbf{F}_x(\omega^{1\dots M}))$  where  $\mathbf{w}_i \sim N(0,\mathbb{I})$ , constitutes a  $(S_0, cS_0, p_1, p_2)$ -ALSH for some  $p_1$  and  $p_2$  if
$$
\omega_ {\max } > \frac {4 \Delta_ {\operatorname {s i m}}}{\pi (1 - c) S _ {0}}; \quad M > \left[ \frac {4 L _ {\cos}}{(1 - c) S _ {0}} \right] ^ {2} K I (\omega_ {\max }) \tag {26}
$$
Proof Define  $\Omega = [-\omega_{\max}, \omega_{\max}]^K$ . For  $\omega \in \Omega$ , we have the following relationships:
$$
\left| \left| \boldsymbol {F} _ {q} \left(\omega^ {1 \dots M}\right) \right| \right| _ {2} ^ {2} = \left| \left| \boldsymbol {F} _ {x} \left(\omega^ {1 \dots M}\right) \right| \right| _ {2} ^ {2} = \sum_ {i \in [ M ], k \in [ K ]} \frac {\left| R e \left(S \left(\omega_ {k} ^ {j}\right)\right) \right| + \left| I m \left(S \left(\omega_ {k} ^ {j}\right)\right) \right|}{p \left(\omega_ {k} ^ {j}\right)} = M K I; \tag {27}
$$
We also define that  $\widehat{\mathrm{sim}}_{\omega_{\max}}(q,x) = \int_{\omega \in \Omega}F_q(\iota \omega)^\top F_x(\iota \omega)p(\omega)d\omega$ . This gives us the following result:
$$
\begin{array}{l} \Pr_ {g, h} [ g (q) = h (x) | \boldsymbol {\omega} ^ {i} ] = 1 - \frac {1}{\pi} \cos^ {- 1} \left(\frac {\boldsymbol {F} _ {q} \left(\boldsymbol {\omega} ^ {1 \dots M}\right) ^ {\top} \boldsymbol {F} _ {x} \left(\boldsymbol {\omega} ^ {1 \dots M}\right)}{\left| \left| \boldsymbol {F} _ {q} \left(\boldsymbol {\omega} ^ {1 \dots M}\right) \right| \right| \left| \left| \boldsymbol {F} _ {x} \left(\boldsymbol {\omega} ^ {1 \dots M}\right) \right| \right|}\right) (28) \\ = 1 - \frac {1}{\pi} \cos^ {- 1} \left(\int_ {\omega \in \Omega} \frac {\boldsymbol {F} _ {q} (\iota \omega) ^ {\top} \boldsymbol {F} _ {x} (\iota \omega) p (\omega) d \omega}{| | \boldsymbol {F} _ {q} (\omega) | | | | \boldsymbol {F} _ {x} (\omega) | |}\right) \\ - \frac {1}{\pi} \cos^ {- 1} \left(\frac {\boldsymbol {F} _ {q} \left(\boldsymbol {\omega} ^ {1 \dots M}\right) ^ {\top} \boldsymbol {F} _ {x} \left(\boldsymbol {\omega} ^ {1 \dots M}\right)}{M K I \left(\omega_ {\max }\right)}\right) + \frac {1}{\pi} \cos^ {- 1} \left(\int_ {\boldsymbol {\omega} \in \Omega} \frac {\boldsymbol {F} _ {q} \left(\iota \boldsymbol {\omega}\right) ^ {\top} \boldsymbol {F} _ {x} \left(\iota \boldsymbol {\omega}\right) p (\boldsymbol {\omega}) d \boldsymbol {\omega}}{\underbrace {\left\| \boldsymbol {F} _ {q} (\boldsymbol {\omega}) \right\| \left\| \boldsymbol {F} _ {x} (\boldsymbol {\omega}) \right\|} _ {K I \left(\omega_ {\max }\right)}}\right) (29) \\ = 1 - \frac {1}{\pi} \cos^ {- 1} \left(\frac {\operatorname {s i m} (q , x)}{K I (\omega_ {\max })}\right) + \underbrace {\frac {1}{\pi} \cos^ {- 1} \left(\frac {\operatorname {s i m} (q , x)}{K I (\omega_ {\max })}\right) - \frac {1}{\pi} \cos^ {- 1} \left(\widehat {\frac {\operatorname {s i m} _ {\omega_ {\max }} (q , x)}{K I (\omega_ {\max })}}\right)} _ {\text {T e r m - 1}} \\ \underbrace {- \frac {1}{\pi} \cos^ {- 1} \left(\frac {\boldsymbol {F} _ {q} \left(\boldsymbol {\omega} ^ {1 \dots M}\right) ^ {\top} \boldsymbol {F} _ {x} \left(\boldsymbol {\omega} ^ {1 \dots M}\right)}{M K I \left(\omega_ {\max }\right)}\right) + \frac {1}{\pi} \cos^ {- 1} \left(\int_ {\omega \in \Omega} \frac {\boldsymbol {F} _ {q} \left(\iota \boldsymbol {\omega}\right) ^ {\top} \boldsymbol {F} _ {x} \left(\iota \boldsymbol {\omega}\right) p (\boldsymbol {\omega}) d \boldsymbol {\omega}}{K I \left(\omega_ {\max }\right)}\right)} _ {\text {T e r m - 2}} (30) \\ \end{array}
$$
Next we bound Term-1 and Term-2. First we bound Term-1 as follows:
$$
\begin{array}{l} \left| \frac {1}{\pi} \left[ \cos^ {- 1} \left(\frac {\operatorname {s i m} (q , x)}{K I \left(\omega_ {\max }\right)}\right) - \cos^ {- 1} \left(\frac {\widehat {\operatorname {s i m}} _ {\omega_ {\max }} (q , x)}{K I \left(\omega_ {\max }\right)}\right) \right] \right| \leq \frac {L _ {\cos}}{\pi K I} \left| \operatorname {s i m} (q, x) - \widehat {\operatorname {s i m}} _ {\omega_ {\max }} (q, x) \right| (31) \\ \leq \frac {\underbrace {K L _ {\cos} \left(6 + \frac {2 T}{T - \operatorname* {m a x} _ {\boldsymbol {x} , \boldsymbol {q}} \left\| \boldsymbol {x} - \boldsymbol {q} \right\| _ {\infty}}\right)} _ {\Delta_ {\mathrm {s i m}}} (32) \\ \end{array}
$$
The last inequality is due to Lemma D.1. Next we bound Term-2 in Eq. (30).
$$
\begin{array}{l} \frac {1}{\pi} \mathbb {E} \left[ \left| - \cos^ {- 1} \left(\frac {\boldsymbol {F} _ {q} \left(\boldsymbol {\omega} ^ {1 \dots M}\right) ^ {\top} \boldsymbol {F} _ {x} \left(\boldsymbol {\omega} ^ {1 \dots M}\right)}{M K I \left(\omega_ {\max }\right)}\right) + \cos^ {- 1} \left(\int_ {\boldsymbol {\omega} \in \Omega} \frac {\boldsymbol {F} _ {q} (\iota \boldsymbol {\omega}) ^ {\top} \boldsymbol {F} _ {x} (\iota \boldsymbol {\omega}) p (\boldsymbol {\omega}) d \boldsymbol {\omega}}{K I \left(\omega_ {\max }\right)}\right) \right| \right] (33) \\ \leq \frac {L _ {\cos}}{\pi K I} \left| \boldsymbol {F} _ {q} \left(\boldsymbol {\omega} ^ {1 \dots M}\right) ^ {\top} \boldsymbol {F} _ {x} \left(\boldsymbol {\omega} ^ {1 \dots M}\right) / M - \int_ {\boldsymbol {\omega} \in \Omega} \boldsymbol {F} _ {q} (\iota \boldsymbol {\omega}) ^ {\top} \boldsymbol {F} _ {x} (\iota \boldsymbol {\omega}) p (\boldsymbol {\omega}) d \boldsymbol {\omega} \right| (34) \\ = \frac {L _ {\cos}}{\pi K I \left(\omega_ {\max }\right)} \left| \sum_ {j \in [ M ]} \boldsymbol {F} _ {q} \left(\boldsymbol {\omega} ^ {j}\right) ^ {\top} \boldsymbol {F} _ {x} \left(\boldsymbol {\omega} ^ {j}\right) / M - \int_ {\boldsymbol {\omega} \in \Omega} \boldsymbol {F} _ {q} (\iota \boldsymbol {\omega}) ^ {\top} \boldsymbol {F} _ {x} (\iota \boldsymbol {\omega}) p (\boldsymbol {\omega}) d \boldsymbol {\omega} \right| (35) \\ \end{array}
$$
Taking expectation with respect to  $\omega$ , we have:
$$
\begin{array}{l} \frac {1}{\pi} \mathbb {E} \left[ \left| - \cos^ {- 1} \left(\frac {\boldsymbol {F} _ {q} \left(\boldsymbol {\omega} ^ {1 \dots M}\right) ^ {\top} \boldsymbol {F} _ {x} \left(\boldsymbol {\omega} ^ {1 \dots M}\right)}{M K I \left(\omega_ {\max }\right)}\right) + \cos^ {- 1} \left(\int_ {\omega} \frac {\boldsymbol {F} _ {q} (\iota \boldsymbol {\omega}) ^ {\top} \boldsymbol {F} _ {x} (\iota \boldsymbol {\omega}) p (\boldsymbol {\omega}) d \boldsymbol {\omega}}{K I \left(\omega_ {\max }\right)}\right) \right| \right] (36) \\ \leq \frac {L _ {\cos}}{\pi K I \left(\omega_ {\max }\right)} \mathbb {E} \left| \sum_ {j \in [ M ]} \boldsymbol {F} _ {q} \left(\boldsymbol {\omega} ^ {j}\right) ^ {\top} \boldsymbol {F} _ {x} \left(\boldsymbol {\omega} ^ {j}\right) / M - \int_ {\omega} \boldsymbol {F} _ {q} (\iota \boldsymbol {\omega}) ^ {\top} \boldsymbol {F} _ {x} (\iota \boldsymbol {\omega}) p (\boldsymbol {\omega}) d \boldsymbol {\omega} \right| (37) \\ \leq \frac {L _ {\cos}}{\pi K I \left(\omega_ {\max }\right)} \left(\operatorname {V a r i a n c e} \left[ \sum_ {j \in [ M ]} \boldsymbol {F} _ {q} \left(\boldsymbol {\omega} ^ {j}\right) ^ {\top} \boldsymbol {F} _ {x} \left(\boldsymbol {\omega} ^ {j}\right) / M \right]\right) ^ {1 / 2} \quad (\mathbb {E} [ | Z | ] \leq \sqrt {\mathbb {E} [ | Z | ^ {2} ]}) (38) \\ = \frac {L _ {\cos}}{\pi K I \left(\omega_ {\max }\right) \sqrt {M}} \left(\text {V a r i a n c e} \left[ \boldsymbol {F} _ {q} (\iota \boldsymbol {\omega}) ^ {\top} \boldsymbol {F} _ {x} (\iota \boldsymbol {\omega}) \right]\right) ^ {1 / 2} (39) \\ = \frac {L _ {\cos}}{\pi K I \left(\omega_ {\max }\right) \sqrt {M}} \sqrt {K} \left(\operatorname {V a r i a n c e} \left[ \boldsymbol {F} _ {q} \left(\omega_ {k}\right) ^ {\top} \boldsymbol {F} _ {x} \left(\omega_ {k}\right) \right]\right) ^ {1 / 2} \leq \frac {L _ {\cos}}{\pi \sqrt {K M}} (40) \\ \end{array}
$$
The last inequality follows from bound on the variance due the following:
$$
\begin{array}{l} \boldsymbol {F} _ {q} \left(\omega_ {k}\right) ^ {\top} \boldsymbol {F} _ {x} \left(\omega_ {k}\right) = \boldsymbol {S} _ {q} \left(\omega_ {k}\right) ^ {\top} \boldsymbol {S} _ {x} \left(\omega_ {k}\right) / p \left(\omega_ {k}\right) (41) \\ = \frac {\operatorname {R e} [ \boldsymbol {S} (\omega) ] \cos \omega_ {k} (q [ k ] - x [ k ]) - \operatorname {I m} [ \boldsymbol {S} (\omega) ] \sin \omega_ {k} (q [ k ] - x [ k ]))}{(1 / I) [ | \operatorname {R e} [ \boldsymbol {S} (\omega) ] | + | \operatorname {I m} [ \boldsymbol {S} (\omega) ] | ]} (42) \\ \leq I \frac {\left| R e [ \boldsymbol {S} (\omega_ {k}) ] \right| + \left| I m [ \boldsymbol {S} (\omega_ {k}) ] \right|}{\left| R e [ \boldsymbol {S} (\omega) ] \right| + \left| I m [ \boldsymbol {S} (\omega) ] \right|} = I (43) \\ \end{array}
$$
Putting Eqs. (32) (40) into Eq. (30), we will have the upper bound for  $\sin(q, x) \leq cS_0$ :
$$
\mathbb {E} _ {g, h} [ g (q) = h (x) | \boldsymbol {\omega} ^ {j} ] ] \leq p _ {2} = 1 - \frac {1}{\pi} \cos^ {- 1} \left(\frac {\operatorname {s i m} (q , x)}{K I \left(\omega_ {\max }\right)}\right) + \frac {L _ {\cos}}{\pi \sqrt {K M}} + \frac {\Delta_ {\operatorname {s i m}}}{\pi^ {2} K I \left(\omega_ {\max }\right) \omega_ {\max }} \tag {44}
$$
Similarly, we have the lower bound for  $\sin (q,x) > S_0$ :
$$
\mathbb {E} \left[ \Pr_ {g, h} [ g (q) = h (x) | \boldsymbol {\omega} ^ {j} ] \right] \geq p _ {1} = 1 - \frac {1}{\pi} \cos^ {- 1} \left(\frac {\sin (q , x)}{K I \left(\omega_ {\max }\right)}\right) - \frac {L _ {\cos}}{\pi \sqrt {K M}} - \frac {\Delta_ {\operatorname* {s i m}}}{\pi^ {2} K I \left(\omega_ {\max }\right) \omega_ {\max }} \tag {45}
$$
Now, note that: