Revisiting Locality-Sensitive Binary Codes from Random Fourier Features

1 INTRODUCTION . Developing efficient machine learning algorithms in large-scale problems has been an important research topic to deal with massive data . In this paper , we focus on efficient retrieval/search methods , specifically , by designing similarity-preserving binary representations of the data . That is , for each data vector x ∈ Rd , we hash it into a length-b binary 0/1 vector h ( x ) ∈ { 0 , 1 } b , where the geometry of the data should be well preserved in the Hamming space . Searching with binary codes has been widely applied in many applications , such as large-scale image retrieval ( Weiss et al. , 2008 ; Gong & Lazebnik , 2011 ; Kulis & Darrell , 2009 ; He et al. , 2013 ; Lin et al. , 2015 ; Liu et al. , 2016 ; 2017 ; Song et al. , 2018 ; Yu et al. , 2018 ; Yan et al. , 2021 ) . The benefits are two-fold . Firstly , it may largely reduce the memory cost for storing massive datasets , especially with high-dimensional data . Secondly , it can significantly speedup the retrieval process . For instance , the binary codes can be used to build hash tables ( e.g. , Indyk & Motwani ( 1998 ) ) for sub-linear time approximate nearest neighbor ( ANN ) search . Moreover , in the Hamming space , we can also apply exhaustive search , which is much faster than computing the Euclidean distances , plus the technique of multi-index hashing ( Norouzi et al. , 2012 ) can further accelerate the exact Hamming search by orders of magnitude . In general , binary hashing methods can be categorized into supervised and unsupervised approaches . In this work , we focus on the unsupervised setting . Locality-Sensitive Hashing ( LSH ) ( Indyk & Motwani , 1998 ) is one of the early hashing methods leading to binary embedding . The LSH targeting at cosine similarity ( Charikar , 2002 ) , also known as SimHash , generates binary hash codes using random hyperplanes by taking the sign of data-independent random Gaussian projections . For the cosine similarity , LSH has strict theoretical guarantee on the approximation error and search efficiency , but typically requires relatively long codes to achieve good performance . Hence , many works have focused on learning data-dependent short binary hash codes , through different objective functions . Examples include Iterative Quantization ( ITQ ) ( Gong & Lazebnik , 2011 ) , Spectral Hashing ( SpecH ) ( Weiss et al. , 2008 ) and Binary Reconstruction Embedding ( BRE ) ( Kulis & Darrell , 2009 ) . Recently , some unsupervised deep learning based methods have been proposed in the computer vision community , many of which are , to some extent , “ task-specific ” for cross-modal/video/image retrieval , implemented based on some deep models like the autoencoder and VGG-16 ( Liu et al. , 2016 ; Do et al. , 2017 ; Chen et al. , 2018 ; Li et al. , 2019 ; Yang et al. , 2019 ; Hansen et al. , 2020 ; Liu et al. , 2020 ; Qiu et al. , 2021 ) . By taking the advantage of the complicated model structures ( e.g. , CNN layers ) , these deep methods show promising performance in many image retrieval tasks . Compared with the data-dependent methods ( including deep methods ) , LSH has three advantages : • Although the data-dependent procedures can provide improved performance with fewer binary codes , a known undesirable effect of many of these mechanisms is the performance degradation when we increase the code length b , as reported in prior literature ( Raginsky & Lazebnik , 2009 ; Joly & Buisson , 2011 ) . In our experiments , we will show that this may also be an issue for deep learning based methods . On the other hand , the performance ( i.e. , search accuracy ) of the data-independent LSH would keep boosting with larger b . In many scenarios , it is often the case that only using short codes ( e.g. , ≤ 128 bits ) can not achieve a desirable level of search accuracy for practical purposes . In these cases , one may need to use longer codes anyway , where LSH could be more favorable . • LSH is very simple to implement ( only with random projections ) , while data-dependent methods require additional optimization/training which might be costly . Moreover , large deep learning models would typically need longer inference time , which might be infeasible in practice where the query speed is important . • It is difficult to characterize the properties of the data-dependent methods theoretically , while LSH enjoys rigorous guarantees on the retrieval/approximation performance . Therefore , LSH is still a popular hashing method with great research interest and many practical applications ( Shrivastava & Li , 2014 ; Qi et al. , 2017 ; Driemel & Silvestri , 2017 ; Chen et al. , 2019 ; Zandieh et al. , 2020 ; Lei et al. , 2020 ; Daghaghi et al. , 2021 ) . 1.1 LOCALITY-SENSITIVE HASHING FROM RANDOM FOURIER FEATURES . Kernel methods have gained great success in many machine learning tasks ( Schölkopf & Smola , 2002 ; Zhu & Hastie , 2001 ; Avron et al. , 2017 ; Sun et al. , 2018 ) . However , standard kernel methods require the n × n kernel matrix , which becomes computationally expensive on large-scale datasets with massive data points . To this end , Rahimi & Recht ( 2007 ) proposes the method of random Fourier feature ( RFF ) , which defines a feature map that approximates shift-invariant kernels by the linear inner products . As such , the RFF preserves the “ non-linear locality structure ” of the data . This leads to numerous applications in large-scale learning where one trains linear models on RFF to approximate training non-linear kernel machines ( Yang et al. , 2012 ; Chwialkowski et al. , 2015 ; Sutherland & Schneider , 2015 ; Avron et al. , 2017 ; Sun et al. , 2018 ; Tompkins & Ramos , 2018 ) . Given the popularity of RFF , one natural step is to apply it to search problems . Raginsky & Lazebnik ( 2009 ) proposes to construct binary codes from the RFF using stochastic binary quantization . We call this method “ SQ-RFF ” , whose convergence ( as b→∞ ) and concentration can be theoretically characterized . The author showed that SQ-RFF can achieve competitive search accuracy with sufficient b , e.g. , b ≥ 512 . Since RFF itself is a widely used tool for large-scale kernel learning , SQ-RFF could be a convenient/useful tool in practical scenarios where RFF has been generated for training kernel machines , and the data scientist wants to further use it for efficient near neighbor retrieval . 1.2 OUR CONTRIBUTIONS . Given the benefits of LSH and RFF , we revisit hashing methods for non-linear kernels , and improve the prior RFF based hashing method . Our approach is named as “ SignRFF ” . Specifically , • We theoretically compare several linear and non-linear LSH methods in terms of a novel measure called the ranking efficiency , which is defined based on the probability of retrieving a wrong/reversed similarity ranking . Under this unified metric , the proposed SignRFF is uniformly better than SQ-RFF . Moreover , the ranking efficiency also indicates that typically one should prefer SignRFF over linear LSH when the near neighbors are close to each other , which is the first systematic comparison of linear vs. non-linear LSH in literature . • Empirically , we conduct near neighbor search experiments on benchmark image datasets to compare the proposed SignRFF with other popular hashing methods , which verifies the superiority of SignRFF over SQ-RFF . We validate that the ranking efficiency metric aligns well with the empirical results . Moreover , SignRFF outperforms various competing methods ( including deep methods ) with moderately large number of hash bits , indicating its advantage in applications to achieve a high search recall/precision . 2 PRELIMINARIES . 2.1 LOCALITY-SENSITIVE HASHING ( LSH ) . In large-scale information retrieval , exhaustive search of the exact nearest neighbors is usually too expensive . A common relaxation in this setting is the Approximate Nearest Neighbor ( ANN ) search , where we return the “ good ” neighbors of a query with high probability . In this paper , we will consider the search problem with data points in Rd . X denotes the database consisting of n data points , and q is a query point . x , y are two data points with ρ = cos ( x , y ) . Definition 2.1 ( S̃-neighbor ) . For a similarity measure S : Rd × Rd 7→ R , the S̃-neighbor set of q is defined as { x ∈X : S ( x , q ) > S̃ } . Definition 2.2 ( ( c , S̃ ) -ANN ) . Assume δ > 0 is a parameter . An algorithm A is a ( c , S̃ ) -ANN method provided the following : with probability at least 1− δ , for 0 < c < 1 , if there exists an S̃-neighbor of q in X , A returns a cS̃-neighbor of q . One popular method satisfying Definition 2.2 is the Locality-Sensitive Hashing ( LSH ) . Definition 2.3 ( Indyk & Motwani ( 1998 ) ) . A family of hash functions H is called ( S̃ , cS̃ , p1 , p2 ) locality-sensitive for similarity measure S and 0 < c < 1 , if for ∀x , y ∈ Rd and hash function h uniformly chosen from H , it holds that : 1 ) If S ( x , y ) ≥ S̃ , then P ( h ( x ) = h ( y ) ) ≥ p1 ; 2 ) If S ( x , y ) ≤ cS̃ , then P ( h ( x ) = h ( y ) ) ≤ p2 , with p2 < p1 . A key intuition of LSH is that , similar data points with have higher chance of hash collision in the Hamming space . In this paper , we will specifically consider the LSH for the cosine similarity ( Charikar , 2002 ) . For a data vector x ∈ Rd , the LSH binary code is given by LSH : hLSH ( x ) = sign ( wTx ) , ( 1 ) where wT is a standard Gaussian random vector . We use b i.i.d . w1 , ... , wb to generate b LSH binary codes . The collision probability between the codes of two data points x , y is P ( hLSH ( x ) = hLSH ( y ) ) = 1− cos−1 ( ρ ) π , ( 2 ) where ρ = cos ( x , y ) is their cosine similarity . Note that P ( hLSH ( x ) = hLSH ( y ) ) is increasing in ρ , which , By Definition 2.3 , is the key to ensure the locality sensitivity . 2.2 KERNELIZED LOCALITY-SENSITIVE HASHING ( KLSH ) . In this paper , we will consider the popular Gaussian kernel function defined for x , y ∈ Rd as k ( x , y ) = exp ( −γ2‖x− y‖2 2 ) , where γ is a hyper-parameter . Let Ψ : Rd 7→ F be the feature map with the kernel induced feature spaceF . To incorporate non-linearity in LSH , Kulis & Grauman ( 2009 ) proposes Kernelized Locality-Sensitive Hashing ( KLSH ) , by applying LSH ( 1 ) in the kernel induced feature space F , i.e. , h ( x ) = sign ( wTΨ ( x ) ) . However , as in many cases ( e.g. , for the Gaussian kernel ) the map Ψ can not be explicitly identified , constructing the random Gaussian projection vector w needs some careful design . KLSH approximates the random Gaussian distribution using a sufficient number of data points , by the Central Limit Theorem ( CLT ) in the Reproducing Kernel Hilbert Space . One first samples m data points from X to form a kernel matrix K , then uniformly picks t points from [ 1 , ... , m ] at random to approximate the Gaussian distribution . The hash code has the form KLSH : hKLSH ( x ) = sign ( m∑ i=1 w ( i ) k ( x , xi ) ) , ( 3 ) where w = K−1/2et , and et ∈ { 0 , 1 } m has ones in the entries with indices of the t selected points . We see that the codes h1 , ... , hb are actually dependent , and the quality of using CLT to approximate Gaussian distribution is not guaranteed , especially in high-dimensional kernel feature space . Jiang et al . ( 2015 ) re-formulates KLSH as applying LSH on the kernel principle components in the kernel induced feature space , resolving a theoretical concern of KLSH . Notably , since KLSH uses a pool of data samples to approximate the Gaussian distribution , the hash codes are in fact dependent in practical implementation . It is observed that the performance of KLSH also drops as b increases ( Joly & Buisson , 2011 ) , similar to the behavior of many data-dependent methods . We provide more detailed explanation in Appendix C .

Summary: The paper revisits the binary hashing from Random Fourier Feature (RFF) for approximate nearest neighbor search. The authors propose a simple and effective RFF-based hashing method SignRFF and demonstrate its locality-sensitive property. They also introduce a new measure called ranking efficiency to evaluate the performance of different LSH methods theoretically. Extensive experiments on three real-life image data sets validate the effectiveness of SignRFF and the consistency of the ranking efficiency from theoretical evaluation.