Revisiting Locality-Sensitive Binary Codes from Random Fourier Features

1 INTRODUCTION . Developing efficient machine learning algorithms in large-scale problems has been an important research topic to deal with massive data . In this paper , we focus on efficient retrieval/search methods , specifically , by designing similarity-preserving binary representations of the data . That is , for each data vector x ∈ Rd , we hash it into a length-b binary 0/1 vector h ( x ) ∈ { 0 , 1 } b , where the geometry of the data should be well preserved in the Hamming space . Searching with binary codes has been widely applied in many applications , such as large-scale image retrieval ( Weiss et al. , 2008 ; Gong & Lazebnik , 2011 ; Kulis & Darrell , 2009 ; He et al. , 2013 ; Lin et al. , 2015 ; Liu et al. , 2016 ; 2017 ; Song et al. , 2018 ; Yu et al. , 2018 ; Yan et al. , 2021 ) . The benefits are two-fold . Firstly , it may largely reduce the memory cost for storing massive datasets , especially with high-dimensional data . Secondly , it can significantly speedup the retrieval process . For instance , the binary codes can be used to build hash tables ( e.g. , Indyk & Motwani ( 1998 ) ) for sub-linear time approximate nearest neighbor ( ANN ) search . Moreover , in the Hamming space , we can also apply exhaustive search , which is much faster than computing the Euclidean distances , plus the technique of multi-index hashing ( Norouzi et al. , 2012 ) can further accelerate the exact Hamming search by orders of magnitude . In general , binary hashing methods can be categorized into supervised and unsupervised approaches . In this work , we focus on the unsupervised setting . Locality-Sensitive Hashing ( LSH ) ( Indyk & Motwani , 1998 ) is one of the early hashing methods leading to binary embedding . The LSH targeting at cosine similarity ( Charikar , 2002 ) , also known as SimHash , generates binary hash codes using random hyperplanes by taking the sign of data-independent random Gaussian projections . For the cosine similarity , LSH has strict theoretical guarantee on the approximation error and search efficiency , but typically requires relatively long codes to achieve good performance . Hence , many works have focused on learning data-dependent short binary hash codes , through different objective functions . Examples include Iterative Quantization ( ITQ ) ( Gong & Lazebnik , 2011 ) , Spectral Hashing ( SpecH ) ( Weiss et al. , 2008 ) and Binary Reconstruction Embedding ( BRE ) ( Kulis & Darrell , 2009 ) . Recently , some unsupervised deep learning based methods have been proposed in the computer vision community , many of which are , to some extent , “ task-specific ” for cross-modal/video/image retrieval , implemented based on some deep models like the autoencoder and VGG-16 ( Liu et al. , 2016 ; Do et al. , 2017 ; Chen et al. , 2018 ; Li et al. , 2019 ; Yang et al. , 2019 ; Hansen et al. , 2020 ; Liu et al. , 2020 ; Qiu et al. , 2021 ) . By taking the advantage of the complicated model structures ( e.g. , CNN layers ) , these deep methods show promising performance in many image retrieval tasks . Compared with the data-dependent methods ( including deep methods ) , LSH has three advantages : • Although the data-dependent procedures can provide improved performance with fewer binary codes , a known undesirable effect of many of these mechanisms is the performance degradation when we increase the code length b , as reported in prior literature ( Raginsky & Lazebnik , 2009 ; Joly & Buisson , 2011 ) . In our experiments , we will show that this may also be an issue for deep learning based methods . On the other hand , the performance ( i.e. , search accuracy ) of the data-independent LSH would keep boosting with larger b . In many scenarios , it is often the case that only using short codes ( e.g. , ≤ 128 bits ) can not achieve a desirable level of search accuracy for practical purposes . In these cases , one may need to use longer codes anyway , where LSH could be more favorable . • LSH is very simple to implement ( only with random projections ) , while data-dependent methods require additional optimization/training which might be costly . Moreover , large deep learning models would typically need longer inference time , which might be infeasible in practice where the query speed is important . • It is difficult to characterize the properties of the data-dependent methods theoretically , while LSH enjoys rigorous guarantees on the retrieval/approximation performance . Therefore , LSH is still a popular hashing method with great research interest and many practical applications ( Shrivastava & Li , 2014 ; Qi et al. , 2017 ; Driemel & Silvestri , 2017 ; Chen et al. , 2019 ; Zandieh et al. , 2020 ; Lei et al. , 2020 ; Daghaghi et al. , 2021 ) . 1.1 LOCALITY-SENSITIVE HASHING FROM RANDOM FOURIER FEATURES . Kernel methods have gained great success in many machine learning tasks ( Schölkopf & Smola , 2002 ; Zhu & Hastie , 2001 ; Avron et al. , 2017 ; Sun et al. , 2018 ) . However , standard kernel methods require the n × n kernel matrix , which becomes computationally expensive on large-scale datasets with massive data points . To this end , Rahimi & Recht ( 2007 ) proposes the method of random Fourier feature ( RFF ) , which defines a feature map that approximates shift-invariant kernels by the linear inner products . As such , the RFF preserves the “ non-linear locality structure ” of the data . This leads to numerous applications in large-scale learning where one trains linear models on RFF to approximate training non-linear kernel machines ( Yang et al. , 2012 ; Chwialkowski et al. , 2015 ; Sutherland & Schneider , 2015 ; Avron et al. , 2017 ; Sun et al. , 2018 ; Tompkins & Ramos , 2018 ) . Given the popularity of RFF , one natural step is to apply it to search problems . Raginsky & Lazebnik ( 2009 ) proposes to construct binary codes from the RFF using stochastic binary quantization . We call this method “ SQ-RFF ” , whose convergence ( as b→∞ ) and concentration can be theoretically characterized . The author showed that SQ-RFF can achieve competitive search accuracy with sufficient b , e.g. , b ≥ 512 . Since RFF itself is a widely used tool for large-scale kernel learning , SQ-RFF could be a convenient/useful tool in practical scenarios where RFF has been generated for training kernel machines , and the data scientist wants to further use it for efficient near neighbor retrieval . 1.2 OUR CONTRIBUTIONS . Given the benefits of LSH and RFF , we revisit hashing methods for non-linear kernels , and improve the prior RFF based hashing method . Our approach is named as “ SignRFF ” . Specifically , • We theoretically compare several linear and non-linear LSH methods in terms of a novel measure called the ranking efficiency , which is defined based on the probability of retrieving a wrong/reversed similarity ranking . Under this unified metric , the proposed SignRFF is uniformly better than SQ-RFF . Moreover , the ranking efficiency also indicates that typically one should prefer SignRFF over linear LSH when the near neighbors are close to each other , which is the first systematic comparison of linear vs. non-linear LSH in literature . • Empirically , we conduct near neighbor search experiments on benchmark image datasets to compare the proposed SignRFF with other popular hashing methods , which verifies the superiority of SignRFF over SQ-RFF . We validate that the ranking efficiency metric aligns well with the empirical results . Moreover , SignRFF outperforms various competing methods ( including deep methods ) with moderately large number of hash bits , indicating its advantage in applications to achieve a high search recall/precision . 2 PRELIMINARIES . 2.1 LOCALITY-SENSITIVE HASHING ( LSH ) . In large-scale information retrieval , exhaustive search of the exact nearest neighbors is usually too expensive . A common relaxation in this setting is the Approximate Nearest Neighbor ( ANN ) search , where we return the “ good ” neighbors of a query with high probability . In this paper , we will consider the search problem with data points in Rd . X denotes the database consisting of n data points , and q is a query point . x , y are two data points with ρ = cos ( x , y ) . Definition 2.1 ( S̃-neighbor ) . For a similarity measure S : Rd × Rd 7→ R , the S̃-neighbor set of q is defined as { x ∈X : S ( x , q ) > S̃ } . Definition 2.2 ( ( c , S̃ ) -ANN ) . Assume δ > 0 is a parameter . An algorithm A is a ( c , S̃ ) -ANN method provided the following : with probability at least 1− δ , for 0 < c < 1 , if there exists an S̃-neighbor of q in X , A returns a cS̃-neighbor of q . One popular method satisfying Definition 2.2 is the Locality-Sensitive Hashing ( LSH ) . Definition 2.3 ( Indyk & Motwani ( 1998 ) ) . A family of hash functions H is called ( S̃ , cS̃ , p1 , p2 ) locality-sensitive for similarity measure S and 0 < c < 1 , if for ∀x , y ∈ Rd and hash function h uniformly chosen from H , it holds that : 1 ) If S ( x , y ) ≥ S̃ , then P ( h ( x ) = h ( y ) ) ≥ p1 ; 2 ) If S ( x , y ) ≤ cS̃ , then P ( h ( x ) = h ( y ) ) ≤ p2 , with p2 < p1 . A key intuition of LSH is that , similar data points with have higher chance of hash collision in the Hamming space . In this paper , we will specifically consider the LSH for the cosine similarity ( Charikar , 2002 ) . For a data vector x ∈ Rd , the LSH binary code is given by LSH : hLSH ( x ) = sign ( wTx ) , ( 1 ) where wT is a standard Gaussian random vector . We use b i.i.d . w1 , ... , wb to generate b LSH binary codes . The collision probability between the codes of two data points x , y is P ( hLSH ( x ) = hLSH ( y ) ) = 1− cos−1 ( ρ ) π , ( 2 ) where ρ = cos ( x , y ) is their cosine similarity . Note that P ( hLSH ( x ) = hLSH ( y ) ) is increasing in ρ , which , By Definition 2.3 , is the key to ensure the locality sensitivity . 2.2 KERNELIZED LOCALITY-SENSITIVE HASHING ( KLSH ) . In this paper , we will consider the popular Gaussian kernel function defined for x , y ∈ Rd as k ( x , y ) = exp ( −γ2‖x− y‖2 2 ) , where γ is a hyper-parameter . Let Ψ : Rd 7→ F be the feature map with the kernel induced feature spaceF . To incorporate non-linearity in LSH , Kulis & Grauman ( 2009 ) proposes Kernelized Locality-Sensitive Hashing ( KLSH ) , by applying LSH ( 1 ) in the kernel induced feature space F , i.e. , h ( x ) = sign ( wTΨ ( x ) ) . However , as in many cases ( e.g. , for the Gaussian kernel ) the map Ψ can not be explicitly identified , constructing the random Gaussian projection vector w needs some careful design . KLSH approximates the random Gaussian distribution using a sufficient number of data points , by the Central Limit Theorem ( CLT ) in the Reproducing Kernel Hilbert Space . One first samples m data points from X to form a kernel matrix K , then uniformly picks t points from [ 1 , ... , m ] at random to approximate the Gaussian distribution . The hash code has the form KLSH : hKLSH ( x ) = sign ( m∑ i=1 w ( i ) k ( x , xi ) ) , ( 3 ) where w = K−1/2et , and et ∈ { 0 , 1 } m has ones in the entries with indices of the t selected points . We see that the codes h1 , ... , hb are actually dependent , and the quality of using CLT to approximate Gaussian distribution is not guaranteed , especially in high-dimensional kernel feature space . Jiang et al . ( 2015 ) re-formulates KLSH as applying LSH on the kernel principle components in the kernel induced feature space , resolving a theoretical concern of KLSH . Notably , since KLSH uses a pool of data samples to approximate the Gaussian distribution , the hash codes are in fact dependent in practical implementation . It is observed that the performance of KLSH also drops as b increases ( Joly & Buisson , 2011 ) , similar to the behavior of many data-dependent methods . We provide more detailed explanation in Appendix C .

Summary: This paper proposed an improved version of Random Fourier Feature (RFF) based binary feature coding method as well as a new measure(ranking efficiency) to compare different Locality-Sensitive Hashing (LSH) methods. Under the new measure, the proposed method is better than previous methods. The authors also validated by visualization the consistency between theoretical comparisons based the proposed measure and the empirical results.