Semantic Hashing with Locality Sensitive Embeddings

1 INTRODUCTION . One of most challenging aspects in many Information Retrieval ( IR ) systems is the discovery and identification of the nearest neighbors of a query element in an vector space . This is typically solved using Approximate Nearest Neighbors ( ANN ) methods as exact solutions typically do not scale well with the dimension of the vector space . ANN methods typically fall into one of three categories : space partitioning trees , such as the kd-tree ( Bentley ( 1975 ) ; Friedman et al . ( 1977 ) ; Arya et al . ( 1998 ) ) , neighborhood graph search ( Chen et al . ( 2018 ) ; Iwasaki & Miyazaki ( 2018 ) ) or Locality Sensitive Hashing ( LSH ) methods ( Charikar ( 2002 ) ; Gionis et al . ( 1999 ) ; Lv et al . ( 2007 ) ) . Despite their theoretical , intuitive , and computational appeal , LSH methods are not as prevalent in modern IR systems as are space-partitioning trees or neighborhood graph methods ( Bernhardsson ( 2013 ) ; Chen et al . ( 2018 ) ; Johnson et al . ( 2017 ) ; Iwasaki & Miyazaki ( 2018 ) ) . Empirical studies demonstrate that LSH techniques frequently do not attain the same level of quality as spacepartitioning trees ( Muja & Lowe ( 2009 ) ) . Nonetheless , space-partitioning and neighborhood graph search methods are expensive , both in data structure construction and in query time , and remain a bottleneck in many modern IR pipelines . As many modern retrieval tasks revolve around solving ANN for vector representations learned from raw , structured data , one might attempt to learn representations which are more suited towards efficient retrieval . Metric learning methods ( Xing et al . ( 2003 ) ; Weinberger et al . ( 2006 ) ; Chechik et al . ( 2010 ) ; Hoffer & Ailon ( 2015 ) ; Kulis et al . ( 2013 ) ) have been proposed for learning linear and non-linear transformations of given representations for improved clustering and retrieval quality . A class of related methods , semantic hashing or hash learning methods ( Salakhutdinov & Hinton ( 2009 ) ) , have also been explored for learning transformations into binary vector spaces . These learned binary representations may then be used in hashing based retrieval methods , typically by retrieving all neighboring elements in the Hamming ball with radius 1 or 2 . Exact hashing retrieval algorithms , that is , Hamming ball “ search ” with radius 0 , have a particular computational appeal in that search data structures are not needed nor is enumeration of all codes within a Hamming ball . In addition , binary representations that are suitable for exact hashing retrieval can also be used to identify groups of related items that can be interpreted as clusters in the traditional sense . As the number of clusters discovered by the algorithm isn ’ t explicitly controlled ( only bounded by 2d , ) algorithms generating binary embeddings suitable for exact hashing retrieval can be viewed as nonparametric clustering methods . To this end , we propose a method for learning continuous representations in which the optimized similarity is the angular similarity . The angular similarity corresponds to the collision probability of SimHash , a hyperplane based LSH function ( Charikar ( 2002 ) ) . Angular distance gives a sharp topology on the embedding space which encourages similar objects have nearly identical embeddings suitable for exact hashing retrieval . Related work on similarity learning , LSH , and hash learning can be found in Section 2 . The proposed models are found in Section 3 . The experimental results , and other technical details , can be found in Sections 4 . Finally , we conclude in Section 5 . 2 PRELIMINARIES . 2.1 SIMILARITY MODELLING . Similarity learning methods are a class of techniques for learning a similarity function between objects . One successful approach for similarity learning are “ twin network ” or “ two tower architecture ” models , in which two neural network architectures are joined to produce a similarity prediction ( Bromley et al . ( 1994 ) ; Chopra et al . ( 2005 ) ; Huang et al . ( 2013 ) ) . The weights of these networks may be shared or not , depending on whether the two input domains are equivalent or not . Let i ∈ U and j ∈ V be the identities of two objects , where U and V are the two domains across which a similarity function is to be learned . Let φu ( i ) and φv ( j ) be the input representations for the objects ( these functions φ may be identity functions if the input domains are discrete . ) These representations are then transformed through parameterized vector-valued functions fu ( ·|θu ) and fv ( ·|θv ) , whose output are typically the learned representations ui = fu ( φu ( i ) |θu ) and vj = fv ( φv ( j ) |θv ) . A loss is then defined using pairwise labels yij and an interaction function s ( ui , vj ) which denotes the similarity or relevancy of the pair . Taking fu to be a mapping for each index i to an independent parameter vector ui ( similarly for fv and vi ) , and taking s ( ui , vj ) = uTi vj with an appropriate loss results in a variety of matrix factorization approaches ( Koren et al . ( 2009 ) ; Lee & Seung ( 2001 ) ; Mnih & Salakhutdinov ( 2008 ) ; Blei et al . ( 2003 ) ; Rendle et al . ( 2012 ) ; Pennington et al . ( 2014 ) ) . Taking fu to be a neural network mapping a context φu ( i ) to a representation ui allows for similarity models that readily make use of complex contextual information . Common choices for the similarity function include transformations of Euclidean distance ( Chopra et al . ( 2005 ) ) , and cosine similarity : s ( ui , vj ) = uTi vj ||ui||||vj || ( Huang et al . ( 2013 ) ) . In addition , the loss can be defined for pairs ( Chopra et al . ( 2005 ) ) , triplets ( one positive pair , one negative pair ) ( Rendle et al . ( 2012 ) ; Chechik et al . ( 2010 ) ) , or on larger sets ( Huang et al . ( 2013 ) ) . 2.2 LOCALITY SENSITIVE HASHING AND ANGULAR SIMILARITY . A Locality Sensitive Hash ( LSH ) family F is a distribution of hashes h on a collection of objects Q such that for qi , qj ∈ Q , ( Indyk & Motwani ( 1998 ) ; Gionis et al . ( 1999 ) ; Charikar ( 2002 ) ) Pr [ h ( qi ) = h ( qj ) ] = s ( qi , qj ) ( 1 ) for some similarity function s on the objects . SimHash ( Charikar ( 2002 ) ) is a LSH technique developed for document deduplication but may be used in other contexts . For a vector representations q ∈ Rd , SimHash draws a random matrix Z ∈ Rd×M with standard Normal entries . The hash h ( qi ) ∈ { 0 , 1 } M is then constructed as h ( qi ) m = 1 [ q T i Z : m > 0 ] . ( 2 ) Intuitively , SimHash draws random hyperplanes intersecting the origin to separate points . A useful property of this hash function , as stated in Charikar ( 2002 ) , is that ψ ( qi , qj ) : = Pr [ h ( qi ) m = h ( qj ) m ] = 1− 1 π cos−1 ( qTi qj ||qi||||qj || ) , where the above probability is measured with respect to Z. ψ ( qi , qj ) , the collision probability for two vectors , is also known as the angular similarity , and ξ = 1 − ψ is the angular distance , which is a proper metric ( unlike the cosine distance 1− q T i qj ||qi||||qj || ) . As the columns of Z are independent , the collision probability for a K bit hash is ψK . 2.3 LEARNING TO HASH . A related approach to similarity learning is hash learning methods , introduced in Salakhutdinov & Hinton ( 2009 ) . These methods train binary embeddings directly and then use hash collisions or Hamming Ball search to retrieve approximate nearest neighbors . Binary representations lead to some technical challenges ; Salakhutdinov & Hinton ( 2009 ) uses contrastive divergence for training , whereas Hubara et al . ( 2016 ) implement binary threshold activation functions with stochastic neurons . Another approach ( and the one followed in this work ) is to avoid explicit binary representations in training and to introduce an quantization loss to penalize embeddings that are not close to binary , and to subsequently threshold these near-binary embeddings to binary ones . This type of quantization loss is distinct from those used in vector quantization methods ( Ahalt et al . ( 1990 ) ; Kohonen ( 1990 ) ; Sato & Yamada ( 1996 ) ) in which the data representations are fixed and the codes are learned ; here the codes are fixed and the representations are learned . The quantization loss introduced in Deep Hashing Networks ( DHN ) Zhu et al . ( 2016 ) is of the form b ( ui|θ ) = ∑ d log cosh ( |uid| − 1 ) ≈ ‖|ui| − 1‖1 . ( 3 ) Other quantization losses based on distances to binary codes have been used in Li et al . ( 2016 ) ; Liu et al . ( 2016 ) . Cao et al . ( 2017 ) utilizes a quantization loss whose strength increases over time . Finally , Deep Cauchy Hashing ( DCH ) ( Cao et al . ( 2018 ) ) has shown improvements by utilizing a heavy-tailed similarity function with a similarly inspired quantization loss . 3 LOCALITY SENSITIVE EMBEDDINGS . Many similarity learning methods utilize dot products or cosine similarity to relate the embeddings of a pair to each other . For example GloVe ( Pennington et al . ( 2014 ) ) minimizes the weighted error between the dot product of the embeddings and a log-coocurrence matrix , and the DSSM model ( Huang et al . ( 2013 ) ) utilizes cosine similarity as the “ crossing ” layer between the two halves of a twin network . In general , embeddings trained in this way are not suitable for SimHash retrieval , as can be seen in Figure 1 . If models are trained so as to minimize the error of a prediction made by cosine similarity , extremely low tolerances are required in order to achieve embeddings with significant collision probability . Similar observations on the misspecifiation of cosine distance for Semantic Hashing were made in Cao et al . ( 2018 ) . In this section , we define models in which collision probabilities of learned representations are directly optimized . 3.1 LOSS DEFINITION . In the following , we define a population loss through a data distribution D of relevant and irrelevant pairs . Each sample from D is a tuple ( y , i , j ) ∈ { 0 , 1 } × U × V , where U and V are the sets across which a similarity is to be learned – for example , “ users ” and “ items ” in a recommender system . y is the relevancy of the pair ( i , j ) . The population losses we consider are expectations over D of a per-tuple loss l with regularization terms r per item : L ( θ ) = E y , i , j∼D l ( y , i , j|θ ) + λr ( i|θ ) + λr ( j|θ ) . ( 4 ) In practice , we minimize the empirical loss L̂ constructed from a finite sample from D , and we use r ( i|θ ) = b ( ui|θ ) defined in equation 3. θ represents all parameters of the model , including any learned representations for the elements of the sets U and V . An embedding ui for element i may either be a vector of free parameters , as would be in a fixed vocabulary embedding model , or may be the output of a model on a raw input : ui = fu ( φu ( i ) ) , as would be in a twin network model . In addition , each half of the pair ( ui , vj ) may represent a different input space , as in the DSSM model .

Summary: The authors consider the problem of learning a hash function such that semantically similar elements have high collision probability.  They modify the approach Deep Hashing Networks (Zhu et al., 2016) with a new loss function. Rather than use a sigmoid based loss function, the authors argue that a loss function based on angular similarity and SimHash would be better. Specifically, they use the probability of SimHash collisions as a loss function. They then experimentally verify their method on synthetic data from a Stochastic Block Model distribution, image data (CIFAR-10 and ImageNet), and text data (OSCAR). They show improvements over related methods.