Bio-Inspired Hashing for Unsupervised Similarity Search

The fruit fly Drosophila ’ s olfactory circuit has inspired a new locality sensitive hashing ( LSH ) algorithm , FlyHash . In contrast with classical LSH algorithms that produce low dimensional hash codes , FlyHash produces sparse high-dimensional hash codes and has also been shown to have superior empirical performance compared to classical LSH algorithms in similarity search . However , FlyHash uses random projections and can not learn from data . Building on inspiration from FlyHash and the ubiquity of sparse expansive representations in neurobiology , our work proposes a novel hashing algorithm BioHash that produces sparse high dimensional hash codes in a data-driven manner . We show that BioHash outperforms previously published benchmarks for various hashing methods . Since our learning algorithm is based on a local and biologically plausible synaptic plasticity rule , our work provides evidence for the proposal that LSH might be a computational reason for the abundance of sparse expansive motifs in a variety of biological systems . We also propose a convolutional variant BioConvHash that further improves performance . From the perspective of computer science , BioHash and BioConvHash are fast , scalable and yield compressed binary representations that are useful for similarity search . 1 INTRODUCTION . Sparse expansive representations are ubiquitous in neurobiology . Expansion means that a highdimensional input is mapped to an even higher dimensional secondary representation . Such expansion is often accompanied by a sparsification of the activations : dense input data is mapped into a sparse code , where only a small number of secondary neurons respond to a given stimulus . A classical example of the sparse expansive motif is the Drosophila fruit fly olfactory system . In this case , approximately 50 projection neurons send their activities to about 2500 Kenyon cells ( Turner et al. , 2008 ) , thus accomplishing an approximately 50x expansion . An input stimulus typically activates approximately 50 % of projection neurons , and less than 10 % Kenyon cells ( Turner et al. , 2008 ) , providing an example of significant sparsification of the expanded codes . Another example is the rodent olfactory circuit . In this system , dense input from the olfactory bulb is projected into piriform cortex , which has 1000x more neurons than the number of glomeruli in the olfactory bulb . Only about 10 % of those neurons respond to a given stimulus ( Mombaerts et al. , 1996 ) . A similar motif is found in rat ’ s cerebellum and hippocampus ( Dasgupta et al. , 2017 ) . From the computational perspective , expansion is helpful for increasing the number of classification decision boundaries by a simple perceptron ( Cover , 1965 ) or increasing memory storage capacity in models of associative memory ( Hopfield , 1982 ) . Additionally , sparse expansive representations have been shown to reduce intrastimulus variability and the overlaps between representations induced by distinct stimuli ( Sompolinsky , 2014 ) . Sparseness has also been shown to increase the capacity of models of associative memory ( Tsodyks & Feigelman , 1988 ) . The goal of the present work is to use this “ biological ” inspiration about sparse expansive motifs , as well as local Hebbian learning , for designing a novel hashing algorithm BioHash that can be used in similarity search . Below we describe the task , the algorithm , and demonstrate that BioHash improves retrieval performance on common benchmark datasets . Similarity search and LSH . In similarity search , given a query q ∈ Rd , a similarity measure sim ( q , x ) , and a database X ∈ Rn×d containing n items , the objective is to retrieve a ranked list of R items from the database most similar to q . When data is high-dimensional ( e.g . images/documents ) and the databases are large ( millions or billions items ) , this is a computationally challenging problem . However , approximate solutions are generally acceptable , with Locality Sensitive Hashing ( LSH ) being one such approach ( Wang et al. , 2014 ) . Similarity search approaches maybe unsupervised or supervised . Since labelled information for extremely large datasets is infeasible to obtain , our work focuses on the unsupervised setting . In LSH ( Indyk & Motwani , 1998 ; Charikar , 2002 ) , the idea is to encode each database entry x ( and query q ) with a binary representation h ( x ) ( h ( q ) respectively ) and to retrieve R entries with smallest Hamming distances dH ( h ( x ) , h ( q ) ) . Intuitively , ( see ( Charikar , 2002 ) , for a formal definition ) , a hash function h : Rd −→ { −1 , 1 } m is said to be locality sensitive , if similar ( dissimilar ) items x1 and x2 are close by ( far apart ) in Hamming distance dH ( h ( x1 ) , h ( x2 ) ) . LSH algorithms are of fundamental importance in computer science , with applications in similarity search , data compression and machine learning ( Andoni & Indyk , 2008 ) . Drosophila olfactory circuit and FlyHash . In classical LSH approaches , the data dimensionality d is much larger than the embedding space dimension m , resulting in low-dimensional hash codes ( Wang et al. , 2014 ; Indyk & Motwani , 1998 ; Charikar , 2002 ) . In contrast , a new family of hashing algorithms has been proposed ( Dasgupta et al. , 2017 ) where m d , but the secondary representation is highly sparse with only a small number k of m units being active , see Figure 1 . We call this algorithm FlyHash in this paper , since it is motivated by the computation carried out by the fly ’ s olfactory circuit . The expansion from the d dimensional input space into an m dimensional secondary representation is carried out using a random set of weights W ( Dasgupta et al. , 2017 ; Caron et al. , 2013 ) . The resulting high dimensional representation is sparsified by k-Winner-Take-All ( k-WTA ) feedback inhibition in the hidden layer resulting in top ∼ 5 % of units staying active ( Lin et al. , 2014 ; Stevens , 2016 ) . While FlyHash uses random synaptic weights , sparse expansive representations are not necessarily random ( Sompolinsky , 2014 ) , perhaps not even in the case of Drosophila ( Gruntman & Turner , 2013 ; Zheng et al. , 2018 ) . Moreover , using synaptic weights that are learned from data might help to further improve the locality sensitivity property of FlyHash . Thus , it is important to investigate the role of learned synapses on the hashing performance . A recent work SOLHash ( Li et al. , 2018 ) , takes inspiration from FlyHash and attempts to adapt the synapses to data , demonstrating improved performance over FlyHash . However , every learning update step in SOLHash invokes a constrained linear program and also requires computing pairwise inner-products between all training points , making it very time consuming and limiting its scalability to datasets of even modest size . These limitations restrict SOLHash to training only on a small fraction of the data ( Li et al. , 2018 ) . Additionally , SOLHash is biologically implausible ( for an extended discussion , see Sec . 5 ) . BioHash also takes inspiration from FlyHash and demonstrates improved performance compared to random weights used in FlyHash , but it is fast , online , scalable and , importantly , BioHash is neurologically plausible . Not only `` biological '' inspiration can lead to improving hashing techniques , but the opposite might also be true . One of the statements of the present paper is that BioHash satisfies locality sensitive property , and , at the same time , utilizes a biologically plausible learning rule for synaptic weights ( Krotov & Hopfield , 2019 ) . This provides evidence toward the proposal that the reason why sparse expansive representations are so common in biological organisms is because they perform locality sensitive hashing . In other words , they cluster similar stimuli together and push distinct stimuli far apart . Thus , our work provides evidence toward the proposal that LSH might be a fundamental computational principle utilized by the sparse expansive circuits Fig . 1 ( right ) . Importantly , learning of synapses must be neurobiologically plausible ( the synaptic plasticity rule should be local ) . Contributions . Building on inspiration from FlyHash and more broadly the ubiquity of sparse , expansive representations in neurobiology , our work proposes a novel hashing algorithm BioHash , that in contrast with previous work ( Dasgupta et al. , 2017 ; Li et al. , 2018 ) , produces sparse high dimensional hash codes in a data-driven manner and with learning of synapses in a neurobiologically plausible way . We provide an existence proof for the proposal that LSH maybe a fundamental computational principle in neural circuits ( Dasgupta et al. , 2017 ) in the context of learned synapses . We incorporated convolutional structure into BioHash , resulting in a hashing with improved performance compared to previously published benchmarks . From the perspective of computer science , we show that BioHash is simple , scalable to large datasets and demonstrates good performance for similarity search . Interestingly , BioHash outperforms a number of recent SOTA deep hashing methods trained via backpropogation . 2 APPROXIMATE SIMILARITY SEARCH VIA BIOHASHING . Formally , if we denote a data point as x ∈ Rd , we seek a binary hash code y ∈ { −1 , 1 } m. We define the hash length of a binary code as k , if the exact Hamming distance computation is O ( k ) . Below we present our bio-inspired hashing algorithm . 2.1 BIO-INSPIRED HASHING ( BIOHASH ) . We adopt a biologically plausible unsupervised algorithm for representation learning from Krotov & Hopfield ( 2019 ) . Denote the synapses from the input layer to the hash layer as W ∈ Rm×d . The learning dynamics for the synapses of an individual neuron µ , denoted by Wµi , is given by τ dWµi dt = g [ Rank ( 〈Wµ , x〉µ ) ] ( xi − 〈Wµ , x〉µWµi ) , ( 1 ) where Wµ = ( Wµ1 , Wµ2 ... Wµd ) , and g [ µ ] = 1 , µ = 1 −∆ , µ = r 0 , otherwise ( 2 ) and 〈x , y〉µ = ∑ i , j η µ i , jxiyj , with η µ i , j = |Wµi|p−2δij , where δij is Kronecker delta and τ is the time scale of the learning dynamics . The Rank operation in equation ( 1 ) sorts the inner products from the largest ( µ = 1 ) to the smallest ( µ = m ) . It can be shown that the synapses converge to a unit ( p−norm ) sphere ( Krotov & Hopfield , 2019 ) . The training dynamics can be shown to minimize the following energy function E = − ∑ A m∑ µ=1 g [ Rank ( 〈Wµ , xA〉µ ) ] 〈Wµ , xA〉µ 〈Wµ , Wµ〉 p−1 p µ , ( 3 ) where A indexes the training example . Note that the training dynamics do not perform gradient descent , i.e Ẇµ 6= ∇WµE . However , time derivative of the energy function under dynamics ( 1 ) is always negative ( we show this for the case ∆ = 0 below ) , τ dE dt = − ∑ A τ ( p− 1 ) 〈Wµ̂ , Wµ̂〉 p−1 p +1 µ̂ [ 〈dWµ̂ dt , xA〉µ̂〈Wµ̂ , Wµ̂〉µ̂ − 〈Wµ̂ , xA〉µ̂〈 dWµ̂ dt , Wµ̂〉µ̂ ] = = − ∑ A τ ( p− 1 ) 〈Wµ̂ , Wµ̂〉 p−1 p +1 µ̂ [ 〈xA , xA〉µ̂〈Wµ̂ , Wµ̂〉µ̂ − 〈Wµ̂ , xA〉2µ̂ ] ≤ 0 , ( 4 ) where Cauchy-Schwartz inequality is used . For every training example A the index of the activated hidden unit is defined as µ̂ = arg max µ [ 〈Wµ , xA〉µ ] . ( 5 ) Thus , the energy function decreases during learning . A similar result can be shown for ∆ 6= 0 . After the learning-phase is complete , the hash code is generated , as in FlyHash , via WTA sparsification : for a given query x we generate a hash code y ∈ { −1 , 1 } m as yµ = { 1 , 〈Wµ , x〉µ is in top k −1 , otherwise . ( 6 ) Thus , the hyperparameters of the method are p , r , m and ∆ . Note that the synapses are updated based only on pre- and post-synaptic activations resulting in Hebbian or anti-Hebbian updates . Many `` unsupervised '' learning to hash approaches provide a sort of `` weak supervision '' in the form of similarities evaluated in the feature space of deep CNNs trained on ImageNet ( Jin et al. , 2019 ) to achieve good performance . BioHash does not assume such information is provided and is completely unsupervised .

Summary: This paper introduces a variant of FlyHash for similarity search in vector space. The basic idea is motivated by the intuition: the original FlyHash method is data-independent, so can we improve FlyHash's locality sensitivity by learning from data. It does so by learning the weights of the projection layer and uses winner-take-all sparsification to generate sparse binary hash code. This leads to the bio-inspired hashing algorithm (BioHash).  The paper argues that by taking into account the density of data, the learned projection helps to further push similar entities to have hash codes that point in similar direction while repelling dissimilar objects in opposite directions. Experiment results are reported on MNIST and CIFAR10, and the proposed approach demonstrates better retrieval precision results compared to several other hashing based methods.