Bio-Inspired Hashing for Unsupervised Similarity Search

The fruit fly Drosophila ’ s olfactory circuit has inspired a new locality sensitive hashing ( LSH ) algorithm , FlyHash . In contrast with classical LSH algorithms that produce low dimensional hash codes , FlyHash produces sparse high-dimensional hash codes and has also been shown to have superior empirical performance compared to classical LSH algorithms in similarity search . However , FlyHash uses random projections and can not learn from data . Building on inspiration from FlyHash and the ubiquity of sparse expansive representations in neurobiology , our work proposes a novel hashing algorithm BioHash that produces sparse high dimensional hash codes in a data-driven manner . We show that BioHash outperforms previously published benchmarks for various hashing methods . Since our learning algorithm is based on a local and biologically plausible synaptic plasticity rule , our work provides evidence for the proposal that LSH might be a computational reason for the abundance of sparse expansive motifs in a variety of biological systems . We also propose a convolutional variant BioConvHash that further improves performance . From the perspective of computer science , BioHash and BioConvHash are fast , scalable and yield compressed binary representations that are useful for similarity search . 1 INTRODUCTION . Sparse expansive representations are ubiquitous in neurobiology . Expansion means that a highdimensional input is mapped to an even higher dimensional secondary representation . Such expansion is often accompanied by a sparsification of the activations : dense input data is mapped into a sparse code , where only a small number of secondary neurons respond to a given stimulus . A classical example of the sparse expansive motif is the Drosophila fruit fly olfactory system . In this case , approximately 50 projection neurons send their activities to about 2500 Kenyon cells ( Turner et al. , 2008 ) , thus accomplishing an approximately 50x expansion . An input stimulus typically activates approximately 50 % of projection neurons , and less than 10 % Kenyon cells ( Turner et al. , 2008 ) , providing an example of significant sparsification of the expanded codes . Another example is the rodent olfactory circuit . In this system , dense input from the olfactory bulb is projected into piriform cortex , which has 1000x more neurons than the number of glomeruli in the olfactory bulb . Only about 10 % of those neurons respond to a given stimulus ( Mombaerts et al. , 1996 ) . A similar motif is found in rat ’ s cerebellum and hippocampus ( Dasgupta et al. , 2017 ) . From the computational perspective , expansion is helpful for increasing the number of classification decision boundaries by a simple perceptron ( Cover , 1965 ) or increasing memory storage capacity in models of associative memory ( Hopfield , 1982 ) . Additionally , sparse expansive representations have been shown to reduce intrastimulus variability and the overlaps between representations induced by distinct stimuli ( Sompolinsky , 2014 ) . Sparseness has also been shown to increase the capacity of models of associative memory ( Tsodyks & Feigelman , 1988 ) . The goal of the present work is to use this “ biological ” inspiration about sparse expansive motifs , as well as local Hebbian learning , for designing a novel hashing algorithm BioHash that can be used in similarity search . Below we describe the task , the algorithm , and demonstrate that BioHash improves retrieval performance on common benchmark datasets . Similarity search and LSH . In similarity search , given a query q ∈ Rd , a similarity measure sim ( q , x ) , and a database X ∈ Rn×d containing n items , the objective is to retrieve a ranked list of R items from the database most similar to q . When data is high-dimensional ( e.g . images/documents ) and the databases are large ( millions or billions items ) , this is a computationally challenging problem . However , approximate solutions are generally acceptable , with Locality Sensitive Hashing ( LSH ) being one such approach ( Wang et al. , 2014 ) . Similarity search approaches maybe unsupervised or supervised . Since labelled information for extremely large datasets is infeasible to obtain , our work focuses on the unsupervised setting . In LSH ( Indyk & Motwani , 1998 ; Charikar , 2002 ) , the idea is to encode each database entry x ( and query q ) with a binary representation h ( x ) ( h ( q ) respectively ) and to retrieve R entries with smallest Hamming distances dH ( h ( x ) , h ( q ) ) . Intuitively , ( see ( Charikar , 2002 ) , for a formal definition ) , a hash function h : Rd −→ { −1 , 1 } m is said to be locality sensitive , if similar ( dissimilar ) items x1 and x2 are close by ( far apart ) in Hamming distance dH ( h ( x1 ) , h ( x2 ) ) . LSH algorithms are of fundamental importance in computer science , with applications in similarity search , data compression and machine learning ( Andoni & Indyk , 2008 ) . Drosophila olfactory circuit and FlyHash . In classical LSH approaches , the data dimensionality d is much larger than the embedding space dimension m , resulting in low-dimensional hash codes ( Wang et al. , 2014 ; Indyk & Motwani , 1998 ; Charikar , 2002 ) . In contrast , a new family of hashing algorithms has been proposed ( Dasgupta et al. , 2017 ) where m d , but the secondary representation is highly sparse with only a small number k of m units being active , see Figure 1 . We call this algorithm FlyHash in this paper , since it is motivated by the computation carried out by the fly ’ s olfactory circuit . The expansion from the d dimensional input space into an m dimensional secondary representation is carried out using a random set of weights W ( Dasgupta et al. , 2017 ; Caron et al. , 2013 ) . The resulting high dimensional representation is sparsified by k-Winner-Take-All ( k-WTA ) feedback inhibition in the hidden layer resulting in top ∼ 5 % of units staying active ( Lin et al. , 2014 ; Stevens , 2016 ) . While FlyHash uses random synaptic weights , sparse expansive representations are not necessarily random ( Sompolinsky , 2014 ) , perhaps not even in the case of Drosophila ( Gruntman & Turner , 2013 ; Zheng et al. , 2018 ) . Moreover , using synaptic weights that are learned from data might help to further improve the locality sensitivity property of FlyHash . Thus , it is important to investigate the role of learned synapses on the hashing performance . A recent work SOLHash ( Li et al. , 2018 ) , takes inspiration from FlyHash and attempts to adapt the synapses to data , demonstrating improved performance over FlyHash . However , every learning update step in SOLHash invokes a constrained linear program and also requires computing pairwise inner-products between all training points , making it very time consuming and limiting its scalability to datasets of even modest size . These limitations restrict SOLHash to training only on a small fraction of the data ( Li et al. , 2018 ) . Additionally , SOLHash is biologically implausible ( for an extended discussion , see Sec . 5 ) . BioHash also takes inspiration from FlyHash and demonstrates improved performance compared to random weights used in FlyHash , but it is fast , online , scalable and , importantly , BioHash is neurologically plausible . Not only `` biological '' inspiration can lead to improving hashing techniques , but the opposite might also be true . One of the statements of the present paper is that BioHash satisfies locality sensitive property , and , at the same time , utilizes a biologically plausible learning rule for synaptic weights ( Krotov & Hopfield , 2019 ) . This provides evidence toward the proposal that the reason why sparse expansive representations are so common in biological organisms is because they perform locality sensitive hashing . In other words , they cluster similar stimuli together and push distinct stimuli far apart . Thus , our work provides evidence toward the proposal that LSH might be a fundamental computational principle utilized by the sparse expansive circuits Fig . 1 ( right ) . Importantly , learning of synapses must be neurobiologically plausible ( the synaptic plasticity rule should be local ) . Contributions . Building on inspiration from FlyHash and more broadly the ubiquity of sparse , expansive representations in neurobiology , our work proposes a novel hashing algorithm BioHash , that in contrast with previous work ( Dasgupta et al. , 2017 ; Li et al. , 2018 ) , produces sparse high dimensional hash codes in a data-driven manner and with learning of synapses in a neurobiologically plausible way . We provide an existence proof for the proposal that LSH maybe a fundamental computational principle in neural circuits ( Dasgupta et al. , 2017 ) in the context of learned synapses . We incorporated convolutional structure into BioHash , resulting in a hashing with improved performance compared to previously published benchmarks . From the perspective of computer science , we show that BioHash is simple , scalable to large datasets and demonstrates good performance for similarity search . Interestingly , BioHash outperforms a number of recent SOTA deep hashing methods trained via backpropogation . 2 APPROXIMATE SIMILARITY SEARCH VIA BIOHASHING . Formally , if we denote a data point as x ∈ Rd , we seek a binary hash code y ∈ { −1 , 1 } m. We define the hash length of a binary code as k , if the exact Hamming distance computation is O ( k ) . Below we present our bio-inspired hashing algorithm . 2.1 BIO-INSPIRED HASHING ( BIOHASH ) . We adopt a biologically plausible unsupervised algorithm for representation learning from Krotov & Hopfield ( 2019 ) . Denote the synapses from the input layer to the hash layer as W ∈ Rm×d . The learning dynamics for the synapses of an individual neuron µ , denoted by Wµi , is given by τ dWµi dt = g [ Rank ( 〈Wµ , x〉µ ) ] ( xi − 〈Wµ , x〉µWµi ) , ( 1 ) where Wµ = ( Wµ1 , Wµ2 ... Wµd ) , and g [ µ ] = 1 , µ = 1 −∆ , µ = r 0 , otherwise ( 2 ) and 〈x , y〉µ = ∑ i , j η µ i , jxiyj , with η µ i , j = |Wµi|p−2δij , where δij is Kronecker delta and τ is the time scale of the learning dynamics . The Rank operation in equation ( 1 ) sorts the inner products from the largest ( µ = 1 ) to the smallest ( µ = m ) . It can be shown that the synapses converge to a unit ( p−norm ) sphere ( Krotov & Hopfield , 2019 ) . The training dynamics can be shown to minimize the following energy function E = − ∑ A m∑ µ=1 g [ Rank ( 〈Wµ , xA〉µ ) ] 〈Wµ , xA〉µ 〈Wµ , Wµ〉 p−1 p µ , ( 3 ) where A indexes the training example . Note that the training dynamics do not perform gradient descent , i.e Ẇµ 6= ∇WµE . However , time derivative of the energy function under dynamics ( 1 ) is always negative ( we show this for the case ∆ = 0 below ) , τ dE dt = − ∑ A τ ( p− 1 ) 〈Wµ̂ , Wµ̂〉 p−1 p +1 µ̂ [ 〈dWµ̂ dt , xA〉µ̂〈Wµ̂ , Wµ̂〉µ̂ − 〈Wµ̂ , xA〉µ̂〈 dWµ̂ dt , Wµ̂〉µ̂ ] = = − ∑ A τ ( p− 1 ) 〈Wµ̂ , Wµ̂〉 p−1 p +1 µ̂ [ 〈xA , xA〉µ̂〈Wµ̂ , Wµ̂〉µ̂ − 〈Wµ̂ , xA〉2µ̂ ] ≤ 0 , ( 4 ) where Cauchy-Schwartz inequality is used . For every training example A the index of the activated hidden unit is defined as µ̂ = arg max µ [ 〈Wµ , xA〉µ ] . ( 5 ) Thus , the energy function decreases during learning . A similar result can be shown for ∆ 6= 0 . After the learning-phase is complete , the hash code is generated , as in FlyHash , via WTA sparsification : for a given query x we generate a hash code y ∈ { −1 , 1 } m as yµ = { 1 , 〈Wµ , x〉µ is in top k −1 , otherwise . ( 6 ) Thus , the hyperparameters of the method are p , r , m and ∆ . Note that the synapses are updated based only on pre- and post-synaptic activations resulting in Hebbian or anti-Hebbian updates . Many `` unsupervised '' learning to hash approaches provide a sort of `` weak supervision '' in the form of similarities evaluated in the feature space of deep CNNs trained on ImageNet ( Jin et al. , 2019 ) to achieve good performance . BioHash does not assume such information is provided and is completely unsupervised .

Summary: This paper studies a new model of locally sensitive hashing (LSH) that is inspired by the fruit fly Drosophila's olfactory circuit. Instead of mapping each input to a low-dimensional space, such LSH methods (FlyHash) map given $d$-dimensional inputs to an $m$-dimensional space such that $m \gg d$. However, these mappings enforce sparsity on the hash maps such that only $k$ out of $m$ coordinates of the output of the hash map are non-zero.