Search Data Structure Learning

1 INTRODUCTION . In many applications , the machines need to perform many searches in a gigantic database where the number of relevant documents is minuscule , e.g . ten in a billion . It is like searching for some needles in a haystack . In those cases , considering every document is extremely inefficient . For productivity , the search should not consider the whole database . Traditionally , this is accomplished by building a search data structure and seeking within it . Those data structures can take many forms . For example , there are tree-based structures such as the B-Tree ( Bayer & McCreight , 1970 ) , the k-d tree ( Friedman et al. , 1977 ) , the R-Tree ( Guttman , 1984 ) or the M-Tree ( Ciaccia et al. , 1997 ) to name a few . In addition to trees , KNNG ( Paredes & Chávez , 2005 ) build a graph designed for the k-nearest neighbour search . Later approaches improve on KNNG , both for construction and search time and for the search quality itself . In those lines , there is Efanna ( Fu & Cai , 2016 ) , HNSW ( Malkov & Yashunin , 2018 ) and ONNG ( Iwasaki & Miyazaki , 2018 ) . One of the most common types of search data structures is the hash table . It is so useful that it is implemented natively in programming languages such as Python ( with the dictionary type ) . Hash table is often the main tool an application will use for efficiency . For example , from a short and noisy song sample , Shazam ( Wang et al. , 2003 ) can retrieve the whole song by using hash tables filled with well-designed fingerprints of each song . Traditionally , the design of a search data structure was for a particular type of search . For example , hash tables can retrieve documents very quickly , even in gigantic databases . However , the query must be equal to the key . This requirement makes the hash table not always applicable . For instance , if the database is indexed by date and time and we seek all documents from a specific day , then it might not be optimal to query every second of that day with an equality search . B-Tree ( Bayer & McCreight , 1970 ) was precisely introduced for applications where a range search is preferable ( and faster insertion than dichotomic search is needed ) . Equality and range are far from being the only types of search . For instance , the k-nearest neighbours is another well-studied type a search . Also , the subset search is a more exotic example that occurs when every queries and documents are sets and when a document is relevant if and only if it is a subset of the query . As a final example , the auto-complete function often uses a Trie data structure ( De La Briandais , 1959 ) to suggest the end of each word . It is easy not to realize how the problem of efficiently finding needles in a haystack was solved multiple times for specific applications . This is the ascertainment that make Search Data Structure Learning ( SDSL ) a significant subject . Machine Learning has been a very flexible paradigm , whether by solving multiple NLP ( Natural Language Processing ) tasks with a unique Transformer ( Vaswani et al. , 2017 ) or solving most Atari games with Reinforcement Learning ( Mnih et al. , 2013 ) , the capacity of a single learning algorithm to perform on multiple tasks is outstanding . Search Data Structure Learning aims at developing generic learning algorithms meant for multiple types of search . Furthermore , what makes a document relevant need not to be described formally or even understood by a human . It might be k-nearest neighbour with a complex metric or something else altogether , the only thing we need for learning is a dataset . While we use the term ” Search Data Structure Learning ” for the first time , algorithms that fall into its paradigm already exist . The large video-hosting platform YouTube implements an SDSL algorithm ( Covington et al. , 2016 ) for its recommendation system ( the user being the query and the videos being the documents ) . Not having a formalized definition of what makes a document relevant and relying on Machine Learning has its challenges , the most important being the evaluation . Traditional search data structure , such as the hash table , the Trie , the B-Tree , are exact , meaning the documents retrieved contains all and uniquely the relevant documents . For comparing those exact search data structures , when possible , the comparison between two exact search data structures is made with the asymptotic time complexity ( the big-O notation ) . However , when the search is not exact , it is unclear how to compare structures with different efficiency and exactitude . The precision at Hamming distance of 2 is an attempt to unify those two properties into a single measure specific to the context of binary encoding . However , as described below , it fails in many aspects . It might seem like it is up to the programmer to decide what is more important between the speed and the quality of the retrieved documents . For example , the recall-queries per second ( Aumüller et al. , 2017 ) helps to visually understand the trade-off between speed and quality . In section 3 , we describe a reliable measure to evaluate the efficiency and quality simultaneously of any search data structure . This metric solidifies the Machine Learning subfield of Search Data Structure Learning . This article presents the SDSL framework that brings two crucial generalization w.r.t . its predecessors ( Li et al. , 2011 ; Cayton & Dasgupta , 2008 ) . First , it allows for dynamic databases , i.e . databases that might change or evolve after training . For example , it is plausible that a company wants to design and train a search engine ready for distribution to multiple clients without further training on each client ’ s database . The current mindset is to retrain each time a new database is given ; however , this is not feasible in many cases . Hopefully , this article motivates the research towards models that can generalize to never seen databases . Secondly , the previous framework does not support relative relations , i.e . when the relevance of a document w.r.t . a query depends on the other documents in the database . The most studied relative relations is probably the KNN task , which is relative since it is impossible to know if a document is in the k-nearest neighbour of a query without knowing the other documents . In contrast , radius search is an example of what we call an absolute relation because it is possible to know if a document is relevant to a query only by looking at the query-document pair . In this work , however , we did not introduce relative relations only for KNN . Many interesting relative relation tasks exist ; for example , another rather exciting example of relative relation is the multiple supporting facts : A harder task is to answer questions where two supporting statements have to be chained to answer the question [ ... ] where to answer the question “ Where is the football ? ” one has to combine information from two sentences “ John is in the playground ” and “ John picked up the football ” . ( Weston et al. , 2015 ) . In this work , we first introduce a general framework to formalize the SDSL task in which we present a novel metric to simultaneously evaluate the efficiency and quality of the search . Then , we inaugurate the field of SDSL with Efficient Learning Binary Access ( ELBA ) ( Section 4 ) that describes a family of models that use a traditional search data structure and parametric functions ( e.g . neural networks ) to create a discrete binary code ( s ) for both the queries and documents . A reader familiar with the field must appreciate the difficulty that has to be overcome when dealing with ( semi ) discrete structure . To instantiate ELBA , we concocted the F-beta Loss used for the training and Multi-Bernoulli Search ( MBS ) , a novel SDS technique designed for probabilistic binary codes . Finally , for comparisons , we will instantiate ELBA with other loss functions and another SDS , namely the MIHash ’ s loss ( Cakir et al. , 2017 ) , the HashNet ’ s loss ( Cao et al. , 2017 ) , and the Hamming Radius Search C.4 . We will then experimentally show the F-beta Loss and MBS ’ s advantage by putting in evidence their synergy . 2 RELATED WORK . In data structure terminology , the concepts of dynamic and static structures describe whether or not the structure can change via insertion , deletion or merge . In SDSL , if the database ( s ) used for training are not the same as the one ( s ) used for evaluation , then the structure has to search for documents only seen once at insertion . From a Machine Learning perspective , this is known as a One-Shot Learning task . For example , Matching Network ( Vinyals et al. , 2016 ) tries to match never seen elements together . However , applying their technique would require a database scan . Hence it is incompatible with a gigantic database . In the same vein , soft addressing ( or attention ) is a differentiable mechanism to select an element from many , thus compatible with gradient descent . Memory Network ( Kumar et al. , 2016 ) , Neural-Turing Machine ( Graves et al. , 2014 ) and Transformer ( Vaswani et al. , 2017 ) all use some kind of soft addressing . It is interesting for training our models but can not be used alone in SDSL . For the same reason as above , it would require considering the whole database . Finding the k-nearest neighbour is trivial with unlimited resources . In this field , the research focuses mainly on the efficiency of both the search and the structure ’ s construction . The exact algorithms are as efficient in higher dimensions than a random search due to the curse of dimensionality . Consequently , the focus has recently been on approximate k-nearest neighbour . The search data structure developed are mostly tree-based , such as the k-d tree ( Friedman et al. , 1977 ) or the K-Means tree ( Nister & Stewenius , 2006 ) , and graph-based , such as the KNNG ( Paredes & Chávez , 2005 ) , Efanna ( Fu & Cai , 2016 ) , HNSW ( Malkov & Yashunin , 2018 ) or ONNG ( Iwasaki & Miyazaki , 2018 ) just to name a few . A good resource for comparing those approaches is the ann-benchmark Aumüller et al . ( 2017 ) . In this work , we generalize the problem to conceive algorithms able to learn what to search efficiently . Efficient Learnable Binary Access , described below , encodes queries and documents into binary vectors . In this work , we will use neural networks as the encoders . Such encoders already exists in the literature . For example , CNNH ( Xia et al. , 2014 ) , DPSH ( Li et al. , 2015 ) , DHN ( Zhu et al. , 2016 ) , GreedyHash ( Su et al. , 2018 ) , PGDH ( Yuan et al. , 2018 ) , HashGan ( Cao et al. , 2018 ) , ADSH ( Jiang & Li , 2018 ) or JMLH ( Shen et al. , 2019 ) , just to name a few . Below we compare different loss functions , the F-beta Loss 4 the MIHash ’ s loss Cakir et al . ( 2017 ) and HashNet ’ s loss Cao et al . ( 2017 ) . Graph learning , introduced in Zhu et al . ( 2003 ) for semi-supervised learning , is a type of data structure learning that has shown experimentally to be a strong idea . Those models learn to do inference from graphs , sometime by generating them first . Some approaches work with static graphs ( static structures ) ( Zhu et al. , 2003 ; Perozzi et al. , 2014 ; Scarselli et al. , 2008 ; Bruna et al. , 2013 ) while other work with dynamic graphs ( dynamic structures ) ( Narayan & Roe , 2018 ; Manessi et al. , 2020 ) . While this literature does not focus on retrieval , they learn to compute using a data structure . To put SDSL in contrast with the Learning to Search framework ( Li et al. , 2011 ) . As mentioned in the introduction , it does not support dynamic databases and relative relation . It is possible to update the framework to deal with dynamic databases by taking an expectation over the databases in the retrieval quality Q ( T ) and computational cost C ( T ) . However , it is not clear how to deal with relative relations because the selection function T ( q , x ) is a ” matching function ” that does not exist for relative tasks . Generalizing the selection function by allowing it to consider the whole database ( i.e . with T ( q , X ) ) does not work because T ( q , X ) could use the ranking function s ( x , q ) on every document and nothing would penalize such exhaustive strategies since the computational cost is the number of candidates . Nevertheless , this is not the main issue . As with the framework proposed in Cayton & Dasgupta ( 2008 ) , the computational cost does not consider the retrieval cost but only the size of the candidates set ( divided by the number of documents in the database for the latter framework ) . Those frameworks fail to quantify the work needed to retrieve the candidates . For example , while proposing the Learning to Search framework , the authors relied on timing to evaluate their model . The SDSL framework , proposed below , provides a unique quantity that quantifies both the cost of retrieval and the candidates ’ quality simultaneously . Finally , while not introduced as such , an SDSL algorithm is used in NLP . In this field , many articles attempt to accelerate the training and inference of the neural network based models , in which the main bottleneck is the normalization over a large vocabulary . Morin & Bengio ( 2005 ) use a precomputed tree and train their model to travel from the root to a leaf , where each leaf corresponds to a word . Doing so accelerates both training and inference . Latter , Mnih & Hinton ( 2009 ) proposed a way to learn the structure of the tree .

Summary: In this paper, the authors proposed Search Data Structure Learning (SDSL), which they claim to be a generalization of the standard Search Data Structure. They also present a new metric called Sequential Search Work Ratio (SSWR) to evaluate the quality and efficiency of the search. They introduced a new loss called F-beta Loss, showing their algorithm is better than two previous results, MIHash (Cakir et al. 2017) and HashNet (Cao et al. 2017).