Meta-Learning Neural Bloom Filters

There has been a recent trend in training neural networks to replace data structures that have been crafted by hand , with an aim for faster execution , better accuracy , or greater compression . In this setting , a neural data structure is instantiated by training a network over many epochs of its inputs until convergence . In many applications this expensive initialization is not practical , for example streaming algorithms — where inputs are ephemeral and can only be inspected a small number of times . In this paper we explore the learning of approximate set membership over a stream of data in one-shot via meta-learning . We propose a novel memory architecture , the Neural Bloom Filter , which we show can be more compressive than Bloom Filters and several existing memory-augmented neural networks . 1 INTRODUCTION . One of the simplest questions one can ask of a set of data is whether or not a given query is contained within it . Is q , our query , a member of S , our chosen set of observations ? This set membership query arises across many computing domains ; from databases , network routing , and firewalls . One could query set membership by storing S in its entirety and comparing q against each element . However , more space-efficient solutions exist . The original and most widely implemented approximate set membership data-structure is the Bloom Filter ( Bloom , 1970 ) . It works by storing sparse distributed codes , produced from randomized hash functions , within a binary vector . The Bloom-filter trades off space for an allowed false positive rate , which arises due to hash collisions . However its error is one-sided ; if an element q is contained in S then it will always be recognized . It never emits false negatives . One can find Bloom Filters embedded within a wide range of production systems ; from network security ( Geravand & Ahmadi , 2013 ) , to block malicious IP addresses ; databases , such as Google ’ s Bigtable ( Chang et al. , 2008 ) , to avoid unnecessary disk lookups ; cryptocurrency ( Hearn & Corallo , 2012 ) , to allow clients to filter irrelevant transactions ; search , such as Facebook ’ s typeahead search ( Adams , 2010 ) , to filter pages which do not contain query prefixes ; and program verification ( Dillinger & Manolios , 2004 ) , to avoid recomputation over previously observed states . While the main appeal of Bloom Filters is favourable compression , another important quality is the support for dynamic updates . New elements can be inserted inO ( 1 ) time . This is not the case for all approximate set membership data structures . For example , perfect hashing saves ≈ 40 % space over Bloom Filters but requires a pre-processing stage that is polynomial-time in the number of elements to store ( Dietzfelbinger & Pagh , 2008 ) . Whilst the static set membership problem is interesting , it limits the applicability of the algorithm . For example , in a database application that is serving a high throughput of write operations , it may be intractable to regenerate the full data-structure upon each batch of writes . We thus focus on the data stream computation model ( Muthukrishnan et al. , 2005 ) , where input observations are assumed to be ephemeral and can only be inspected a constant number of times — usually once . This captures many real-world applications : network traffic analysis , database query serving , and reinforcement learning in complex domains . Devising an approximate set membership data structure that is not only more compressive than Bloom Filters , but can be applied to either dynamic or static sets , could have a significant performance impact on modern computing applications . In this paper we investigate this problem using memory-augmented neural networks and meta-learning . We build upon the recently growing literature on using neural networks to replace algorithms that are configured by heuristics , or do not take advantage of the data distribution . For example , Bloom Filters are indifferent to the data distribution . They have near-optimal space efficiency when data is drawn uniformly from a universe set ( Carter et al. , 1978 ) ( maximal-entropy case ) but ( as we shall show ) are sub-optimal when there is more structure . Prior studies on this theme have investigated compiler optimization ( Cummins et al. , 2017 ) , computation graph placement ( Mirhoseini et al. , 2017 ) , and data index structures such as b-trees ( Kraska et al. , 2018 ) . In the latter work , Kraska et al . ( 2018 ) explicitly consider the problem of static set membership . By training a neural network over a fixed S ( URLs from Google ’ s Transparency Report ) with negative examples in the form of held-out URLs , they observe 36 % space reduction over a conventional Bloom Filter1 . Crucially this requires iterating over the storage set S a large number of times to embed its salient information into the weights of a neural network classifier . For a new S this process would have to be repeated from scratch . Instead of learning from scratch , we draw inspiration from the few-shot learning advances obtained by meta-learning memory-augmented neural networks ( Santoro et al. , 2016 ; Vinyals et al. , 2016 ) . In this setup , tasks are sampled from a common distribution and a network learns to specialize to ( learn ) a given task with few examples . This matches very well to applications where many Bloom Filters are instantiated over different subsets of a common data distribution . For example , a Bigtable database usually contains one Bloom Filter per SSTable file . For a large table that contains Petabytes of data , say , there can be over 100 , 000 separate instantiated data-structures which share a common row key format and query distribution . Meta-learning allows us to exploit this common redundancy . The main contributions of this paper are ( 1 ) A new sparse memory-augmented neural network architecture , the Neural Bloom Filter , which learns to write to memory using a distributed write scheme , and ( 2 ) An empirical evaluation of the Neural Bloom Filter meta-learned on one-shot approximate set membership problems of varying structure . We compare with the classical Bloom Filter alongside other memory-augmented neural networks such as the Differentiable Neural Computer ( Graves et al. , 2016 ) and Memory Networks ( Sukhbaatar et al. , 2015 ) . We find when there is no structure , that differentiates the query set elements and queries , the Neural Bloom Filter learns a solution similar to a Bloom Filter derivative ( a Bloom-g filter ( Qiao et al. , 2011 ) ) , but when there is a lot of structure the solution can be considerably more space-efficient . 2 BACKGROUND . 2.1 APPROXIMATE SET MEMBERSHIP . The problem of exact set membership is to state whether or not a given query q belongs to a set of n distinct observations S = { x1 , . . . , xn } where xi are drawn from a universe set U . By counting the number of distinct subsets of size n it can be shown that any such exact set membership tester requires at least log2 ( |U | n ) bits of space . To mitigate the space dependency on |U | , which can be prohibitively large , one can relax the constraint on perfect correctness . Approximate set membership allows for a false positive rate of at most . Specifically we answer q ∈ A ( S ) where A ( S ) ⊇ S and p ( q ∈ A ( S ) − S ) ≤ . It can be shown2 the space requirement for approximate set membership of uniformly sampled observations is at least n log2 ( 1 ) bits ( Carter et al. , 1978 ) which can be achieved with perfect hashing . So for a false positive rate of 1 % , say , this amounts to 6.6 bits per element . In contrast to storing raw or compressed elements this can be a huge space saving , for example ImageNet images require 108 KB per image on average when compressed with JPEG , an increase of over four orders of magnitude . 2.2 BLOOM FILTER . The Bloom Filter ( Bloom , 1970 ) is a data structure which solves the dynamic approximate set membership problem with near-optimal space complexity . It assumes access to k uniform hash functions 1The space saving increases to 41 % when an additional trick is incorporated , in discretizing and re-scaling the classifier outputs and treating the resulting function as a hash function to a bit-map . 2By counting the minimal number of A ( S ) sets required to cover all S ⊂ U . hi : U → { 1 , . . . , m } , i = 1 , . . . , k such that p ( hi ( x ) = j ) = 1/m independent of prior hash values or input x . The Bloom Filter ’ s memory M ∈ [ 0 , 1 ] m is a binary string of length m which is initialized to zero . Writes are performed by hashing an input x to k locations in M and setting the corresponding bits to 1 , M [ hi ( x ) ] ← 1 ; i = 1 , . . . , k. For a given query q the Bloom Filter returns true if all corresponding hashed locations are set to 1 and returns false otherwise : Query ( M , q ) : = M [ h1 ( q ) ] ∧M [ h2 ( q ) ] ∧ . . . ∧M [ hk ( q ) ] . This incurs zero false negatives , as any previously observed input must have enabled the corresponding bits in M , however there can be false positives due to hash collisions . To achieve a false positive rate of with minimal space one can set k = log2 ( 1/ ) and m = n log2 ( 1/ ) log2 e , where e is Euler ’ s number . The resulting space is a factor of log2 e ≈ 1.44 from the optimal static lower bound given by Carter et al . ( 1978 ) . 2.3 MEMORY-AUGMENTED NEURAL NETWORKS . Recurrent neural networks such as LSTMs retain a small amount of memory via the recurrent state . However this is usually tied to the number of trainable parameters in the model . There has been recent interest in augmenting neural networks with a larger external memory . The method for doing so , via a differentiable write and read interface , was first popularized by the Neural Turing Machine ( NTM ) ( Graves et al. , 2014 ) and its successor the Differentiable Neural Computer ( DNC ) ( Graves et al. , 2016 ) in the context of learning algorithms , and by Memory Networks ( Sukhbaatar et al. , 2015 ) in the context of question answering . Memory Networks store embeddings of the input in separate rows of a memory matrix M . Reads are performed via a differentiable content-based addressing operation . Given a query embedding q we take some similarity measure D ( e.g . cosine similarity , or negative euclidean distance ) against each row in memory and apply a softmax to obtain a soft address vector a ∝ eD ( q , M ) . A read is then a weighted sum over memory r ← aTM . The NTM and DNC use the same content-based read mechanism , but also learns to write . These models can arbitrate whether to write to slots in memory with similar content ( content-based writes ) , temporally ordered locations , or unused memory . When it comes to capacity , there has been consideration to scaling both the DNC and Memory Networks to very large sizes using sparse read and write operations ( Rae et al. , 2016 ; Chandar et al. , 2016 ) . However another way to increase the capacity is to increase the amount of compression which occurs in memory . Memory Nets can create compressive representations of each input , but can not compress jointly over multiple inputs because they are hard-wired to write one slot per timestep . The NTM and DNC can compress over multiple slots in memory because they can arbitrate writes across multiple locations , but in practice seem to choose very sharp read and write addresses . The Kanerva Machine ( Wu et al. , 2018 ) tackles memory-wide compression using a distributed write scheme to jointly compose and compress its memory contents . The model uses content-based addressing over a separate learnable addressing matrix A , instead of the memory M , and thus learns where to write . We take inspiration from this scheme .

Summary: The paper proposes a method whereby a neural network is trained and used as a data structure to assess approximate set membership. Unlike the Bloom filter, which uses hand-constructed hash functions to store data and a pre-specified method for answering queries, the Neural Bloom Filter learns both the Write function and the Read function (both are "soft" values rather than the hard binary values used in the Bloom filter). Experiments show that, when there is structure in the data set, the Neural Bloom Filter can achieve the same false positive rate with less space.