Learning-Based Frequency Estimation Algorithms

1 . INTRODUCTION . Classical algorithms provide formal guarantees over their performance , but often fail to leverage useful patterns in their input data to improve their output . On the other hand , deep learning models are highly successful at capturing and utilizing complex data patterns , but often lack formal error bounds . The last few years have witnessed a growing effort to bridge this gap and introduce algorithms that can adapt to data properties while delivering worst case guarantees . Deep learning modules have been integrated into the design of Bloom filters ( Kraska et al. , 2018 ; Mitzenmacher , 2018 ) , caching algorithms ( Lykouris & Vassilvitskii , 2018 ) , graph optimization ( Dai et al. , 2017 ) , similarity search ( Salakhutdinov & Hinton , 2009 ; Weiss et al. , 2009 ) and compressive sensing ( Bora et al. , 2017 ) . This paper makes a significant step toward this vision by introducing frequency estimation streaming algorithms that automatically learn to leverage the properties of the input data . Estimating the frequencies of elements in a data stream is one of the most fundamental subroutines in data analysis . It has applications in many areas of machine learning , including feature selection ( Aghazadeh et al. , 2018 ) , ranking ( Dzogang et al. , 2015 ) , semi-supervised learning ( Talukdar & Cohen , 2014 ) and natural language processing ( Goyal et al. , 2012 ) . It has been also used for network measurements ( Estan & Varghese , 2003 ; Yu et al. , 2013 ; Liu et al. , 2016 ) and security ( Schechter et al. , 2010 ) . Frequency estimation algorithms have been implemented in popular data processing libraries , such as Algebird at Twitter ( Boykin et al. , 2016 ) . They can answer practical questions like : what are the most searched words on the Internet ? or how much traffic is sent between any two machines in a network ? The frequency estimation problem is formalized as follows : given a sequence S of elements from some universe U , for any element i ∈ U , estimate fi , the number of times i occurs in S. If one could store all arrivals from the stream S , one could sort the elements and compute their frequencies . However , in big data applications , the stream is too large ( and may be infinite ) and can not be stored . This challenge has motivated the development of streaming algorithms , which read the elements of S in a single pass and compute a good estimate of the frequencies using a limited amount of space.1 Over the last two decades , many such streaming algorithms have been developed , including 1 Specifically , the goal of the problem is as follows . Given a sequence S of elements from U , the desired algorithm reads S in a single pass while writing into memory C ( whose size can be much smaller than the length of S ) . Then , given any element i ∈ U , the algorithm reports an estimation of fi based only on the content of C. Count-Sketch ( Charikar et al. , 2002 ) , Count-Min ( Cormode & Muthukrishnan , 2005b ) and multistage filters ( Estan & Varghese , 2003 ) . The performance guarantees of these algorithms are wellunderstood , with upper and lower bounds matching up to O ( · ) factors ( Jowhari et al. , 2011 ) . However , such streaming algorithms typically assume generic data and do not leverage useful patterns or properties of their input . For example , in text data , the word frequency is known to be inversely correlated with the length of the word . Analogously , in network data , certain applications tend to generate more traffic than others . If such properties can be harnessed , one could design frequency estimation algorithms that are much more efficient than the existing ones . Yet , it is important to do so in a general framework that can harness various useful properties , instead of using handcrafted methods specific to a particular pattern or structure ( e.g. , word length , application type ) . In this paper , we introduce learning-based frequency estimation streaming algorithms . Our algorithms are equipped with a learning model that enables them to exploit data properties without being specific to a particular pattern or knowing the useful property a priori . We further provide theoretical analysis of the guarantees associated with such learning-based algorithms . We focus on the important class of “ hashing-based ” algorithms , which includes some of the most used algorithms such as Count-Min , Count-Median and Count-Sketch . Informally , these algorithms hash data items into B buckets , count the number of items hashed into each bucket , and use the bucket value as an estimate of item frequency . The process can be repeated using multiple hash functions to improve accuracy . Hashing-based algorithms have several useful properties . In particular , they can handle item deletions , which are implemented by decrementing the respective counters . Furthermore , some of them ( notably Count-Min ) never underestimate the true frequencies , i.e. , f̃i ≥ fi holds always . However , hashing algorithms lead to estimation errors due to collisions : when two elements are mapped to the same bucket , they affect each others ’ estimates . Although collisions are unavoidable given the space constraints , the overall error significantly depends on the pattern of collisions . For example , collisions between high-frequency elements ( “ heavy hitters ” ) result in a large estimation error , and ideally should be minimized . The existing algorithms , however , use random hash functions , which means that collisions are controlled only probabilistically . Our idea is to use a small subset of S , call it S′ , to learn the heavy hitters . We can then assign heavy hitters their own buckets to avoid the more costly collisions . It is important to emphasize that we are learning the properties that identify heavy hitters as opposed to the identities of the heavy hitters themselves . For example , in the word frequency case , shorter words tend to be more popular . The subset S′ itself may miss many of the popular words , but whichever words popular in S′ are likely to be short . Our objective is not to learn the identity of high frequency words using S′ . Rather , we hope that a learning model trained on S′ learns that short words are more frequent , so that it can identify popular words even if they did not appear in S′ . Our main contributions are as follows : • We introduce learning-based frequency estimation streaming algorithms , which learn the properties of heavy hitters in their input and exploit this information to reduce errors • We provide performance guarantees showing that our algorithms can deliver a logarithmic factor improvement in the error bound over their non-learning counterparts . Furthermore , we show that our learning-based instantiation of Count-Min , a widely used algorithm , is asymptotically optimal among all instantiations of that algorithm . See Table 4.1 in section 4.1 for the details . • We evaluate our learning-based algorithms using two real-world datasets : traffic load on an Internet backbone link and search query popularity . In comparison to their non-learning counterparts , our algorithms yield performance gains that range from 18 % to 71 % . 2 . RELATED WORK . Frequency estimation in data streams . Frequency estimation , and the closely related problem of finding frequent elements in a data stream , are some of the most fundamental and well-studied problems in streaming algorithms , see Cormode & Hadjieleftheriou ( 2008 ) for an overview . Hashingbased algorithms such as Count-Sketch ( Charikar et al. , 2002 ) , Count-Min ( Cormode & Muthukrishnan , 2005b ) and multi-stage filters ( Estan & Varghese , 2003 ) are widely used solutions for these problems . These algorithms also have close connections to sparse recovery and compressed sens- ing ( Candès et al. , 2006 ; Donoho , 2006 ) , where the hashing output can be considered as a compressed representation of the input data ( Gilbert & Indyk , 2010 ) . Several “ non-hashing ” algorithms for frequency estimation have been also proposed ( Misra & Gries , 1982 ; Demaine et al. , 2002 ; Karp et al. , 2003 ; Metwally et al. , 2005 ) . These algorithms do not possess many of the properties of hashing-based methods listed in the introduction ( such as the ability to handle deletions ) , but they often have better accuracy/space tradeoffs . For a fair comparison , our evaluation focuses only on hashing algorithms . However , our approach for learning heavy hitters should be useful for non-hashing algorithms as well . Some papers have proposed or analyzed frequency estimation algorithms customized to data that follows Zipf Law ( Charikar et al. , 2002 ; Cormode & Muthukrishnan , 2005a ; Metwally et al. , 2005 ; Minton & Price , 2014 ; Roy et al. , 2016 ) ; the last algorithm is somewhat similar to the “ lookup table ” implementation of the heavy hitter oracle that we use as a baseline in our experiments . Those algorithms need to know the data distribution a priori , and apply only to one distribution . In contrast , our learning-based approach applies to any data property or distribution , and does not need to know that property or distribution a priori . Learning-based algorithms . Recently , researchers have begun exploring the idea of integrating machine learning models into algorithm design . In particular , researchers have proposed improving compressed sensing algorithms , either by using neural networks to improve sparse recovery algorithms ( Mousavi et al. , 2017 ; Bora et al. , 2017 ) , or by designing linear measurements that are optimized for a particular class of vectors ( Baldassarre et al. , 2016 ; Mousavi et al. , 2015 ) , or both . The latter methods can be viewed as solving a problem similar to ours , as our goal is to design “ measurements ” of the frequency vector ( f1 , f2 . . . , f|U | ) tailored to a particular class of vectors . However , the aforementioned methods need to explicitly represent a matrix of size B × |U | , where B is the number of buckets . Hence , they are unsuitable for streaming algorithms which , by definition , have space limitations much smaller than the input size . Another class of problems that benefited from machine learning is distance estimation , i.e. , compression of high-dimensional vectors into compact representations from which one can estimate distances between the original vectors . Early solutions to this problem , such as Locality-Sensitive Hashing , have been designed for worst case vectors . Over the last decade , numerous methods for learning such representations have been developed ( Salakhutdinov & Hinton , 2009 ; Weiss et al. , 2009 ; Jegou et al. , 2011 ; Wang et al. , 2016 ) . Although the objective of those papers is similar to ours , their techniques are not usable in our applications , as they involve a different set of tools and solve different problems . More broadly , there have been several recent papers that leverage machine learning to design more efficient algorithms . The authors of ( Dai et al. , 2017 ) show how to use reinforcement learning and graph embedding to design algorithms for graph optimization ( e.g. , TSP ) . Other learning-augmented combinatorial optimization problems are studied in ( He et al. , 2014 ; Balcan et al. , 2018 ; Lykouris & Vassilvitskii , 2018 ) . More recently , ( Kraska et al. , 2018 ; Mitzenmacher , 2018 ) have used machine learning to improve indexing data structures , including Bloom filters that ( probabilistically ) answer queries of the form “ is a given element in the data set ? ” As in those papers , our algorithms use neural networks to learn certain properties of the input . However , we differ from those papers both in our design and theoretical analysis . Our algorithms are designed to reduce collisions between heavy items , as such collisions greatly increase errors . In contrast , in existence indices , all collisions count equally . This also leads to our theoretical analysis being very different from that in ( Mitzenmacher , 2018 ) .

Summary: This paper introduces the study of the problem of frequency estimation algorithms with machine learning advice. The problem considered is the standard frequency estimation problem in data streams where the goal is to estimate the frequency of the i-th item up to an additive error, i.e. the |\tilde f_i - f_i| should be minimized where \tilde f_i is the estimate of the true frequency f_i.