Learning-Augmented Data Stream Algorithms

1 INTRODUCTION . Processing data streams has been an active research field in the past two decades . This is motivated by the increasingly common scenario where the size of the data far exceeds the size of the available storage , and the only feasible access to the data is to make a single or a few passes over the data . This situation occurs , for example , in network applications with high-speed packets being sent through routers with limited resources . Other examples include processing internet search logs , sensor networks , and scientific data streams . For an introduction to data streams , see , e.g. , Muthukrishnan ( 2005 ) . Formally , in the data stream model , we assume there is an underlying frequency vector x ∈ Zn , initialized to 0n , which evolves throughout the course of a stream . The stream consists of updates of the form ( i , w ) , meaning xi ← xi + w. Suppose M is such that ‖x‖∞ = maxi∈ { 1 , ... , n } |xi| ≤ M throughout the stream . At the end of the stream , we are asked to approximate f ( x ) for a function f : Zn → R. Because of the sheer size of the stream , most algorithms are approximate and randomized . Typically , such algorithms output an estimate Z for which Pr { ( 1 − ) f ( x ) ≤ Z ≤ ( 1 + ) f ( x ) } ≥ 2/3 , where the probability is over the randomness used by the algorithm , and not of the input stream , which may be a worst-case stream . The success probability 2/3 can be replaced with any constant greater than 1/2 by independently repeating the algorithm a constant number of times and taking the median estimate . The space complexity of the algorithm is measured in bits and the goal is to use much less than the trivial n bits of space required to store x . A common difficulty in designing algorithms in the data stream model is that the coordinates i for which |xi| is large , i.e. , the heavy hitters , are not known in advance . Such coordinates usually require very accurate estimates in order to achieve an accurate final estimate of the function value . A number of techniques have emerged for identifying such coordinates , such as the Count-Min and CountSketch . These techniques are often used at multiple scales by subsampling the input coordinates and finding heavy items in the subsample , and have found applications to estimating frequency moments , cascaded aggregates , earthmover distance , and empirical entropy , among other statistics ( Indyk & Woodruff , 2005 ; Chakrabarti et al. , 2006 ; Jayram & Woodruff , 2009 ; Andoni et al. , 2009 ; McGregor et al. , 2016 ) . In many applications , however , the worst-case input is rarely seen ; in fact , there is usually a pattern inside the underlying vector . For instance , in network traffic monitoring , the large or “ elephant ” flows fluctuate on a daily basis , but the actual source and destination servers of those flows do not vary much . In our formulation , this implies the value xi fluctuates while whether or not xi is a heavy coordinate remains stable . Motivated by this as well as related work on learning oracles for algorithm design ( Gupta & Roughgarden , 2016 ; Dick et al. , 2017 ; Balcan et al. , 2018a ; b ) , Hsu et al . ( 2019 ) introduced the notion of a heavy hitter oracle into the data stream model . The heavy hitter oracle , trained on past data , receives a coordinate index i and returns a prediction of whether xi will be heavy or not . They showed that such an oracle simplifies classical heavy hitter algorithms such as COUNT-MIN and COUNT-SKETCH . We note that similar oracles were also studied in previous work , e.g. , membership oracles for Bloom filters by Kraska et al . ( 2017 ) . These works demonstrate the feasibility of creating such an oracle using machine learning techniques . Recently , learned oracles for low rank approximation , which can be viewed as an analogue of heavy hitters for heavy directions , was explored by Indyk et al . ( 2019 ) . The work of Hsu et al . ( 2019 ) is the starting point of our work . While Hsu et al . ( 2019 ) showed applications of the heavy hitter oracle to frequency estimation , the full power of such an oracle remained largely unexplored . Our main goal is to understand if such an oracle is more broadly useful , and whether it can give memory and time improvements to the large number of data stream problems mentioned above . Accurately estimating the heavy hitters in a vector is the bottleneck of existing algorithms for numerous data stream problems , and one can not help but ask : Can a heavy hitter oracle be applied to a large number of problems in the data stream model to obtain significantly improved bounds for these problems ? We consider estimating the following common functions f ( x ) in the data stream model : • Distinct Elements : f ( x ) = ‖x‖0 , where ‖x‖0 is the number of nonzero coordinates of x , defined as ‖x‖0 = | { i : xi 6= 0 } | . This quantity is useful for detecing denial of service attacks and database query optimization ( Kane et al. , 2010b ) . • the Fp-Moment : f ( x ) = ‖x‖pp , where ‖x‖p is the usual ` p-norm of x , defined as ‖x‖p = ( ∑ i |xi|p ) 1/p . For 0 < p < 2 , these are often more robust than the Euclidean norm ( Indyk , 2000 ) . For p > 2 these are used to estimate skewness and kurtosis ( Indyk & Woodruff , 2005 ) . • the ( k , p ) -Cascaded Norm : in this problem x is an n × d matrix that receives updates to its individual entries , and f ( x ) = ‖x‖k , p : = ( ∑ i ( ∑ j |xij |p ) k/p ) 1/k . These are useful for executing multiple databse queries ( Cormode & Muthukrishnan , 2005 ; Jayram & Woodruff , 2009 ) . These problems have also been studied for non-numerical data , such as geometric data . For example , the Fp-moment problem was studied on a stream of rectangles in Tirthapura & Woodruff ( 2012 ) . One can think of an underlying ∆d-dimensional vector x , identified with a d-dimensional rectangle { 1 , . . . , ∆ } d. Without loss of generality , we assume that ∆ is a power of 2 . Each item in the stream is a pair of the form ( R , w ) , where R is a d-dimensional rectangle and w is an integer , meaning xi ← xi +w for all points i ∈ R. The goal is to estimate f ( x ) = ‖x‖pp . This is called the Rectangle Fp-Moment problem , which occurs in spatial data streams and constraint databases . Our Results We show that a heavy hitter oracle can greatly improve the complexity of a wide array of commonly studied problems in the data stream model , leading to the first optimal bounds for several important problems , and shattering lower bounds that have stood in the way of making further progress on important problems . We note that not only do we give new algorithms , we also give several lower bounds for algorithms equipped with the oracle , that show optimality of our algorithms even with an oracle . Our algorithms not only give practically better , theoreticallygrounded improvements to these problems with an oracle , they also shed light on what exactly the difficulties are of making further progress on data stream problems without an oracle . We consider both perfect oracles and oracles that may sometimes make mistakes . We summarize in Table 1 our improvements to existing algorithms using appropriate heavy hitter oracles , whose conditions are summarized in Table 2 . Throughout , we make the conventional assumption that m , M = poly ( n ) . Distinct Elements : For the problem of estimating ‖x‖0 , we improve the prior memory bound of O ( 1 2 log n ( log 1 + log log n ) ) to O ( 1 2 log n log 1 ) . For constant , this gives an optimal O ( log n ) bits of memory , breaking a recent Ω ( log n log log n ) bit lower bound shown in Woodruff & Yang ( 2019 ) . Examining the lower bound argument in Woodruff & Yang ( 2019 ) , the hard instance is precisely when there are many items of large frequency , namely , the product of the first small number of primes . This prevents hashing-based algorithms , such as the one in Kane et al . ( 2010b ) , from working with O ( log n ) bits of space , since hash values are typically taken modulo a small prime and such heavy items necessitate the use of a large enough prime . Surprisingly , we show that with a heavy hitter oracle , we can separately handle such items and thus bypass this lower bound . We note that our O ( log n ) -bit algorithm is optimal even given a heavy hitter oracle ; see the Ω ( log n ) lower bound in Alon et al . ( 1999 ) , which holds even if all item frequencies are in the set { 0 , 1 , 2 } . Fp-Moment Estimation , 0 < p < 2 : There is a long line of work on this problem with the best known bounds given in Kane et al . ( 2011 ) , which achieve an optimal O ( −2 log n ) bits of space , and O ( log2 ( 1/ ) log log 1/ ) time to process each element . The existing algorithm is based on separately handling those elements i for which |xi|p ≥ 2‖x‖pp , i.e. , the heavy hitters , from the remaining “ light items ” . Although we can simply store the heavy hitters given a heavy hitter oracle , unfortunately , the large O ( log2 ( 1/ ) log log 1/ ) processing time in the algorithm of Kane et al . ( 2011 ) also comes from running a fast multipoint evaluation on the light elements . We observe that if we instead train two oracles , one for detecting heavy hitters , and one for detecting items xi with |xi|p ≥ 2 log2 ( 1/ ) log log ( 1/ ) ‖x‖pp , then we can also separate out “ medium items ” , i.e. , items that are detected with this oracle but not our heavy hitter oracle . There are only O ( −2 log2 ( 1/ ) log log ( 1/ ) ) medium items and so assuming log n ≥ poly ( log ( 1/ ) ) , one could perfectly hash all such items and maintain their hashed identities in the optimal O ( −2 log n ) bit of space . However , one can not maintain their counts , since each count is O ( log n ) bits , and approximate counts do not work if the stream can increment or decrement the coordinate values xi . This is too much memory , as it would lead to O ( −2 ( log n ) log2 ( 1/ ) log log ( 1/ ) ) bits of space , even worse than without the oracle . We could also maintain a so-called p-stable random variable for each medium item , and use it to estimate the Fp-moment of the medium items , but this would again be O ( −2 ( log n ) log2 ( 1/ ) log log ( 1/ ) ) bits of space to store one random variable per medium item . We instead observe that most of the p-stable random variables are small , and so can be stored with many fewer than log n bits , and therefore globally we can get by with only O ( −2 log n ) bits in total for estimating the medium items . Finally , for the light items , the critical observation is that we can sub-sample a 1 log2 ( 1/ ) log log ( 1/ ) fraction of them , and still accurately estimate their contribution . But this means we can run the multipoint evaluation algorithm of Kane et al . ( 2011 ) on such items , since while that has update time O ( log2 ( 1/ ) log log ( 1/ ) ) , on average it is only executed on one out of every log2 ( 1/ ) log log ( 1/ ) items . In summary , with the aid of our oracles and by separately handling these three types of items , we are able to achieve the optimal O ( 1 ) amount of update time per item , while retaining the optimal O ( −2 log n ) bits of memory . Fp-Moment Estimation for p > 2 : It is well-known ( Bar-Yossef et al. , 2004 ) that a hard input distribution for this problem is when a random coordinate of x has absolute value n1/p , and all other coordinates have value 0 or 1 . In this case , one needs to certify that there is indeed a coordinate of value n1/p , as it contributes a constant fraction of ‖x‖pp . There is a known Ω ( n1−2/p ) bits of space lower bound for this . However , a heavy hitter oracle exactly allows us to overcome this , as it can be used to separately identify such items . Our idea is to separate out every item i for which |xi|p ≥ 1n1/2 ‖x‖ p p with the oracle , and run an existing Fp-estimation algorithm on such items . Since there are at most n1/2 items , existing algorithms give ( n1/2 ) 1−2/p poly ( log n ) = n1/2−1/p poly ( log n ) bits of memory for this problem . For the remaining light elements , one can show that if one subsamples only a 1/n1/2 fraction of them and then runs an existing Fp-estimation algorithm on them , one can use this to accurately estimate their contribution . This again takes n1/2−1/p poly ( log n ) bits of memory , balancing the memory with the other part of our algorithm and giving a quadratic improvement in memory in the case without a heavy hitter oracle . Overall our algorithm uses O ( n1/2−1/p poly ( log n ) ) bits with an oracle and we can in fact show that this is tight up to logarithmic factors as there exists a lower bound of Ω ( n1/2−1/p ) bits provided the same oracle . We describe our algorithms for cascaded norms and rectangle Fp-moments in the appendix . We conduct experiments for the distinct elements and the Fp moment ( p > 2 ) problems , on both real-world and synthetic data , which demonstrate significant practical benefits .

Summary: The paper presents algorithms for solving computational problems in a datastream model augmented with an oracle learned from data. The authors show that under this model, there exist algorithms that have significantly better time and space complexity than the current best known algorithms that do not use an oracle. The authors support their theoretical analysis with experiments in which the oracle is represented by a deep neural network and demonstrate improvement over classical algorithms that do not use machine learning.