Learning-based Support Estimation in Sublinear Time

log ( 1/ε ) · n1−Θ ( 1/ log ( 1/ε ) ) . We evaluate the proposed algorithms on a collection of data sets , using the neuralnetwork based estimators from Hsu et al , ICLR ’ 19 as predictors . Our experiments demonstrate substantial ( up to 3x ) improvements in the estimation accuracy compared to the state of the art algorithm . 1 INTRODUCTION . Estimating the support size of a distribution from random samples is a fundamental problem with applications in many domains . In biology , it is used to estimate the number of distinct species from experiments ( Fisher et al. , 1943 ) ; in genomics to estimate the number of distinct protein encoding regions ( Zou et al. , 2016 ) ; in computer systems to approximate the number of distinct blocks on a disk drive ( Harnik et al. , 2016 ) , etc . The problem has also applications in linguistics , query optimization in databases , and other fields . Because of its wide applicability , the problem has received plenty of attention in multiple fields1 , including statistics and theoretical computer science , starting with the seminal works of Good and Turing Good ( 1953 ) and Fisher et al . ( 1943 ) . A more recent line of research pursued over the last decade ( Raskhodnikova et al. , 2009 ; Valiant & Valiant , 2011 ; 2013 ; Wu & Yang , 2019 ) focused on the following formulation of the problem : given access to independent samples from a distribution ∗Authors listed in alphabetical order 1A partial bibliography from 2007 contains over 900 references . It is available at https : //courses.cit.cornell.edu/jab18/bibliography.html . P over a discrete domain { 0 , . . . , n − 1 } whose minimum non-zero mass2 is at least 1/n , estimate the support size of P up to ±εn . The state of the art estimator , due to Valiant & Valiant ( 2011 ) ; Wu & Yang ( 2019 ) , solves this problem using only O ( n/ log n ) samples ( for a constant ε ) . Both papers also show that this bound is tight . A more straightforward linear-time algorithm exists , which reports the number of distinct elements seen in a sample of size N = O ( n log ε−1 ) ( which is O ( n ) for constant ε ) , without accounting for the unseen items . This algorithm succeeds because each element i with non-zero mass ( and thus mass at least 1/n ) appears in the sample with probability at least 1 − ( 1 − 1/n ) N > 1 − ε , so in expectation , at most ε · n elements with non-zero mass will not appear in the sample . Thus , in general , the number of samples required by the best possible algorithm ( i.e. , n/ log n ) is only logarithmically smaller than the complexity of the straightforward linear-time algorithm . A natural approach to improve over this bound is to leverage the fact that in many applications , the input distribution is not entirely unknown . Indeed , one can often obtain rough approximations of the element frequencies by analyzing different but related distributions . For example , in genomics , frequency estimates can be obtained from the frequencies of genome regions of different species ; in linguistics they can be inferred from the statistical properties of the language ( e.g. , long words are rare ) , or from a corpus of writings of a different but related author , etc . More generally , such estimates can be learned using modern machine learning techniques , given the true element frequencies in related data sets . The question then becomes whether one can utilize such predictors for use in support size estimation procedures in order to improve the estimation accuracy . Our results In this paper we initiate the study of such “ learning-based ” methods for support size estimation . Our contributions are both theoretical and empirical . On the theory side , we show that given a “ good enough ” predictor of the distribution P , one can solve the problem using much fewer than n/ log n samples . Specifically , suppose that in the input distribution P the probability of element i is pi , and that we have access to a predictor Π ( i ) such that Π ( i ) ≤ pi ≤ b ·Π ( i ) for some constant approximation factor b ≥ 1.3 Then we give an algorithm that estimates the support size up to ±εn using only log ( 1/ε ) · n1−Θ ( 1/ log ( 1/ε ) ) samples , assuming the approximation factor b is a constant ( see Theorem 1 for a more detailed bound ) . This improves over the bound of Wu & Yang ( 2019 ) for any fixed values of the accuracy parameter ε and predictor quality factor b . Furthermore , we show that this bound is almost tight . Our algorithm is presented in Algorithm 1 . On a high level , it partitions the range of probability values into geometrically increasing intervals . We then use the predictor to assign the elements observed in the sample to these intervals , and produce a Wu-Yang-like estimate within each interval . Specifically , our estimator is based on Chebyshev polynomials ( as in Valiant & Valiant ( 2011 ) ; Wu & Yang ( 2019 ) ) , but the finer partitioning into intervals allows us to use polynomials with different , carefully chosen parameters . This leads to significantly improved sample complexity if the predictor is sufficiently accurate . On the empirical side , we evaluate the proposed algorithms on a collection of real and synthetic data sets . For the real data sets ( network traffic data and AOL query log data ) we use neural-network based predictors from Hsu et al . ( 2019 ) . Although those predictors do not always approximate the true distribution probabilities up to a small factor , our experiments nevertheless demonstrate that the new algorithm offers substantial improvements ( up to 3x reduction in relative error ) in the estimation accuracy compared to the state of the art algorithm of Wu & Yang ( 2019 ) . 1.1 RELATED WORK . Estimating support size As described in the introduction , the problem has been studied extensively in statistics and theoretical computer science . The best known algorithm , due to Wu & Yang 2This constraint is naturally satisfied e.g. , if the distribution P is an empirical distribution over a data set of n items . In fact , in this case all probabilities are multiples of 1/n so the support size is equal to the number of distinct elements in the data set . 3Our results hold without change if we modify the assumption to r ·Π ( i ) ≤ pi ≤ r · b ·Π ( i ) , for any r > 0 . We use r = 1 for simplicity . ( 2019 ) , uses O ( log2 ( 1/ε ) · n/ log n ) samples . Because of the inherent limitations of the model that uses only random samples , Canonne & Rubinfeld ( 2014 ) considered an augmented model where an algorithm has access to the exact probability of any sampled item . The authors show that this augmentation is very powerful , reducing the sampling complexity to only O ( 1/ε2 ) . More recently , Onak & Sun ( 2018 ) proved that Canonne and Rubinfeld ’ s algorithm works as long as the probabilities accessed are accurate up to a ( 1 ± ε3 ) -multiplicative factor . However , this algorithm strongly relies on the probabilities being extremely accurate , and the predicted probabilities even being off by a small constant factor can cause the support size estimate to become massively incorrect . As a result , their algorithm is not robust to mispredicted probabilities , as our experiments show . A different line of research studied streaming algorithms for estimating the number of distinct elements . Such algorithms have access to the whole data set , but must read it in a single pass using limited memory . The best known algorithms for this problem compute a ( 1 + ε ) -approximate estimation to the number of distinct elements using O ( 1/ε2 + log n ) bits of storage ( Kane et al. , 2010 ) . See the discussion in that paper for a history of the problem and further references . Learning-based algorithms Over the last few years , there has been a growing interest in using machine learning techniques to improve the performance of “ classical ” algorithms . This methodology found applications in similarity search ( Wang et al. , 2016 ; Sablayrolles et al. , 2019 ; Dong et al. , 2020 ) , graph optimization ( Khalil et al. , 2017 ; Balcan et al. , 2018 ) , data structures ( Kraska et al. , 2018 ; Mitzenmacher , 2018 ) , online algorithms ( Lykouris & Vassilvitskii , 2018 ; Purohit et al. , 2018 ) , compressed sensing ( Mousavi et al. , 2015 ; Baldassarre et al. , 2016 ; Bora et al. , 2017 ) and streaming algorithms ( Hsu et al. , 2019 ; Jiang et al. , 2019 ) . The last two papers are closest to our work , as they solve various computational problems over data streams , including distinct elements estimation in Jiang et al . ( 2019 ) using frequency predictors . Furthermore , in our experiments we are using the neural-network-based predictors developed in Hsu et al . ( 2019 ) . However , our algorithm operates in a fundamentally different model , using a sublinear ( in n ) number of samples of the input , as opposed to accessing the full input via a linear scan . Thus , our algorithms run in sublinear time , in contrast to streaming algorithms that use sublinear space . Distribution property testing This work can be seen more broadly in the context of testing properties of distributions over large discrete domains . Such questions are studied at the crossroads of social networks , statistics , information theory , database algorithms , and machine learning algorithms . Examples of specific properties that have been extensively considered include testing whether the distribution is uniform , Gaussian , high entropy , independent or monotone increasing ( see e.g . Rubinfeld ( 2012 ) ; Canonne ( 2015 ) ; Goldreich ( 2017 ) for surveys on the topic ) . 2 LEARNING-BASED ALGORITHM . 2.1 PRELIMINARIES . Problem setting and notation . The support estimation problem is formally defined as follows . We are given sample access to an unknown distribution P over a discrete domain of size n. For simplicity , we identify the domain with [ n ] = { 1 , . . . , n } . Let pi denote the probability of element i . Let S ( P ) = { i : pi > 0 } be the support of P . Our goal is to estimate the support size S = |S ( P ) | using as few samples as possible . In particular , given ε > 0 , the goal is to output an estimate S̃ that satisfies S̃ ∈ [ S − εn , S + εn ] . We assume that the minimal non-zero mass of any element is at least 1/n , namely , that pi ≥ 1/n for every i ∈ S ( P ) . This is a standard promise in the support estimation problem ( see , e.g. , Raskhodnikova et al . ( 2009 ) ; Valiant & Valiant ( 2011 ) ; Wu & Yang ( 2019 ) ) , and as mentioned earlier , it naturally holds in the context of counting distinct elements , where pi is defined as the count of element i in the sample divided by n. Furthermore , a lower bound on the minimum non-zero probability is a necessary assumption without which no estimation algorithm is possible , even if the number of samples is allowed to be an arbitrarily large function of n , i.e. , not just sublinear algorithms . The reason is that there could be arbitrarily many elements with exceedingly small probabilities that would never be observed . See for example the discussion in the supplementary Section 5 of Orlitsky et al . ( 2016 ) . In the learning-based setting , we furthermore assume we have a predictor Π that can provide information about pi . In our analysis , we will assume that Π ( i ) is a constant factor approximation of each pi . In order to bound the running time of our algorithms , we assume we are given access to a ready-made predictor and ( as in Canonne & Rubinfeld ( 2014 ) ) that evaluating Π ( i ) takes unit time . In our experiments , we use neural network based predictors from Hsu et al . ( 2019 ) . In general , predictors need to be trained ( or otherwise produced ) in advance . This happens in a preprocessing stage , before the input distribution is given to the algorithm , and this stage is not accounted for in the sublinear running time . We also note that training the predictor needs to be done only once for all future inputs ( not once per input ) . The Wu & Yang ( 2019 ) estimator . In the classical setting ( without access to a predictor ) , Wu & Yang ( 2019 ) gave a sample-optimal algorithm based on Chebyshev polynomials . We now describe it briefly , as it forms the basis for our learning-based algorithm . Suppose we draw N samples , and let Ni be the number of times element i is observed . The output estimate of Wu & Yang ( 2019 ) is of the form S̃WY = ∑ i∈ [ n ] ( 1 + f ( Ni ) ) , where f ( Ni ) is a correction term intended to compensate for the fact that some elements in the support do not appear in the sample at all . If pi = 0 , then necessarily Ni = 0 ( as i can not appear in the sample ) . Thus , choosing f ( 0 ) = −1 ensures that unsupported elements contribute nothing to S̃WY . On the other hand , if pi > lognN , then by standard concentration we have Ni > Ω ( log n ) with high probability ; thus choosing f ( Ni ) = 0 for all Ni > L = Ω ( log n ) ensures that high-mass elements are only counted once in S̃WY . It remains to take care of elements i with pi ∈ [ 1n , logn N ] . By a standard Poissonization trick , the expected additive error |S − E [ S̃WY ] | can be bounded by∑ i∈ [ n ] |PL ( pi ) | , where PL is the degree-L polynomial PL ( x ) = L∑ k=0 E [ N ] k k ! · f ( k ) · xk . To make the error as small as possible , we would like to choose f ( 1 ) , . . . , f ( L ) so as to minimize |PL ( pi ) | on the interval pi ∈ [ 1n , logn N ] , under the constraint PL ( 0 ) = −1 ( which is equivalent to f ( 0 ) = −1 ) . This is a well-known extremal problem , and its solution is given by Chebyshev polynomials , whose coefficients have a known explicit formula . Indeed , Wu & Yang ( 2019 ) show that choosing f ( 1 ) , . . . , f ( L ) such that E [ N ] k k ! f ( k ) are the coefficients of an ( appropriately shifted and scaled ) Chebyshev polynomial leads to an optimal sample complexity ofO ( log2 ( 1/ε ) ·n/ log n ) .

Summary: Estimation of the size of the support of a distribution over a discreet domain is a fundamental problem. In the standard setting, this problem is theoretically well-understood with matching upper and lower bounds. The authors assume additional access to a constant approximation of the density function at each point, and then show that this can provably reduces the sample complexity. In particular, they offer matching upper and lower bounds in this setting. While the upper bound is a twist on the existing state-of-the-art method, the lower bound seem to deviate from that.