Improved Coresets and Sublinear Algorithms for Power Means in Euclidean Spaces

We consider these problem in the big data regime . Here , we are interested in sampling as few points as possible such that we can accurately estimate m. More specifically , we consider sublinear algorithms as well as coresets for these problems . Sublinear algorithms have a random query access to the set A and the goal is to minimize the number of queries . Here , we show that Õ ( ε−z−3 ) samples are sufficient to achieve a ( 1+ε ) -approximation , generalizing the results from Cohen , Lee , Miller , Pachocki , and Sidford [ STOC ’ 16 ] and Inaba , Katoh , and Imai [ SoCG ’ 94 ] to arbitrary z . Moreover , we show that this bound is nearly optimal , as any algorithm requires at least Ω ( ε−z+1 ) queries to achieve said approximation . The second contribution are coresets for these problems , where we aim to find find a small , weighted subset of the points which approximates cost of every candidate point c ∈ Rd up to a ( 1 ± ε ) factor . Here , we show that Õ ( ε−2 ) points are sufficient , improving on the Õ ( dε−2 ) bound by Feldman and Langberg [ STOC ’ 11 ] and the Õ ( ε−4 ) bound by Braverman , Jiang , Krauthgamer , and Wu [ SODA 21 ] . 1 Introduction . Large data sets have shifted the focus of algorithm design . In the past , an algorithm might have been deemed feasible if its running time was polynomial in the input size and so a textbook fast algorithm can have time complexity for example quadratic . For truly gargantuan data sets , even linear time or nearly linear time algorithms could be considered too slow or requiring too much memory . This led to the emergence of the field of sublinear algorithms : How well can we solve a problem without reading the entire input ? Except for trivial problems , deterministic time sublinear algorithms do not exist . Our primary tool in designing sublinear algorithms is thus the following basic approach : • Take a uniform sample of the input . • Run an algorithm on the sample . Hence , the performance of a sublinear algorithm is often measured in terms of its query complexity , i.e . the number of samples required such that we can extract a high quality solution in the second ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . step above that generalizes to the entire input . Sublinear algorithms have close ties to questions in learning theory and estimation theory , where we are similarly interested in a quality to sample size tradeoff . A perhaps very fundamental problem of primary importance in machine learning and data analysis is to efficiently estimate the parameters of a distribution . For example , given a distribution D , how many samples do we need to estimate the mean ? Even such a simple and basic question has surprisingly involved answers and are still subject to ongoing research ( Lugosi & Mendelson ( 2019 ) ; Lee & Valiant ( 2021 ) ) . In this paper , we investigate the possibility of estimating power means in high dimensional Euclidean spaces . Specifically , given an arbitrary set of points A , we wish to determine the number of uniform queries S such that we can extract a power mean m with cost ( m ) : = ∑ p∈A ‖p−m‖z ≤ ( 1 + ε ) · min µ ∑ p∈A ‖p− µ‖z , where ‖p‖ denotes the Euclidean norm of a vector p. The power mean problem captures a number of important problems in computational geometry and multivariate statistics . For example , for z = 1 , this corresponds to the Fermat-Weber problem also known as the geometric median . For z = 2 , the problem is to determine the mean or centroid of the data set . Letting z →∞ , we have the Minimum Enclosing Ball ( MEB ) , where one needs to find the Euclidean sphere of smallest radius containing all input points . For z > 2 , the problem is not as well studied , but it still has many applications . First , higher powers allows us to interpolate between z = 2 and z →∞ , which is interesting as the latter admits no sublinear algorithms2 . Skewness ( a measure of the asymmetry of the probability distribution of a real-valued random variable about its mean ) and kurtosis ( a measure of the ” tailedness ” of the probability distribution ) are the centralized moments with respect to the three and the four norms and are frequently used in statistics . The power mean is a way of estimating these values for multivariate distributions . Another application is when dealing with non-Euclidean distances , such as the Hamming metric , coreset constructions for powers of z can be reduced to coreset constructions for powers 2z . So for example if we want the mean in Hamming space , we can reduce it to the z = 4 case /in squared Euclidean spaces Huang & Vishnoi ( 2020 ) . These problems are convex and thus can be approximated in the near-linear time efficiently via convex optimization techniques . However , aside from the mean ( z = 2 ) , doing so in a sublinear setting is challenging and to the best of our knowledge , only the mean and the geometric median ( z = 1 ) are currently known to admit nearly linear time algorithms . Our main result is : Theorem 1 . There exists an algorithm that , with query complexity O ( ε−z−3 · polylog ( ε−1 ) log2 1/δ ) , computes a ( 1 + ε ) approximate solution to the high dimensional power mean problem with probability at least 1− δ . A key component in designing this algorithm is a novel analysis for coresets for these problems . Coresets are succinct summaries that approximate cost for any center solution c. Theorem 2 . For any set of points in d-dimensional Euclidean space , there exists an ε-coreset for the high dimensional power mean problem of size Õ ( ε−2 · 2O ( z ) ) . With the exception of the mean , previous coresets for these problems achieved boundsO ( ε−2 ·2O ( z ) · min ( d , ε−2 ) ) ( Cohen-Addad et al . ( 2021 ) ; Braverman et al . ( 2021 ) ; Feldman & Langberg ( 2011 ) ) , or had weaker guarantees such as merely approximating an optimal solution or requiring removal of outliers from the data set . Comparing the bounds in Theorem 1 and Theorem 2 , one may question whether the exponential dependency in z is necessary for computing an approximation . Indeed , previous sublinear algorithms 2To see this , we place n− 1 points at 0 and one point with probability 1/2 at 1 and with probability 1/2 at 0 . Any 2 − ε approximation can distinguish between the two cases , but this clearly requires us to query Ω ( n ) points . for both the geometric median and the mean had a query complexity of O ( ε−2 ) , and thus matched our coresets bounds . Unfortunately , we show that the exponential dependency in the power is indeed necessary even in a single dimension : Theorem 3 . For any ε > 0 and z , any algorithm that computes with probability more than 4/5 a ( 1 + ε ) -approximation for a one-dimensional power mean has query complexity Ω ( ε−z+1 ) . Hence , up to constants in the exponent , our sublinear algorithm is tight . Moreover , the algorithm is very simple to implement and performs well empircally . 1.1 Techniques . While stochastic gradient descent has been used for a variety of center-based problems ( Clarkson et al . ( 2012 ) ; Cohen et al . ( 2016 ) ) , it is difficult to apply it for higher powers . Indeed , Cohen et al . ( 2016 ) remark in their paper that even for the mean3 ( z = 2 ) their analysis does not work as the objective function is neither Lipschitz , nor strictly convex . Hence , one needs to use new tools . A natural starting point is to use techniques from coresets , as they allow us to preserve most of the relevant information , using a substantially smaller number of points . Unfortunately , coresets have a drawback : the sampling distributions used to construct coresets is non-uniform and therefore difficult to use in a sublinear setting . The first step is therefore to design coresets from uniform sampling . To do this , we use and improve upon a technique originally introduced by Chen ( 2009 ) . Chen showed that , given a sufficiently good initial solution q , one can partition the points into rings exponentially increasing radii such that the points cost the same , up to constant factors . Thereafter , taking a uniform sample of size Õ ( d · ε−2 ) from each ring produces a coreset . Since there are at most O ( log n ) rings in the worst case , this yields a coreset of size Õ ( d · ε−2 · log n ) . To realize these ideas in a sublinear setting , we are now faced with a number of challenges . First , rings that are particularly far from q may contain very few points . This makes is difficult for a sublinear algorithm to access them . Second , partitioning the points into rings depends on the cost of q . It is very simple to construct examples where estimating the cost of an optimal power center requires Ω ( n ) many queries . Finally , this analysis loses both factors log n and d , which we aim to avoid . We improve and extend this framework in two ways . First , we show that it is sufficient to only consider O ( log ε−1 ) many rings , which , in of itself , already removes the dependency on log n. Moreover , we show that it is possible to simply ignore any ring containing too few points , i.e . any ring with less than εz+O ( 1 ) · n points may be discarded . The intuition is that , while rings with few point may contribute significantly to the cost , these points do not influence the position of the optimal center by much . Thus , using a number of carefully chosen pruning steps , we show how to reduce both the problem of obtaining a sublinear algorithm as well as obtaining coresets to sampling from a select few rings containing many points . The second improvement directly considers an improved analysis of sampling from rings . The standard way to prove a coreset guarantee is to show that using s log 1/δ samples we preserve the cost of a single solution with probability 1 − delta . If there exist T solutions then we set δ = 1/T and have obtained a coreset , which we call the ” naive ” union bound . This works in certain cases such as finite metrics , but is insufficient if we have infinitely many solutions such as Euclidean spaces . The simplest way to improve over the naive union bound is to discretize the space and then apply the naive union bound on the discritization . In literature this is sometimes called an ε-net bound . This can be optimal or close to optimal for certain metrics , but so far these arguments have only lead to Õ ( ε−2 min ( d , ε−2 ) ) bounds for coresets in Euclidean spaces . Instead of applying a union bound over the discretization in ” one shot ” , we apply a union bounded over a nested sequence of increasingly better discretizations . Essentially , instead of only using a set of centers Cε as a substitute for all centers is Rd , we use centers ch ∈ Ch for different values of h ∈ { 1 , . . . , log 1/ε } . In literature , this is known as chaining , see Nelson ( 2016 ) for a survey . We can then write , for any input point set P , ∑ p∈P cost ( p , c ) = ∑ p∈P ∑ h≥0 cost ( p , c h+1 ) − cost ( p , ch ) , where cost ( p , c0 ) is defined to be 0 and |cost ( p , ch ) − cost ( p , c ) | ≤ 2−hcost ( p , c ) . We now only 3For the special case of the mean , Inaba et al . Inaba et al . ( 1994 ) observed that O ( ε−2 ) samples are nevertheless sufficient . apply the naive union bound for successive summands , i.e . we approximate ∑ p∈P cost ( p , c h+1 ) − cost ( p , ch ) up to an error of ε · ∑ p∈P cost ( p , c ) . Essentially , the idea is to use a sequence of solutions ch that approximate a candidate solution c with increasing accuracy as h → ∞ . Approximating the cost of a candidate center c can then be written as a telescoping sum of solutions ch , i.e . cost ( c ) = ∑∞ h=0 cost ( ch+1 ) − cost ( ch ) , where ch is a solution that has distance to c of the order 2−h and cost ( c0 ) : = 0 . The notion of distance between candidate solutions is perhaps most easily understood by imagining a solution to be a cost vector vc where the ith entry corresponds to the cost of the ith point of A when using c as a candidate center , i.e. , vci = cost ( pi , c ) . Informally , we consider two solutions c and c ′ to have distance at most 2−h if |‖pi − c‖z − ‖pi − c′‖z| ≤ 2−h · ( ‖pi − c‖z + ‖pi − c′‖z ) for all points pi ∈ A .

Summary: Given n points in R^d, the goal is to compute a center x in R^d that minimizes the sum (dist(p,x))^z of distance to the power of z over every input point p. The main (sub-linear time) algorithm uniformly sample ~1/eps^z points and use smart processing to obtain a (1+epsilon) approximation to the desired center x. Nice application is differential privacy. Lower bounds are also provided. More generally, the authors suggest such coreset that approximates every query center.