Coresets for Clustering with Missing Values

1 Introduction . We consider coresets and approximation algorithms for k-clustering problems , particularly k-MEANS1 and more generally ( k , z ) -CLUSTERING ( see Definition 2.1 ) , for points in Rd with missing values ( coordinates ) . The presence of missing values in data sets is a common phenomenon , and dealing with it is a fundamental challenge in data science . While data imputation is a very popular method for handling missing values , it often requires prior knowledge which might not be available , or statistical assumptions on the missing values that might be difficult to verify [ All01 , LR19 ] . In contrast , our worst-case approach does not requires any prior knowledge . Specifically , in our context of clustering , the distance dist ( x , c ) between a clustering center point c and a data point x is evaluated only on the available ( i.e. , non-missing ) coordinates . Similar models that aim to minimize clustering costs using only the available coordinates have been proposed in previous work [ HB01 , Wag04 , CCB16 , WLH+19 ] , and some other relevant works were discussed in a survey [ HC10 ] . Clustering under this distance function , which is evaluated only on the available coordinates , is a formidable computational challenge , because distances do not satisfy the triangle inequality , and 1In the usual k-MEANS problem ( without missing coordinates ) , the input is a data set X ⊂ Rd and the goal is to find a center set C ⊂ Rd , |C| = k that minimizes the sum of squared distances from every x ∈ X to C. 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . therefore many classical and effective clustering algorithms , such as k-MEANS++ [ AV07 ] , can not be readily applied or even be defined properly . Despite the algorithmic interest in clustering with missing values , the problem is still not well understood and only a few results are known . In a pioneering work , Gao , Langberg and Schulman [ GLS08 ] initiated the algorithmic study of the k-CENTER problem with missing values . They took a geometric perspective and interpreted the k-CENTER with missing values problem as an affine-subspace clustering problem , and followup work [ GLS10 , LS13 ] has subsequently improved and generalized their algorithm . Only very recently , approximation algorithms for objectives other than k-CENTER , particularly k-MEANS , were obtained for the limited case of at most one missing coordinate in each input point [ MF19 ] or for constant number of missing coordinates [ EFG+21 ] . We focus on designing coresets for clustering with missing values . Roughly speaking , an -coreset is a small proxy of the data set , such that the clustering objective is preserved within ( 1± ) factor for all center sets ( see Definition 2.2 for formal definition ) . Efficient constructions of small -coresets usually lead to efficient approximations schemes , since the input size is reduced to that of the coreset , see e.g . [ HJLW18 , FRS19 , MF19 ] . Moreover , apart from speeding up approximation algorithms in the classical setting ( offline computation ) , coresets can also be applied to design streaming [ HM04 , FS05 , BFL+17 ] , distributed [ BEL13 , RPS15 , BLK18 ] , and dynamic algorithms [ Cha09 , HK20 ] , which are effective methods/models for dealing with big data , and recently coresets were used even in neural networks [ MOB+20 ] . 1.1 Our Results . Coresets . Our main result , stated in Theorem 1.1 , is a near-linear time construction of coresets for k-MEANS with missing values . Here , an -coreset for k-MEANS for a data set X in Rd with missing coordinates is a weighted subset S ⊆ X with weights w : S → R+ , such that ∀C ⊂ Rd , |C| = k , ∑ x∈S w ( x ) · dist2 ( x , C ) ∈ ( 1± ) ∑ x∈X dist2 ( x , C ) , where dist ( x , c ) : = √∑ i : xi not missing ( xi − ci ) 2 , and dist ( x , C ) : = minc∈C dist ( x , c ) ; note that the center set C does not contain missing values . More generally , our coreset also works for ( k , z ) CLUSTERING , which includes k-MEDIAN ( see Definition 2.1 and Definition 2.2 ) . Throughout , we use Õ ( f ) to denote O ( f poly log f ) . Theorem 1.1 ( Informal version of Theorem 3.1 ) . There is an algorithm that , given 0 < < 1/2 , integers d , j , k ≥ 1 , and a set X ⊂ Rd of n points each having at most j missing values , it constructs with constant probability an -coreset for k-MEANS onX of sizem = ( jk ) O ( min { j , k } ) · ( −1d log n ) 2 , and runs in time Õ ( ( jk ) O ( min { j , k } ) · nd+m ) . Our coreset size is only a low-degree polynomial of d , and log n , and can thus deal with moderatelyhigh dimension or large data set . The dependence on k ( number of clusters ) and j ( maximum number of missing values per point ) is also a low-degree polynomial as long as at least one of k and j is small . Actually , we justify in Theorem 1.2 that this exponential dependence in min { j , k } can not be further improved , as long as the coreset size is in a similar parameter regime , i.e. , the coreset size is of the form f ( j , k ) · poly ( −1d log n ) . We provide the proof of Theorem 1.2 in the full version . Theorem 1.2 . Consider the k-MEANS with missing values problem in Rd ? where each point can have at most j missing coordinates . Assume there is an algorithm that constructs an -coreset of size f ( j , k ) · poly ( −1d log n ) , then f ( j , k ) can not be as small as 2o ( min ( j , k ) ) . Furthermore , the space complexity of our construction algorithm is near-linear , and since our coreset is clearly mergeable , it is possible to apply the merge-and-reduce method [ HM04 ] to convert our construction into a streaming algorithm of space poly log n. Prior to our result , the only known coreset construction for clustering with missing values is for the special case j = 1 [ MF19 ] 2 and has size kO ( k ) · ( −2d log n ) . Since our coreset has size poly ( k −1d log n ) when j = 1 , it improves the dependence on k over that of [ MF19 ] by a factor of kO ( k ) . 2In fact , [ MF19 ] considers a slightly more general setting where the input are arbitrary lines that are not necessarily axis-parallel . Near-linear time PTAS for k-MEANS with missing values . Very recently , a PTAS for kMEANS with missing values , was obtained by Eiben , Fomin , Golovach , Lochet , Panolan , and Simonov [ EFG+21 ] . Its time bound is quadratic , namely O ( 2poly ( jk/ ) · n2d ) , and since our coreset can be constructed in near-linear time , we can speedup this PTAS to near-linear time by first constructing our coreset and then running this PTAS on the coreset . Corollary 1.3 ( Near-linear time PTAS for k-MEANS with missing values ) . There is an algorithm that , given 0 < < 1/2 , integers d , j , k ≥ 1 , and a set X ⊂ Rd of n points each having at most j missing values , it finds with constant probability a ( 1 + ) -approximation for k-MEANS on X , and runs in time Õ ( ( jk ) O ( min { j , k } ) · nd+ 2poly ( jk/ ) · dO ( 1 ) ) . Experiments . We implement our algorithm and validate its performance on various real and synthetic data sets in Section 4 . Our coreset exhibits flexible tradeoffs between coreset size and accuracy , and generally outperforms a uniform-sampling baseline and a baseline that is based on imputation , in both error rate and stability , especially when the coreset size is relatively small . In particular , on each data set , a coreset of moderate size 2000 ( which is 0.5 % -5 % of the data sets ) achieves low empirical error ( 5 % -20 % ) . We further demonstrate an application and use our coresets to accelerate a Lloyd ’ s-style heuristic adapted to the missing-values setting . The experiments suggest that running the heuristic on top of our coresets gives equally good solutions ( error < 1 % relative to running on the original data set ) but is much faster ( speedup > 5x ) . 1.2 Technical Overview . Our coreset construction is based on the importance sampling framework introduced by Feldman and Langberg [ FL11 ] and subsequently improved and generalized by [ FSS20 , BJKW21 ] . In the framework , one first computes an importance score σx for every data point x ∈ X , and then draws independent samples with probabilities proportional to these scores . When no values are missing , the importance scores can be computed easily , even for general metric spaces [ VX12b , FSS20 , BJKW21 ] . However , a significant challenge with missing values is that distances do not satisfy the triangle inequality , hence importance scores can not be easily computed . We overcome this hurdle using a method introduced by Varadarajan and Xiao [ VX12a ] for projective clustering ( where the triangle inequality similarly does not hold ) . They reduce the importancescore computation to the construction of a coreset for k-CENTER objective ; this method is quite different from earlier approaches , e.g . [ FL11 , VX12b , FSS20 , BJKW21 ] , and yields a coreset for k-MEANS whose size depends linearly on log n and of course on the size of the k-CENTER coreset . ( Mathematically , this arises from the sum of all importance scores . ) We make use of this reduction , and thus focus on constructing ( efficiently ) a small coreset for k-CENTER with missing values . An immediate difficulty is how to deal with the missing values . We show that it is possible to find a collection of subsets of coordinates I ( so each I ∈ I is a subset of [ d ] ) , such that if we construct k-CENTER coresets SI on the data set “ restricted ” to each I ∈ I , then the union of these SI ’ s is a k-CENTER coreset for the original data set with missing values . Crucially , we ensure that each “ restricted ” data set does not contain any missing value , so that it is possible to use a classical coreset construction for k-CENTER . Finally , we show in a technical lemma how to find a collection as necessary of size |I| ≤ ( jk ) O ( min { j , k } ) . Since a “ restricted ” data set does not contain any missing values , we can use a classical k-CENTER coreset construction , and a standard construction has size O ( k −d ) [ AP02 ] , which is known to be tight . We bypass this −d limitation by observing that actually Õ ( 1 ) -coreset for k-CENTER suffices , even though the final coreset error is . We observe that an Õ ( 1 ) -coreset can be constructed using a variant of Gonzalez ’ s algorithm [ Gon85 ] . To implement Gonzalez ’ s algorithm , a key step is to find the furthest neighbor of a given subset of at most O ( k ) points , and a naive implementation of this runs in linear time , which overall yields a quadratic-time coreset construction , because the aforementioned reduction of [ VX12a ] actually requires Θ ( n/k ) successive runs of Gonzalez ’ s algorithm . To resolve this issue , we propose a fully-dynamic implementation of Gonzalez ’ s algorithm so that a furthest-point query is answered in time poly ( k log n ) , and the point-set is updated between successive runs instead of constructed from scratch . Our dynamic algorithm is based on a random-projection method that was proposed for furthest-point queries in the streaming setting [ Ind03 ] . Specifically , we project the ( restricted ) data set onto several random directions , and on each projected ( one-dimensional ) data set we apply a data structure for intervals .

Summary: The paper describes the first algorithms for coresets for k-means clustering with missing data coordinates for some points in d-dimensions, with provably guarantees.  The algorithm runs in linear time O(nd) times factor depending only on j (most possible missing values per coordinate) and k (number of allowed clusters).  A SODA 2021 paper provides a PTAS for this problem, but it has worse dependence on j, k, epsilon and has a base cost of O(n^2 d), so quadratic in n.  This paper demonstrates their approach is implementable, and practical.