Coresets for Clustering with Missing Values

1 Introduction . We consider coresets and approximation algorithms for k-clustering problems , particularly k-MEANS1 and more generally ( k , z ) -CLUSTERING ( see Definition 2.1 ) , for points in Rd with missing values ( coordinates ) . The presence of missing values in data sets is a common phenomenon , and dealing with it is a fundamental challenge in data science . While data imputation is a very popular method for handling missing values , it often requires prior knowledge which might not be available , or statistical assumptions on the missing values that might be difficult to verify [ All01 , LR19 ] . In contrast , our worst-case approach does not requires any prior knowledge . Specifically , in our context of clustering , the distance dist ( x , c ) between a clustering center point c and a data point x is evaluated only on the available ( i.e. , non-missing ) coordinates . Similar models that aim to minimize clustering costs using only the available coordinates have been proposed in previous work [ HB01 , Wag04 , CCB16 , WLH+19 ] , and some other relevant works were discussed in a survey [ HC10 ] . Clustering under this distance function , which is evaluated only on the available coordinates , is a formidable computational challenge , because distances do not satisfy the triangle inequality , and 1In the usual k-MEANS problem ( without missing coordinates ) , the input is a data set X ⊂ Rd and the goal is to find a center set C ⊂ Rd , |C| = k that minimizes the sum of squared distances from every x ∈ X to C. 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . therefore many classical and effective clustering algorithms , such as k-MEANS++ [ AV07 ] , can not be readily applied or even be defined properly . Despite the algorithmic interest in clustering with missing values , the problem is still not well understood and only a few results are known . In a pioneering work , Gao , Langberg and Schulman [ GLS08 ] initiated the algorithmic study of the k-CENTER problem with missing values . They took a geometric perspective and interpreted the k-CENTER with missing values problem as an affine-subspace clustering problem , and followup work [ GLS10 , LS13 ] has subsequently improved and generalized their algorithm . Only very recently , approximation algorithms for objectives other than k-CENTER , particularly k-MEANS , were obtained for the limited case of at most one missing coordinate in each input point [ MF19 ] or for constant number of missing coordinates [ EFG+21 ] . We focus on designing coresets for clustering with missing values . Roughly speaking , an -coreset is a small proxy of the data set , such that the clustering objective is preserved within ( 1± ) factor for all center sets ( see Definition 2.2 for formal definition ) . Efficient constructions of small -coresets usually lead to efficient approximations schemes , since the input size is reduced to that of the coreset , see e.g . [ HJLW18 , FRS19 , MF19 ] . Moreover , apart from speeding up approximation algorithms in the classical setting ( offline computation ) , coresets can also be applied to design streaming [ HM04 , FS05 , BFL+17 ] , distributed [ BEL13 , RPS15 , BLK18 ] , and dynamic algorithms [ Cha09 , HK20 ] , which are effective methods/models for dealing with big data , and recently coresets were used even in neural networks [ MOB+20 ] . 1.1 Our Results . Coresets . Our main result , stated in Theorem 1.1 , is a near-linear time construction of coresets for k-MEANS with missing values . Here , an -coreset for k-MEANS for a data set X in Rd with missing coordinates is a weighted subset S ⊆ X with weights w : S → R+ , such that ∀C ⊂ Rd , |C| = k , ∑ x∈S w ( x ) · dist2 ( x , C ) ∈ ( 1± ) ∑ x∈X dist2 ( x , C ) , where dist ( x , c ) : = √∑ i : xi not missing ( xi − ci ) 2 , and dist ( x , C ) : = minc∈C dist ( x , c ) ; note that the center set C does not contain missing values . More generally , our coreset also works for ( k , z ) CLUSTERING , which includes k-MEDIAN ( see Definition 2.1 and Definition 2.2 ) . Throughout , we use Õ ( f ) to denote O ( f poly log f ) . Theorem 1.1 ( Informal version of Theorem 3.1 ) . There is an algorithm that , given 0 < < 1/2 , integers d , j , k ≥ 1 , and a set X ⊂ Rd of n points each having at most j missing values , it constructs with constant probability an -coreset for k-MEANS onX of sizem = ( jk ) O ( min { j , k } ) · ( −1d log n ) 2 , and runs in time Õ ( ( jk ) O ( min { j , k } ) · nd+m ) . Our coreset size is only a low-degree polynomial of d , and log n , and can thus deal with moderatelyhigh dimension or large data set . The dependence on k ( number of clusters ) and j ( maximum number of missing values per point ) is also a low-degree polynomial as long as at least one of k and j is small . Actually , we justify in Theorem 1.2 that this exponential dependence in min { j , k } can not be further improved , as long as the coreset size is in a similar parameter regime , i.e. , the coreset size is of the form f ( j , k ) · poly ( −1d log n ) . We provide the proof of Theorem 1.2 in the full version . Theorem 1.2 . Consider the k-MEANS with missing values problem in Rd ? where each point can have at most j missing coordinates . Assume there is an algorithm that constructs an -coreset of size f ( j , k ) · poly ( −1d log n ) , then f ( j , k ) can not be as small as 2o ( min ( j , k ) ) . Furthermore , the space complexity of our construction algorithm is near-linear , and since our coreset is clearly mergeable , it is possible to apply the merge-and-reduce method [ HM04 ] to convert our construction into a streaming algorithm of space poly log n. Prior to our result , the only known coreset construction for clustering with missing values is for the special case j = 1 [ MF19 ] 2 and has size kO ( k ) · ( −2d log n ) . Since our coreset has size poly ( k −1d log n ) when j = 1 , it improves the dependence on k over that of [ MF19 ] by a factor of kO ( k ) . 2In fact , [ MF19 ] considers a slightly more general setting where the input are arbitrary lines that are not necessarily axis-parallel . Near-linear time PTAS for k-MEANS with missing values . Very recently , a PTAS for kMEANS with missing values , was obtained by Eiben , Fomin , Golovach , Lochet , Panolan , and Simonov [ EFG+21 ] . Its time bound is quadratic , namely O ( 2poly ( jk/ ) · n2d ) , and since our coreset can be constructed in near-linear time , we can speedup this PTAS to near-linear time by first constructing our coreset and then running this PTAS on the coreset . Corollary 1.3 ( Near-linear time PTAS for k-MEANS with missing values ) . There is an algorithm that , given 0 < < 1/2 , integers d , j , k ≥ 1 , and a set X ⊂ Rd of n points each having at most j missing values , it finds with constant probability a ( 1 + ) -approximation for k-MEANS on X , and runs in time Õ ( ( jk ) O ( min { j , k } ) · nd+ 2poly ( jk/ ) · dO ( 1 ) ) . Experiments . We implement our algorithm and validate its performance on various real and synthetic data sets in Section 4 . Our coreset exhibits flexible tradeoffs between coreset size and accuracy , and generally outperforms a uniform-sampling baseline and a baseline that is based on imputation , in both error rate and stability , especially when the coreset size is relatively small . In particular , on each data set , a coreset of moderate size 2000 ( which is 0.5 % -5 % of the data sets ) achieves low empirical error ( 5 % -20 % ) . We further demonstrate an application and use our coresets to accelerate a Lloyd ’ s-style heuristic adapted to the missing-values setting . The experiments suggest that running the heuristic on top of our coresets gives equally good solutions ( error < 1 % relative to running on the original data set ) but is much faster ( speedup > 5x ) . 1.2 Technical Overview . Our coreset construction is based on the importance sampling framework introduced by Feldman and Langberg [ FL11 ] and subsequently improved and generalized by [ FSS20 , BJKW21 ] . In the framework , one first computes an importance score σx for every data point x ∈ X , and then draws independent samples with probabilities proportional to these scores . When no values are missing , the importance scores can be computed easily , even for general metric spaces [ VX12b , FSS20 , BJKW21 ] . However , a significant challenge with missing values is that distances do not satisfy the triangle inequality , hence importance scores can not be easily computed . We overcome this hurdle using a method introduced by Varadarajan and Xiao [ VX12a ] for projective clustering ( where the triangle inequality similarly does not hold ) . They reduce the importancescore computation to the construction of a coreset for k-CENTER objective ; this method is quite different from earlier approaches , e.g . [ FL11 , VX12b , FSS20 , BJKW21 ] , and yields a coreset for k-MEANS whose size depends linearly on log n and of course on the size of the k-CENTER coreset . ( Mathematically , this arises from the sum of all importance scores . ) We make use of this reduction , and thus focus on constructing ( efficiently ) a small coreset for k-CENTER with missing values . An immediate difficulty is how to deal with the missing values . We show that it is possible to find a collection of subsets of coordinates I ( so each I ∈ I is a subset of [ d ] ) , such that if we construct k-CENTER coresets SI on the data set “ restricted ” to each I ∈ I , then the union of these SI ’ s is a k-CENTER coreset for the original data set with missing values . Crucially , we ensure that each “ restricted ” data set does not contain any missing value , so that it is possible to use a classical coreset construction for k-CENTER . Finally , we show in a technical lemma how to find a collection as necessary of size |I| ≤ ( jk ) O ( min { j , k } ) . Since a “ restricted ” data set does not contain any missing values , we can use a classical k-CENTER coreset construction , and a standard construction has size O ( k −d ) [ AP02 ] , which is known to be tight . We bypass this −d limitation by observing that actually Õ ( 1 ) -coreset for k-CENTER suffices , even though the final coreset error is . We observe that an Õ ( 1 ) -coreset can be constructed using a variant of Gonzalez ’ s algorithm [ Gon85 ] . To implement Gonzalez ’ s algorithm , a key step is to find the furthest neighbor of a given subset of at most O ( k ) points , and a naive implementation of this runs in linear time , which overall yields a quadratic-time coreset construction , because the aforementioned reduction of [ VX12a ] actually requires Θ ( n/k ) successive runs of Gonzalez ’ s algorithm . To resolve this issue , we propose a fully-dynamic implementation of Gonzalez ’ s algorithm so that a furthest-point query is answered in time poly ( k log n ) , and the point-set is updated between successive runs instead of constructed from scratch . Our dynamic algorithm is based on a random-projection method that was proposed for furthest-point queries in the streaming setting [ Ind03 ] . Specifically , we project the ( restricted ) data set onto several random directions , and on each projected ( one-dimensional ) data set we apply a data structure for intervals .

Summary: The paper provides the first coreset for clustering tasks, where each of the input d-dimensional points might contain up to j missing values. The coreset construction time is near-linear. The main technical result of the paper is a reduction from the above problem, to the problem of computing a dynamic coreset for the k-center problem with missing values. The authors utilize the sensitivity sampling framework for this task. Using this coreset, the authors improve the running time of an existing PTAS for the problem, from quadratic, to near-linear time. The authors provide experimental results which demonstrate the effectiveness of the coreset in practice, as compared to a uniform random sample.