A Constant Approximation Algorithm for Sequential Random-Order No-Substitution k-Median Clustering

1 Introduction . Clustering is a fundamental unsupervised learning task used for various applications , such as anomaly detection [ Leung and Leckie , 2005 ] , recommender systems [ Shepitsen et al. , 2008 ] and cancer diagnosis [ Zheng et al. , 2014 ] . In recent years , the problem of sequential clustering has been actively studied , motivated by applications in which data arrives sequentially , such as online recommender systems [ Nasraoui et al. , 2007 ] and online community detection [ Aggarwal , 2003 ] . In this work , we study k-median clustering in the sequential no-substitution setting , a term first introduced in Hess and Sabato [ 2020 ] . In this setting , a stream of data points is sequentially observed , and some of these points are selected by the algorithm as cluster centers . However , a point can be selected as a center only immediately after it is observed , before observing the next point . In addition , a selected center can not be substituted later . This setting is motivated by applications in which center selection is mapped to a real-world irreversible action . For instance , consider a stream of website users , where the goal is to instantaneously identify users that serve as social cluster centers and provide them with a promotional gift while they are still on the website . As another example , consider recruiting representative participants for a clinical trial out of a stream of incoming patients . ∗This work was done while the author was at the University of California San Diego . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The goal in the no-substitution k-median setting is to obtain a near-optimal k-median cost value , while selecting a number of centers that is as close as possible to k. For an adversarially ordered stream , it has been shown [ Moshkovitz , 2021 ] that an algorithm that selects a number of centers that is sublinear in the stream length can not obtain a constant approximation guarantee , unless some structural assumptions are imposed on the input data . In this work , we show that in contrast , when the stream order is random , a constant approximation guarantee can be obtained without imposing structural assumptions on the data . Previous works on the random-order setting provide either an algorithm with an almost-constant approximation assuming a bounded metric space [ Hess and Sabato , 2020 ] or an algorithm with an approximation factor that is exponential in k [ Moshkovitz , 2021 ] . Thus , our guarantees are stronger and more general than the previous guarantees . Moreover , the number of centers selected by our algorithm is only quasi-linear in k and does not depend on the stream length . Main result . We propose a new algorithm , MUNSC ( Multiscale No-Substitution Clustering ) , that obtains a constant approximation factor with probability 1− δ over the random order of the stream , while selecting only O ( k log2 ( k/δ ) ) centers . MUNSC operates by executing several instances of a new center-selection procedure , called SelectProc , where each instance is applied to a stream prefix of a different scale . SelectProc decides which centers to select using a novel technique , in which a small stream prefix is used to estimate a truncated version of the optimal cost of the entire stream . This truncated version ignores outliers , and so can be estimated reliably . The value of the estimate is used to determine which centers to select from the rest of the stream . The multiscale use of SelectProc by MUNSC allows MUNSC to select only a quasi-linear number of centers . As part of our analysis , we propose a new concept of a linear bin division , which allows a highprobability association of the costs of stream subsets , while selecting a number of centers that is independent of the stream length . This improves a previous construction of Meyerson et al . [ 2004 ] . The guarantees for MUNSC are provided in our main theorem , Theorem 4.1 , stated in Section 4 . Table 1 below compares our guarantees to previous works ; See Section 2 for additional details . We note that our algorithm and guarantees are easily extendable to k-means clustering , as well as to any other clustering objective that satisfies the weak triangle inequality . MUNSC uses as a black box an offline k-median approximation algorithm with a constant approximation factor . Whenever the offline algorithm is efficient [ e.g. , Charikar et al. , 2002 , Li and Svensson , 2016 ] , then so is MUNSC . Limitations . The proposed algorithm is the first constant-approximation no-substitution clustering algorithm that makes no structural assumptions on the input data set . The only limitation is the assumption that the input order is random . This assumption is useful in cases where there is no adversary and any order is as likely , as in the example applications mentioned above . Moreover , a random order is a standard assumption in many streaming settings , satisfied also whenever the input stream is drawn i.i.d . from a distribution . In some cases , the randomness assumption might hold only approximately . In these cases , we expect a graceful degradation of the guarantees . We leave for future work a comprehensive treatment of input orders that are neither random nor adversarial . We expect the techniques of the current work to serve as a cornerstone in such a treatment . Paper structure . We discuss related work in Section 2 . The formal setting and notations are defined in Section 3 . The algorithm and its guarantees are given in Section 4 . In Section 5 , we outline the proof of the main result . Some parts of the proof are deferred to appendices , which can be found in the supplementary material . We conclude with a discussion in Section 6 . 2 Related Work . Several works have studied settings related to the no-substitution k-median clustering setting . Table 1 summarizes the upper bounds mentioned below . First , some works studied related settings under an adversarial arrival order . Liberty et al . [ 2016 ] studied online k-means clustering , in which centers are sequentially selected , and each observed point is allocated to one of the previously selected centers . The proposed algorithm can be applied to the no-substitution setting , yielding an approximation factor of O ( log ( n ) ) , where n is the stream length . For this algorithm to select a sublinear number of centers , the aspect ratio of the input data must be bounded . Bhaskara and Rwanpathirana [ 2020 ] studied the same setting as Liberty et al . [ 2016 ] , improving the approximation factor to a constant , under the same assumptions . Bhattacharjee and Moshkovitz [ 2021 ] explicitly studied the no-substitution setting , and provided an approximation factor of O ( k3 ) , under a different assumption on the input data set . The setting of a random arrival order has been studied in several recent works . Hess and Sabato [ 2020 ] proposed an algorithm that selects exactly k centers . They obtained a constant approximation factor , with an additional additive term that vanishes for large streams , under the assumption that the metric space has a bounded diameter . Their guarantee is with high probability . Moshkovitz [ 2021 ] obtained an approximation factor that is exponential in k , with a number of centers that is linear in k and logarithmic in the stream length n. Assuming a known stream length , the same work also obtained an approximation factor which is exponential in k while selecting O ( k5 ) centers . Both guarantees hold only with a constant probability . As can be seen in Table 1 , to date , the only known constant approximation algorithms for no-substitution k-median clustering impose structural assumptions on the input data . A related sequential clustering setting is the streaming setting [ e.g. , Guha et al. , 2000 , Ailon et al. , 2009 , Chen , 2009 , Ackermann et al. , 2012 , Braverman et al. , 2016 ] , in which the main restriction is the amount of memory available to the algorithm . This setting allows substituting selected centers , but algorithms in this setting can be used in the no-substitution setting , by collecting all the centers ever selected . However , we are not aware of any algorithm in this setting with a competitive bound on the total number of selected centers . 3 Setting and Notation . For an integer i , denote [ i ] : = { 1 , . . . , i } . Let ( X , ρ ) be a finite metric space , where X is a set of size n and ρ : X × X → R+ is a metric . For a point x ∈ X and a set T ⊆ X , we denote ρ ( x , T ) : = miny∈T ρ ( x , y ) . For an integer k ≥ 2 , a k-clustering of X is a set of ( at most ) k points from X which represent cluster centers . Throughout this work , whenever an item is selected based on minimizing ρ , we assume that ties are broken based on a fixed arbitrary ordering . Given a set S ⊆ X , the k-median cost of T on S is cost ( S , T ) : = ∑ x∈S ρ ( x , T ) . The k-median clustering problem aims to select a k-clustering T of X with a minimal overall cost cost ( X , T ) . We denote by OPT an optimal solution to this problem : OPT ∈ argminT⊆X , |T |≤k cost ( X , T ) . In the sequential no-substitution k-median setting that we study , X is not known a priori . The points from X are presented to the algorithm sequentially , in a random order . We assume that the stream length , n , is provided as input to the algorithm ; see Section 6 for a discussion on supporting an unknown stream length . The algorithm may select an observed point as a center only before observing the next point . Any selected point can not later be removed or substituted . The goal of the algorithm is to select a small number of centers , such that with a high probability , the overall cost of the selected set on the entire X approximates the optimal k-median cost , cost ( X , OPT ) . In other words , we seek an algorithm with a low competitive ratio , in the sense of competitive analysis [ Borodin and El-Yaniv , 2005 ] . An offline k-median algorithm A is an algorithm that takes as input a finite set Algorithm 1 The sequential clustering algorithm : MUNSC input δ ∈ ( 0 , 1 ) , k ∈ N , n ∈ N ( stream length ) , A ( an offline k-median algorithm ) , sequential access to the input stream . 1 : α1 ← δ/ ( 4k ) ; For i ∈ N , αi ← α1 · 2i−1 . 2 : I ← blog2 ( 1/ ( 6α1 ) ) c , δ′ ← δ/ ( I + 1 ) . 4 I is set so that αI+1 ∈ ( 1/12 , 1/6 ] . 3 : Prepare I + 1 copies of SelectProc , indexed by i ∈ [ I + 1 ] , as follows : In all copies , use inputs k , n , A , τ ← φαI+1 , and set the confidence parameter to δ′ . In copy i ∈ [ I + 1 ] , set α← αi . In all but the last copy , set M ← 1 and γ ← 2αi . In the last copy ( index I + 1 ) , set M ← log ( 8k+/δ′ ) , and γ ← 1− 2αI+1 . 4 : Read each point from the input stream and feed it to each of the copies of SelectProc . If any of the copies selected the point , then select it as a center . of points S and the parameter k , and outputs a k-clustering of S , denoted A ( S , k ) . For β ≥ 1 , we say that A is a β-approximation offline k-median algorithm on ( X , ρ ) , if for all input sets S ⊆ X , cost ( S , A ( S , k ) ) ≤ β · cost ( S , OPTS ) , where OPTS is an optimal solution with centers from S. Formally , OPTS ∈ argminT⊆S , |T |≤k cost ( S , T ) .

Summary: This is a theoretical paper, studying sequentially assigning points as k-median cluster centers.   It assumes a random arrival order, but has no other structural assumptions.  It offers a constant factor approximation using O(k log^2 k) centers.  This improves on a recent line of work in this model removing structural assumptions and the approximation factor.    The main idea is using geometrically increasing prefixes of the stream to estimate the k-median solution and risk, and for each using this estimate to choose centers.  A key insight is that by increasing the size of centers maintained a bit, the risk can be approximated after excluding outliers -- which turns out to be the key step in the analysis.