Title: Near-Optimal $k$-Clustering in the Sliding Window Model
Venue: NeurIPS 2023
Accepted: yes
Keywords: clustering

streaming algorithms

sliding window model
Abstract: Clustering is an important technique for identifying structural information in large-scale data analysis, where the underlying dataset may be too large to store. In many applications, recent data can provide more accurate information and thus older data past a certain time is expired. The sliding window model captures these desired properties and thus there has been substantial interest in clustering in the sliding window model. In this paper, we give the first algorithm that achieves near-optimal $(1+\varepsilon)$-approximation to $(k,z)$-clustering in the sliding window model. Our algorithm uses $\frac{k}{\min(\varepsilon^4,\varepsilon^{2+z})}\,\text{polylog}\frac{n\Delta}{\varepsilon}$ words of space when the points are from $[\Delta]^d$, thus significantly improving on works by Braverman et. al. (SODA 2016), Borassi et. al. (NeurIPS 2021), and Epasto et. al. (SODA 2022).

Along the way, we develop a data structure for clustering called an online coreset, which outputs a coreset not only for the end of a stream, but also for all prefixes of the stream. Our online coreset samples $\frac{k}{\min(\varepsilon^4,\varepsilon^{2+z})}\,\text{polylog}\frac{n\Delta}{\varepsilon}$ points from the stream. We then show that any online coreset requires $\Omega\left(\frac{k}{\varepsilon^2}\log n\right)$ samples, which shows a separation between the problem of constructing an offline coreset, i.e., constructing online coresets is strictly harder. Our results also extend to general metrics on $[\Delta]^d$ and are near-optimal in light of a $\Omega\left(\frac{k}{\varepsilon^{2+z}}\right)$ lower bound for the size of an offline coreset.

Near-Optimal k -Clustering in the Sliding Window Model

Abstract

Clustering is an important technique for identifying structural information in largescale data analysis, where the underlying dataset may be too large to store. In many applications, recent data can provide more accurate information and thus older data past a certain time is expired. The sliding window model captures these desired properties and thus there has been substantial interest in clustering in the sliding window model.

In this paper, we give the first algorithm that achieves near-optimal (1 + ε ) -approximation to ( k, z ) -clustering in the sliding window model, where z is the exponent of the distance function in the cost. Our algorithm uses k min( ε 4 ,ε 2+ z ) polylog n ∆ ε words of space when the points are from [∆] d , thus significantly improving on works by Braverman et. al. (SODA 2016), Borassi et. al. (NeurIPS 2021), and Epasto et. al. (SODA 2022).

Along the way, we develop a data structure for clustering called an online coreset, which outputs a coreset not only for the end of a stream, but also for all prefixes of the stream. Our online coreset samples k min( ε 4 ,ε 2+ z ) polylog n ∆ ε points from the stream. We then show that any online coreset requires Ω ( k ε 2 log n ) samples, which shows a separation from the problem of constructing an offline coreset, i.e., constructing online coresets is strictly harder. Our results also extend to general metrics on [∆] d and are near-optimal in light of a Ω ( k ε 2+ z ) lower bound for the size of an offline coreset.

1 Introduction

Clustering is a fundamental procedure frequently used to help extract important structural information from large datasets. Informally, the goal of clustering is to partition the data into k clusters so that the elements within each cluster have similar properties. Classic formulations of clustering include the k -median and k -means problems, which have been studied since the 1950's [60, 50]. More generally, for a set X of n points in R d , along with a metric dist , a cluster parameter k > 0 , and an exponent z > 0 that is a positive integer, the clustering objective can be defined by

When dist is the Euclidean distance, the problem is known as ( k, z ) -clustering and more specifically, k -median clustering and k -means clustering, when z is additionally set to 1 and 2 , respectively.

As modern datasets have significantly increased in size, attention has shifted to large-scale computational models, such as the streaming model of computation, that do not require multiple passes over the data. In the (insertion-only) streaming model, the points x 1 , . . . , x n of X arrive sequentially,

and the goal is to output an optimal or near-optimal clustering of X while using space sublinear in n , ideally space k polylog( n, d ) , since outputting the cluster centers uses k words of space, where each word of space is assumed to be able to store an entire input point in R d . There exist slight variants of the insertion-only streaming model and a long line of active research has been conducted on clustering in these models [42, 21, 44, 43, 23, 17, 37, 39, 1, 11, 59, 46, 14, 27, 10, 25, 61, 29].

The sliding window model. Unfortunately, an important shortcoming of the streaming model is that it ignores the time at which a specific data point arrives and thus it is unable to prioritize recent data over older data. Consequently, the streaming model cannot capture applications in which recent data is more accurate and therefore considered more important than data that arrived prior to a certain time, e.g., Census data or financial markets. Indeed, it has been shown that for a number of applications, the streaming model has inferior performance [4, 52, 57, 62] compared to the sliding window model [33], where only the most recent W updates in the stream comprise the underlying dataset. Here, W > 0 is a parameter that designates the window size of the active data, so that all updates before the W most recent updates are considered expired, and the goal is to aggregate statistics about the active data using space sublinear in W . In the setting of clustering, where the data stream is x 1 , . . . , x n ⊂ R d , the active data set is X = { x n -W +1 , . . . , x n } for n ≥ W and X = { x 1 , . . . , x n } otherwise. Thus the sliding window model is a generalization of the streaming model, depending on the choice of W , and is especially relevant for time-sensitive settings, such as data summarization [22, 34], event detection in social media [56], and network monitoring [32, 31, 30].

The sliding window model is especially relevant for applications in which computation must be restricted to data that arrived after a certain time. Data privacy laws such as the General Data Protection Regulation (GDPR) mandate that companies cannot retain specific user data beyond a certain duration. For example, the Facebook data policy [36] states that user search histories are retained for 6 months, the Apple differential privacy overview [3] states that collected user information is retained for 3 months, and the Google data retention policy states that browser information may be stored for up to 9 months [41]. These retention polices can be modeled by the sliding window model with the corresponding setting of the window parameter W and thus the sliding window model has been subsequently studied in a wide range of applications [48, 49, 18, 19, 12, 13, 9, 20, 63, 2, 47, 7].

Clustering in the sliding window model. Because the clustering objective is not well-suited to popular frameworks such as the exponential histogram or the smooth histogram, there has been significant interest in clustering in the sliding window model. We now describe the landscape of clustering algorithms in the sliding window model; these results are summarized in Table 1. In 2003, [5] first gave a 2 O (1 /ε ) -approximation algorithm for k -median clustering in the sliding window model using O ( k ε 4 W 2 ε log 2 W ) words of space, where ε ∈ ( 0 , 1 2 ) is an input parameter. Subsequently, [15] gave an O (1) -approximate bicriteria algorithm using 2 k centers and k 2 polylog( W ) space for the k -median problem in the sliding window model. The question of whether there exists a poly( k log W ) space algorithm for k -clustering on sliding windows remained open until [16] gave constant-factor approximation sliding window algorithms for k -median and k -means using O ( k 3 log 6 W ) space and [28] gave constant-factor approximation algorithms for k -center clustering using O ( k log ∆) space, where ∆ is the aspect ratio, i.e., the ratio of the largest to smallest distances between any pair of points. Afterwards, [8] gave a C -approximation algorithm for some constant C > 2 14 , though it should be noted that their main contribution was the first constant-factor approximation algorithm for k -clustering using space linear in k , i.e., k polylog( W, ∆) space, and thus they did not attempt to optimize the constant C . Recently, [35] gave the first (1 + ε ) -approximation algorithm for ( k, z ) -clustering using ( kd + d C ) ε 3 polylog ( W, ∆ , 1 ε ) words of space, for some constant C ≥ 7 . Using known dimensionality reduction techniques, i.e., [51], the algorithm's dependence on d C can be removed in exchange for a 1 ε 14 polylog ( W, 1 ε ) overhead. However, neither the d C dependency nor the 1 ε 14 polylog ( W, 1 ε ) trade-off is desirable for realistic settings of d and ε for applications of k -clustering on sliding windows. In particular, recent results have achieved efficient summarizations, i.e., coresets, for k -median and k -means clustering in the offline setting using ˜ O ( k ε 4 log n ) words of space [27, 25] when the input is from [∆] d and it is known that this is near-optimal, i.e., Ω ( k ε 2+ z log n ) samples are necessary to form coresets for ( k, z ) -clustering [45] in that setting. Thus a natural question is to ask whether such near-optimal space bounds can be achieved in the sliding window model.

1.1 Our Contributions

In this paper, we answer the question in the affirmative. That is, we give near-optimal space algorithms for k -median and k -means clustering in the sliding window model. In fact, we give more general algorithms for ( k, z ) -clustering in the sliding window that nearly match the space used by the offline coreset constructions of [27, 25, 26]:

Theorem 1.1. There exists an algorithm that samples k min( ε 4 ,ε 2+ z ) polylog n ∆ ε points and with high probability, outputs a (1 + ε ) -approximation to ( k, z ) -clustering for the Euclidean distance on [∆] d in the sliding window model.

In particular, our bounds in Theorem 1.1 achieve k ε 4 polylog n ∆ ε words of space for k -median clustering and k -means clustering, i.e., z = 1 and z = 2 , respectively, matching the lower bounds of [25, 45] up to polylogarithmic factors.

Moreover, our algorithm actually produces a coreset, i.e., a data structure that approximately answers the clustering cost of the underlying dataset with respect to any set of k centers, not just the optimal k centers.

Theorem 1.2. There exists an algorithm that samples k min( ε 4 ,ε 2+ z ) polylog n ∆ ε points and with high probability, outputs a (1 + ε ) -coreset to ( k, z ) -clustering in the sliding window model for general metrics on [∆] d .

We emphasize that the guarantees of Theorem 1.2 are for general metrics on [∆] d , such as L p metrics. Note that in light of the properties of coresets, the guarantee of Theorem 1.1 follows from taking a coreset for ( k, z ) -clustering on Euclidean distances and then using an offline algorithm for ( k, z ) -clustering for post-processing after the data stream, i.e., see Theorem 2.4.

Along the way, we provide a construction for a (1 + ε ) -online coreset for ( k, z ) -clustering for general metrics on [∆] d . An online coreset for ( k, z ) -clustering is a data structure on a data stream that will not only approximately answer the clustering cost of the underlying dataset with respect to any set of k centers, but also approximately answer the clustering cost of any prefix of the data stream with respect to any set of k centers.

Theorem 1.3. There exists an algorithm that samples k min( ε 4 ,ε 2+ z ) polylog n ∆ ε points and with high probability, outputs a (1 + ε ) -online coreset for ( k, z ) -clustering.

We remark that Theorem 1.3 further has the attractive property that once a point is sampled into the online coreset at some point in the stream, then the point irrevocably remains in the online coreset. That is, the online coreset essentially satisfies two different definitions of online: 1) the data structure is a coreset for any prefix of the stream and 2) points sampled into the data structure will never be deleted from the data structure.

We further remark that due to leveraging the coreset construction of [27, 25, 26], we can similarly trade a factor of 1 ε z for a poly( k ) in the guarantees of Theorem 1.1, Theorem 1.2, and Theorem 1.3.

By contrast, the lower bound by [25] states that any offline coreset construction for k -means clustering only requires Ω ( k ε 2 ) points. This lower bound was later strengthened to Ω ( k ε 2+ z ) points by [45], for which matching upper bounds are given by [27, 25]. Thus our online coreset constructions are near-optimal in the k and 1 ε dependencies for z > 1 and nearly match the best known offline constructions for z = 1 .

It is thus a natural question to ask whether our polylogarithmic overheads in Theorem 1.3 are necessary for an (1 + ε ) -online coreset. We show that in fact, a logarithmic overhead is indeed necessary to maintain a (1 + ε ) -online coreset.

Theorem 1.4. Let ε ∈ (0 , 1) . For sufficiently large n , d , and ∆ , there exists a set X ⊂ [∆] d of n points x 1 , . . . , x n such that any (1 + ε ) -online coreset for k -means clustering on X requires Ω ( k ε 2 log n ) points.

We emphasize that combined with existing offline coreset constructions [25, 26], Theorem 1.4 shows a separation between the problems of constructing offline coresets and online coresets. That is, the problem of maintaining a data structure that recovers coresets for all prefixes of the stream is provably harder than maintaining a coreset for an offline set of points.

1.2 Technical Overview

In this section, we give a high-level overview of our techniques. We also describe the limitations of many natural approaches.

Shortcomings of histograms and sensitivity sampling. A first attempt at clustering in the sliding window model might be to adapt the popular exponential histogram [33] and smooth histogram techniques [18]. These frameworks convert streaming algorithms to sliding window algorithms in the case that the objective function is smooth, which informally means that once a suffix of a data stream becomes a good approximation of the overall data stream, then it always remains a good approximation, regardless of the values of new elements that arrive in the stream. Unfortunately, [16] showed that the k -clustering objective function is not smooth and thus these histogram-based frameworks cannot work. Nevertheless, they gave the first constant-factor approximation by showing that the k -clustering objective function is almost-smooth using a generalized triangle inequality, which inherently loses constant factors and thus will not suffice for our goal of achieving a (1 + ε ) -approximation.

Another approach might be to adapt the popular sensitivity sampling framework of coreset construction [37, 39, 10, 29]. The sensitivity sampling framework assigns a value to each point, called the sensitivity, which intuitively quantifies the 'importance' of that point, and then samples each point with probability proportional to its sensitivity. [9] observed that sliding window algorithms can be achieved from online sensitivity sampling, where the importance of each point is measured against the prefix of the stream, and then running the process in reverse at each time, so that more emphasis is placed on the suffix of the sliding window. At a high level, this is the intuition taken by [34, 35], which leverage data structures that prioritize more recent elements of the data stream. However, it is not known how to achieve optimal bounds simply using sensitivity sampling, and indeed the optimal coreset constructions use slightly more nuanced sampling schemes [27, 25].

Sliding window algorithms from online coresets. Instead, we recall an observation by [9], who noted that deterministic constructions for online coresets for linear algebraic problems can be utilized to obtain sliding window algorithms for the corresponding linear algebraic problems. We first extend this observation to randomized constructions for online coresets for k -clustering problem.

The intuition is quite simple. Given an (1 + ε ) -online coreset algorithm for a k -clustering problem on a data stream of length n from R d that stores S ( n, d, k, ε, δ ) weights points and succeeds with probability 1 -δ , we store the S ( n, d, k, ε ′ , δ ′ ) most recent points in the stream, where ε ′ = O ( ε log n ) and δ ′ = δ poly( n ) . Wethen feed the S ( n, d, k, ε ′ , δ ′ ) points to the online coreset construction in reverse order of their arrival . Since the online coreset preserves all costs for all prefixes of its input, then the resulting data structure will preserve all costs for all suffixes of the data stream. To extend this guarantee to the entire stream, including the sliding window, we can then use a standard merge-andreduce framework. It thus remains to devise a (1 + ε ) -online coreset construction for k -clustering with near-optimal sampling complexity.

Online coreset construction. To that end, our options are quite limited, as to the best of our knowledge, the only offline coreset constructions using ˜ O ( k ε 4 log n ) words of space when the input is from [∆] d are due to [27, 25]. Fortunately, although the analyses of correctness for these sampling

schemes are quite involved, the constructions themselves are quite accessible. For example, [27] first uses an ( α, β ) -approximation, i.e., a clustering that achieves α -approximation to the optimal cost but uses βk centers, to partition the underlying dataset X into disjoint concentric rings around each of the βk centers. These rings are then gathered into groups and it is shown that by independently sampling a fixed number of points with replacement from each of the groups suffices to achieve a (1 + ε ) -coreset. Their analysis argues that the contribution of each of the groups toward the overall k -clustering cost is preserved through an expectation and variance bounding argument, and then taking a sophisticated union bound over a net over the set of possible centers. Thus their argument still holds when each point of the dataset is independently sampled by the data structure with probability proportional to the probability it would have been sampled by the group. Moreover, independently sampling each point with a higher probability can only decrease the variance, so that correctness is retained, though we must also upper bound the number of sampled points. Crucially, independently sampling each point can be implemented in the online setting and the probability of correctness can be boosted to union bound over all times in the stream, which facilitates the construction of our (1 + ε ) -online coreset, given an ( α, β ) -approximation.

Consistent ( α, β ) -approximation. It seemingly remains to find ( α, β ) -approximations for k -clustering at all times in the stream. A natural approach would be to use an algorithm that achieves a ( α, β ) -approximation at a certain time in the stream with constant probability, e.g., [59], boost the probability of success to 1 -1 poly( n ) , and the union bound to argue correctness over all times in the stream. However, a subtle pitfall here is that the rings and groups in the offline coreset construction of [27] are with respect to a specific ( α, β ) -approximation. Hence their analysis would no longer hold if a point x t was assigned to cluster i 1 at time t when the sampling process occurs but then assigned to cluster i 2 at the end of the stream. Therefore, we require a consistent ( α, β ) -approximation, so that once the algorithm assigns point x t to cluster i , then the point x t will always remain in cluster i even if a newer and closer center is subsequently opened later in the stream. To that end, we invoke a result of [34] that analyzes the popular Meyerson online facility location algorithm, along with a standard guess-and-double approach for estimating the input parameter to the Meyerson subroutine.

Lower bound. The intuition for our lower bound that any (1+ ε ) -online coreset for ( k, z ) -clustering requires Ω ( k ε 2 ) is somewhat straightforward and in a black-box manner. We first observe that [25] showed the existence of a set X of Ω ( k ε 2 ) unit vectors in R d such that any coreset with o ( k ε 2 ) samples provably cannot accurately estimate the ( k, z ) -clustering cost for a set C of k unit vectors.

Since an online (1 + ε ) -coreset must answer queries on all prefixes of the stream, we embed Ω(log n ) instances of X . We first increase the dimension by a log n factor so that each of these instances can have disjoint support. We then give each of the instances increasingly exponential weight to force the data structure to sample Ω ( k ε 2 ) points for each instance. Specifically, we insert τ i copies of the i -th instance of X , where τ > 1 is some constant. Because the weight of the i -th instance is substantially greater than the sum of the weights of all previous instances, then any (1 + ε ) -online coreset must essentially be a (1 + ε ) -offline coreset for the i -th instance, thus requiring Ω ( k ε 2 ) points for the i -th instance. This reasoning extends to all Ω(log n ) instances, thus showing that any online (1 + ε ) -coreset requires Ω ( k ε 2 log n ) points.

2 Algorithm

In this section, we describe our sliding window algorithm for k -clustering. We first overview the construction of an online (1 + ε ) coreset for ( k, z ) -clustering under general discrete metrics. We then describe how our online coreset construction for ( k, z ) -clustering on general discrete metric spaces can be used to achieve near-optimal space algorithms for ( k, z ) -clustering in the sliding window model.

Online (1 + ε ) -coreset. We first recall the following properties from the Meyerson sketch, which we formally introduce in Appendix A.

Theorem 2.1. [8] Given an input stream x 1 , . . . , x n ∈ R d defining a set X ⊂ [∆] d , there exists an online algorithm MULTMEYERSON that with probability at least 1 -1 poly( n ) :

on the arrival of each point x i , assigns x i to a center in C through a mapping π : X → C , where C contains at most O ( 2 2 z k log n log ∆ ) centers

∑ x ∈ X ∥ x i -π ( x i ) ∥ z 2 ≤ 2 z +7 Cost | S |≤ k ( X,S )

MULTMEYERSON uses O ( 2 z k log 3 ( nd ∆) ) words of space

We also use the following notation, adapted from [27] to the online setting.

Let A be an ( α, β ) -approximation for a k -means clustering on an input set X ⊆ [∆] d and let C 1 , . . . , C βk be the clusters of X induced by A . Suppose the points of X arrive in a data stream S . For a fixed ε > 0 , define the following notions of rings and groups:

The average cost of cluster C i is denoted by κ C i := Cost( C i , A ) | C i | .

For any i, j , the ring R i,j is the set of points x ∈ C i such that 2 j κ C i ≤ Cost( x, A ) < 2 j +1 κ C i . For any j , R j = ∪ R i,j .

The inner ring R I ( C i ) = ∪ j ≤ 2 z log ε 2 R i,j is the set of points of C i with cost at most ( ε z ) 2 z κ C i . More generally for a solution S , let R S I denote the union of the inner rings induced by S .

The outer ring R O ( C i ) = ∪ j ≥ 2 z log z ε R i,j is the set of points of C i with cost at least ( z ε ) 2 z κ C i . More generally for a solution S , let R S O denote the union of the outer rings induced by S .

The main ring R M ( C i ) is the set of points of C i that are not in the inner or outer rings, i.e., R M ( C i ) = C i \ ( R I ( C i ) ∪ R O ( C i ) .

For any j , the group G j,b consists of the (2 b -1 +1) -th to (2 b ) -th points of each ring R i,j that arrive in S .

For any j , we use G j, min to denote the union of the groups with the smallest costs, i.e.,

The outer groups G O b partition the outer rings R A O so that

We define G O min = ∪ b ≤ 0 G O b and G O max = ∪ b ≥ z log 4 z ε G O b .

Algorithm 1 RINGSAMPLE

We then adapt the offline coreset construction of [27] to an online setting at the cost of logarithmic overheads, which suffice for our purpose. The algorithm (Algorithm 1) has the following guarantees:

Lemma 2.2. Let C be an A -approximate centroid set for a fixed group G . There exists an algorithm RINGSAMPLE that samples

points and with high probability, outputs a (1 + ε ) -online coreset for the k -means clustering problem.

Informally, an approximate centroid set is a set of possible points so that taking the centers from this set generates an approximately accurate solution (see Appendix B for a formal definition). To bound log | C | , we construct and apply a terminal embedding to project each point to a lower dimension and then appeal to known bounds for approximate centroid sets in low-dimensional Euclidean, thereby giving our online coreset algorithm with the guarantees of Theorem 1.3.

Sliding window model. We first recall a standard approach for using offline coreset constructions for insertion-only streaming algorithms. Suppose there exists a randomized algorithm that produces an online coreset algorithm that uses S ( n, ε, δ ) points for an input stream of length n , accuracy ε , and failure probability δ , where for the ease of discussion, we omit additional dependencies. A standard approach for using coresets on insertion-only streams is the merge-and-reduce approach, which partitions the stream into blocks of size S ( n, ε 2 log n , δ poly( n ) ) and builds a coreset for each block. Each coreset is then viewed as the leaves of a binary tree with height at most log n , since the binary tree has at most n leaves. Then at each level of the binary tree, for each node in the level, a coreset of size S ( n, ε 2 log n , δ poly( n ) ) is built from the coresets representing the two children of the node. Due to the mergeability property of coresets, the coreset at the root of the tree will be a coreset for the entire stream with accuracy ( 1 + ε 2 log n ) log n ≤ (1 + ε ) and failure probability δ .

This approach fails for sliding window algorithms because the elements at the beginning of the data stream can expire, and so coresets corresponding to earlier blocks of the stream may no longer accurate, which would result in the coreset at the root of the tree also no longer being accurate. On the other hand, suppose we partition the stream into blocks consisting of S ( n, ε 2 log n , δ poly( n ) ) elements as before, but instead of creating an offline coreset for each block, we can create an online coreset for the elements in reverse . That is, since the elements in each block are explicitly stored, we can create offline an artificial stream consisting of the elements in the block in reverse and then give the artificial stream as input to the online coreset construction. Note that if we also first consider the 'latter' coreset when merging two coresets, then this effectively reverses the stream. Moreover, by the correctness of the online coreset, our data structure provides correctness over any prefix of the reversed stream, or equivalently, any suffix of the stream and specifically, correctness over the sliding window. We thus further adapt the merge-and-reduce framework to show that randomized online coresets for problems in clustering can also be used to achieve randomized algorithms for the corresponding problems in the sliding window model. We formalize this approach in Algorithm 2.

Theorem 2.3. Let x 1 , . . . , x n be a stream of points in [∆] d , ε > 0 , and let X = { x n -W +1 , . . . , x n } be the W most recent points. Suppose there exists a randomized algorithm that with probability at least 1 -δ , outputs an online coreset algorithm for a k -clustering problem with S ( n, d, k, ε, δ ) points. Then there exists a randomized algorithm that with probability at least 1 -δ , outputs a coreset for the k -clustering problem in the sliding window model with O ( S ( n, d, k, ε log n , δ n 2 ) log n ) points.

By Theorem 1.3 and Theorem 2.3, we have:

Theorem 2.4. There exists an algorithm that samples k min( ε 4 ,ε 2+ z ) polylog n ∆ ε points and with high probability, outputs a (1 + ε ) -coreset to ( k, z ) -clustering in the sliding window model.

Using an offline algorithm for ( k, z ) -clustering for post-processing after the data stream, we have Theorem 1.1.

3 Experimental Evaluations

In this section, we conduct simple empirical demonstrations as proof-of-concepts to illustrate the benefits of our algorithm. Our empirical evaluations were conducted using Python 3.10 using a 64-bit

Algorithm 2 Merge-and-reduce framework for randomized algorithms in the sliding window model, using randomized constructions of online coresets

Input: A clustering function f , a set of points x 1 , . . . , x n ⊆ R d , accuracy parameter ε > 0 , failure probability δ ∈ (0 , 1) , and window size W > 0 Output: An approximation of f on the W most recent points 1: Let CORESET ( X,n,d,k,ε, δ ) be an online coreset construction with S ( n, d, k, ε, δ ) points on a set X ⊆ R d 2: m ← O ( S ( n, d, k, ε log n , δ n ) log n ) 3: Initialize blocks B 0 , B 1 , . . . , B log n ←∅ 4: for each point x t with t ∈ [ n ] do 5: if B 0 does not contain m points then 6: Prepend x t to B 0 , i.e., B 0 ←{ x t } ∪ B 0 7: else 8: Let i be the smallest index such that B i = ∅ 9: B i ← CORESET ( Y, n, d, k, ε log n , δ n 2 ) for Y = B 0 ∪ . . . ∪ B i -1 ▷Y is an ordered set of weighted points 10: for j = 0 to j = i -1 do 11: B j ←∅ 12: B 0 ←{ x t } 13: return the ordered set B log n ∪ . . . ∪ B 0

operating system on an AMD Ryzen 7 5700U CPU, with 8GB RAM and 8 cores with base clock 1.80 GHz. The general approach to our experiments is to produce a data stream S that defines dataset X , whose generation we describe below, as well as in Appendix F. We then compare the performance of a simplified version of our algorithm with various state-of-the-art baselines.

Baselines. Our first baseline (denoted off for offline) is the simple Lloyd's algorithm on the entire dataset X , with multiple iterations using the k-means++ initialization. This is a standard approach for finding a good approximation to the optimal clustering cost, because finding the true optimal centers requires exponential time. Because this offline Lloyd's algorithm has access to the entire dataset, the expected behavior is that this algorithm will have the best objective, i.e., smallest clustering cost. However, we emphasize that this algorithm requires storing the entire dataset X in memory and thus its input size is significantly larger than the sublinear space algorithms.

To compare with the offline Lloyd's algorithm, we run a number of sublinear space algorithms. These algorithms generally perform some sort of processing on the datastream X to create a coreset C . We normalize the space requirement of these algorithms by permitting each algorithm to store m points across specific ranges of m . We then run Lloyd's algorithm on the coreset C , with the same number of iterations using the k-means++ initialization.

Our first sublinear space algorithm is uniform sampling on the dataset X . That is, we form C by uniformly sampling m points from X , before running Lloyd's algorithm. We use uni to denote this algorithm whose first step is based on uniformly sampling. Our second sublinear space algorithm is the importance sampling approach used by histogram-based algorithms, e.g., [15, 11, 8]. These algorithms perform importance sampling, i.e., sample points into the coreset C with probability proportional to their distances from existing samples and delete points once the clustering cost of C is much higher than the clustering cost of the dataset X . We use hist(ogram) to denote this algorithm that is based on the histogram frameworks for sliding windows.

Our final algorithm is a simplification of our algorithm. As with the histogram-based algorithm, we perform importance sampling on the stream S to create the coreset C of size m . Thus we do not implement the ring and group sampling subroutines in our full algorithm. However, the crucial difference compared to the histogram-based approach is that we forcefully discard points of C that have expired. We use imp to denote this algorithm whose first step is based on importance sampling.

Dataset. We first describe the methodology and experimental setup of our empirical evaluation on a real-world dataset with an amount of synthetic noise before detailing the experimental results. The

first component of our dataset consists of the points of the SKIN (Skin Segmentation) dataset X ′ from the publicly available UCI repository [6], which was also used in the experiments of [8]. The dataset X ′ consists of 245 , 057 points with four features, where each point refers to a separate image, such that the first three features are constructed over BGR space, and the fourth feature is the label for whether or not the image refers to a skin sample. We subsequently pre-process each dataset to have zero mean and unit standard deviation in each dimension.

We then form our dataset X by augmenting X ′ with 201 points in four-dimensional space, where 100 of these points were drawn from a spherical Gaussian with unit standard deviation in each direction and centered at ( -10 , 10 , 0 , 0) and 100 of these points were drawn from a spherical Gaussian with unit standard deviation in each direction and centered at (10 , -10 , 0 , 0) . The final point of X was drawn from a spherical Gaussian with unit standard deviation centered at (500 , 500 , 0 , 0) . Thus our dataset X has dimensions n = 245 , 258 and d = 4 . We then create the data stream S by prepending two additional points drawn from spherical Gaussians with standard deviation 2 . 75 centered at ( -10 , 10 , 0 , 0) and ( -10 , -10 , 0 , 0) respectively, so that the stream has length 245 , 260 . We set the window length to be 245 , 258 in accordance with the 'true' data set, so that the first two points of the stream will be expired by the data stream.

Experimental setup. For each of the instances of Lloyd's algorithm, either on the entire dataset X or the sampled coreset C , we use 10 iterations using the k-means++ initialization. While the offline Lloyd's algorithm stores the entire dataset X of 245 , 258 points in memory, we only allow each of the sublinear-space algorithms to store a fixed m points. We compare the algorithms across m ∈ { 5 , 10 , 15 , 20 , 25 , 30 } and k ∈ { 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 } . Note that in the original dataset, each of the points has a label for either skin or non-skin, which would be reasonable for k = 2 . However, due to the artificial structure possibly induced by the synthetic noise, it also makes sense to other values of k . In particular, preliminary experiments from uniform sampling by the elbow method indicated that k = 3 would be a reasonable setting. Thus we fix k = 3 while varying m ∈ { 5 , 10 , 15 , 20 , 25 , 30 } and we arbitrarily fix m = 25 while varying k ∈ { 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 } .

Experimental results. For each choice of m and k , we ran each algorithm 30 times and tracked the resulting clustering cost. Our algorithm demonstrated superior performance than the other sublinearspace algorithms across all values of m ∈ { 5 , 10 , 15 , 20 , 25 , 30 } and k ∈ { 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 } , and was even quite competitive with the offline Lloyd's algorithm, even though our algorithm only used memory size m ≤ 30 , while the offline algorithm used memory 245 , 258 .

Uniform sampling performed well for k = 2 , which in some case captures the structure imposed on the data through the skin vs. non-skin label, but for larger k , the optimal solutions start placing centers to handle the synthetic noise, which may not be sampled by uniform sampling. Thus uniform sampling performed relatively poorly albeit quite stably for larger k . In contrast, the histogram-based algorithm performed poorly for small k across all our ranges of m , due to sampling the extra points in S \ X , so that the resulting Lloyd's algorithm on C moved the centers far away from the optimal centers of X . On the other hand, the histogram-based algorithm performed well for larger k , likely due to additional centers that could be afforded to handle the points in S \ X . We plot our results in Figure 1 and defer additional experiments to Appendix F.

4 Conclusion

In this paper, we give an algorithm outputs a (1 + ε ) -approximation to ( k, z ) -clustering in the sliding window model, while using k min( ε 4 ,ε 2+ z ) polylog n ∆ ε words of space when the points are from [∆] d . Our algorithm not only improves on a line of work [5, 16, 34, 8, 35], but also nearly matches the space used by the offline coreset constructions of [27], which is known to be near-optimal in light of a Ω ( k ε 2+ z ) lower bound for the size of an offline coreset [45].

We also give a lower bound that shows a logarithmic overhead in the number of points is needed to maintain a (1 + ε ) -online coreset compared to a (1 + ε ) -coreset. That is, we gave in Theorem 1.4 a set X ⊂ [∆] d of n points x 1 , . . . , x n such that any (1 + ε ) -online coreset for k -means clustering on X requires Ω ( k ε 2 log n ) points. However, this does not rule out whether the log n overhead is necessary for ( k, z ) -clustering in the sliding window model, since a sliding window algorithm does

(a) Comparisons for varying

k

.

(b) Comparisons for varying

m

.

uniformrange

uniformrange

avg.uni. errs.

avg.uni.errs.

220

histogramrange

223

histogramrange

avg.hist. errs

avg.hist. errs

importancerange

importancerange

Clustering cost

avg.imp.errs

oflinerange

Clustering cost

222

oflinerange

avg. imp. errs

219

avg. offl. errs

avg.offl. errs

221

218

220

217

219

2

3

4

5

6

7

8

6

10

5

10

15

20

25

30

Number of clusters

Number of samples

not necessarily need to maintain an online coreset. We leave this question as a possible direction for future work.

Acknowledgments

David P. Woodruff and Samson Zhou were partially supported by a Simons Investigator Award and by the National Science Foundation under Grant No. CCF-1815840. This work was done in part while Samson Zhou was at Carnegie Mellon University, UC Berkeley, and Rice University.

References

Marcel R. Ackermann, Marcus Märtens, Christoph Raupach, Kamil Swierkot, Christiane Lammersen, and Christian Sohler. Streamkm++: A clustering algorithm for data streams. ACM J. Exp. Algorithmics , 17(1), 2012. 2

Miklós Ajtai, Vladimir Braverman, T. S. Jayram, Sandeep Silwal, Alec Sun, David P. Woodruff, and Samson Zhou. The white-box adversarial data stream model. In PODS '22: International Conference on Management of Data, 2022 , pages 15-27, 2022. 2

Apple. https://images.apple.com/privacy/docs/Differential_Privacy_Overview.pdf. 2

Brian Babcock, Shivnath Babu, Mayur Datar, Rajeev Motwani, and Jennifer Widom. Models and issues in data stream systems. In Proceedings of the Twenty-first ACM SIGACT-SIGMODSIGART Symposium on Principles of Database Systems , pages 1-16, 2002. 2

Brian Babcock, Mayur Datar, Rajeev Motwani, and Liadan O'Callaghan. Maintaining variance and k-medians over data stream windows. In Proceedings of the Twenty-Second ACM SIGACTSIGMOD-SIGART Symposium on Principles of Database Systems , pages 234-243, 2003. 2, 3, 9

Rajen Bhatt and Abhinav Dhall. Skin segmentation dataset. UCI Learning Repository. 9

Jeremiah Blocki, Seunghoon Lee, Tamalika Mukherjee, and Samson Zhou. Differentially private l 2 -heavy hitters in the sliding window model. In The Eleventh International Conference on Learning Representations, ICLR , 2023. 2

Michele Borassi, Alessandro Epasto, Silvio Lattanzi, Sergei Vassilvitskii, and Morteza Zadimoghaddam. Sliding window algorithms for k-clustering problems. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, NeurIPS , 2020. 2, 5, 8, 9, 16

Vladimir Braverman, Petros Drineas, Cameron Musco, Christopher Musco, Jalaj Upadhyay, David P. Woodruff, and Samson Zhou. Near optimal linear algebra in the online and sliding window models. In 61st IEEE Annual Symposium on Foundations of Computer Science, FOCS , pages 517-528, 2020. 2, 4, 21

Vladimir Braverman, Dan Feldman, Harry Lang, Adiel Statman, and Samson Zhou. Efficient coreset constructions via sensitivity sampling. In Asian Conference on Machine Learning, ACML , pages 948-963, 2021. 2, 4

Vladimir Braverman, Gereon Frahling, Harry Lang, Christian Sohler, and Lin F. Yang. Clustering high dimensional dynamic data streams. In Proceedings of the 34th International Conference on Machine Learning, ICML , pages 576-585, 2017. 2, 8

Vladimir Braverman, Ran Gelles, and Rafail Ostrovsky. How to catch l 2 -heavy-hitters on sliding windows. Theor. Comput. Sci. , 554:82-94, 2014. 2

Vladimir Braverman, Elena Grigorescu, Harry Lang, David P. Woodruff, and Samson Zhou. Nearly optimal distinct elements and heavy hitters on sliding windows. In Approximation, Randomization, and Combinatorial Optimization. Algorithms and Techniques, APPROX/RANDOM , pages 7:1-7:22, 2018. 2

Vladimir Braverman, Avinatan Hassidim, Yossi Matias, Mariano Schain, Sandeep Silwal, and Samson Zhou. Adversarial robustness of streaming algorithms through importance sampling. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems, NeurIPS , pages 3544-3557, 2021. 2

Vladimir Braverman, Harry Lang, Keith Levin, and Morteza Monemizadeh. Clustering on sliding windows in polylogarithmic space. In 35th IARCS Annual Conference on Foundation of Software Technology and Theoretical Computer Science, FSTTCS , pages 350-364, 2015. 2, 8

Vladimir Braverman, Harry Lang, Keith Levin, and Morteza Monemizadeh. Clustering problems on sliding windows. In Proceedings of the Twenty-Seventh Annual ACM-SIAM Symposium on Discrete Algorithms, SODA , pages 1374-1390, 2016. 2, 3, 4, 9

Vladimir Braverman, Adam Meyerson, Rafail Ostrovsky, Alan Roytman, Michael Shindler, and Brian Tagiku. Streaming k-means on well-clusterable data. In Proceedings of the Twenty-Second Annual ACM-SIAM Symposium on Discrete Algorithms, SODA , pages 26-40, 2011. 2

Vladimir Braverman and Rafail Ostrovsky. Smooth histograms for sliding windows. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS), Proceedings , pages 283-293, 2007. 2, 4

Vladimir Braverman, Rafail Ostrovsky, and Carlo Zaniolo. Optimal sampling from sliding windows. J. Comput. Syst. Sci. , 78(1):260-272, 2012. 2

Vladimir Braverman, Viska Wei, and Samson Zhou. Symmetric norm estimation and regression on sliding windows. In Computing and Combinatorics - 27th International Conference, COCOON, Proceedings , pages 528-539, 2021. 2

Moses Charikar, Liadan O'Callaghan, and Rina Panigrahy. Better streaming algorithms for clustering problems. In Proceedings of the 35th Annual ACM Symposium on Theory of Computing , pages 30-39, 2003. 2

Jiecao Chen, Huy L. Nguyen, and Qin Zhang. Submodular maximization over sliding windows. CoRR , abs/1611.00129, 2016. 2

Ke Chen. On coresets for k-median and k-means clustering in metric and euclidean spaces and their applications. SIAM J. Comput. , 39(3):923-947, 2009. 2

Vincent Cohen-Addad, 2022. Private communication. 17

Vincent Cohen-Addad, Kasper Green Larsen, David Saulpic, and Chris Schwiegelshohn. Towards optimal lower bounds for k-median and k-means coresets. In STOC '22: 54th Annual ACM SIGACT Symposium on Theory of Computing , pages 1038-1051, 2022. 2, 3, 4, 5, 22

Vincent Cohen-Addad, Kasper Green Larsen, David Saulpic, Chris Schwiegelshohn, and Omar Ali Sheikh-Omar. Improved coresets for euclidean k-means. In NeurIPS , 2022. 3, 4

Vincent Cohen-Addad, David Saulpic, and Chris Schwiegelshohn. A new coreset framework for clustering. In STOC: 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 169-182, 2021. 2, 3, 4, 5, 6, 9, 16, 17, 19, 24, 25, 26

Vincent Cohen-Addad, Chris Schwiegelshohn, and Christian Sohler. Diameter and k-center in sliding windows. In 43rd International Colloquium on Automata, Languages, and Programming, ICALP , pages 19:1-19:12, 2016. 2

Vincent Cohen-Addad, David P. Woodruff, and Samson Zhou. Streaming euclidean k-median and k-means with o(log n) space. CoRR , abs/2310.02882, 2023. 2, 4

Graham Cormode. The continuous distributed monitoring model. SIGMOD Rec. , 42(1):5-14, 2013. 2

Graham Cormode and Minos N. Garofalakis. Streaming in a connected world: querying and tracking distributed data streams. In EDBT 2008, 11th International Conference on Extending Database Technology, Proceedings , page 745, 2008. 2

Graham Cormode and S. Muthukrishnan. What's new: finding significant differences in network data streams. IEEE/ACM Trans. Netw. , 13(6):1219-1232, 2005. 2

Mayur Datar, Aristides Gionis, Piotr Indyk, and Rajeev Motwani. Maintaining stream statistics over sliding windows. SIAM J. Comput. , 31(6):1794-1813, 2002. 2, 4

Alessandro Epasto, Silvio Lattanzi, Sergei Vassilvitskii, and Morteza Zadimoghaddam. Submodular optimization over sliding windows. In Proceedings of the 26th International Conference on World Wide Web, WWW , pages 421-430, 2017. 2, 3, 4, 5, 9

Alessandro Epasto, Mohammad Mahdian, Vahab S. Mirrokni, and Peilin Zhong. Improved sliding window algorithms for clustering and coverage via bucketing-based sketches. In Proceedings of the 2022 ACM-SIAM Symposium on Discrete Algorithms, SODA , pages 30053042, 2022. 2, 3, 4, 9

Facebook. https://www.facebook.com/policy.php. 2

Dan Feldman and Michael Langberg. A unified framework for approximating and clustering data. In Proceedings of the 43rd ACM Symposium on Theory of Computing, STOC , pages 569-578, 2011. 2, 4

Dan Feldman, Melanie Schmidt, and Christian Sohler. Turning big data into tiny data: Constantsize coresets for k-means, pca, and projective clustering. SIAM J. Comput. , 49(3):601-657, 2020. 19

Dan Feldman and Leonard J. Schulman. Data reduction for weighted and outlier-resistant clustering. In Proceedings of the Twenty-Third Annual ACM-SIAM Symposium on Discrete Algorithms, SODA , pages 1343-1354, 2012. 2, 4

Zhili Feng, Praneeth Kacham, and David P. Woodruff. Strong coresets for subspace approximation and k-median in nearly linear time. CoRR , abs/1912.12003, 2019. 19

Google. https://policies.google.com/technologies/retention. 2

Sudipto Guha, Nina Mishra, Rajeev Motwani, and Liadan O'Callaghan. Clustering data streams. In 41st Annual Symposium on Foundations of Computer Science, FOCS , pages 359-366, 2000. 2

Sariel Har-Peled and Akash Kushal. Smaller coresets for k-median and k-means clustering. Discret. Comput. Geom. , 37(1):3-19, 2007. 2

Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-median clustering. In Proceedings of the 36th Annual ACM Symposium on Theory of Computing , pages 291-300, 2004. 2

Lingxiao Huang, Jian Li, and Xuan Wu. Towards optimal coreset construction for (k, z)clustering: Breaking the quadratic dependency on k. CoRR , abs/2211.11923, 2022. 2, 3, 9

Lingxiao Huang and Nisheeth K. Vishnoi. Coresets for clustering in euclidean spaces: importance sampling is nearly optimal. In Proccedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, STOC , pages 1416-1429, 2020. 2, 19

Rajesh Jayaram, David P. Woodruff, and Samson Zhou. Truly perfect samplers for data streams and sliding windows. In PODS '22: International Conference on Management of Data , pages 29-40, 2022. 2

Lap-Kei Lee and H. F. Ting. Maintaining significant stream statistics over sliding windows. In Proceedings of the Seventeenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA , pages 724-732, 2006. 2

Lap-Kei Lee and H. F. Ting. A simpler and more efficient deterministic scheme for finding frequent items over sliding windows. In Proceedings of the Twenty-Fifth ACM SIGACTSIGMOD-SIGART Symposium on Principles of Database Systems , pages 290-297, 2006. 2

J MacQueen. Classification and analysis of multivariate observations. In 5th Berkeley Symp. Math. Statist. Probability , pages 281-297, 1967. 1

Konstantin Makarychev, Yury Makarychev, and Ilya P. Razenshteyn. Performance of johnsonlindenstrauss transform for k -means and k -medians clustering. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC , pages 1027-1038, 2019. 2

Gurmeet Singh Manku and Rajeev Motwani. Approximate frequency counts over data streams. Proc. VLDB Endow. , 5(12):1699, 2012. 2

Jirí Matousek. On approximate geometric k-clustering. Discret. Comput. Geom. , 24(1):61-84, 2000. 17

Adam Meyerson. Online facility location. In 42nd Annual Symposium on Foundations of Computer Science, FOCS , pages 426-431. IEEE Computer Society, 2001. 15

Shyam Narayanan and Jelani Nelson. Optimal terminal dimensionality reduction in euclidean space. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, STOC , pages 1064-1069, 2019. 20

Miles Osborne, Sean Moran, Richard McCreadie, Alexander von Lünen, Martin D. Sykora, Amparo Elizabeth Cano, Neil Ireson, Craig Macdonald, Iadh Ounis, Yulan He, Tom Jackson, Fabio Ciravegna, and Ann O'Brien. Real-time detection, tracking, and monitoring of automatically discovered events in social media. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL , pages 37-42, 2014. 2

Odysseas Papapetrou, Minos N. Garofalakis, and Antonios Deligiannakis. Sketching distributed sliding-window data streams. VLDB J. , 24(3):345-368, 2015. 2

Christian Sohler and David P. Woodruff. Strong coresets for k-median and subspace approximation: Goodbye dimension. In 59th IEEE Annual Symposium on Foundations of Computer Science, FOCS , pages 802-813, 2018. 19

Zhao Song, Lin F. Yang, and Peilin Zhong. Sensitivity sampling over dynamic geometric data streams with applications to k-clustering. CoRR , abs/1802.00459, 2018. 2, 5

Hugo Steinhaus et al. Sur la division des corps matériels en parties. Bull. Acad. Polon. Sci , 1(804):801, 1956. 1

Murad Tukan, Xuan Wu, Samson Zhou, Vladimir Braverman, and Dan Feldman. New coresets for projective clustering and applications. In International Conference on Artificial Intelligence and Statistics, AISTATS , pages 5391-5415, 2022. 2

Zhewei Wei, Xuancheng Liu, Feifei Li, Shuo Shang, Xiaoyong Du, and Ji-Rong Wen. Matrix sketching over sliding windows. In Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference , pages 1465-1480. ACM, 2016. 2

David P. Woodruff and Samson Zhou. Tight bounds for adversarially robust streams and sliding windows via difference estimators. In 62nd IEEE Annual Symposium on Foundations of Computer Science, FOCS , pages 1183-1196, 2021. 2

A Preliminaries

For a positive integer n , we use the notation [ n ] to denote the set { 1 , . . . , n } . Similarly, we use [∆] d to denote { 1 , . . . , ∆ } d . We use poly( n ) to denote a fixed polynomial in n with degree determined as necessary by setting the appropriate constants in corresponding variables. Similarly, we use polylog( n ) to denote poly(log n ) . We suppress polylogarithmic dependencies by writing ˜ O ( f ( · )) = O ( f ( · )) polylog f ( · ) .

For ( k, z ) -clustering on a set X = { x 1 , . . . , x n } ⊂ R d using a set C of k centers and a distance function dist( · , · ) , we define the notation Cost( X,C ) = ∑ n i =1 min c ∈ C dist( x i , c ) z . We also define the notation Cost | S |≤ k ( X,S ) := min S : | S |≤ k Cost( X,S ) , so that Cost | S |≤ k is the cost of an optimal ( k, z ) -clustering.

Definition A.1 ( ( α, β ) -approximation) . We say a set of centers C provides an ( α, β ) -approximation to the optimal k -means clustering on a set X if | C | ≤ βk and

Definition A.2 (Coreset) . A coreset for ( k, z ) -clustering on an approximation parameter ε > 0 and a set X of points x 1 , . . . , x n ∈ R d with distance function dist is a subset S of weighted points of X with weight function w such that for any set C of k points, we have

Definition A.3 (Online Coreset) . An online coreset for ( k, z ) -clustering on an approximation parameter ε > 0 and a set X of points x 1 , . . . , x n ∈ R d with distance function dist is a subset S of weighted points of X with weight function w such that for any set C of k points and for any t ∈ [ n ] , we have

where S t = S ∩ { X 1 , . . . , X t } , i.e., the subset of S that has arrived at time t .

Theorem A.4 (Bernstein's inequality) . Let X 1 , . . . , X n be independent random variables such that E [ X 2 i ] < ∞ and X i ≥ 0 for all i ∈ [ n ] . Let X = ∑ i X i and γ > 0 . Then

If X i -E [ X i ] ≤ ∆ for all i , then for σ 2 i = E [ X 2 i ] -E [ X i ] 2 ,

Meyerson sketch. We briefly review the Meyerson sketch [54] and the relevant properties that we need from the Meyerson sketch. The Meyerson sketch provides an ( α, β ) -approximation to ( k, z ) -clustering on a data stream of points x 1 , . . . , x n ∈ [∆] d with α = 2 z +7 and β = O ( 2 2 z log n log ∆ ) . Moreover, for our purposes, it provides the crucial property that on the arrival of each point x i , the algorithm irrevocably assigns x i to one of the βk centers. Specifically, the clustering cost at the end of the stream is computed with respect to the center that x i is assigned at time i , which may not be the closest center to x i because the closer center can be opened at a later time.

For the ease of discussion, we describe the Meyerson sketch for z = 1 ; the intuition generalizes naturally to other values of z . The Meyerson sketch performs via a guess-and-double approach, where it first attempts to guess the cost of the optimal clustering cost. Using the guess of the cost, it then turns each point into a center with probability proportional to the distance of that point from the existing centers. This subroutine is illustrated in Algorithm 3. If too many centers have been opened, then the Meyerson sketch determines that the guess for the optimal clustering cost must have been too low and increases the guess. The overall algorithm is given in Algorithm 4.

We require the following properties from the Meyerson sketch.

Algorithm 4 High probability MULTMEYERSON sketch

Output: A coreset for k -means clustering on X if ˜ OPT upper bounds the cost of the optimal clustering

2: for i ∈ [ γ ] do

3: Run MEYERSON ( X, 2 i , α = 1 2 , δ, ∆ , z, k ) in parallel

4: Let j be the minimal index in [ γ ] such that MEYERSON with input 2 j has size smaller than 8 k log 1 δ (1 + log ∆) ( 2 2 z +3 +1 ) and cost smaller than 2 z +6+ j

5: return the output for MEYERSON ( X, 2 j , α = 1 2 , δ, ∆ , z, k )

Theorem 2.1. [8] Given an input stream x 1 , . . . , x n ∈ R d defining a set X ⊂ [∆] d , there exists an online algorithm MULTMEYERSON that with probability at least 1 -1 poly( n ) :

on the arrival of each point x i , assigns x i to a center in C through a mapping π : X → C , where C contains at most O ( 2 2 z k log n log ∆ ) centers

MULTMEYERSON uses O ( 2 z k log 3 ( nd ∆) ) words of space

B Online (1 + ε ) -Coreset

In this section, we describe how to construct an online (1 + ε ) coreset for ( k, z ) -clustering under general discrete metrics. We first describe the offline coreset construction of [27] and then argue that the construction can be adapted to an online setting at the cost of logarithmic overheads, which suffice for our purpose.

Let A be an ( α, β ) -approximation for a ( k, z ) -clustering on an input set X ⊆ [∆] d and let C 1 , . . . , C βk be the clusters of X induced by A . Suppose the points of X arrive in a data stream S . For a fixed ε > 0 , [27] define the following notions of rings and groups:

The average cost of cluster C i is denoted by κ C i := Cost( C i , A ) | C i | .

For any i, j , the ring R i,j is the set of points x ∈ C i such that

For any j , R j = ∪ R i,j .

The inner ring R I ( C i ) = ∪ j ≤ 2 z log ε z R i,j is the set of points of C i with cost at most ( ε z ) 2 z κ C i . More generally for a solution S , let R S I denote the union of the inner rings induced by S .

The outer ring R O ( C i ) = ∪ j ≥ 2 log z ε R i,j is the set of points of C i with cost at least ( z ε ) 2 z κ C i . More generally for a solution S , let R S O denote the union of the outer rings induced by S .

The main ring R M ( C i ) is the set of points of C i that are not in the inner or outer rings, i.e., R M ( C i ) = C i \ ( R I ( C i ) ∪ R O ( C i ) .

For any j , the group G j,b consists of the (2 b -1 +1) -th to (2 b ) -th points of each ring R i,j that arrive in S .

For any j , we use G j, min to denote the union of the groups with the smallest costs, i.e.,

The outer groups G O b partition the outer rings R A O so that

We define G O min = ∪ b ≤ 0 G O b and G O max = ∪ b ≥ z log 4 z ε G O b .

We remark that unlike the definition of [27], G j, min is a subset of the groups G j,b with b ≥ 1 , but we shall nevertheless show that our sampling procedure preserves the cost contributed by each group. We also require the following slight variation of the definition of A -approximate centroid set from [53] due to [27].

Definition B.1 ( A -approximate centroid set) . Let X ⊆ R d be a set of points, let k, z be two positive integers, and let ε > 0 be an accuracy parameter. Given a set A of centers, we say a set C is an A -approximate centroid set for ( k, z ) -clustering on X if for every set of k centers S ⊆ R d , there exists ˜ S ⊆ R d of k points such that for all x ∈ X with Cost( x, S ) ≤ ( 8 z ε ) z Cost( x, A ) or Cost( x, ˜ S ) ≤ ( 8 z ε ) z Cost( x, A ) ,

The following statement is implied by the proof of Theorem 1 in [27].

Theorem B.2. [27, 24] Let z > 0 be a constant. Let x ∈ G for a group induced by an ( α, β ) -bicriteria assignment A . For each cluster C i with i ∈ [ βk ] , let D i = C i ∩ G . Let C be an A -approximate centroid set for G and let

for some sufficiently large constant C > 0 . Let

Suppose each point x ∈ X is sampled and reweighted independently into a set Ω 0 with probability p x , where

Suppose Ω 2 is the set of centers in A , where each center c i with i ∈ [ βk ] has weight w i , where w i is a (1 + ε ) -approximation to | R I ( C i ) | + | C i ∩∪ j G j, min | + | R O ( C i ) ∩ G O min | . Then (Ω 1 \ Ω 2 ) ∪ Ω 2 is (1 + ε ) -coreset for the ( k, z ) -clustering problem with probability 1 -1 poly( n ) .

We first show that the sampling probabilities for each point in the stream by RINGSAMPLE in Algorithm 1 satisfies the conditions of Theorem B.2.

Lemma B.3. Let x ∈ G for a group induced by an ( α, β ) -bicriteria assignment A at a time t , with t ∈ [ n ] . For each cluster C i with i ∈ [ βk ] , let D i = C i ∩ G . Let C be an A -approximate centroid set for G and let

for some sufficiently large constant C > 0 Let

Then the probability p x that RINGSAMPLE (Algorithm 1) samples each point x satisfies

Proof. Suppose that x ∈ R i,j and x ∈ G j,b at time t , for some i ∈ [ βk ] in an assignment by A from MULTMEYERSON. Let u be the time that x arrived in the stream. By the properties of the Meyerson sketch, i.e., MULTMEYERSON in Theorem 2.1, x is irrevocably assigned to a cluster C i with i ∈ [ βk ] at time u . Hence, x must also be assigned to ring R i,j at time u . Moreover, since the stream is insertion-only, then the number of points in all rings R i,j for a fixed j across all i ∈ [ βk ] is monotonically non-decreasing. Thus x must also be assigned to group G j,b at time u .

Let p x be the sampling probability of x by RINGSAMPLE in Algorithm 1 at time u . We have that

where r u is the number of points in G j,b at time u . Let G ( u ) j,b be the subset of G j,b that have arrived at time u and let G ( t ) j,b be the subset of G j,b that have arrived at time t . Let c i be the center assigned to point x , so that Cost( x, c i ) = Cost( x, A ) and let C ( u ) i be the points assigned to c i at time u . Similarly, let D ( u ) i = C ( u ) i ∩ G ( u ) j,b . By the definition of R i,j and G j,b ,

Since both the cost of group G j,b and the number of points in D i is monotonically non-decreasing over time, then at time t , we have

Similarly, we have that due to the monotonicity of the cost of group G j,b over time,

Thus for sufficiently large constant C in the definition of γ in RINGSAMPLE, we have that

We next justify the space complexity of Algorithm 1, i.e., showing that with high probability, an upper bound of the number of samples can be determined.

Lemma B.4. RINGSAMPLE (Algorithm 1) samples

points with high probability.

Proof. Recall that by definition, the groups G j,b partition the points X = x 1 , . . . , x n ⊆ [∆] d . For a fixed j and b , let Y i be an indicator random variable for whether the i -th point of G j,b is sampled by RINGSAMPLE. Then we have E [ Y i ] ≤ 4 i · γ log n and similarly E [ Y 2 i ] ≤ 4 i · γ log n . By Bernstein's inequality, Theorem A.4, we have that

and more generally, we have that ∑ Y i = O ( γ log 2 n ) with high probability. Thus by a union bound over all j and b , we have that the number of sampled points is at most

Moreover, note that we can for all t ∈ [ n ] , we can explicitly track both | G ( t ) j,b | and Cost( G ( t ) j,b , A ) as the stream is updated, because once the bicriteria algorithm assigns a point to a center in A , the assignment will remain the same for the rest of the stream. Thus, we have the following:

LemmaB.5. For each j and b , there exists an algorithm that maintains both | G ( t ) j,b | and Cost( G ( t ) j,b , A ) for all t ∈ [ n ] using O (log( nd ∆)) space.

Putting things together, we give the full guarantees of RINGSAMPLE in Algorithm 1.

Lemma 2.2. Let C be an A -approximate centroid set for a fixed group G . There exists an algorithm RINGSAMPLE that samples

points and with high probability, outputs a (1 + ε ) -online coreset for the k -means clustering problem.

Proof. Consider RINGSAMPLE. Before claiming the algorithm gives an (1 + ε ) -online coreset, we first consider a fixed time t ∈ [ n ] . Then correctness at time t follows from applying Theorem B.2, given Lemma B.3 and Lemma B.5. We then observe that once a center is formed by RINGSAMPLE, i.e., once a point is sampled, then it irrevocably remains a center in the data structure. Therefore, conditioned on the correctness at time t , then the data structure will always correctly give an (1 + ε ) -coreset to the prefix of t points in the stream at any later point t ′ in the stream, t ′ ∈ [ n ] with t ′ > t . It thus suffices to argue correctness over all t ∈ [ n ] , which requires a simple union bound. The space complexity follows from Lemma B.4 and Lemma B.5.

To apply Lemma 2.2, we require upper bounding the term log | C | . To that end, we first require the following definition of doubling dimension.

Definition B.6 (Doubling dimension) . The doubling dimension of a metric space X with metric d is the smallest integer ℓ such that for any x ∈ X , it is possible to cover the ball of radius 2 r around x with 2 ℓ balls of radius r .

Observe that general discrete metric spaces with n points have doubling dimension O (log n ) since all points can be covered by 2 log n balls.

We then recall the following result that upper bounds the size log | C | for metric spaces with doubling dimension d .

Lemma B.7. [27] Given a subset X from a metric space with doubling dimension d , ε > 0 , and an α -approximate solution A with at most k polylog( n ) centers, there exists an A -approximate centroid set for X of size | X | · ( α ε ) O ( d ) .

It is known that the Euclidean space has doubling dimension Θ( d ) , which would give a d dependency on our coreset size. However, [38] showed that the d dependency can be replaced with k ε 2 , which was subsequently improved by a line of works, e.g., [58, 40, 46], ultimately down to a dependency of 1 ε 2 log k ε using the following notion of terminal embeddings:

Stream:

Level 1:

Level 2:

Level 3:

Definition B.8 (Terminal embedding) . Let ε ∈ (0 , 1) and X ⊆ R d be a set of n points. Then a mapping f : R d → R m is a terminal embedding if for all x ∈ X and all y ∈ R d ,

[55] gave a construction of a terminal embedding with m = O ( 1 ε 2 log n ) that can be applied in linear space through exhaustive search when polynomial runtime is not required. Thus Lemma 2.2 nows give the following:

Theorem 1.3. There exists an algorithm that samples k min( ε 4 ,ε 2+ z ) polylog n ∆ ε points and with high probability, outputs a (1 + ε ) -online coreset for ( k, z ) -clustering.

For the purpose of clarity, we emphasize that the algorithm does not use sublinear space, even though the sample complexity is sublinear. Namely, for each stream update, we construct and apply a terminal embedding to project each point to a lower dimension. We then compute the appropriate sampling probability of the projected point, but then sample the original point with the computed sampling probability.

C Sliding Window Model

In this section, we describe how our online coreset construction for ( k, z ) -clustering on general discrete metric spaces can be used to achieve near-optimal space algorithms for ( k, z ) -clustering in the sliding window model.

We first recall a standard approach for using offline coreset constructions for insertion-only streaming algorithms. Suppose there exists a randomized algorithm that produces an online coreset algorithm that uses S ( n, ε, δ ) points for an input stream of length n , accuracy ε , and failure probability δ , where for the ease of discussion, we omit additional dependencies, such as on the dimension d , the clustering constraint k , the parameter z , or additional parameters for whatever problem the coreset construction may approximate. A standard approach for using coresets on insertion-only streams is the merge-and-reduce approach, which partitions the stream into blocks of size S ( n, ε 2 log n , δ poly( n ) ) and builds a coreset for each block. Each coreset is then viewed as the leaves of a binary tree with height at most log n , since the binary tree has at most n leaves. Then at each level of the binary tree, for each node in the level, a coreset of size S ( n, ε 2 log n , δ poly( n ) ) is built from the coresets representing the two children of the node. Due to the mergeability property of coresets, the coreset at the root of the tree will be a coreset for the entire stream with accuracy ( 1 + ε 2 log n ) log n ≤ (1 + ε ) and failure probability δ . We give an illustration of this approach in Figure 2.

This approach fails for sliding window algorithms because the elements at the beginning of the data stream can expire, and so coresets corresponding to earlier blocks of the stream may no longer accurate, which would result in the coreset at the root of the tree also no longer being accurate. On the other hand, suppose we partition the stream into blocks consisting of S ( n, ε 2 log n , δ poly( n ) ) elements as before, but instead of creating an offline coreset for each block, we can create an online coreset

for the elements in reverse . That is, since the elements in each block are explicitly stored, we can create offline an artificial stream consisting of the elements in the block in reverse and then give the artificial stream as input to the online coreset construction. Note that if we also first consider the 'latter' coreset when merging two coresets, then this effectively reverses the stream. Moreover, by the correctness of the online coreset, our data structure provides correctness over any prefix of the reversed stream, or equivalently, any suffix of the stream and specifically, correctness over the sliding window.

Indeed, [9] showed that deterministic online coresets for problems in randomized numerical linear algebra can be used to achieve deterministic algorithms for the corresponding problems in the sliding window model. We thus further adapt the merge-and-reduce framework to show that randomized online coresets for problems in clustering can also be used to achieve randomized algorithms for the corresponding problems in the sliding window model. We formalize this approach in Algorithm 2, duplicated below:

Algorithm 5 Merge-and-reduce framework for randomized algorithms in the sliding window model, using randomized constructions of online coresets

Input: A clustering function f , a set of points x 1 , . . . , x n ⊆ R d , accuracy parameter ε > 0 , failure probability δ ∈ (0 , 1) , and window size W > 0 Output: An approximation of f on the W most recent points 1: Let CORESET ( X,n,d,k,ε, δ ) be an online coreset construction with S ( n, d, k, ε, δ ) points on a set X ⊆ R d 2: m ← O ( S ( n, d, k, ε log n , δ n ) log n ) 3: Initialize blocks B 0 , B 1 , . . . , B log n ←∅ 4: for each point x t with t ∈ [ n ] do 5: if B 0 does not contain m points then 6: Prepend x t to B 0 , i.e., B 0 ←{ x t } ∪ B 0 7: else 8: Let i be the smallest index such that B i = ∅ 9: B i ← CORESET ( Y, n, d, k, ε log n , δ n 2 ) for Y = B 0 ∪ . . . ∪ B i -1 ▷Y is an ordered set of weighted points 10: for j = 0 to j = i -1 do 11: B j ←∅ 12: B 0 ←{ x t } 13: return the ordered set B log n ∪ . . . ∪ B 0

Theorem 2.3. Let x 1 , . . . , x n be a stream of points in [∆] d , ε > 0 , and let X = { x n -W +1 , . . . , x n } be the W most recent points. Suppose there exists a randomized algorithm that with probability at least 1 -δ , outputs an online coreset algorithm for a k -clustering problem with S ( n, d, k, ε, δ ) points. Then there exists a randomized algorithm that with probability at least 1 -δ , outputs a coreset for the k -clustering problem in the sliding window model with O ( S ( n, d, k, ε log n , δ n 2 ) log n ) points.

Proof. Consider Algorithm 2. Let CORESET ( X,n,d,k,ε, δ ) be a randomized algorithm that, with probability at least 1 -δ , computes an online coreset for a k -clustering problem f with S ( n, d, k, ε, δ ) points.

We first claim that for each B i is a ( 1 + ε log n ) i online coreset for 2 i -1 m points. To that end, observe that B i can only be non-empty if at some time, B 0 contains m points and B 1 , . . . , B i -1 are all non-empty. By the correctness of the subroutine CORESET, B i is a ( 1 + ε log n ) online coreset for the points in B 0 ∪ . . . ∪ B i -1 at some point during the stream. Hence by induction, B i is a ( 1 + ε log n )( 1 + ε log n ) i -1 = ( 1 + ε log n ) i coreset for m + ∑ i -1 j =1 2 j -1 m = 2 i -1 m points.

Now, because Algorithm 2 inserts the newest points at the beginning of B 0 , then the stream is fed in reverse to the merge-and-reduce procedure. Thus, for any W ∈ [2 i -1 , 2 i ) , B 0 ∪ . . . ∪ B i provides an online coreset k -clustering for the W most recent points in the stream.

To analyze the probability of failure, we remark that there are at most n points in the stream. For each point, there are at most n coresets constructed by the subroutine CORESET (in fact, the number of coreset constructions is upper bounded by O (log n ) ). Since each subroutine is called with failure probability δ n 2 , then by a union bound, the total failure probability is at most δ .

To analyze the space complexity, note that there are at most O (log n ) coreset constructions B 0 , . . . , B log n maintained by the algorithm. Each coreset construction samples S ( n, d, k, ε log n , δ n 2 ) points. Hence, the total number of sampled points is O ( S ( n, d, k, ε log n , δ n 2 ) log n ) .

By Theorem 1.3 and Theorem 2.3, we have:

Theorem 2.4. There exists an algorithm that samples k min( ε 4 ,ε 2+ z ) polylog n ∆ ε points and with high probability, outputs a (1 + ε ) -coreset to ( k, z ) -clustering in the sliding window model.

Using an offline algorithm for ( k, z ) -clustering for post-processing after the data stream, we have:

Theorem 1.1. There exists an algorithm that samples k min( ε 4 ,ε 2+ z ) polylog n ∆ ε points and with high probability, outputs a (1 + ε ) -approximation to ( k, z ) -clustering for the Euclidean distance on [∆] d in the sliding window model.

D Lower Bounds

In this section, we show that any (1 + ε ) -online coreset for ( k, z ) -clustering requires Ω ( k ε 2 log n ) points. The intuition is somewhat straightforward and in a black-box manner. [25] showed the existence of a set X of Ω ( k ε 2 ) unit vectors such that any sublinear space data structure would not be able to accurately determine Cost( C, X ) for a set of k unit vectors C . They thus showed that any offline (1 + ε ) -coreset construction for ( k, z ) -clustering required Ω ( k ε 2 ) points.

Because an online (1 + ε ) -coreset must answer queries on all prefixes of the stream, our goal is to essentially embed Ω(log n ) instances of the hard instance of [25] into the stream, which would require Ω ( k ε 2 log n ) points. To enforce the data structure to sample Ω ( k ε 2 ) points for each of the hard instance, we give each of the instances increasingly exponential weight. That is, we give the points in the i -th instance τ i weight for some constant τ > 1 , by inserting τ i copies of each of the points. Because the weight of the i -th instance is substantially greater than the sum of the weights of the previous instances, any (1 + ε ) -online coreset must essentially be a (1 + ε ) -coreset for the i -th instance, thus requiring Ω ( k ε 2 ) points for the i -th instance. This reasoning extends to all of the Ω(log n ) instances, thereby giving a lower bound of Ω ( k ε 2 log n ) points.

We first recall the following offline coreset lower bound by [25].

Theorem D.1. [25] For d = Θ ( k ε 2 ) , let X = e 1 , . . . , e d ∈ R 2 d be the set of elementary vectors. Let z be a constant and let a 1 , . . . , a m ∈ R 2 d with corresponding weights w 1 , . . . , w m ∈ R be a weighted set P of points. Then there exists a set of k unit vectors C = c 1 , . . . , c k ∈ R 2 d such that for m = o ( k ε 2 ) ,

We remark that the first property is due to Lemma 31 pf [25] and the second property is due to Lemma 33 and Lemma 34 of [25].

Let γ = Θ ( log n d ′ ) . Let d ′ = Θ ( k ε 2 ) be the dimension of the hard instance in Theorem D.1 and set d = γd ′ , so that we can partition the space R 2 d into γ groups of 2 d ′ coordinates.

We define a stream by creating γ weighted instances of the hard instance defined in Theorem D.1. Each of the γ hard instances will be embedded into a separate partition of 2 d ′ coordinates of R 2 d . Namely, the first instance consists of the vectors e 1 , . . . , e d ′ being inserted into the stream. By Theorem D.1, any (1 + ε ) -coreset must contain Ω ( k ε 2 ) points. The next instance consists of the

vectors e 1+2 d ′ , . . . , e 3 d ′ each being inserted τ = 100 times into the stream. That is, after the vector e d ′ arrives in the stream from the first hard instance, then t copies of e 1+2 d ′ arrive in the stream, followed by then t copies of e 2+2 d ′ , and so forth. Due to the weights of these vectors, any (1 + ε ) -coreset must essentially be a (1 + ε ) -coreset for the second hard instance and thus contain Ω ( k ε 2 ) points with support in the second group of 2 d ′ coordinates.

More generally, for each i ∈ [ γ ] , the stream inserts t i -1 copies of e 1+2( i -1) d ′ , followed by τ i -1 copies of e 2+2( i -1) d ′ , and so on. The main intuition is that due to the weights of the i -th group of d ′ elementary vectors, an (1 + ε ) -online coreset must contain a (1 + ε ) -coreset for the i -th hard instance. Moreover, since the (1 + ε ) -online coreset must be a coreset for any prefix of the stream, then it needs to be a (1 + ε ) -coreset for each of the hard instances. Hence, the online coreset must contain γ · Ω ( k ε 2 ) = Ω ( k ε 2 · log n log 1 ε ) points.

Lemma D.2. Let τ = 100 . For each integer i > 0 , let S i be the stream that consists of τ i -1 consecutive copies of e 1+2( i -1) d ′ , followed by τ i -1 copies of e 2+2( i -1) d ′ , and so on. Let S be the stream that consists of S 1 ◦ S 2 ◦ . . . . Then for each i , any (1 + ε ) -online coreset after the arrival of S i must consist of i · Ω ( k ε 2 ) points.

Proof. We prove the claim by induction on i . The base case of i = 1 follows from Theorem D.1.

Now suppose the claim holds for a fixed i -1 . Let X i be the set of points that have arrived after S i , i.e., X i = S 1 ◦ . . . ◦ S i . Let C i -1 be any (1 + ε ) -online coreset for S after the arrival of S i -1 . Let P i be a set of weighted points sampled during stream S i , so that C i = C i -1 ∪ P i . Since each point in S i has weight τ i , then by scaling the first property of Theorem D.1, we have that there exists a set of k unit vectors U i = c 1 , . . . , c k ∈ R 2 d such that

In particular, the unit vectors U i = c 1 , . . . , c k have support entirely in the i -th group of 2 d ′ coordinates in R 2 d . By the same argument, there exists a set U i -1 with the same properties in the ( i -1) -th group of 2 d ′ coordinates in R 2 d .

By the correctness of the online coreset, we have

Since U i -1 consists of unit vectors and each substream S a consists of unit vectors, then we have

Thus for ε ∈ (0 , 1) ,

since τ = 100 . On the other hand, since U i -1 has support entirely in the ( i -1) -th group of 2 d ′ coordinates and S i has support entirely in the i -th group of 2 d ′ coordinates in R 2 d , then

Thus for C i to be a (1 + ε ) -online coreset for ε ∈ (0 , 1) , C i must sample additional points from X i on top of C i -1 . Hence, P i = ∅ .

̸

̸

In particular, let P i consist of vectors y 1 , . . . , y m with weights w 1 , . . . , w m . Since P i = ∅ , then

If | P i | = o ( k ε 2 ) , then by the second property of Theorem D.1, we have

which together with Equation 1 contradicts the fact that C i is an (1 + ε ) -online coreset for X i .

Therefore, we have | P i | = Ω ( k ε 2 ) . Moreover, since P i has disjoint support from C i -1 , then by induction,

Theorem 1.4. Let ε ∈ (0 , 1) . For sufficiently large n , d , and ∆ , there exists a set X ⊂ [∆] d of n points x 1 , . . . , x n such that any (1 + ε ) -online coreset for k -means clustering on X requires Ω ( k ε 2 log n ) points.

Proof. Let γ = Θ ( log n d ′ ) . For each i ∈ [ γ ] , construct the stream S i as in the statement of Lemma D.2. Observe that | S i | = d ′ · t i for t = 100 and so under the settings of the parameter γ with the appropriate constant, the total length of the stream S = S 1 ◦ . . . ◦ S γ is precisely n . Moreover, by Lemma D.2, any (1 + ε ) -online coreset must store γ · Ω ( k ε 2 ) = Ω ( k ε 2 log n ) points for n = poly( d ) .

E On the Proof of Theorem B.2

We remark that Theorem 1 of [27] is stated for sampling a fixed number of points with replacement from each group, rather than sampling each point independently without replacement. By contrast, Theorem B.2 is stated for sampling each point independently without replacement. In this section, we briefly outline the proof of Theorem 1 of [27] and how the analysis translates to the statement of Theorem B.2.

At a high level, the coreset construction of [27] first collects rings of an approximate solution A of k points into groups, using a similar approach to that described in Appendix B with β = 1 . The algorithm then computes a coreset for each group first using a procedure GROUPSAMPLE and then using a procedure SENSITIVITYSAMPLE for some of the points not considered by the first procedure. We briefly describe both procedures, as well as how to adapt them to the setting where each point is sampled independently and without replacement.

E.1 Adaptation of Group Sampling

The GROUPSAMPLE procedure of [27] samples a fixed Λ 1 number of points from each group G with probability proportional to the contribution of each corresponding cluster of the point to the group. That is, given clusters ˜ C 1 , . . . , ˜ C k induced by A on G , GROUPSAMPLE then performs Λ 1 rounds of sampling. Each round samples a single point, where a point p ∈ ˜ C i is sampled proportional to Cost( ˜ C i , A ) | ˜ C i |· Cost( G, A ) and rescaled appropriately. Then GROUPSAMPLE offers the following guarantees:

Lemma E.1 (Lemma 2 of [27]) . Let ( X, dist) be a metric space, k , z be positive integers, G be a group of clients and A be an α -approximate solution to ( k, z ) -clustering on G so that:

For every cluster ˜ C induced by A on G , all points of ˜ C contribute the same cost in A up to a factor of 2 .

For all clusters ˜ C induced by A on G , we have that Cost( G, A ) 2 k ≤ Cost( ˜ C, A ) .

Let C be an A -approximate centroid set for ( k, z ) -clustering on G .

Then there exists a procedure GROUPSAMPLE that constructs a set Ω of size

such that with high probability, it simultaneously holds for all sets S of k centers that

We outline the high-level approach of the proof of Lemma E.1 and how can it can adjusted for an ( α, β ) -approximate solution A , as well as a process that samples each point independently without replacement, rather than using Λ 1 rounds as GROUPSAMPLE.

The proof of Lemma E.1 involves further partitioning the points of G into three subsets, based on the cost induced by the point. Namely, given a set S of k centers, a point p in group G is categorized as tiny, interesting, or huge, depending on Cost( p, S ) (though the interesting and huge points actually have a small overlap to allow slack in the proof). [27] applies standard Chernoff bounds to show that the number of sampled points is well-concentrated around its expectation and then applies Bernstein's inequality to show that the clustering costs of the tiny points, the interesting points are well-concentrated around their expectations. In particular, they show that the expected number of sampled points from each cluster ˜ C i is

due to the assumption that for all clusters ˜ C induced by A on G , we have that Cost( G, A ) 2 k ≤ Cost( ˜ C, A ) .

We first remark that if A is an ( α, β ) -approximate solution rather than an α -approximate solution, i.e., if A has βk centers rather than k centers, then the definition of the rings and groups would instead insist that for all clusters ˜ C induced by A on G , we have that Cost( G, A ) 2 βk ≤ Cost( ˜ C, A ) . Then by oversampling Λ 1 by a factor of β , i.e., sampling β Λ 1 points would ensure that the expected number of sampled points from each cluster ˜ C i would be

It then remains to argue the correctness of sampling each point independently without replacement rather than a fixed β Λ 1 number of points, which simply holds by adjusting the applications of the Chernoff bounds and Bernstein's inequality so that there is a separate random variable for each point in the input rather than for each of the Λ 1 rounds.

E.2 Adaptation of Sensitivity Sampling

The SENSITIVITYSAMPLE procedure of [27] samples a fixed Λ 2 number of points from each group G with probability proportional to the contribution of the point. Specifically, SENSITIVITYSAMPLE then performs Λ 2 rounds of sampling, where each round samples a point p in the group G with probability proportional to Cost( p, A ) Cost( G, A ) and rescales the sampled point appropriately. Then SENSITIVITYSAMPLE offers the following guarantee:

Lemma E.2 (Lemma 3 of [27]) . Let ( X, dist) be a metric space, k , z be positive integers, and A be an α -approximate solution to ( k, z ) -clustering on G . Let C be an A -approximate centroid set for ( k, z ) -clustering on G . Let G be either a group G O b or G O max . Then there exists a procedure SENSITIVITYSAMPLE that constructs a set Ω of size

such that with high probability, it simultaneously holds for all sets S of k centers that

We outline the high-level approach of the proof of Lemma E.2 and how can it can adjusted for an ( α, β ) -approximate solution A , as well as a process that samples each point independently without replacement, rather than using Λ 2 rounds as SENSITIVITYSAMPLE.

The proof of Lemma E.2 partitions the points of G into two categories, based on the cost induced by the point. Given a set S of k centers, the close points are the points p in G that have Cost( p, S ) ≤ 4 z · Cost( p, A ) . The far points are the remaining points in G , i.e., the points p in G with Cost( p, S ) > 4 z · Cost( p, A ) .

[27] applies Bernstein's inequality to show that the clustering cost of the close points is wellconcentrated around their expectations. We can again adjust the application of Bernstein's inequality so that there is a separate random variable for each point in the input rather than for each of the Λ 2 samples.

To handle the far points, [27] again uses Bernstein's inequality to show that with high probability, the clustering points of these points with respect to S can be replaced with the distance to the closest center c ∈ A plus the distance from c to the closest center in S . Conditioned on this event, the latter distance can then be charged to the remaining points of the cluster from the original dataset, i.e., the remaining points of the cluster not necessarily restricted to group G , which are significantly more numerous and already paying a similar value in S . In particular, Bernstein's inequality utilizes the fact that the second moment of the estimated cost of a cluster C is at most

for β = 1 . Thus for general β , we recover the same guarantee by oversampling Λ 2 by a factor of β , i.e., sampling β Λ 2 points would ensure that the second moment would be at most 2 k Λ 2 2 Cost 2 ( C ∩ G, A ) . It then remains to argue the correctness of sampling each point independently without replacement rather than a fixed β Λ 2 number of points, which again holds by adjusting the application of Bernstein's inequality so that there is a separate random variable for each point in the input rather than for each of the Λ 2 rounds.

F Additional Experiments on Synthetic Data

We first describe the methodology and experimental setup of our empirical evaluation on a synthetic dataset before detailing the experimental results. To emphasize the benefits of our algorithm against worst-case input, we generate a synthetic dataset that would fully capture the failure cases of previous baselines.

Dataset. We generated our dataset X consisting of 200 , 001 points on two-dimensional space so that 100 , 000 points were drawn from a spherical Gaussian with standard deviation 2 . 75 centered at ( -10 , 10) and 100 , 000 points were drawn from a spherical Gaussian with standard deviation 2 . 75 centered at (10 , -10) . The final point of X was drawn from a spherical Gaussian with standard deviation 2 . 75 centered at (100000 , 100000) . Thus by construction of our synthetic dataset for k = 3 , the optimal centers should be close to ( -10 , 10) , (10 , -10) , and (100000 , 100000) . We then create the data stream S by prepending two additional points drawn from spherical Gaussians with standard deviation 2 . 75 centered at ( -100000 , 100000) and ( -100000 , -100000) respectively. We set the window length to be 200 , 001 in accordance with the 'true' data set, so that the first two points of the stream of length 200 , 003 will be expired by the data stream.

Experimental setup. For each of the instances of Lloyd's algorithm, either on the entire dataset X or the sampled coreset C , we use 3 iterations using the k-means++ initialization. In this case, the offline Lloyd's algorithm requires storing the entire dataset X in memory and thus its input size is 200 , 001 points. By comparison, we normalize the space requirement of the sublinear-space algorithms by permitting each algorithm to store m ∈ { 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 } points. Note that since k = 3 , it would not be reasonable for C to have fewer than 3 points. We then run Lloyd's algorithm on the coreset C , with 3 iterations using the k-means++ initialization.

By construction of our dataset, we generally expect the uniform sampling algorithm uni to be stable across the various values of m but perform somewhat poorly, as it will sample points from the large clusters but it will miss the point generated from the Gaussian centered at (100000 , 100000) . Since in our construction the stream S only contains two more points than the dataset X , the histogrambased algorithm hist will not delete any points. Thus, the resulting coreset C generated by hist is somewhat likely contain the points generated from the Gaussians centered at ( -100000 , 100000)

and ( -100000 , -100000) and can perform poorly on the synthetic dataset in these cases. Finally, since we allow the last point of the stream to be the single point of X far from the two large clusters, then the importance sampling based algorithm imp will sample the last point with high probability once any points of C have been expired. Hence by the construction of our stream, we expect imp to perform well.

(a) Comparisons for varying

k

.

(b) Comparisons for varying

m

.

avg.u

1016

242

avg.h

avg.i1

239

avg.0

1014

236

Clustering cost

233

Clustering cost

1012

avg.uni. errs.

avg.hist.errs

histogramrange

230

avg.imp.errs

1010

avg.offl.errs

227

224

108

221

2

3

4

5

6

7

8

9

4

6

8

10

12

Numberof clusters

Experimental results. For each choice of m and k , we ran each algorithm 50 times and tracked the resulting clustering cost. As expected by our construction, our algorithm performed significantly better than the other sublinear-space algorithms. In fact, even though our algorithm was only permitted memory size m ∈ { 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 } , our algorithm was quite competitive with the offline Lloyd's algorithm, which used memory size 200 , 001 , i.e., the entire dataset. For k ≥ 3 , uniform sampling performed relatively poorly but quite stably, because although it never managed to sample the point generated from the Gaussian centered at (100000 , 100000) , the two other Gaussian distributions were sufficiently close that any sampled point would serve as a relatively good center for points generated from the two distributions. Similarly, for fixed k = 3 in Figure 3b, the importance sampling approach used by histogram-based algorithms performed the worse, by multiple orders of magnitude. We expect this is because we did not delete the points in S \ X from C and thus the resulting Lloyd's algorithm on C moved the centers far away from the centers of the Gaussian distributions that induced X . A more optimized fine-tuned histogram-based algorithm would have searched for parameters that govern when to delete points from S \ X , which have reduced the algorithm down to our main algorithm. We plot our results in Figure 3.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the first near-optimal $(1+\\varepsilon)$-approximation algorithm for $(k, z)$-clustering problem in the sliding window model. The core part is the $(1+\\varepsilon)$-coreset of $(k, z)$-clustering in the sliding window model which is based on an online coreset algorithm for $k$-clustering problem. This paper gives strict proof for the proposed algorithm and experimental results manifest its efficiency. In short, the theory aspect of this paper is solid enough.\n\nSTRENGTHS:\n(1) This paper gives the first $(1+\\varepsilon)$-approximation algorithm for $(k, z)$-clustering in the sliding window, which improves the existing results in two aspects including accuracy and space.  \n(2) The space $\\frac{k}{\\min (\\varepsilon^4, \\varepsilon^{2+z}) }$ of $(1+\\varepsilon)$-coreset for $(k, z)$-clustering in the sliding window almost matches the lower bound $\\Omega \\left(\\frac{k}{\\varepsilon^2} \\log{n}\\right)$.  \n(3) Strict proof is provided and experimental results are sufficient.  \n(4) The paper is well-written and well-organized.\n\nWEAKNESSES:\n(1) Some symbol is not explained, like $[ \\Delta ]^d$.  \n(2) The merge-and-reduce framework lacks some citations.\n\nQUESTIONS:\nIn Line 92-93, Theorem 1.1 presents the word space $\\frac{k}{\\min(\\varepsilon^4, \\varepsilon^{2+z})} \\text{poly} \\log \\frac{n \\Delta}{\\varepsilon}$. For $k$-median and $k$-means, $z$ is equal to $1$ and $2$ respectively. No matter $z$ is $1$ or $2$, the space should be $\\frac{k}{\\varepsilon^4} \\text{poly} \\log \\frac{n \\Delta}{\\varepsilon}$ since $\\varepsilon \\in (0, 1)$. How can you achieve $\\frac{k}{\\varepsilon^2} \\text{poly} \\log \\frac{n \\Delta}{\\varepsilon}$? In addition, for $(k, z)$-clustering, $z=1$ does not correspond to $k$-median when $\\mathrm{dist}$ represents the Euclidean distance, since $k$-median is $\\ell_1$-norm.\n\nLIMITATIONS:\nSee Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the $k$-clustering problem in sliding window model. In sliding window model, a window of size $W$ is given to capture the most recent $W$ updates in the stream, where good clustering approximation should be maintained in the window with small space complexity. In previous work, in order to achieve an $(1+\\epsilon)$ approximation for $(k,z)$-clustering problem in sliding window, a space complexity of $\\frac{kd+d^{Cz}}{\\epsilon^3}polylog(W,\\Delta,\\frac{1}{\\epsilon})$ for some constant $C \\ge 7$. In this work, the authors propose a coreset-based method, which gives an $(1+\\epsilon)$-approximation for the $(k,z)$-clustering problem in sliding window model with space complexity $\\frac{k}{min(\\epsilon^4,\\epsilon^{2+z})}polylog(\\frac{n\\Delta}{\\epsilon})$, which is independent of the window size and nearly matches the lower bound of the space used by the offline coreset construction.\n\nSTRENGTHS:\n1. The presented result significantly improves the space complexity of previous work with $(1+\\epsilon)$-approximation guarantee on the clustering quality in sliding window model. Besides, this is the first approximation result that can achieve a space complexity independent of the window size $W$. In practical settings, since $\\Delta$ is usually assumed to be bounded by $poly(n)$, the presented result can nearly match the lower bound space complexity required for any $(1+\\epsilon)$-online coreset construction for $(k,z)$-clustering problem.\n\n2. The extensive experimental results on real world datasets show that the proposed method is more efficient than previous ones.\n\nWEAKNESSES:\n1. The techniques used in this paper seem to rely heavily on the current SOTA method for offline coreset construction [1] (using ring structures and independent sampling method) and the consistent approximation scheme proposed by Meyerson et al. [2].\n\n2. The challenges for merge and reduce operations is not discussed comprehensively in the literature.\n\n[1] Adam Meyerson. Online facility location. In 42nd Annual Symposium on Foundations of Computer Science, FOCS, pages 426–431. IEEE Computer Society, 2001.\n\n[2] Vincent Cohen-Addad, Kasper Green Larsen, David Saulpic, and Chris Schwiegelshohn. Towards optimal lower bounds for k-median and k-means coresets. In STOC ’22: 54th Annual ACM SIGACT Symposium on Theory of Computing, pages 1038–1051, 2022.\n\nQUESTIONS:\n1. It seems that the $O(log\\Delta)$ dependence on the space complexity comes from the Meyerson approach (for obtaining the consistent approximation), where a guess-and-double process is used to obtain an estimation of the cost of an optimal solution. Is the $O(log\\Delta)$ term necessary in the proposed method and is there any other method that can avoid guessing $O(log\\Delta)$ times for the optimal clustering cost (for example by calling a single-criteria approximation algorithm with large approximation ratio to serve as the upper bound for optimal clustering cost)? \n\n2. Can the authors explain more about the challenges when directly applying the reduce and merge method for each block of the divided data points in a stream without using an \"inverse\" operation?\n\n3. I am curious about whether, in the sliding window model, the space complexity is more critical to consider compared to the time complexity. As I am not very familiar with this model, I would appreciate some insights on this matter.\n\nLIMITATIONS:\nSince this is mainly a theoretical work, I don't think there is potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies $k$-means and $k$-median clustering in the sliding window, and proposes an $(1+\\varepsilon$)-approximation algorithm on the top of an coreset maintained through the stream. The space complexity is roughly $k/\\text{poly}(\\varepsilon) \\cdot \\text{poly}\\log n$ where $n$ is the number of points, coming together with an nearly-tight lower bound.\n\nSTRENGTHS:\n- The paper is well-written in general, most parts are clear and well-organized.\n- The results are interesting and significant. I did not read the entire proof, but checking the key steps convince me the result should be correct. I am happy to go back to proof details if any concern comes up.\n- It is a plus for a theory paper to have experiments. The space consumption is reduced a lot with some compromise on the clustering performance.\n\nWEAKNESSES:\n- I am confused by line 388 389, it seems that the window size is too close to the stream size. Though theoretically the window size is not an important parameter, but doing this converges back to the streaming model.\n- After staring at it for a while, I do not think either of the algorithm or lower bound applies to the $k$-center clustering. Then is it OK to have $k$-clustering as in the title? The authors can either convince me this result holds / has impact on $k$-center clustering, or show me a bunch of work that does not include $k$-center in $k$-clustering.\n- I think it is good to make clear of the notation, especially when it first appears. In the abstract, maybe clarify what $z, \\Delta$ is.\n\nQUESTIONS:\n- Will the codes be made public?\n\n- Can you add an open problem section?\n\nLIMITATIONS:\nI would appreciate this as a basically-complete theory work. There are some limitations on the experiments, but not a hurdle on assessing the contribution of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an near-optimal algorithm to build (1+$\\epsilon$)-online coreset for k-clustering problem, and apply it to the sliding window model. The (near) optmality is established by a matching lower bound, also proved in the paper.\n\nA coreset is a compression of a dataset, such that any clustering centers will induce similar cost over the coreset and the original dataset. The (insertion-only) online coreset problem aims to build a coreset incrementally for a data stream, such that the coreset property (i.e., cost approximation) holds at any time point\n\nThe main algorithm borrows lots of ideas from paper [24] (\"*A New Coreset Framework for Clustering*\"), which proposes a framework for building (offline) coresets and can be roughly described as follows:\n1. Given a dataset, find a constant-approx clustering solution $A$ on it.\n2. For each cluster induced by $A$, divide it into rings by distance to the center, such that all points in a same ring have roughly the same distance to the center. \n3. Group rings at the same level from each cluster, then do importance sampling on the group. The sampled points (with scaled weights) will be the output coreset.\nFor analysis, [24] shows that: as long as you can find a small \"$A$-approximate centroid\" $C$ (which is essentially something like a $\\epsilon$-net restricted to part of the dataset), then you can have a small coreset.\n\nThe paper essentially adapts the algorithm from [24] to the online setting\n1. To get an constant-approx clustering in the online setting, the paper adopts a bi-criteria variant of Meyerson's algorithm for online facility location. Here the algorithm could produce more than k centers, but this is not an issue for bounding the coreset size since it only adds a constant factor.\n2. The importance sampling step is mostly the same as described before, just in an online fashion. This gives the online coreset construction.\n3. Finally, to apply it in the sliding window model, the paper breaks the input stream into blocks, then apply the online coreset construction in the reverse order on each block, and merge each block's coreset also in the reverse order (and keep only those within the window).\n\nI don't have time to look into the details of the lowerbound construction, however, it appears to be an adaptation of the offline bound from [23] (\"*Towards optimal lower bounds for k-median and k-means coresets*\").\n\nSTRENGTHS:\nI think the main contribution is mostly in the theory side. The problem studied is important and well-motivated. It's nice to see that the result of [23] and [24] can be ported to the online setting. This requires quite some non-trivial work and the paper is able to identify an optimal choice of parameters, which results in an asymptotically optimal algorithm.\n\nWEAKNESSES:\nAs mentioned in the Summary, the main idea seems to be an (arguably) straightforward adaptation of [24], which somewhat limits the novelty.\n\nAlso, I feel the experiment section is kinda redundant or even harmful to the paper's contribution. The proposed algorithm is not acutally implemented (in particular, the RingSample Alg, which is a core part of the proposed algorithm). It seems that the performance gain comes only from discarding points outside the sliding window, especially when compared with the histogram-based algorithm.\n- If the authors are able to implement their algorithm in a more meaningful way, and if it indeed produces better result on some realworld dataset, then I would be happy to increase my rating. Otherwise, I feel it's better to just remove the experiment section completely and present the paper as a theoretical contribution.\n\nLastly, I believe the writing can be improved a lot. It seems the draft may have been finalized hastily, with instances of noticeable cutting and pasting to meet the page limit. This renders the main body insufficiently self-contained and challenging for readers unfamiliar with [24] (and not dive into the appendix proof). For example, it's hard to get much intuition why RingSample algorithm (Alg. 1) works unless a reader already knows the result of [24].\n\nQUESTIONS:\nSome other comments.\n- I would recommend the authors to at least give a sketch of [24]'s idea before describing the RingSample algorithm.\n- I recommend to have \"coreset\" formally defined somewhere.\n- The notation $\\mathrm{Cost}(X,Y)$ is never formally defined in the paper. Although I can guess what it is, it's better to define it somewhere as it is used frequently.\n- line 224: Same as above for $\\mathrm{Cost}_{|S|\\leq k}(X, S)$.\n- line 227-245: Many symbols defined here are not used at all in the main body. Please move them to the supplementary where they are actually used in the proof. Here they only distracts readers' attention.\n- line 233-236: $R_I(C_i)$ and $R_O(C_i)$ have the same definition.\n- line 243: I would write $G_{j, \\mathrm{min}}$ and $G^O_b$ as union of some $R_{i,j}$'s rather than introducing a \"$x$\" there. It feels like you can take some $x$ from a $R_{i,j}$ while in fact it's all-or-nothing.\n- line 248: Lemma 2.2: what is $G$?\n- Section 3 (Experiment): My suggestion to the authors is to throw the experiment section to appendix or simply remove it, so you can have more space to enhance the readability of the theory part.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-14"}