SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Near-Optimal  $k$ -Clustering in the Sliding Window Model
Samson Zhou
samsonzhou@gmail.com
Clustering is an important technique for identifying structural information in large-scale data analysis, where the underlying dataset may be too large to store. In many applications, recent data can provide more accurate information and thus older data past a certain time is expired. The sliding window model captures these desired properties and thus there has been substantial interest in clustering in the sliding window model.
In this paper, we give the first algorithm that achieves near-optimal  $(1 + \varepsilon)$ -approximation to  $(k,z)$ -clustering in the sliding window model, where  $z$  is the exponent of the distance function in the cost. Our algorithm uses  $\frac{k}{\min(\varepsilon^4,\varepsilon^{2 + z})}$  polylog  $\frac{n\Delta}{\varepsilon}$  words of space when the points are from  $[\Delta]^d$ , thus significantly improving on works by Braverman et. al. (SODA 2016), Borassi et. al. (NeurIPS 2021), and Epasto et. al. (SODA 2022).
Along the way, we develop a data structure for clustering called an online coreset, which outputs a coreset not only for the end of a stream, but also for all prefixes of the stream. Our online coreset samples  $\frac{k}{\min(\varepsilon^4,\varepsilon^{2 + z})}$  polylog  $\frac{n\Delta}{\varepsilon}$  points from the stream. We then show that any online coreset requires  $\Omega \left(\frac{k}{\varepsilon^2}\log n\right)$  samples, which shows a separation from the problem of constructing an offline coreset, i.e., constructing online coresets is strictly harder. Our results also extend to general metrics on  $[\Delta ]^d$  and are near-optimal in light of a  $\Omega \left(\frac{k}{\varepsilon^{2 + z}}\right)$  lower bound for the size of an offline coreset.
Clustering is a fundamental procedure frequently used to help extract important structural information from large datasets. Informally, the goal of clustering is to partition the data into  $k$  clusters so that the elements within each cluster have similar properties. Classic formulations of clustering include the  $k$ -median and  $k$ -means problems, which have been studied since the 1950's [60, 50]. More generally, for a set  $X$  of  $n$  points in  $\mathbb{R}^d$ , along with a metric dist, a cluster parameter  $k > 0$ , and an exponent  $z > 0$  that is a positive integer, the clustering objective can be defined by
$$
\min  _ {C \subset \mathbb {R} ^ {d}, | C | = k} \sum_ {i = 1} ^ {n} \min  _ {c \in C} \operatorname {d i s t} (x _ {i}, c) ^ {z}.
$$
When dist is the Euclidean distance, the problem is known as  $(k,z)$ -clustering and more specifically,  $k$ -median clustering and  $k$ -means clustering, when  $z$  is additionally set to 1 and 2, respectively.
As modern datasets have significantly increased in size, attention has shifted to large-scale computational models, such as the streaming model of computation, that do not require multiple passes over the data. In the (insertion-only) streaming model, the points  $x_{1}, \ldots, x_{n}$  of  $X$  arrive sequentially,
and the goal is to output an optimal or near-optimal clustering of  $X$  while using space sublinear in  $n$ , ideally space  $k$  polylog( $n, d$ ), since outputting the cluster centers uses  $k$  words of space, where each word of space is assumed to be able to store an entire input point in  $\mathbb{R}^d$ . There exist slight variants of the insertion-only streaming model and a long line of active research has been conducted on clustering in these models [42, 21, 44, 43, 23, 17, 37, 39, 1, 11, 59, 46, 14, 27, 10, 25, 61, 29].
The sliding window model. Unfortunately, an important shortcoming of the streaming model is that it ignores the time at which a specific data point arrives and thus it is unable to prioritize recent data over older data. Consequently, the streaming model cannot capture applications in which recent data is more accurate and therefore considered more important than data that arrived prior to a certain time, e.g., Census data or financial markets. Indeed, it has been shown that for a number of applications, the streaming model has inferior performance [4, 52, 57, 62] compared to the sliding window model [33], where only the most recent  $W$  updates in the stream comprise the underlying dataset. Here,  $W > 0$  is a parameter that designates the window size of the active data, so that all updates before the  $W$  most recent updates are considered expired, and the goal is to aggregate statistics about the active data using space sublinear in  $W$ . In the setting of clustering, where the data stream is  $x_{1},\ldots ,x_{n}\subset \mathbb{R}^{d}$ , the active data set is  $X = \{x_{n - W + 1},\dots,x_n\}$  for  $n\geq W$  and  $X = \{x_{1},\dots,x_{n}\}$  otherwise. Thus the sliding window model is a generalization of the streaming model, depending on the choice of  $W$ , and is especially relevant for time-sensitive settings, such as data summarization [22, 34], event detection in social media [56], and network monitoring [32, 31, 30].
The sliding window model is especially relevant for applications in which computation must be restricted to data that arrived after a certain time. Data privacy laws such as the General Data Protection Regulation (GDPR) mandate that companies cannot retain specific user data beyond a certain duration. For example, the Facebook data policy [36] states that user search histories are retained for 6 months, the Apple differential privacy overview [3] states that collected user information is retained for 3 months, and the Google data retention policy states that browser information may be stored for up to 9 months [41]. These retention policies can be modeled by the sliding window model with the corresponding setting of the window parameter  $W$  and thus the sliding window model has been subsequently studied in a wide range of applications [48, 49, 18, 19, 12, 13, 9, 20, 63, 2, 47, 7].
Clustering in the sliding window model. Because the clustering objective is not well-suited to popular frameworks such as the exponential histogram or the smooth histogram, there has been significant interest in clustering in the sliding window model. We now describe the landscape of clustering algorithms in the sliding window model; these results are summarized in Table 1. In 2003, [5] first gave a  $2^{O(1 / \varepsilon)}$ -approximation algorithm for  $k$ -median clustering in the sliding window model using  $O\left(\frac{k}{\varepsilon^4} W^{2\varepsilon}\log^2 W\right)$  words of space, where  $\varepsilon \in (0,\frac{1}{2})$  is an input parameter. Subsequently, [15] gave an  $O(1)$ -approximate bicriteria algorithm using  $2k$  centers and  $k^2$  polylog(W) space for the  $k$ -median problem in the sliding window model. The question of whether there exists a poly  $(k\log W)$  space algorithm for  $k$ -clustering on sliding windows remained open until [16] gave constant-factor approximation sliding window algorithms for  $k$ -median and  $k$ -means using  $O\left(k^3\log^6 W\right)$  space and [28] gave constant-factor approximation algorithms for  $k$ -center clustering using  $O(k\log \Delta)$  space, where  $\Delta$  is the aspect ratio, i.e., the ratio of the largest to smallest distances between any pair of points. Afterwards, [8] gave a  $C$ -approximation algorithm for some constant  $C > 2^{14}$ , though it should be noted that their main contribution was the first constant-factor approximation algorithm for  $k$ -clustering using space linear in  $k$ , i.e.,  $k$  polylog(W,  $\Delta$ ) space, and thus they did not attempt to optimize the constant  $C$ . Recently, [35] gave the first  $(1 + \varepsilon)$ -approximation algorithm for  $(k,z)$ -clustering using  $\frac{(kd + d^C)}{\varepsilon^3}$  polylog  $(W,\Delta,\frac{1}{\varepsilon})$  words of space, for some constant  $C \geq 7$ . Using known dimensionality reduction techniques, i.e., [51], the algorithm's dependence on  $d^C$  can be removed in exchange for a  $\frac{1}{\varepsilon^{14}}$  polylog  $(W,\frac{1}{\varepsilon})$  overhead. However, neither the  $d^C$  dependency nor the  $\frac{1}{\varepsilon^{14}}$  polylog  $(W,\frac{1}{\varepsilon})$  trade-off is desirable for realistic settings of  $d$  and  $\varepsilon$  for applications of  $k$ -clustering on sliding windows. In particular, recent results have achieved efficient summarizations, i.e., coresets, for  $k$ -median and  $k$ -means clustering in the offline setting using  $\tilde{O}\left(\frac{k}{\varepsilon^4}\log n\right)$  words of space [27, 25] when the input is from  $[\Delta]^d$  and it is known that this is near-optimal, i.e.,  $\Omega \left(\frac{k}{\varepsilon^{2 + z}}\log n\right)$  samples are necessary to form coresets for  $(k,z)$ -clustering [45] in that setting. Thus a natural question is to ask whether such near-optimal space bounds can be achieved in the sliding window model.
In this paper, we answer the question in the affirmative. That is, we give near-optimal space algorithms for  $k$ -median and  $k$ -means clustering in the sliding window model. In fact, we give more general algorithms for  $(k,z)$ -clustering in the sliding window that nearly match the space used by the offline coreset constructions of [27, 25, 26]:
Theorem 1.1. There exists an algorithm that samples  $\frac{k}{\min(\varepsilon^4, \varepsilon^{2 + z})}$  polylog  $\frac{n\Delta}{\varepsilon}$  points and with high probability, outputs a  $(1 + \varepsilon)$ -approximation to  $(k, z)$ -clustering for the Euclidean distance on  $[\Delta]^d$  in the sliding window model.
In particular, our bounds in Theorem 1.1 achieve  $\frac{k}{\varepsilon^4}$  polylog  $\frac{n\Delta}{\varepsilon}$  words of space for  $k$ -median clustering and  $k$ -means clustering, i.e.,  $z = 1$  and  $z = 2$ , respectively, matching the lower bounds of [25, 45] up to polylogarithmic factors.
Reference	Accuracy	Space	Setting
[5]	2O(1/ε)	O(k/ε4W2ε log2W)	k-median, ε ∈ (0, 1/2)
[16]	C>2	O(k3 log6W)	k-median and k-means
[34]	C>214	k polylog(W, Δ)	(k, z)-clustering
[35]	(1+ε)	(kd+dCz/ε3) polylog(W, Δ, 1/ε), C≥7	(k, z)-clustering
Our work	(1+ε)	k/min(ε4,ε2+z) polylogn(ε)	(k, z)-clustering
Table 1: Summary of  $(k,z)$ -clustering results in the sliding window model for input points in  $[\Delta]^d$  on a window of size  $W$
Moreover, our algorithm actually produces a coreset, i.e., a data structure that approximately answers the clustering cost of the underlying dataset with respect to any set of  $k$  centers, not just the optimal  $k$  centers.
Theorem 1.2. There exists an algorithm that samples  $\frac{k}{\min(\varepsilon^4, \varepsilon^{2 + z})}$  polylog  $\frac{n\Delta}{\varepsilon}$  points and with high probability, outputs a  $(1 + \varepsilon)$ -coreset to  $(k, z)$ -clustering in the sliding window model for general metrics on  $[\Delta]^d$ .
We emphasize that the guarantees of Theorem 1.2 are for general metrics on  $[\Delta]^d$ , such as  $L_{p}$  metrics. Note that in light of the properties of coresets, the guarantee of Theorem 1.1 follows from taking a coreset for  $(k,z)$ -clustering on Euclidean distances and then using an offline algorithm for  $(k,z)$ -clustering for post-processing after the data stream, i.e., see Theorem 2.4.
Along the way, we provide a construction for a  $(1 + \varepsilon)$ -online coreset for  $(k,z)$ -clustering for general metrics on  $[\Delta]^d$ . An online coreset for  $(k,z)$ -clustering is a data structure on a data stream that will not only approximately answer the clustering cost of the underlying dataset with respect to any set of  $k$  centers, but also approximately answer the clustering cost of any prefix of the data stream with respect to any set of  $k$  centers.
Theorem 1.3. There exists an algorithm that samples  $\frac{k}{\min(\varepsilon^4, \varepsilon^{2 + z})}$  polylog  $\frac{n\Delta}{\varepsilon}$  points and with high probability, outputs a  $(1 + \varepsilon)$ -online coreset for  $(k, z)$ -clustering.
We remark that Theorem 1.3 further has the attractive property that once a point is sampled into the online coreset at some point in the stream, then the point irrevocably remains in the online coreset. That is, the online coreset essentially satisfies two different definitions of online: 1) the data structure is a coreset for any prefix of the stream and 2) points sampled into the data structure will never be deleted from the data structure.
We further remark that due to leveraging the coreset construction of [27, 25, 26], we can similarly trade a factor of  $\frac{1}{\varepsilon^z}$  for a poly  $(k)$  in the guarantees of Theorem 1.1, Theorem 1.2, and Theorem 1.3.
By contrast, the lower bound by [25] states that any offline coreset construction for  $k$ -means clustering only requires  $\Omega \left(\frac{k}{\varepsilon^2}\right)$  points. This lower bound was later strengthened to  $\Omega \left(\frac{k}{\varepsilon^{2 + z}}\right)$  points by [45], for which matching upper bounds are given by [27, 25]. Thus our online coreset constructions are near-optimal in the  $k$  and  $\frac{1}{\varepsilon}$  dependencies for  $z > 1$  and nearly match the best known offline constructions for  $z = 1$ .
It is thus a natural question to ask whether our polylogarithmic overheads in Theorem 1.3 are necessary for an  $(1 + \varepsilon)$ -online coreset. We show that in fact, a logarithmic overhead is indeed necessary to maintain a  $(1 + \varepsilon)$ -online coreset.
Theorem 1.4. Let  $\varepsilon \in (0,1)$ . For sufficiently large  $n$ ,  $d$ , and  $\Delta$ , there exists a set  $X \subset [\Delta]^d$  of  $n$  points  $x_1, \ldots, x_n$  such that any  $(1 + \varepsilon)$ -online coreset for  $k$ -means clustering on  $X$  requires  $\Omega\left(\frac{k}{\varepsilon^2} \log n\right)$  points.
We emphasize that combined with existing offline coreset constructions [25, 26], Theorem 1.4 shows a separation between the problems of constructing offline coresets and online coresets. That is, the problem of maintaining a data structure that recovers coresets for all prefixes of the stream is provably harder than maintaining a coreset for an offline set of points.
1.2 Technical Overview
In this section, we give a high-level overview of our techniques. We also describe the limitations of many natural approaches.
Shortcomings of histograms and sensitivity sampling. A first attempt at clustering in the sliding window model might be to adapt the popular exponential histogram [33] and smooth histogram techniques [18]. These frameworks convert streaming algorithms to sliding window algorithms in the case that the objective function is smooth, which informally means that once a suffix of a data stream becomes a good approximation of the overall data stream, then it always remains a good approximation, regardless of the values of new elements that arrive in the stream. Unfortunately, [16] showed that the  $k$ -clustering objective function is not smooth and thus these histogram-based frameworks cannot work. Nevertheless, they gave the first constant-factor approximation by showing that the  $k$ -clustering objective function is almost-smooth using a generalized triangle inequality, which inherently loses constant factors and thus will not suffice for our goal of achieving a  $(1 + \varepsilon)$ -approximation.
Another approach might be to adapt the popular sensitivity sampling framework of coreset construction [37, 39, 10, 29]. The sensitivity sampling framework assigns a value to each point, called the sensitivity, which intuitively quantifies the "importance" of that point, and then samples each point with probability proportional to its sensitivity. [9] observed that sliding window algorithms can be achieved from online sensitivity sampling, where the importance of each point is measured against the prefix of the stream, and then running the process in reverse at each time, so that more emphasis is placed on the suffix of the sliding window. At a high level, this is the intuition taken by [34, 35], which leverage data structures that prioritize more recent elements of the data stream. However, it is not known how to achieve optimal bounds simply using sensitivity sampling, and indeed the optimal coreset constructions use slightly more nuanced sampling schemes [27, 25].
Sliding window algorithms from online coresets. Instead, we recall an observation by [9], who noted that deterministic constructions for online coresets for linear algebraic problems can be utilized to obtain sliding window algorithms for the corresponding linear algebraic problems. We first extend this observation to randomized constructions for online coresets for  $k$ -clustering problem.
The intuition is quite simple. Given an  $(1 + \varepsilon)$ -online coreset algorithm for a  $k$ -clustering problem on a data stream of length  $n$  from  $\mathbb{R}^d$  that stores  $S(n, d, k, \varepsilon, \delta)$  weights points and succeeds with probability  $1 - \delta$ , we store the  $S(n, d, k, \varepsilon', \delta')$  most recent points in the stream, where  $\varepsilon' = O\left(\frac{\varepsilon}{\log n}\right)$  and  $\delta' = \frac{\delta}{\mathrm{poly}(n)}$ . We then feed the  $S(n, d, k, \varepsilon', \delta')$  points to the online coreset construction in reverse order of their arrival. Since the online coreset preserves all costs for all prefixes of its input, then the resulting data structure will preserve all costs for all suffixes of the data stream. To extend this guarantee to the entire stream, including the sliding window, we can then use a standard merge-and-reduce framework. It thus remains to devise a  $(1 + \varepsilon)$ -online coreset construction for  $k$ -clustering with near-optimal sampling complexity.
Online coreset construction. To that end, our options are quite limited, as to the best of our knowledge, the only offline coreset constructions using  $\tilde{O}\left(\frac{k}{\varepsilon^4} \log n\right)$  words of space when the input is from  $[\Delta]^d$  are due to [27, 25]. Fortunately, although the analyses of correctness for these sampling
schemes are quite involved, the constructions themselves are quite accessible. For example, [27] first uses an  $(\alpha, \beta)$ -approximation, i.e., a clustering that achieves  $\alpha$ -approximation to the optimal cost but uses  $\beta k$  centers, to partition the underlying dataset  $X$  into disjoint concentric rings around each of the  $\beta k$  centers. These rings are then gathered into groups and it is shown that by independently sampling a fixed number of points with replacement from each of the groups suffices to achieve a  $(1 + \varepsilon)$ -coreset. Their analysis argues that the contribution of each of the groups toward the overall  $k$ -clustering cost is preserved through an expectation and variance bounding argument, and then taking a sophisticated union bound over a net over the set of possible centers. Thus their argument still holds when each point of the dataset is independently sampled by the data structure with probability proportional to the probability it would have been sampled by the group. Moreover, independently sampling each point with a higher probability can only decrease the variance, so that correctness is retained, though we must also upper bound the number of sampled points. Crucially, independently sampling each point can be implemented in the online setting and the probability of correctness can be boosted to union bound over all times in the stream, which facilitates the construction of our  $(1 + \varepsilon)$ -online coreset, given an  $(\alpha, \beta)$ -approximation.
Consistent  $(\alpha, \beta)$ -approximation. It seemingly remains to find  $(\alpha, \beta)$ -approximations for  $k$ -clustering at all times in the stream. A natural approach would be to use an algorithm that achieves a  $(\alpha, \beta)$ -approximation at a certain time in the stream with constant probability, e.g., [59], boost the probability of success to  $1 - \frac{1}{\mathrm{poly}(n)}$ , and the union bound to argue correctness over all times in the stream. However, a subtle pitfall here is that the rings and groups in the offline coreset construction of [27] are with respect to a specific  $(\alpha, \beta)$ -approximation. Hence their analysis would no longer hold if a point  $x_{t}$  was assigned to cluster  $i_{1}$  at time  $t$  when the sampling process occurs but then assigned to cluster  $i_{2}$  at the end of the stream. Therefore, we require a consistent  $(\alpha, \beta)$ -approximation, so that once the algorithm assigns point  $x_{t}$  to cluster  $i$ , then the point  $x_{t}$  will always remain in cluster  $i$  even if a newer and closer center is subsequently opened later in the stream. To that end, we invoke a result of [34] that analyzes the popular Meyerson online facility location algorithm, along with a standard guess-and-double approach for estimating the input parameter to the Meyerson subroutine.
Lower bound. The intuition for our lower bound that any  $(1 + \varepsilon)$ -online coreset for  $(k,z)$ -clustering requires  $\Omega\left(\frac{k}{\varepsilon^2}\right)$  is somewhat straightforward and in a black-box manner. We first observe that [25] showed the existence of a set  $X$  of  $\Omega\left(\frac{k}{\varepsilon^2}\right)$  unit vectors in  $\mathbb{R}^d$  such that any coreset with  $o\left(\frac{k}{\varepsilon^2}\right)$  samples provably cannot accurately estimate the  $(k,z)$ -clustering cost for a set  $C$  of  $k$  unit vectors.
Since an online  $(1 + \varepsilon)$ -coreset must answer queries on all prefixes of the stream, we embed  $\Omega (\log n)$  instances of  $X$ . We first increase the dimension by a log  $n$  factor so that each of these instances can have disjoint support. We then give each of the instances increasingly exponential weight to force the data structure to sample  $\Omega \left(\frac{k}{\varepsilon^2}\right)$  points for each instance. Specifically, we insert  $\tau^i$  copies of the  $i$ -th instance of  $X$ , where  $\tau > 1$  is some constant. Because the weight of the  $i$ -th instance is substantially greater than the sum of the weights of all previous instances, then any  $(1 + \varepsilon)$ -online coreset must essentially be a  $(1 + \varepsilon)$ -offline coreset for the  $i$ -th instance, thus requiring  $\Omega \left(\frac{k}{\varepsilon^2}\right)$  points for the  $i$ -th instance. This reasoning extends to all  $\Omega (\log n)$  instances, thus showing that any online  $(1 + \varepsilon)$ -coreset requires  $\Omega \left(\frac{k}{\varepsilon^2} \log n\right)$  points.
In this section, we describe our sliding window algorithm for  $k$ -clustering. We first overview the construction of an online  $(1 + \varepsilon)$  coreset for  $(k,z)$ -clustering under general discrete metrics. We then describe how our online coreset construction for  $(k,z)$ -clustering on general discrete metric spaces can be used to achieve near-optimal space algorithms for  $(k,z)$ -clustering in the sliding window model.
Online  $(1 + \varepsilon)$ -coreset. We first recall the following properties from the Meyerson sketch, which we formally introduce in Appendix A.
Theorem 2.1. [8] Given an input stream  $x_{1},\ldots ,x_{n}\in \mathbb{R}^{d}$  defining a set  $X\subset [\Delta ]^d$ , there exists an online algorithm MULTMEYERSON that with probability at least  $1 - \frac{1}{\mathrm{poly}(n)}$ :
(1) on the arrival of each point  $x_{i}$ , assigns  $x_{i}$  to a center in  $C$  through a mapping  $\pi : X \to C$ , where  $C$  contains at most  $O\left(2^{2z}k\log n\log \Delta\right)$  centers
(2)  $\sum_{x\in X}\| x_i - \pi (x_i)\| _2^z\leq 2^{z + 7}\operatorname{Cost}_{|S|\leq k}(X,S)$
(3) MULTMEYERSON uses  $O\left(2^z k \log^3 (nd\Delta)\right)$  words of space
We also use the following notation, adapted from [27] to the online setting.
Let  $\mathcal{A}$  be an  $(\alpha, \beta)$ -approximation for a  $k$ -means clustering on an input set  $X \subseteq [\Delta]^d$  and let  $C_1, \ldots, C_{\beta k}$  be the clusters of  $X$  induced by  $\mathcal{A}$ . Suppose the points of  $X$  arrive in a data stream  $S$ . For a fixed  $\varepsilon > 0$ , define the following notions of rings and groups: