Coresets for Time Series Clustering

1 Introduction . A multivariate time series dataset , represented as X = { Xi = ( xi1 , . . . , xi , Ti ) ⊂ RTi×d | i ∈ [ N ] } , where N is the number of entities , Ti is the number of time periods corresponding to entity i and d is the number of features , tracks features of a cross-section of entities longitudinally over time . Such data is also referred to as panel data [ 8 ] and has seen rapid growth due to proliferation of sensors , IOT and wearables that facilitate real time measurement of various features associated with entities and the rapid drop in storage costs . Specific examples of time series data include biomedical measurements ( e.g. , blood pressure and electrocardiogram ) , epidemiology and diffusion through social networks , weather , consumer search , content browsing and purchase behaviors , mobility through cell phone and GPS locations , stock prices and exchange rates in finance [ 2 ] . Computational problems of interest with time series data include pattern discovery [ 49 ] , regression/prediction [ 39 ] , forecasting [ 9 ] , and clustering [ 47 , 31 , 2 ] which arises in applications such as anomaly detection [ 46 ] . Though clustering is a central and well-studied problem in unsupervised learning , most standard treatments of clustering tend to be on static data with one observation per entity [ 35 , 11 ] . Time series clustering introduces a number of additional challenges and is an active area of research , with many types of clustering methods proposed [ 15 ] ; direct methods on raw data , indirect methods based on features generated from the raw data , and model based clustering where the data are assumed to be generated from a model . For surveys of time series clustering , see [ 47 , 2 ] . We focus on model-based time series clustering using a likelihood framework that naturally extends clustering with static data [ 11 ] to time series data [ 47 ] , where each multivariate time series is generated by one of k different parametrically specified models . For a survey of the importance of this subliterature and its various real-world applications , see [ 30 ] . A prevalent approach is to assume a finite mixture of data generating models , with Gaussian mixture models being a common specification [ 50 , 53 ] . Roughly , the problem is to partition X probabilistically into k clusters , grouping those series 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . generated by the same time series model into one cluster . Generically , the model-based k-clustering problem can be formulated as : arg maxα , θ ( 1 ) , ... , θ ( k ) ∑ i∈ [ N ] ln ∑ l∈ [ k ] αl ·pi ( X|θ ( l ) ) , where αl is the probability of a time series belonging to cluster l and pi ( X|θ ( l ) ) is the likelihood of the data of entity i , given that the data is generated from model l ( with parameters θ ( l ) ) . Temporal relationships in time series are commonly modeled as autocorrelations ( AR ) /moving averages ( MA ) [ 20 , 61 ] that account for correlations across observations , and hidden Markov models ( HMM ) where the underlying data generating process is allowed to switch periodically [ 21 , 62 ] . This paper focuses on the case when the data generation model for each segment l ∈ [ k ] is a multivariate Gaussian with autocorrelations . More formally , the generative model for each cluster is from a Gaussian mixture : xit : = µ ( l ) + eit , where eit = N ( 0 , Σ ( l ) ) + Λ ( l ) ei , t−1 , where N ( 0 , Σ ( l ) ) captures the mixture of Gaussian distributions from which entity level observations are drawn , and Λ captures the correlation between two successive observations through an AR ( 1 ) process [ 32 , 44 ] . Overall , this model can be represented with cluster level model parameters θ ( l ) = { µ ( l ) , Σ ( l ) , Λ ( l ) } and cluster probability αl . Time series datasets are also much larger than static datasets [ 47 ] . For instance , as noted in [ 42 ] , ECG ( electrocardiogram ) data requires 1 GB/hour , a typical weblog requires 5 GB/week . Such large sizes lead to high storage costs and also make it necessary to work with a subset of the data to conduct the clustering analysis to be computationally practical . Further , long time series data on entities also entails significant data security and privacy risks around the entity as it requires longer histories of entity behaviors to be stored and maintained [ 25 , 41 ] . Thus , there is significant value for techniques that can produce approximately similar clustering results with smaller samples as with complete data . Coresets have emerged as an effective tool to both speed up and reduce data storage by taking into account the objective and carefully sampling from the full dataset in a way that any algorithm run on the sampled set returns an approximate solution for the original objective with guaranteed bounds on approximation error [ 34 ] . Coresets have been developed for both unsupervised clustering ( e.g. , k-means , Gaussian mixture models ) and supervised machine learning ( e.g. , regression ) methods for static data ; for surveys on coresets for static data , see [ 7 , 22 ] . A natural question is whether coresets can be also useful for addressing the time series clustering problem . Recently , there is evidence that small coresets can be efficiently constructed for time series data , albeit for regressions with time series ( panel ) data [ 37 ] . However , as we explain in Section 2 , there are challenges in extending coresets for static data clustering and time series ( panel ) data regressions for time series clustering . Our contributions . We study coresets for a general class of time series clustering in which all entities are drawn from Gaussian mixtures with autocorrelations ( as described above ; see Problem 3.1 ) . We first present a definition of coresets for the log-likelihood k-clustering objective for this mixture model . One issue is that this objective can not be decomposed into a summation of entity-time objectives due to the interaction of the ln term from log-likelihood and the exp term from Gaussians ( Problem 3.1 ) . To estimate this objective using a coreset , we introduce an analogous clustering objective on a subset of data ( Definition 3.3 ) . Our main result is an algorithm to construct coresets for this objective for the aforementioned mixture model ( Theorem 4.3 ) . Our algorithm is efficient ( linear on both N and ∑ i∈ [ N ] Ti ) and , assuming that the condition number of the covariance matrices of the Gaussians ( Σ ( l ) ) are bounded and the eigenvalues of the autocorrelation matrices ( Λ ( l ) ) are bounded away from 1 ( Assumption 4.1 ) , the size of our coreset is independent of the number of entities N and the number of observations Tis ; it only polynomially depends on k , d and 1/ε , where ε is the error parameter ( Theorem 4.3 ) . Our coreset construction ( Algorithm 1 ) leverages the Feldman-Langberg ( FL ) framework [ 27 ] . Due to multiple observations for each entity and associated autocorrelations , the objective is non-convex and quite complicated in contrast to the static setting where the objective only contains one observation drawn from Gaussian mixtures for each entity . Thus , bounding the “ sensitivity ” ( Lemma 4.6 ) of each entity , which is a key step in coreset construction using the FL-framework , becomes challenging . We handle this technical difficulty by 1 ) upper-bounding the maximum effect of covariances and autocorrelations by using the observation that the gap between the clustering objectives with and without Σ and Λ is always constant , and 2 ) reducing the Gaussian mixture time series clustering problem to a certain k-means clustering problem whose total sensitivity is upper bounded . Empirically , we assess the performance of our coreset on synthetic data for a three cluster problem ( Section 5 ) , and compare the performance with two benchmarks : uniform sampling and a static coreset benchmark ( LFKF [ 48 ] ) in which we do not consider autocorrelations and regard time series data as static . We find that our coreset performs better relative to uniform sampling and LFKF on both data storage and computation speed for a range of accuracy guarantees : To achieve a similar fit with the full data , our coreset needs fewer entity-time observations ( < 40 % ) ; the computation time for a given level of accuracy reduces by a 3x-22x factor when compared to uniform sampling and LFKF . Moreover , our coreset speeds up the computation time relative to the full data by 14x-171x . We note that the performance advantage of our coreset is greater when there is more time series data relative to entities ( Ti N ) . 2 Related work . Time series or panel data analysis are important in many fields– biological [ 18 ] , engineering [ 3 ] , economics [ 51 ] and the social sciences [ 59 ] . Clustering has been a central problem in unsupervised learning and also has a long history of use across many fields ; for a historical overview of applications across fields , see [ 12 ] . While most standard treatments of clustering in machine learning tend to be on static data [ 35 , 11 ] , there is by now a significant and rapidly growing literature on time series clustering ; recent surveys include [ 47 , 2 ] . While there are many coresets for algorithms on static data ( including for clustering ) , there has been little work on coresets on time series data ( see below ) . Coresets for clustering . Coresets for various clustering objectives on static data have been well studied , including k-median [ 34 , 17 , 55 , 38 , 19 ] , k-means [ 34 , 17 , 28 , 13 , 10 , 55 , 38 , 19 ] , and k-center [ 1 , 33 ] . For surveys of static coresets , see [ 52 , 23 ] . [ 48 , 26 ] present coreset constructions for clustering problems on static data generated using Gaussian mixture models . [ 48 ] constructs a coreset of size poly ( k , d , 1/ε ) under the same boundedness assumption as ours . [ 26 ] remove this assumption and construct a coreset whose size additionally depends on poly logN , but require another assumption that all xis are integral and in a bounded range . In contrast , we consider a generalized problem where each entity has multiple observations over time and accounts for autocorrelations across successive observations . The new idea is to reduce the GMM time series clustering problem ( Problem 3.1 ) to a k-means clustering problem whose points are linear combinations of those in Xi ( Lemma A.12 ) , and show that the effect of autocorrelation parameters Λ on the entity objectives can be upper bounded ( Lemma A.11 ) . Regression with time series data . [ 37 ] construct coresets for a regression problem , called GLSEk , for time-series data including autocorrelations and a k-partition of entities . This is a supervised learning task that includes an additional label yit for each entity-time pair . It aims to find a kpartition of entities and additional regression vectors β ( l ) ∈ Rd for each partition l ∈ [ k ] such that a certain linear combination of terms ‖yit − ( β ( l ) ) > xit‖22 is minimized . In contrast , our problem is an unsupervised learning task , estimates points xit by Gaussian means µ instead of β , and includes covariances Σ between different features . The definitions of coresets in [ 37 ] and ours consist of entity-time pairs but our coresets need to include multiple weight functions ( w for entities and w ( i ) for time periods of selected entity i ; see Definition 3.3 ) instead of one weight function as in the regression problem . This is to deal with the interaction of the ln term from the log-likelihood and the exp term from the Gaussians . [ 37 ] ’ s result relies on an assumption that the regression objective of each entity is in a bounded range over the parameter space . This may not be satisfied in our setting since the clustering objective of an entity may be unbounded when all Gaussian means µ ( l ) are far away from observations of this entity . To bypass this , we reduce our problem to a k-means clustering problem ( Definition 4.2 ) , whose total sensitivity is provably O ( k ) ( Lemma A.12 ) , by upper bounding the affect of covariance and autocorrelation parameters on the clustering objectives of entities ( Lemma A.11 ) under the assumption that the condition number of covariance matrix is bounded ( Assumption 4.1 ) , which provides an upper bound for the total sensitivity of entities on our problem ( Lemma 4.6 ) . Another related direction is to consider coreset construction for k-segmentation with time series data [ 54 , 29 ] , which aims to estimate the trace of an entity by a k-piecewise linear function ( ksegment ) . Note that the case of k = 1 is equivalent to the linear regression problem . [ 54 , 29 ] proposed efficient coreset construction algorithms which accelerate the computation time of the k-segmentation problem . The main difference from our setting is that they consider a single entity observed at multiple time periods , and the objective is an additive function on all time periods . This enables them to relate the k-segmentation problem to the static setting . It is interesting to investigate coreset construction for more optimization problems with time series data .

Summary: The paper considers the problem of clustering time series data. The authors provide a coreset for this problem, under some assumptions. The coreset size depends polynomially on the number of clusters k, the dimension d, and 1/\varepsilon and the parameters related to the assumption on the data, and requires near-linear time to compute.  The authors validate their coreset empirically on synthetic data, as compared to a uniform random sample. They show that the coreset: (i) achieves better results can the random sample, and (i) can help accelerate the computational time, relative to the full data, by 1-2 orders of magnitude, with only a small compromise in the accuracy.