Coresets for Time Series Clustering

1 Introduction . A multivariate time series dataset , represented as X = { Xi = ( xi1 , . . . , xi , Ti ) ⊂ RTi×d | i ∈ [ N ] } , where N is the number of entities , Ti is the number of time periods corresponding to entity i and d is the number of features , tracks features of a cross-section of entities longitudinally over time . Such data is also referred to as panel data [ 8 ] and has seen rapid growth due to proliferation of sensors , IOT and wearables that facilitate real time measurement of various features associated with entities and the rapid drop in storage costs . Specific examples of time series data include biomedical measurements ( e.g. , blood pressure and electrocardiogram ) , epidemiology and diffusion through social networks , weather , consumer search , content browsing and purchase behaviors , mobility through cell phone and GPS locations , stock prices and exchange rates in finance [ 2 ] . Computational problems of interest with time series data include pattern discovery [ 49 ] , regression/prediction [ 39 ] , forecasting [ 9 ] , and clustering [ 47 , 31 , 2 ] which arises in applications such as anomaly detection [ 46 ] . Though clustering is a central and well-studied problem in unsupervised learning , most standard treatments of clustering tend to be on static data with one observation per entity [ 35 , 11 ] . Time series clustering introduces a number of additional challenges and is an active area of research , with many types of clustering methods proposed [ 15 ] ; direct methods on raw data , indirect methods based on features generated from the raw data , and model based clustering where the data are assumed to be generated from a model . For surveys of time series clustering , see [ 47 , 2 ] . We focus on model-based time series clustering using a likelihood framework that naturally extends clustering with static data [ 11 ] to time series data [ 47 ] , where each multivariate time series is generated by one of k different parametrically specified models . For a survey of the importance of this subliterature and its various real-world applications , see [ 30 ] . A prevalent approach is to assume a finite mixture of data generating models , with Gaussian mixture models being a common specification [ 50 , 53 ] . Roughly , the problem is to partition X probabilistically into k clusters , grouping those series 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . generated by the same time series model into one cluster . Generically , the model-based k-clustering problem can be formulated as : arg maxα , θ ( 1 ) , ... , θ ( k ) ∑ i∈ [ N ] ln ∑ l∈ [ k ] αl ·pi ( X|θ ( l ) ) , where αl is the probability of a time series belonging to cluster l and pi ( X|θ ( l ) ) is the likelihood of the data of entity i , given that the data is generated from model l ( with parameters θ ( l ) ) . Temporal relationships in time series are commonly modeled as autocorrelations ( AR ) /moving averages ( MA ) [ 20 , 61 ] that account for correlations across observations , and hidden Markov models ( HMM ) where the underlying data generating process is allowed to switch periodically [ 21 , 62 ] . This paper focuses on the case when the data generation model for each segment l ∈ [ k ] is a multivariate Gaussian with autocorrelations . More formally , the generative model for each cluster is from a Gaussian mixture : xit : = µ ( l ) + eit , where eit = N ( 0 , Σ ( l ) ) + Λ ( l ) ei , t−1 , where N ( 0 , Σ ( l ) ) captures the mixture of Gaussian distributions from which entity level observations are drawn , and Λ captures the correlation between two successive observations through an AR ( 1 ) process [ 32 , 44 ] . Overall , this model can be represented with cluster level model parameters θ ( l ) = { µ ( l ) , Σ ( l ) , Λ ( l ) } and cluster probability αl . Time series datasets are also much larger than static datasets [ 47 ] . For instance , as noted in [ 42 ] , ECG ( electrocardiogram ) data requires 1 GB/hour , a typical weblog requires 5 GB/week . Such large sizes lead to high storage costs and also make it necessary to work with a subset of the data to conduct the clustering analysis to be computationally practical . Further , long time series data on entities also entails significant data security and privacy risks around the entity as it requires longer histories of entity behaviors to be stored and maintained [ 25 , 41 ] . Thus , there is significant value for techniques that can produce approximately similar clustering results with smaller samples as with complete data . Coresets have emerged as an effective tool to both speed up and reduce data storage by taking into account the objective and carefully sampling from the full dataset in a way that any algorithm run on the sampled set returns an approximate solution for the original objective with guaranteed bounds on approximation error [ 34 ] . Coresets have been developed for both unsupervised clustering ( e.g. , k-means , Gaussian mixture models ) and supervised machine learning ( e.g. , regression ) methods for static data ; for surveys on coresets for static data , see [ 7 , 22 ] . A natural question is whether coresets can be also useful for addressing the time series clustering problem . Recently , there is evidence that small coresets can be efficiently constructed for time series data , albeit for regressions with time series ( panel ) data [ 37 ] . However , as we explain in Section 2 , there are challenges in extending coresets for static data clustering and time series ( panel ) data regressions for time series clustering . Our contributions . We study coresets for a general class of time series clustering in which all entities are drawn from Gaussian mixtures with autocorrelations ( as described above ; see Problem 3.1 ) . We first present a definition of coresets for the log-likelihood k-clustering objective for this mixture model . One issue is that this objective can not be decomposed into a summation of entity-time objectives due to the interaction of the ln term from log-likelihood and the exp term from Gaussians ( Problem 3.1 ) . To estimate this objective using a coreset , we introduce an analogous clustering objective on a subset of data ( Definition 3.3 ) . Our main result is an algorithm to construct coresets for this objective for the aforementioned mixture model ( Theorem 4.3 ) . Our algorithm is efficient ( linear on both N and ∑ i∈ [ N ] Ti ) and , assuming that the condition number of the covariance matrices of the Gaussians ( Σ ( l ) ) are bounded and the eigenvalues of the autocorrelation matrices ( Λ ( l ) ) are bounded away from 1 ( Assumption 4.1 ) , the size of our coreset is independent of the number of entities N and the number of observations Tis ; it only polynomially depends on k , d and 1/ε , where ε is the error parameter ( Theorem 4.3 ) . Our coreset construction ( Algorithm 1 ) leverages the Feldman-Langberg ( FL ) framework [ 27 ] . Due to multiple observations for each entity and associated autocorrelations , the objective is non-convex and quite complicated in contrast to the static setting where the objective only contains one observation drawn from Gaussian mixtures for each entity . Thus , bounding the “ sensitivity ” ( Lemma 4.6 ) of each entity , which is a key step in coreset construction using the FL-framework , becomes challenging . We handle this technical difficulty by 1 ) upper-bounding the maximum effect of covariances and autocorrelations by using the observation that the gap between the clustering objectives with and without Σ and Λ is always constant , and 2 ) reducing the Gaussian mixture time series clustering problem to a certain k-means clustering problem whose total sensitivity is upper bounded . Empirically , we assess the performance of our coreset on synthetic data for a three cluster problem ( Section 5 ) , and compare the performance with two benchmarks : uniform sampling and a static coreset benchmark ( LFKF [ 48 ] ) in which we do not consider autocorrelations and regard time series data as static . We find that our coreset performs better relative to uniform sampling and LFKF on both data storage and computation speed for a range of accuracy guarantees : To achieve a similar fit with the full data , our coreset needs fewer entity-time observations ( < 40 % ) ; the computation time for a given level of accuracy reduces by a 3x-22x factor when compared to uniform sampling and LFKF . Moreover , our coreset speeds up the computation time relative to the full data by 14x-171x . We note that the performance advantage of our coreset is greater when there is more time series data relative to entities ( Ti N ) . 2 Related work . Time series or panel data analysis are important in many fields– biological [ 18 ] , engineering [ 3 ] , economics [ 51 ] and the social sciences [ 59 ] . Clustering has been a central problem in unsupervised learning and also has a long history of use across many fields ; for a historical overview of applications across fields , see [ 12 ] . While most standard treatments of clustering in machine learning tend to be on static data [ 35 , 11 ] , there is by now a significant and rapidly growing literature on time series clustering ; recent surveys include [ 47 , 2 ] . While there are many coresets for algorithms on static data ( including for clustering ) , there has been little work on coresets on time series data ( see below ) . Coresets for clustering . Coresets for various clustering objectives on static data have been well studied , including k-median [ 34 , 17 , 55 , 38 , 19 ] , k-means [ 34 , 17 , 28 , 13 , 10 , 55 , 38 , 19 ] , and k-center [ 1 , 33 ] . For surveys of static coresets , see [ 52 , 23 ] . [ 48 , 26 ] present coreset constructions for clustering problems on static data generated using Gaussian mixture models . [ 48 ] constructs a coreset of size poly ( k , d , 1/ε ) under the same boundedness assumption as ours . [ 26 ] remove this assumption and construct a coreset whose size additionally depends on poly logN , but require another assumption that all xis are integral and in a bounded range . In contrast , we consider a generalized problem where each entity has multiple observations over time and accounts for autocorrelations across successive observations . The new idea is to reduce the GMM time series clustering problem ( Problem 3.1 ) to a k-means clustering problem whose points are linear combinations of those in Xi ( Lemma A.12 ) , and show that the effect of autocorrelation parameters Λ on the entity objectives can be upper bounded ( Lemma A.11 ) . Regression with time series data . [ 37 ] construct coresets for a regression problem , called GLSEk , for time-series data including autocorrelations and a k-partition of entities . This is a supervised learning task that includes an additional label yit for each entity-time pair . It aims to find a kpartition of entities and additional regression vectors β ( l ) ∈ Rd for each partition l ∈ [ k ] such that a certain linear combination of terms ‖yit − ( β ( l ) ) > xit‖22 is minimized . In contrast , our problem is an unsupervised learning task , estimates points xit by Gaussian means µ instead of β , and includes covariances Σ between different features . The definitions of coresets in [ 37 ] and ours consist of entity-time pairs but our coresets need to include multiple weight functions ( w for entities and w ( i ) for time periods of selected entity i ; see Definition 3.3 ) instead of one weight function as in the regression problem . This is to deal with the interaction of the ln term from the log-likelihood and the exp term from the Gaussians . [ 37 ] ’ s result relies on an assumption that the regression objective of each entity is in a bounded range over the parameter space . This may not be satisfied in our setting since the clustering objective of an entity may be unbounded when all Gaussian means µ ( l ) are far away from observations of this entity . To bypass this , we reduce our problem to a k-means clustering problem ( Definition 4.2 ) , whose total sensitivity is provably O ( k ) ( Lemma A.12 ) , by upper bounding the affect of covariance and autocorrelation parameters on the clustering objectives of entities ( Lemma A.11 ) under the assumption that the condition number of covariance matrix is bounded ( Assumption 4.1 ) , which provides an upper bound for the total sensitivity of entities on our problem ( Lemma 4.6 ) . Another related direction is to consider coreset construction for k-segmentation with time series data [ 54 , 29 ] , which aims to estimate the trace of an entity by a k-piecewise linear function ( ksegment ) . Note that the case of k = 1 is equivalent to the linear regression problem . [ 54 , 29 ] proposed efficient coreset construction algorithms which accelerate the computation time of the k-segmentation problem . The main difference from our setting is that they consider a single entity observed at multiple time periods , and the objective is an additive function on all time periods . This enables them to relate the k-segmentation problem to the static setting . It is interesting to investigate coreset construction for more optimization problems with time series data .

Summary: The authors propose a novel coreset construction framework for the problem of clustering multivariate time series. To my knowledge, this is the first construction of coresets in this setting. The framework, as well as the proof scheme, is close to the work in [45], in which Lucic et al. studied the problem of coreset constructions for the MLE task under the GMM model.   Here, the authors add to the GMM model a temporality and thus an additional autocorrelation in the underlying model. The proof's scheme is similar to that of [45] in the sense that they also reduce the GMM time series clustering problem to a $k$-means clustering problem (and then use more classical results for $k$-means from the state-of-the-art).