Latent Space Semi-Supervised Time Series Data Clustering

1 INTRODUCTION . Time series data can be defined as any data which contains multiple sequentially ordered measurements . Real world examples of time series data are abundant throughout many domains , including finance , weather , and medicine . One common learning task is to partition a set of time series into clusters . This unsupervised learning task can be used to learn more about the underlying structure of a dataset , without the need for a supervised learning objective or ground-truth labels . Clustering time series data is a challenging problem because time series data may be high-dimensional , and is not always segmented cleanly , leading to issues with alignment and noise . The most basic methods for time series clustering apply general clustering algorithms to raw time series data . Familiar clustering algorithms like hierarchical clustering or k-Means clustering algorithms may be applied using Euclidean Distance ( ED ) for comparisons . Although ED can perform well in some cases , it is susceptible to noise and temporal shifting . The improved Dynamic Time Warping ( DTW ) ( Berndt & Clifford , 1994 ) metric provides invariance to temporal shifts , but is expensive to compute for clustering tasks . A more scalable alternative to DTW exists in k-Shape , a measure based on the shape-based distance ( SBD ) metric for comparing whole time series ( Paparrizos & Gravano , 2017 ) . Shapelet-based approaches such as Unsupervised Shapelets ( Zakaria et al. , 2012 ) can mitigate issues with shifting and noise but are limited to extracting a single pattern/feature from each time series . One alternative approach for clustering time series data is to apply dimensionality reduction through the use of an autoencoder . Autoencoders are capable of learning low-dimensional projections of high-dimensional data . Both LSTM and convolutional autoencoders have been shown to be successful at learning latent representations of time series data . These models can extract a large number of features at each time step . After training an autoencoder model , the learned low-dimensional latent representation can then be fed to an arbitrary clustering algorithm to perform the clustering task . Because autoencoder models reduce the dimensionality of the data , they naturally avoid issues with noise , and provide a level of invariance against temporal shifting . Recently , the field of semi-supervised learning has shown great success at boosting the performance of unsupervised models using small amounts of labeled data . Dau et al . ( 2016 ) proposes a solution for semi-supervised clustering using DTW . However , this solution is still based on DTW , and as such suffers from scalability issues . He et al . ( 2019 ) proposes a constraint-propagation approach for semi-supervised clustering , which may ( but is not required to ) be used in conjunction with DTW . However , this solution still performs time series comparisons within the raw data space , which may cause issues with scalability for large datasets . In this paper , we present a semi-supervised deep learning model based on a convolutional autoencoder ( CAE ) , which may be used to perform clustering on time series datasets . We also present new semi-supervised learning objectives , adapted from well-known internal clustering metrics , which can significantly improve clustering performance when provided with a small number of labeled time series . We perform experiments to show that our semi-supervised model can improve performance relative to an unsupervised model when applied for clustering tasks . We also implement a lightly modified batch-based version of the semi-supervised learning solution shown presented in Ren et al . ( 2018 ) , and show that our proposed solutions are competitive . In the best case , our model semi-supervised model shows a best-case improvement in ARI of 140 % over an unsupervised CAE model when applying k-Means clustering , and a best-case improvement of 44 % over a similar model . In the remainder of this paper , Section 2 reviews the related work on time series clustering , Section 3 presents our proposed method for semi-supervised time series clustering , and in Section 4.1 we discuss our experimental methodology and present our experimental results . Finally , Section 5 details our conclusions and avenues for future research . 2 RELATED WORK . One of the most common ways to perform time series clustering is to apply the k-Means algorithm . By default , k-Means uses Euclidean Distance ( ED ) . ED is efficient to calculate , and in many cases shows good results ( Ding et al. , 2008 ) . However , ED comparison will fail when two similar time series are shifted temporally relative to one another . Additionally , ED comparisons are sensitive to noisy data . The Dynamic Time Warping ( DTW ) metric ( Berndt & Clifford , 1994 ) improves on ED by computing a warping path between a pair of time series . This approach solves issues with temporal shifting , but requiresO ( N2 ) time to compute for two time series of lengthN . Recent work has provided bounds for this computation ( Keogh & Ratanamahatana , 2005 ) , ( Lemire , 2009 ) , but the scalability of DTW remains an issue for large datasets and long time series . The k-Shape algorithm ( Paparrizos & Gravano , 2015 ) is a scalable and performant alternative to DTW , and offers similar performance to DTW at a lower computational cost . The Unsupervised Shapelets ( Zakaria et al. , 2012 ) clustering method operates by forming clusters around common subsequnces extracted from the data . This approach provides invariance against shifts since the shapelet may appear anywhere within each time series , and also provides some invariance against noise or outliers within the data , since elementwise comparisons only occur between shapelets , rather than the full time series . In this regard , the UShapelet algorithm has some advantages over DTW and k-Shape . However , this method is constrained to extracting a single shapelet/feature from each time series . Recently , semi-supervised learning has shown the benefit of augmenting a large unlabeled dataset with a small amount of labeled data . There is some existing work for applying semi-supervised learning to time series clustering . The semi-supervised time series clustering solution presented in Dau et al . ( 2016 ) proposes a modified version of DTW , which operates in a semi-supervised manner using supervised constraints . However , this method still relies on performing DTW comparison within the original data space , and as such is not a scalable solution for large datasets or long time series . Another methodology for semi-supervised time series clustering is He et al . ( 2019 ) which is a graph-based approach using supervised examples to generate positive and negative constraints between points . This approach does not rely on DTW , but the algorithm still performs comparisons in the original data space , which can be problematic as the length of the time series grows . Popular deep learning frameworks such as LSTMs and CNNs may also be applied to time series data . Both LSTM and CNN networks may be arranged as autoencoders , allowing for unsupervised feature learning for clustering , compression , or anomaly detection tasks . Holden et al . ( 2015 ) use a Convolutional Autoencoder ( CAE ) model to learn a featurized representation of gait data . Autoencoder architectures may also be applied for anomaly detection , as is shown in Bao et al . ( 2017 ) , where the authors use autoencoders for anomaly detection . Performing comparisons on embedded samples avoids many of the issues of direct pairwise comparisons . Since autoencoders reduce di- mensionality of the data , distance comparisons will be performed in a lower-dimensional space than the original data , which improves scalability . Embedding the data before comparison also reduces sensitivity to noise , since raw data are not compared directly . One work that takes advantage of this approach is McConville et al . ( 2019 ) , which learns a manifold for clustering on the trained latent space of an autoencoder . Recent work in the generative model field has produced the Variational Autoencoder ( VAE ) architecture ( Kingma & Welling , 2013 ) , which learns probability distributions for latent features within the data and may also be applied for clustering as in Dilokthanakul et al . ( 2016 ) . This architecture has also been successfully applied in Fortuin et al . ( 2018 ) , which uses a VAE based method to learn feature representations of time series . We base our model on a CAE , as the CAE architecture is simple and well-known , and provides a good platform for evaluating the performance of our proposed objective functions . 3 SEMI-SUPERVISED LEARNING OF LATENT SPACE . Our proposed framework is a semi-supervised CAE model , which combines an unsupervised reconstruction loss Lrc with a semi-supervised loss Lsup , as shown in Figure 3.1 . We show how an existing semi-supervised learning objective function may be adapted to fit into our model , and also propose two new objective functions based on well-known internal clustering metrics . We focus on optimizing the autoencoder ’ s latent space for distance-based clustering algorithms , and specifically perform our experimentation using the k-Means algorithm . The spherical , centroid-based clusters generated by k-Means are a good fit for the proposed semi-supervised losses , which encourage each cluster to converge around a single high-density point . 3.1 CAE MODEL ARCHITECTURE . We base our model off of a two-layer Convolutional Autoencoder ( CAE ) architecture . The CAE uses two 1-D convolutional layers to featurize the time series . The filter widths for both layers are calculated at runtime from the length of the time series . Each layer of the encoder contains a 1D convolution operation , followed by a non-linear activation . These layers are paired with transpose convolution layers in the decoder . After each convolution layer in the encoder , we also apply a max-pooling operation to further reduce the length of the featurized sequence . Each max-pooling operation is “ reversed “ using a nearest-neighbor upsampling operation in the decoder . Alternatively , large strides in the convolutional layer may be used instead of pooling operations . This accomplishes the same goal of reducing the length of the featurized sequence , but does not require any up-sampling operations in the decoder , since the transpose convolution operation with a stride will perform the upsampling . We found that the max-pooling and large stride methods produced similar results in practice . 3.2 SEMI-SUPERVISED LOSS FUNCTIONS . 3.2.1 PROTOTYPE LOSS . Snell et al . ( 2017 ) present a system for performing few-shot learning , where a small number of labeled examples for each class in the dataset are embedded using an encoder network . These points are divided into two sets , query examples and support examples . In the latent space , the support embeddings are averaged to determine a centroid or ” prototype ” for each class . Training occurs by measuring the distance from each query example to the support centroid for the class . Each query point is labeled with the class of the closest support prototype . The probability of a query point i belonging to Class J is calculated as : pij = e−d ( i , Cj ) ∑ j′ e −d ( i , Cj′ ) ( 1 ) In Equation 1 , −d ( i , j ) is the negative distance between query point i and the support prototype for Class J . Any distance metric may be used for d , but for the remainder of this paper , we treat d as the squared euclidean distance . Training proceeds by minimizing the cross entropy between the labels for the query points and the probability distribution p. The objective of this approach is to learn a latent space where similarities between examples are quantifiable using a distance metric like Euclidean distance . Examples from the same class will be close to one another in the embedded space , while examples from separate classes will have larger distances . In addition to the few-shot learning scenario , a later work by Ren et al . ( 2018 ) demonstrates that the prototypical network objective can be modified to support semi-supervised few-shot classification as well . The first objective function proposed in Section 3.1.1 of Ren et al . ( 2018 ) is calculated in a two-stage process . In the first stage , the labeled data prototypes are calculated as the centroid of the Nj embedded labeled data points h ( x ) for each class J , as shown in Equation 2a . The second stage assigns each unlabeled point a soft cluster weighting , based on the distances to each of the calculated prototypes Cj , shown in Equation 2b . Finally , the original prototypes are updated to include the unlabeled data using the soft cluster assignments from Equation 2b . In Equation 2c , the final prototype calculation is performed by adding the embedded values for the labeled class J along with the weighted unlabeled values from set U . This sum is then divided by the sum of Nj and ∑ j wi , j which capture the number of labeled examples , and the sum of weights for class J , respectively . Cj = 1 Nj ∑ i∈J h ( xi ) wij = e−||h ( xi ) −Cj || 2∑ j′ e −||h ( xi ) −C′j ||2 Ĉj = ∑ i∈J h ( xi ) + ∑ i∈U h ( xi ) wij Nj + ∑ j wij ( 2a , b , c ) We extend our vanilla CAE model into a semi-supervised learning context by calculating refined prototypes from the labeled and unlabeled embeddings within the CAE ’ s embedded space . The semi-supervised loss objective Lproto can be written as Equation 3 . In Equation 3 , N represents the total number of samples in the batch ( unlabeled and labeled ) , yij is the ground truth indicator for Class j , and p̂ij is probability that sample i belongs to class J by performing the calculation in Equation 1 using the refined prototypes from Equation 2c . Figure 1 presents the full architecture of our model . Lproto = 1 N N∑ i K∑ j yij ∗ log ( p̂ij ) ( 3 )

Summary: This paper proposes a semi-supervised architecture for clustering time-series data based on Convolutional Autoencoders (AEs). The model combines the regular reconstruction loss usually employed in AEs with two new losses based on the intrinsic clustering evaluation metrics, Silhouette and DBIndex. The experiments show that this setup can achieve good clustering results (ARI) even when very few labeled examples of each class (4-28) are provided.