Latent Space Semi-Supervised Time Series Data Clustering
Authors: Andrew Hill
Katerina Kechris
Russell Bowler
Farnoush Kashani
Venue:  (2021)
Keywords: Semi-supervised clustering
clustering
deep learning
autoencoder

Abstract: Time series data is abundantly available in the real world, but there is a distinct lack of large, labeled datasets available for many types of learning tasks. Semi-supervised models, which can leverage small amounts of expert-labeled data along with a larger unlabeled dataset, have been shown to improve performance over unsupervised learning models. Existing semi-supervised time series clustering algorithms suffer from lack of scalability as they are limited to perform learning operations within the original data space. We propose an autoencoder-based semi-supervised learning model along with multiple semi-supervised objective functions which can be used to improve the quality of the autoencoder’s learned latent space via the addition of a small number of labeled examples. Experiments on a variety of datasets show that our methods can usually improve k-Means clustering performance. Our methods achieve a maximum average ARI of 0.897, a 140% increase over an unsupervised CAE model. Our methods also achieve a maximum improvement of 44% over a semi-supervised model.


Under review as a conference paper at ICLR 2021
LATENT-SPACE SEMI-SUPERVISED TIME SERIES DATA CLUSTERING
Anonymous authors Paper under double-blind review
ABSTRACT
Time series data is abundantly available in the real world, but there is distinct lack of large, labeled datasets available for many types of learning tasks. Semi- supervised models, which can leverage small amounts of expert-labeled data along with larger unlabeled dataset, have been shown to improve performance over unsupervised learning models. Existing semi- supervsper time series clustering algorithmss suffer from lack of scalability as they are limited to perform learning operations within the original data space. We propose an autoencoder- based semi- supervised learning model along with multiple semi-supervised objective func- tions which can be used to improve the quality of the autoencoder's learned latent space via the addition of small number of labeled examples. Experiments a variety of datasets show that our methods can usually improve k-Means clustering performance. Our methods achieve maximum average ARI of 0.897, 140% in- crease over an unsupervised CAE model. Our methods also achieve a maximum improvement of 44% over semi-supervised model.
INTRODUCTION
Time series data can be defined as any data which contains multiple sequentially ordered measure- ments. Real world examples of time series data are abundant throughout many domains, including finance, weather, and medicine. One common learning task is to partition set of time series into clusters. This unsupervised learning task can be used to learn more about the underlying structure of dataset, without the need for supervised learning objective or ground-truth labels. Clustering time series data a challenging problem because time series data may be high-dimensional, and not always segmented cleanly, leading to issues with alignment and noise.
The most basic methods for time series clustering apply general clustering algorithms to raw time series data. Familiar clustering algorithms like hierarchical clustering or k-Means clustering algo- rithms may be applied using Euclidean Distance (ED) for comparisons. Although ED can perform well in some cases, it is susceptible noise and temporal shifting. The improved Dynamic Time Warping (DTW) (Berndt & Clifford 1994) metric provides invariance to temporal shifts, but is ex- pensive compute for clustering tasks. A more scalable alternative DTW exists in k-Shape, measure based on the shape-based distance (SBD) metric for comparing whole time series (Paparri- & Gravano 2017). Shapelet-based approaches such as Unsupervised Shapelets (Zakaria al., 2012) can miligate issues with shifting and noise but are limited to extracting a single pattern/icature from each time series.
One alternative approach for clustering time series data is to apply dimensionality reduction through the use of an autoencoder. Autoencoders are capable of learning low-dimensional projections of high-dimensional data. Both L.TT and convolutional autoencoders have been shown to be success- ful at learning latent representations of time series data. These models can extract large number of features at each time step. After training an autoencoder model, the learned low-dimensional latent representation can then be fed to an arbitrary clustering algorithm to perform the clustering task. Because autoencoder models reduce the dimensionality of the data, they naturally avoid issues with noise, and provide level of invariance against temporal shifting.
Recently, the field of semi-supervised learning has shown great success at boosting the performance of unsupervised models using small amounts of labeled data. Dau et al. (2016) proposes a solution for semi-supervised clustering using DTW. However, this solution is still based on DTW, and as

---

Under review as a conference paper at ICLR 2021
such suffers from scalability issues. He et al. 2019 proposes constraint-propaaaalie approach for -um-iviper clustering, which may (but is not required to) be used in conjunction with DTW. However, this solution still performs time series comparisons within the raw data space, which may cause issues with scalability for large datasets.
In this paper, present semi-superviised deep learning model based convolutional autoen- coder (CAE), which may be used to perform clustering on time series datasets. We also present new semi-supervised learning objectives, adapted from well-known internal clustering metries, which can significantly improve clustering performance when provided with a small number of labeled time series. We perform experiments to show that our cemi-supervised model can improve perfor- mance relative to an unsupervised model when applied for clustering tasks. We also implement a lightly modified batch-based version of the semi-supervised learning solution shown presented in Ren et al. (2018), and show that our proposed solutions are competitive. In the best case, our model semi-supervised model shows a best-case improvement in ARI of 140% over an unsuper- vised CAE model when applying k-Means clustering, and best-case improvement of 44% over similar model.
In the remainder of this paper, Section revieews the related work on time series clustering, Section 3 presents our proposed method for semi-supervised time series clustering, and in Section 4.] we discuss our experimental methodology and present experimental results. Finally, Section details our conclusions and avenues for future research.
2 RELATED WORK
One of the most common ways to perform time series clustering is to apply the k-Means algorithm. By default, k-Means uses Euclidean Distance (ED). ED is efficient to calculate, and in many cases shows good results (Ding et al.) 2008). However, ED comparison will fail when two similar time series are shifted temporally relative to one another. Additionally, ED comparisons are sensitive to noisy data. The Dynamic Time Warping (DTW) metric (Berndi & Clifford 1994) improves on ED by computing warping path between pair of time series. This approach solves issues with temporal shifting. but requires O(N²) time to compute for two time series of length N. Recent work has provided bounds for this computation (Keogh Ratanamahatana 2005), (Lemire 2009), but the scalability of DTW remains an issue for large datasets and long time series. The k-Shape algorithm (Paparrizos & Gravano 2015) is a scalable and performant alternative to DTW, and offers similar performance to DTW at a lower computational cost. The Unsupervised Shapelets (Zakaria et al.) 2012) clustering method operates by forming clusters around common subsequnces extracted from the data. This approach provides invariance against shifts since the shapelet may appear anywhere within each time series, and also provides some invariance against noise or outliers within the data, since elementwise comparisons only occur between shapelets, rather than the full time series. In this regard, the UShapelet algorithm has some advantages over DTTW and k-Shape. However, this method is constrained to extracting a single shapelet/fcature from each time series.
Recently, semi-supervised learning has shown the benefit of augmenting a large unlabeled dataset with a small amount of labeled data. There is some existing work for applying semi-supervised learning to time series clustering. The semi-supervised time series clustering solution presented in Dau et al. (2016) proposes a modified version of DTW, which operates in a semi-supervised manner using supervised constraints. However, this method still relies on performing DTW comparison within the original data space, and as such not a scalable solution for large datasets or long time series. Another methodology for semi-supervised time series clustering is He et al. (2019) which is a graph-based approach using supervised examples to generate positive and negative constraints between points. This approach does not rely on DTW, but the algorithm still performs comparisons in the original data space, which can be problematic as the length of the time series grows.
Popular deep learning frameworks such as LSTMs and CNNs may also be applied to time series data. Both LSTM and CNN networks may be arranged as autoencoders, allowing for unsupervised feature learning for clustering, compression, anomaly detection tasks. Holden al. (2015) use a Convolutional Autoencoder (CAE) model to learn a featurized representation of gait data. Au- toencoder architectures may also be applied for anomaly detection, is shown in Bao et al. 2017). where the authors use autoencoders for anomaly detection. Performing comparisons on embedded samples avoids many of the issues of direct pairwise comparisons. Since autoencoders reduce di-
2

---

Under review as a conference paper at ICLR 2021
mensionality of the data, distance comparisons will be performed in lower- divee-ddmensionall space than the original data, which improves scalability. Embedding the data before comparison also reduces sensitivity to noise, since raw data not compared directly. One work that takes advantage of this approach is McConville et al. (2019), which learns a manifold for clustering on the trained latent space of autoencoder. Recent work in the generative model field has produced the Variational Autoencoder (VAE) architecture (Kingma & Welling, 2013). which learns probability distributions for latent features within the data and may also be applied for clustering as in Dilokthanakul et al. (216)). This architecture has also been successfully applied in Fortuin et al. 2018), which uses VAE based method to learn feature representations time series. We base our model on CAE, as the CAE architecture is simple and well-known, and provides a good platform for evaluating the performance of our proposed objective functions.
SEMI-SUPERVISED LEARNING OF LATENT SPACE
Our proposed framework semi-supervised CAE model, which combines an unsupervised re- construction loss Tre with semi-supervised loss Teupp shown in Figure We show how an existing semi-supervised learning objective function may be adapted to fit into our model, and also propose two new objective functions based well-known internal clustering metries. We focus on optimizing the autoencoder's latent space for distance- based clustering algorithms, and specifically perform our experimentation using the k-Means algorithm. The spherical, centroid-based clusters generated by k-Means are a good fit for the proposed semi-supervised losses, which encourage each cluster to converge around single high-density point.
3.1
CAE MODEL ARCHITECTURE
Embedding Space
Labeled Data
Labeled Embaddings
Samiee pervissd Loss (L,ya)
CAE
CAE
Reconstructed Data
Summed
Unlabeled
Uniaboled Embediings
Reconstructio Loss
Data
Figure I: Architecture of Proposed Semi-supervised CAE
We base our model off of two-layer Convolutional Autoencoder (CAE) architecture. The CAE uses two 1-D convolutional layers to featurize the time series. The filter widths for both layers calculated at runtime from the length of the time series. Each layer of the encoder contains a convolution operation, followed by non-linear activation. These layers convolution layers in the decoder. After each convolution layer in the encoder, we also apply a max-pooling operation to further reduce the length of the featurized sequence. Each max-pooling operation "reversed" using a nearest-neighbor upsampling operation the decoder. Alternatively, large strides in the convolutional layer may be used instead of pooling operations. This accomplishes the same goal of reducing the length of the featurized sequence, but does not require any up-sampling operations in the decoder, since the transpose convolution operation with stride will perform the upsampling. We found that the max-pooling and large stride methods produced similar results in practice.
paired with transpose
3.2
SEMI-SUPERVISED Loss FUNCTIONS
3.2.1 PROTOTYPE Loss
Snell al. (011) present a system for performing few-shot learning, where small number of labeled examples for each class in the dataset are embedded using an encoder network. These points are divided into two sets, query examples and support examples. In the latent space, the support embeddings are averaged to determine& a centroid or "prototype" for each class. Training occurs by
3

---

Under review as a conference paper at ICLR 2021
measuring the distance from each query example the support centroid for the class. Each query point is labeled with the class of the closest support prototype. The probability of a query point belonging to Class J is calculated as:
d(i,C())
Púj Eg C(),C(p)
In Equation 1 -d( is the negative distance between query point and the support prototype for Class J. Any distance metric may be used for d, but for the remainder of this paper, we treat as the squared euclidean distance. Training proceeds by minimizing the cross entropy between the labels for the query points and the probability distribution p. The objective of this approach is to learn a latent space where similarities between examples are quantifiable using distance metric like Eu- clidean distance. Examples from the same class will be close to one another in the embedded space, while examples from separate classes will have larger distances. In addition the few-shot learning scenario, later work by Ren et al. (2018) demonstrates that the prototypical network objective can be modified to support semi- semi-superiieed few-shot classification as well. The first objective function proposed in Section 3.1.1 of Ren et al. (2018) is calculated in a two-stage process. In the first stage, the labeled data prototypes are calculated as the centroid of the Nj embedded labeled data points M(x) for each class J, as shown in Equation 2a. The second stage assigns each unlabeled point a soft cluster weighting, based on the distances each of the calculated prototypes C, shown in Equation 2b. Finally, the original prototypes are updated to include the unlabeled data using the soft cluster assignments from Equation 2b. In Equation 2c, the final prototype calculation is performed by adding the embedded values for the labeled class J along with the weighted unlabeled values from set U. This sum is then divided by the sum of N, and Σ, way which capture the number of labeled examples, and the sum of weights for class J, respectively.
e -111((a)-C,11
Õ Eiesh(r) h(z;) + Eiev h(z))ws)
Cj N
M(x)) VEJ
Wij =
N₂ Eg+ Why
(2a.b.c)
We extend our vanilla CAE model into a semi-supervised learning context by calculating refined prototypes from the labeled and unlabeled embeddings within the CAE's embedded space. The semi-supervised loss objective Lyroto can be written as Equation 3 In Equation 3 N represents the total number of samples in the batch (unlabeled and labeled), Vig is the ground truth indicator for Class j. and Piy is probability that sample i belongs to class J by performing the calculation in Equation using the refined prototypes from Equation 2c, Figure presents the full architecture of our model.
Lypoto
N Σ Σ Vij log(pi))
(3)
3.2.2 SILHOUETTE LOSS
When applying k-Means clustering to an unlabeled dataset, one must first choose the correct k, or number of clusters to fit using the model. One metric for determining the correct number of clusters is the Silhouette score Rousseeuw (1987). Silhouette score belongs to the family of internal clustering metrics (Maulik & Bandyopadhyay 2002), which provide method for evaluating the performance of a clustering algorithm when no ground truth labels are available. In the absence of labels, internal clustering metrics instead evaluate the partitioning's ability separate data into clusters which have low intra-cluster distance, but high inter-cluster distance. Silhouette is per- sample metric calculated using the following formulae:
b(1) a(i) max a()), b(1)
a(i)
Σ d(i,)) LEC2
b(t) min
Σ d(i,t)
s(1)
(4a,b.c)
CC
C&
LEC2
As mentioned in Section 3.2.1 is an arbitrary distance metric, but set d as the squared euclidean distance for all experiments. Equation 4a represents the average intra-cluster distance from point i to all other points with the same cluster label. Equation 4b represents the average distance from i to the second closest cluster, or the inter-cluster distance. The second closest cluster is defined as the cluster having the second lowest average distance. The silhouette score is then calculated as
4

---

Under review as a conference paper at ICLR 2021
the difference between the inter-cluster distance and the intra-cluster distance in Equation 4c. To normalize the term, the difference is divided by the maximum of a(t) and b(i). In our configuration, silhouette score semi-supervised training objective, by providing ground-truth cluster assignments for the labeled points, and calculating cluster assignments for the unlabeled points. In this configuration, Silhouette score will represent the separability of our labeled points within the embedded space. Labeled points will be encouraged to have similar embeddings in the latent space by Equation 4a, and will also be encouraged separate themselves from embeddings with a dif- ferent label by Equation 4b. The silhouette values for the labeled points can be generated directly from equations [4a.bcc] To calculate the silhouette values for the unlabeled points, we calculate the closest embedded centroid for each unlabeled point, then calculate Equations 4a and 4b using dis- tances between the unlabeled point and the labeled points in the closest and second-closest clusters respectively. After the silhouette scores are calculated for both the labeled and unlabeled points, we concatenate both groups and take the mean over the batch. We can rewrite Equation 4c as as training objective:
1
Litth
Σ - si) N
(5)
To formulate Equation 5 take 1(i) = st) loss term, which has minimum s(i) and a maximum at s() 1. We take the mean over all [(i) to produce a scalar loss value, representing the separability of our labeled examples within the embedded space. We can train Equation using gradient descent methods either alone, or in combination with CAE reconstruction method.
3.2.3 DB INDEX LOSS
The Davies-Bouus Bouldin index is another example of a internal clustering metric. Similar to the Silhou- ette score, the DB Index value is measure of cluster quality, with lower values indicating higher quality clusters. Like the Silhoutte index, the DB Index is comprised of two terms, which are com- bined to form the metric value.
Σ dli, Cx) M(Ck, Ci) = d(Ckk. CJ) R(C,, S(C,) + S(Ci) M(C,, Ci)
S(C*)
(6a,b,c)
LEC%
Equations 6a and 6b capture notions intra- and inter-cluster similarity, respectively. Equation 6e metric which captures the quality of the separation between clusters C₁ and C₁ defined by their individual densities, as well as the distances between their centroids C* and Cz. Lower values of R indicate higher quality of separation between clusters C₁ amd C₁, thus R should be minimized. DB Index differs from Silhouette in that the DB Index methods are calculated on each pair of clusters, whereas the Silhouette index is calculated for each sample individually. Equation forms our trainable loss function. As with our implementation for Silhouette loss, we calculate the Equation for both labeled and unlabeled points by assigning unsupervised points label based on the closest labeled cluster centroid.
Lab C.C, Cÿ)
4 PERFORMANCE EVALUATION
In this following section, we evaluate the performance of our method using our unsupervised CAE as the unsupervised baseline result, and the lightly modified Prototype Loss as a comparable semi- supervised result.
4.1
EXPERIMENTAL METHODOLOGY
4.1.1 MODEL SETUP
For our experimental setup, we use a two-layer CAE model, where convolutional operations are applied along the temporal dimension to featurize the data. order to perform fair comparison on multiple datasets, we chose the parameersseserss for the convolutional layers as follows. For both convolutional layers, we set the filter size as f = [. where T' represents the length of the
5

---

Under review as a conference paper at ICLR 2021
time series. In this way, we determined the filter widths for both layers automatically from the data. For optimization, we use the Adam optimizer (Kingma & Ba 2014) as implemented in Tensorflow 2.1.0. We use the default learning rate of ir 0.001 for all experiments, and train for 200 epochs. In experiments which apply two loss functions simultaneously. such as the experiments using a semi- supervised loss along with the reconstruction loss, we optimize the sum of the losses. In practice, found that weighting the losses was not necessary as the model was able to optimize both objectives simultaneously.
4.1.2 DESIGN OF EXPERIMENTS
In our experiments, we integrate both the adapted Prototype Loss from Section 3.2.1 and the pro- posed Silhouette and DB Index Losses from Sections [3..2] and [32.33 into our CAE architecture as presented in Figure I and measure the clustering performance indicated by the Adjusted Rand In- dex (ARI) (Hubert& & Arabic 1985), using the scikit-learn implementation of ARI (Pedregosa et al.) 2011 for our experiments. In order to perform fair comparison, we performed tests using the proposed "combined"' architectures, which integrate both the semi-supervised losses suu and the CAE's reconstruction loss L,e and also measured the effect of disabling the CAE's reconstruction loss, and training solely on the proposed semi-supervisee architectures. This allows us to isolate the performance improvement for the semi-supervised architectures. Finally, we present a baseline comparison against the CAE's unsupervised performance, using only the reconstruction loss value. The goal of these tests is to demonstrate the performance of each model as the number of super- vised examples per-class increases. We perform 4 groups of tests, with each group using fixed number of supervised examples per-class in the range (4, 28]. We initialize 5 training runs for each model within each group. Within group ization to ensure that the supervised examples chosen, as well as the parameter initializations are all identical within the group. After training, we use the latent space of the trained model to perform k-Means clustering, and record the ARI. We use the k-Means algorithm because the centroid- nature of k-Means is a natural fit for the proposed losses. Notably, the Prototype Loss corresponds almost exactly to a k-Means objective, and both Silhouette and DB Index loss also rely on notions of cluster density around a centroid. However, any other general clustering method may be applied. Labeled examples are included when fitting k-Means, but are not included in the ARI metric, to avoid inflating ART artificially as the number of labeled examples increases.
the same 5 random seeds for each model initial-
4.1.3 DATASETS
For our testing, we utilize three datasets chosen from the UCR Archive (Dau et al. 2018 All UCR Archive datasets are labeled, which is useful for our evaluation since we may experiment with differing amounts of labeled data. In a real-world scenario with unlabeled data, domain ex- perts provide label information for a small subset of the data. The three datasets that we chose are some of the largest within the UCR Archive. In the case of trainable architecture like our proposed model, large datasets are advantageous, as larger numbers of samples will increase the quality of the latent fcaturization, and help to improve generalization of the features for unseen samples. All three datasets contains samples which are of the same length. In general the CAE architecture requires that all samples be the same length, although datasets with variable-length samples can still be used by first applying interpolation zero- sero-padding normalize the samples to a consistent length. FacesU'R is a dataset containing face outlines from 14 different individuals, represented as ID time series. ECG5000 is dataset containing ECG readings of heartbeats from a congestive heart-failure patient. L'WaveGestureLibraryyl is a dataset containing accelerometer recordings of subjects performing different types of hand gestures. Table in Appendix (A presents summary of the characteristics of each dataset.
4.2 EXPERIMENTAL RESULTS
The results for the tests on all three datasets are presented in Figure 3 To provide a reference for the performance of our models relative to the unsupervised models, also present Table which provides ARI performance figures for the k-Means and k-Shape unsupervised clustering algorithms, as applied to the raw data for each of our chosen datasets.

---

Under review as a conference paper at ICLR 2021
k-Means
k-Means PCA
k-Shape
Dataset
μ
0
μ
0
ECG5000
0.555
0.092
0.515
0.011
0.441
0.085
FacesUCR
0.223
0.037
0.199
0.007
0.441
0.037
UWaveGessuuree/bbaryyyll
0.632
0.055
0.612
0.045
0.238
0.027
Table 1: Unsupervised methods performance
0.e
Method
AE
ARI Wi8 Ground Truh Labele
AR winh Ground Truh Labela
0.8
ARI Wibh Ground Truth Labolla
0.6
DB Index AE
Protolype
0.7
Prototypp Loss
ass
Sihouette Loes
Sthaworto
as
32:
0.1
Labaled Examples
Labeled Examples
Labeled Per
(a) FacesUCR
(b) ECG5000
(c) UWaweGGstureeibbaayylll
Figure 2: Overall performance results for selected datasets
4.2.1 OVERALL PERFORMANCE
The results for FCcesUCR are presented in Figure 2a As shown in the figure, the semi-supervised approaches significantly improve the performance of the model, relative to the baseline CAE ARI of 0.35. According to Table I the CAE's performance here is much better than the k-Means per- formance on raw data, but slightly worse than k-Shape, which achieves an average ARI of 0.441. Out of all the semi supervise models, the Silhouette Loss + AE model performs slightly better average than the other models, including the Silhouette candidate for this type of semi-supervised model, Loss without AE, This dataset is an excellent providing 4 labels per class an ARI of 0.7 when using the Silhouette Loss. We can see that as the number of supervised ex- amples per-class increases, the ARI achieved for all of the semi-supervised models also increases, approaching a 0.9 ARI. All semi- sumi-supeeiiee models perform well on this dataset, although the two Silhouette Loss models seem to have a slight edge over the others. The results for ECG5000 in Fig- ure 2b show that both DB Index methods perform poorly on this dataset. Both the DB Index and DB Index + AE methods perform worse than the baseline CAE at numbers of labeled examples. The baseline CAE model performs similarly to k-Means clustering on ECG5000, and better than k-Shape. The Prototype and Prototype + AE Methods do show some improvement over the baseline CAE, but these improvements are relatively minor. The Silhouette and Silhouette + AE methods outperform all other methods here, achieving a ARI 0.8 for numbers of labeled examples. However, two of the Silhouette Loss trials at 28 examples seem to diverge, and produce poor results. The Silhouette Loss + AE models do not suffer this same divergence, and provide a stable ARI of around 0.8 for all trials 28 examples per-class. We suspect that the AE model and associated Lye was able to help mediate the effect of the divergence in the Silhouette + AE model. The Sil- houette + AE model does encounter one minor divergence achieves a ARI of 0.5. However, we expect that smaller numbers of labeled examples will tend be noisier, since the model performance depends heavily on the choice of supervised examples. In this case, the Silhouette models are the winners, but do suffer from some divergence issues as mentioned before. We believe that this issue is caused part by the extremely unbalanced nature of the ECG5000 dataset. Table 3 in Appendix A shows the distribution of classes. Two of the classes are very sparse, and must be over-sampled during training in order to provide the correct number of labeled examples. Additionally, most of the cluster points concentrated in Clusters and 2. During training, the clusters with smaller numbers of ground-truth labels tend to "steal" some of the true members of Clusters and leading converged result where Clusters and 5 much
achieve
examples per-class. where it only

---

Under review as a conference paper at ICLR 2021
larger than in the ground-truti data. The results for UWKaveestureLLivarrylll in Figure 2c show that DB index performs consistently well over all numbers of labeled examples. At 4 labeled examples per-class, the DB Index and DB Index AE methods are the only methods which outperform the standard baseline CAF. Starting at 12 examples per-class, the Silhouette and Prototype models do start to outperform the CAE baseline, but perform noticeably worse than the DB Index models until 28 examples per-class. For this dataset, even the baseline CAE outperforms k-Means on the raw data, which performs the best of the three unsupervised models in Table I In this experiment, the DB Index models are the clear winner since they provide excellent performance for any number of labeled examples tested.
4.2.2
HYPERPARAMETER STUDY
Method
Method
Creand Truh Labada
0.
0B Index + RE
AE
AR/wen Ground trap Lacaes
0.-
DB Indox
AE
Pholecypp Lose AC
DB Indec AC
0.6
Soluoutt
0.
Protskypo
Protstype
ARH With Wieaadd
Blhouutt
0.4
944
Sihouett Loos AE
0.2
82
0.0
0.0
100
Gracient Updalc
Filor Bizo
(a) Gradient Update
(b) Filter Size
Figure 3: Hyperparameter Study Results
we perform parameter updates at the end of each batch. In order to better understand how the frequency of parameter updates affects the overall performance, we also experiment by applying the update for the semi- cumi-sueervieed loss at the end of each epoch, while still updating parameters from the autoencoder loss at the end of each batch. In order to accomplish this, we calculate the cemi-supervised gradients at each batch, accumulating them and applying the sum the gradient update at the end of each epoch. For this experiment, train the model on the UWaveGestturLibbrayy dataset and choose 12 supervised examples per class. Since autoencoder updates are performed at the end of each batch, the autoencoder result does not change based on gradient update method, and the results are only provided for comparison. We train all versions of the model using the same random seed, only varying the gradient update method between the two model instances. Figure 3a shows the result of the experiment. As expected, the AE model obtains identical performance between the two update methods. The DB Index and Silhouette methods see marginal improvement when training using the per- methodology. The Prototype Loss method sees marginally better performance when updating at the end of the epoch. In Section 4.1 we describe our method for determining the convolutional filter size dynamically based on data input. this experiment, we test the same model setup as the gradient test, but vary the filter size. Gradient updates are performed at the end of each epoch. Figure 3b shows the result of this experiment. Most models perform their best with the filter size of 50, but in general performance does not differ much with different choices for filter size. In the real-world use case, sub-optimal choice for convolutional filter size should not degrade the performance of the model.
In the performance evaluation results from Section 4.2.1
We also explored the usability of the learned latent space for classification by applying KNN classification. For this series of tests, we treat the randomly chosen "labeled" examples for each class as the training points for KNN classifier, then predict the class of the unlabeled points. We calculate the accuracy of the KNN classifier for each the three datasets described in 4.1.3
8

---

Under review as a conference paper at ICLR 2021
KNN Acsuray wi Labeled Exunutes
KNN Aosuracy wi Labaled Exensies
KNN Aguracy wi Labete Exingles
0.66
Milhhd
DBlirdex
DB index AE
0.7
0.7
2.66-
Profodype Lose AE
Prototype
0.80
Silhouetic Loss
06
Sinouetle Loss AE
26
0.75
0.70
12
20
28
12
20
28
20
28
# Lobeled Exampics Per Class
# Lobeled Exampics Por Class
A Laboled Examples Por Glass
(a) FCC
(b) ECG5000
(c) UWaveGestuuelabbraay.ll
Figure 4: Classification performance for three selected datasets
In the results for FacesUCR as seen in Figure 4a we see clear distinction performance between the semi-supervised models and the unsupervised autoencoder. Even for low numbers of supervised examples, all semi-supervised models outperform the autoencoder by large margin, and the au- toencoder never closes the gap in performance, even for larger numbers of supervised examples. The results for ECG5000 show that all models struggle to perform better than the standard CAE. We suspect that the model performs poorly here because of the imbalanced class size, mentioned previously. The results for U'aveGesture as shown in Figure Ac show more mixed performance. At low numbers of samples, the Sihouette and Prototype losses perform marginally better than the baseline AE, but show a large variance in performance. This demonstrates that both these models' performance is highly sensitive the choice for labeled examples. The DB Index models per- form distinctly better than any other on this dataset, and show little variance in performance even for lower numbers of labeled examples. We suspect that this is because the embedding clusters in U'Wveeeesture are more distinct from each other, so the DB Index approach, which is based on op- timizing distances between pairs of clusters, performs the best. The baseline AE also shows large improvement here as the number of labeled examples increases, although it does not outperform the semi-supervised models.
CONCLUSION
In this paper, we proposed framework for semi-supervised time series clustering, including three alternative supe-supe loss functions. Our experiments show that all three implemented semi- supervised models can improve clustering performance after training. Experiments also show that training the cemi-supervised losses in combination with the reconstruction loss from the autoencoder does provide a slight boost in performance, although this difference is usually small. Although all solutions have generally stable performance across multiple parameter initialization and choices for supervised examples, we do sec occasional model divergences. Because these models rely on the labeled examples for training, the quality of these labels is exceedingly important. In real-world usage scenario, we expect data domain expert providing labels would be able to choose the most relevant examples label. Additionally, datasets with significantly unbalanced class sizes may cause performance issues, as are exhibited by our models' performance on the ECG5000 dataset. The results Tables 6 show that Silhouette Loss on average outperforms the baseline CAE, except in the 4-smmlle case on the U'ave dataset. In addition, the DB Index loss average outperforms the CAE on both the UWave and FacesUCR datasets.
In future work, plan to explore methods for combining the predictions of these models by train- ing multiple instances of the same model in parallel, then using a consensus clustering system to generate the final set of labels. We expect that this will reduce the severity of model divergences. In a similar vein, we will explore a way to determine an optimal weighting between the reconstruc- tion and semi-supervised losses, since our method currently applies no weighting. Additionally, believe that training multiple models simultaneously and applying mutli-view learning constraints like those proposed in Wang et al. (2015) could improve the quality of the model's generated latent space.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Wei Bao, Jun Yue, and Yulei Rao. A deep learning framework for financial time series using stacked autoencoders and long-short term memory. PloS one, ((7)):0180944: 2017.
Donald Berndt and James Clifford. Using dynamic time warping to find patterns in time series. In KDD workshop, volume 10, pp. 359-370. Seattle, WA, 1994.
Hoang Anh Dau, Nurjahan Begum, and Eamonn Keogh. Semi- Semi-suuerviiion dramatically improves time series clustering under dynamic time warping. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 999-1008, 2016.
Hoang Anh Dau, Eamonn Keogh, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, Yanping, Bing Hu, Nurjahan Begum, Anthony Bag- nall, Abdullah Mueen, Gustavo Batista, and Hexagon-Mi MLL The ucr time series classification archive, October 2018. https: /wwwwww..... ucr edu / -eamonn/tt http://timeeeeeriieeeeeeee 20187
Nat Dilokthamakul, Pedro AM Mediano, Marta Gamelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture varia- tional autoencoders. arXiv preprint arXiv:1611.02644 2016.
Hui Ding, Goce Trajcevski, Peter Scheuermann, Xiaoyue Wang, and Eamonn Keogh. Querying and mining of time series data: experimental comparison of representations and distance measures. 2008.
Proceedings of the VLDB Endowment,
1(2):1542-155)) (()):
Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, and Gunnar Rätsch. Som-vae: Interpretable discrete representation learning on time series. arXiv preprint arXiv:1806.02199 2018.
Guoliang He, Yanzhou Pan, Xuewen Xia, Jinrong He, Rong Peng, and Neal N Xiong. A fast semi- supervised clustering framework for large-scale time series data. JEEE Transactions on Systems, Man, and Cybernetics: Systems, 2019.
Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. Learning motion manifolds with convolutional autoencoders. In SIGGRAPH Asia 2015 Technical Briefs, pp. 1-4. 2015.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 2(1):193-218 1985.
Eamonn Keogh and Chotirat Ann Ratanamahatana. Exact indexing of dynamic time warping. Knowledge and information systems, 7(3):358-386, 2005.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.698). 2014.
Diederik p Kingma and Max Welling. 1rXiv:1312.6114, 2013.
Auto-encoding variational bayes. arXiv preprint
Daniel Lemire. Faster retrieval with a two-pass dynamic-time-wwrping lower bound. nition, +2(9):2169-2180) 2009.
Pattern recog-
Ujjwal Maulik and Sanghamitra Bandyopadhyay. Performance evaluation some clustering algo- rithms and validity indices. IEEE Transactions on pattern analysis and machine intelligence, 24 (12):1650-1654, 2002.
Ryan McConville, Raul Santos-Rodriguss Robert J Piechocki, and Ian Craddock. N2d:(not too) deep clustering via clustering the local manifold of an autoencoded embedding. arXiv preprint arXiv:1908.0S5: 05968, 2019.
John Paparrizos and Luis Gravano. k-shape: Efficient and accurate clustering of time series. In Proceedings of the 2015 ACM SIGMOD International Conference Management of Data, pp- 1855-1870, 2015.
10

---

Under review as a conference paper at ICLR 2021
John Paparrizos and Luis Gravano. Fast and accurate time-series clustering. ACM Transactions Database Systems (TODS), 122,):1499, 2017.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830 2011.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classifica- tion. arXiv preprint arXiv: (0t::/800..0000 803.00676, 2018.
Peter Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analy- sis. Journal of computational and applied mathematics, 20:53 -5, 1987.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi- view representation learning. International conference on machine learning, pp. 1083-1092, 2015.
Jesin Zakaria, Abdullah Mueen, and Eamonn Keogh. Clustering time series using unsupervised- shapelets. In 2012 IEEE 12th International Conference Data Mining, pp. 785-794. IEEE, 2012.
A APPENDIX
Dataset
Samples
Classes
Length
FacesUCR
2250
14
131
ECG5000
5000
5
140
('aaveeesuueLLibarrAAl
4477
8
315
Table 2: Specifications of the selected datasets
Cluster
N (%)
2919 (58.38%)
1767 (35.34%)
3
96 (1.92%)
194 (3.88%)
5
24 (0.48%)
Table Specifications of ECG5000
11

---

Under review as a conference paper at ICLR 2021
ARI
Examples
Method
μ
d
AE
0.694
0.008
DB Index
0.796
0.034
DB Index + AB
0.785
0.020
4 Prototype Loss
0.560
0.057
Prototype Loss + AE
0.591
0.056
Silhouette Loss
0.592
0.073
Silhouette Loss AE
0.602
0.045
AE
0.690
0.013
DB Index
0.845
0.011
DB Index + AE
0.842
0.027
Prototype Loss
0.715
0.020
Prototype Loss AE
0.767
0.021
Silhouette Loss
0.793
0.044
Silhouette Loss + AE
0.835
0.029
AE
0.673
0.036
DB Index
0.860
0.013
DB Index AE
0.874
0.018
20
Prototype Loss
0.764
0.038
Prototype Loss + AE
0.814
0.014
Silhouette Loss
0.839
0.022
Silhouette Loss + AE
0.853
0.034
AE
0.687
0.026
DB Index
0.876
0.021
DB Index + AE
0.878
0.019
28
Prototype Loss
0.819
0.007
Prototype Loss AE
0.847
0.011
Silhouette Loss
0.861
0.024
Silhouette Loss AE
0.871
0.021
Table UWaveGeestureLLbbrary Result Table
12

---

Under review as a conference paper at ICLR 2021
ARI
Examples
Method
AE
0.537
0.051
DB Index
0.302
0.009
DB Index AE
0.297
0.009
4
Prototype Loss
0.556
0.114
Prototype Loss + AE
0.558
0.125
Silhouette Loss
0.730
0.168
Silhouette Loss AE
0.736
0.147
AE
0.537
0.034
DB Index
0.307
0.019
DB Index AE
0.306
0,014
12
Prototype Loss
0.577
0.100
Prototype Loss AE
0.598
0.090
Silhouette Loss
0.832
0.033
Silhouette Loss + AE
0.821
0.027
AE
0.565
0.006
DB Index
0.324
0.030
DB Index AE
0.318
0.017
20
Prototype Loss
0.594
0.020
Prototype Loss + AE
0.626
0.035
Silhouette Loss
0.758
0.142
Silhouette Loss + AE
0.822
0.025
AE
0,504
0.055
DB Index
0.308
0.054
DB Index AE
0.368
0.022
28
Prototype Loss
0.554
0.131
Prototype Loss AE
0.636
0.071
Silhouette Loss
0.596
0,311
Silhouette Loss AF
0.806
0.035
Table 5: ECG5000 Result Table
13

---

Under review as a conference paper at ICLR 2021
ARI
Examples
Method
AE
0.366
0,023
DB Index
0.644
0.034
DB Index AE
0.645
0.037
4
Prototype Loss
0.539
0.042
Prototype Loss + AE
0.666
0.014
Silhouette Loss
0.715
0.044
Silhouette Loss AE
0.679
0.051
AE
0.357
0.025
DB Index
0.735
0.038
DB Index AE
0.722
0.049
12
Prototype Loss
0.750
0.023
Prototype Loss AE
0.802
0.027
Silhouette Loss
0.798
0.040
Silhouette Loss + AE
0.809
0.036
AE
0.370
0.043
DB Index
0.777
0.055
DB Index AE
0.786
0.019
20
Prototype Loss
0.819
0.029
Prototype Loss + AE
0.858
0.026
Silhouette Loss
0.875
0.009
Silhouette Loss + AE
0.872
0.022
AE
0.346
0.023
DB Index
0.832
0.022
DB Index AE
0.832
0.026
28
Prototype Loss
0.827
0.014
Prototype Loss AE
0.864
0.017
Silhouette Loss
0.882
0,014
Silhouette Loss AF
0.897
0.020
Table FacesUCR Result Table
14


Reviews:
[{"review_id": "dm5jdr3tlOU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a semi-supervised architecture for clustering time-series data based on Convolutional Autoencoders (AEs). The model combines the regular reconstruction loss usually employed in AEs with two new losses based on the intrinsic clustering evaluation metrics, Silhouette and DBIndex. The experiments show that this setup can achieve good clustering results (ARI) even when very few labeled examples of each class (4-28) are provided.\n\nPositive points:\n\nOverall, the presentation of the paper is good, it is well organized and easy to understand.\nThe experiments are plenty and well thought to evaluate the proposed model in three different datasets.\n\nQuestions:\n\nIn Equation 2b, shouldn’t be C_j and C'_j instead of p_j and p'_j? Otherwise, what p_j means?\nIs the number of clusters defined by the number of classes? What happens if a class is originally split into multiple clusters?\n\nPoints to improve:\n\nIn “Both LSTM and convolutional autoencoders have been show to be successful at learning latent representations of time series data.” Please, provide at least one reference for each type of model.\n\nEvaluating clustering quality is a tricky task. Using only ARI might be misleading, as ARI and most clustering metrics, has its problems. Therefore, I would suggest adding other metrics such as Normalized Mutual Information (NMI), Purity, and Clustering Accuracy.\n\nA comparison with previous methods is lacking. Although I’m not aware of other methods for semi-supervised clustering of time-series data. There several options to compare with in the unsupervised clustering case. The closes one might be Fortuin et. al. (2019) “SOM-VAE: Interpretable Discrete Representation Learning on Time Series”.\n\nA few typos and bad formating to fix: \n“have been show“-> \"have been show\" (section 1)\n“an semi” -> “a semi” (section 1)\n“squnces” -> “sequences”\nfix reversed quotes as the first quote in “reversed\" (section 3.1)\n“equation (5)” -> “Equation (5)” (section 3.3.2)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PW6vdbzUuiC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an autoencoder-based approach for semi-supervised time-series clustering. Specifically, the paper exploits convolutional autoencoder (CAE) and integrates internal validity indexes to evaluate clustering quality in addition to small number of labeled data. Results on three datasets show improvement over the unsupervised case.\n\nPros\n\n- Well-written paper that tackles an important problem\n- Study of validity clustering indexes and their integration to current CAE architectures\n- Results on three datasets showing the potential of this methodology\n\nCons\n\n- Missing related work\n- No comparison against unsupervised time-series clustering methods\n- No comparison against semi-supervised time-series clustering methods\n- Only three datasets used despite 100+ available\n\nDetails:\n\n- The paper is missing a decade of progress in the area of unsupervised time-series clustering. \n\n[a] Paparrizos, John, and Luis Gravano. \"Fast and accurate time-series clustering.\" ACM Transactions on Database Systems (TODS) 42.2 (2017): 1-49.\n\n- Comparison against unsupervised methods such as k-means and k-shape [a] is necessary to understand the effectiveness of CAE at the first place.\n\n- Comparison against semi-supervised methods reviewed in related work is needed. \n\n- Evaluation on all 100+ datasets of the UCR archive is necessary. To advance the state of the art, the method needs to show improvement in the entire benchmark and not just on three datasets, which we are not sure how exactly they were picked and why.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "PMEYgqXAIEK", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper benchmarked three different existing losses, DB/Propotype/Sihouette, on time series clustering. Although there is no technique innovation, this is an interesting topic and the results look good to me. However, there are some concerns: \n\n1. I am not an expert in time series clustering, especially in the semi-supervised clustering case. As there is a labeled dataset, the reason to solve this problem as two steps is not clear to me: 1. learning the semi-supervised representation and 2. do the clustering. Why not just benchmarking by semi-supervised learning metric, like accuracy? \n\n2. The effect of the clustering method looks unclear. In other words, will the conclusion be held on different clustering methods, rather than K-means. \n\n3. Different losses have different performance rankings in different numbers of labels and datasets (Figure 2). Picking up the best and claiming the advantage does not form a fair comparison. It will be more interesting if the author can propose a method, which can consistently win other methods. \n\nMinor concerns: \n\n1. the bar plot should have space between each category.\n2. it will be more convincing if baselines from other papers are considered. \n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-jtD34iXp9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Disclaimer: I am not an expert in the time-series domain, although I did some literature review while performing this review.\n\n#####################################\nSummary:\nThe work investigates semi-supervised (SS) clustering of time-series data (i.e., clustering with few labelled points, which can also be seen as semi-supervised classification here). It specifically investigates how to train a convolutional autoencoder (CAE), which is applied on the time-series data creating an embedding for each timeseries, so that CAE’s embedding clusters samples appropriately. It investigates 3 semi-supervised (SS) losses for this purpose: 1 is the loss proposed in (Ren 2018), and 2 novel losses, the “Silhouette loss” and the “DB index” loss, inspired by the corresponding internal clustering metrics. The work evalutes on 3 databases whether these losses improve clustering of the CAE’s embedding, over vanilla training of the CAE (just reconstruction loss). The work also performs a study of whether the size of the convolution filters affect results and whether performing the gradient updates by the SS loss every batch or every epoch makes a difference.\n\n######################################\nReasons for score:\nI am recommending a rejection, because of limited literature review, problems with writing/clarity, limited evaluation and analysis of the experimental setup (configuration etc), and the results are not too strong either. I would not mind much about the performance, if the other weaknesses were not there, as at least the Silhouette loss seems reasonable. But overall, the weaknesses overcame the good points.\n\n#########################################\n\nPros: \n+ Exploration of semi-supervision for clustering is interesting as it’s of practical importance, and rather timely, as there is general interest in the community on SS.\n\n+ The work explores 3 different losses for inducing clustering in the embedding space, 2 of which seem novel. The 2 new losses are quite intuitive as they are inspired by well established metrics. They are generic (not Time-series specific) and could potentially be interesting to parts of the community that look into how to create more discriminative models by clustering the embedding space of deep networks better, such as in semi-supervised learning.\n\n+ Performance of Silhouette loss seems to be interesting, as in all three databases it performs ok. (although I am not 100% convinced due to some unclear points about experimental setup and not too extensive evaluation, see below).\n\n#########################################\nCons:  \n\nCon.1:\nThe work is not well positioned within the literature. The review is limited both from the scope of unsupervised clustering, as well as from the point of view of semi-supervised learning (both in general and specifically to time-series). Details on literature review:\n\nOne one hand, the work discusses only “shallow” unsupervised clustering methods, but misses deep models for unsupervised clustering. As a result, it considers as the *only* baseline a vanilla Auto-Encoder (AE). However, for unsupervised clustering, AE has been long surpassed by more appropriate models for clustering (AE has by itself no incentive to cluster the latent space). Examples include Contrastive AEs, extensions of VAE and GANs, etc. I below give a citation to a recent work that extends VAEs for clustering [1], which also contains references to other models, which the authors can consult. Also, some old and recent works that apply such models to time-series are [2, 3, 4]. Hopefully references therein will be of interest to authors. I wouldn’t expect all of them to be evaluated, but at least some discussed, to give a good picture of where the field is on time-series clustering. (A suggestion: Perhaps the authors could improve the work by arguing that they chose AE for its simplicity, to focus specifically on the effect of the loss. But at least this discussion should be made.)\n\n[1] Dilokthanakul et al, Deep unsupervised clustering with gaussian mixture variational autoencoders, 2016.\n\n[2] Rifai et al, Contractive Auto-Encoders: Explicit Invariance During Feature Extraction, 2011.\n\n[3] Fortuin et al, SOM-VAE: Interpretable Discrete Representation Learning on Time Series, ICLR 2019\n\n[4] McConville et al, N2D: (Not Too) Deep Clustering via Clustering the Local Manifold of an Autoencoded Embedding, ICPR 2020 (arxiv Aug 2019)\n\nWithin the scope of the time-series application, authors mention deep approaches based approaches (via LSTM and CNN) but they don’t cite any. I would expect some important works to have been cited.\n\nFrom the point of view of semi-supervised learning (SSL) in deep-learning, the literature review is also limited, constrained to Ren et al (2018). This should be improved. I below provide a reference to a work that proposed an SSL loss for improved clustering of latent-space via label propagation, which is in the same spirit as the losses discussed in this paper and hence should be discussed [5]. I also provide a reference to a more recent work with cluster-inducing properties, in case the authors find it useful to discuss [6]. ([6] is not SSL, but it may be of interest, according to your judgement).\n\n[5] Kamnitsas et al, Semi-Supervised Learning via Compact Latent Space Clustering, ICML 2018.\n\n[6] Kenyon-Dean et al, Clustering-Oriented Representation Learning with Attractive-Repulsive Loss, AAAI 2019 workshop.\n\nFinally, I below provide a reference to a recent work on SSL in time-series, where the references therein can also help identify important related work [7].\n\n[7]  Jawed et al, Self-supervised Learning for Semi-supervised Time Series Classification, PAKDD 2020.\n\n\nCon.2: \nEvaluation shows that no method is consistently better than the baseline AE. Which diminishes a lot the value in the technical contribution (e.g. the interest in the 2 novel losses). Even more, central claim of the paper about performance of the methods does not hold.\n\nSpecificially, the claim from the abstract “our methods can consistently improve k-Means clustering” is not True. In Fig 2 and 4 from evaluation, we clearly see that no method “consistently” improves over baseline AE across all settings. For example, DB index is clearly worse than all on ECH5000. Prototype loss also falls short, especially in UWave and Fig 4 ECG500. Silhouete loss also falls below AE on UWave in the little-labels regime in Fig2.a and across all settings in Fig.4.b. This actually shows that no model “consistently” improves AE.\nSimilarly, these claims about performance are accompanied in Abstract & Sec 1 with comments about the “maximum increase” in performance. But the maximum increase does not really convey the actual picture, as these are only the extrema. Instead, the authors should try to derive some statistic that represents the overall performance better.\n\nCon.3: \nEvaluation is limited:\n\nMethod is only against unsupervised CAE and Ren 2018. This is a result of limited literature review I guess (weak point 1). The authors could expand the evaluation by comparing to:\n* other “shallow” approaches that are already applied in time-series. The authors mention some of these in the related literature, arguing they are slow. The authors could at least try some of them, and report performance and time-for-running. This is very important, as the time-to-run of these approaches should be compared with the time-to-train of the proposed approach, which is never mentioned. (to my understanding the authors train and evaluate on the same samples, ala transductive SSL, hence training time plays the role of time-to-run of the traditional approaches)\n* other related losses for SSL in deep models (e.g. [5], or other similar easy to apply ones like VAT, see references in [5]), or other baseline generative models (e.g. VAE/GAN variants). \n\nCon.4: \nThe hyperparameter study in Sec 4.2.2 is not really useful. The authors explore whether different number of layers in the AE affect performance, and whether updating the weights of the AE with the gradients of the SSL loss at every batch step or only every epoch makes a difference. This is not insightful about the proposed losses themselves.\n\nCon.5: \nThere is currently no mention of an attempt to configure the model and training configuration for each loss best. E.g., learning rate, number of steps till convergence, etc. So, I am guessing, same learning rate etc is used in all settings, which is unclear at which settings they have been configured. But, I think that the losses may have gradients at different scale (I m not sure, but seems so). For example, L_proto in Eq.3 has the form ylogp and may have gradients that are at a different scale than Eq 5 and Eq 6 that directly take the gradient of the distances. This means that for optimal convergence, and hence fair comparison, gradients of each loss should be scaled differently. E.g. using different learning rate. Or, better, using a different weight multiplier in front of L_supp, so that the same learning rate can be used for the reconstruction loss across all settings and only scale the L_supp gradients independently. What I would normally expect is a study of the performance’s sensitivity on such as hyperparameter.\n\nCon.6: \nReproducibility is not high. There is no code, and various parameters for training (learning rates, number of sgd steps, etc) are not included.\n\nCon.7: \nClarity of the paper, especially Section 3 (method) is not good. It is largely due to not good mathematical notation, which could help define certain things well, coupled with a problem in Fig 1:\n\nSec 3.1 describes the CAE architecture, which is a very small model (2 layers of conv+pool) but still I don’t think it does it adequately. I am particularly confused about the description of the 1D conv filters, their width, the dimension along which the conv is applied etc. This description is done in natural language and I think it’s currently ambiguous. The authors could introduce some more formal definitions of the input tensor (X \\in ??), weight tensors, the output tensor of a convolutions, define dimensionality of these tensors, which would clarify better the model’s operation.\n\nMath notation should be improved. E.g. in Eq.1, C_j is not formally defined. Eq.2, p_j undefined (I think it’s the prototype for class j, where in Eq.1 is C_j, while p has been previously defined as a probality).\n\nEq 3, L_proto, is a semi-supervised loss (L + U). \n\nFig 1, shows only a loss that is called “semi-supervised” and symbolised as L_sup, while the figure shows that this loss is only dependent on embeddings of labelled data, not those of the unlabelled data. Hence it confuses, as it looks like a supervised loss.\n\nj is sometimes a class (Sec. 3.2.1), sometimes a sample (Eq4), and vice versa for C_j. Also, C_j in 4.b. represents a set of samples, and not just a class number, as in Eq.1. Please, repass math notation and define carefully each term. The current state makes for a difficult reading experience.\n\n\\bar{C} in Eq.6 undefined.\n\n##############################################\nQuestions for rebuttal period:  \n\nPlease address the weak points I raised above, as well as the following (some are related to above anyway):\n\nCan you please clarify whether all the time-series within a database (e.g. FacesUCR) are of the same length? I guess so, but I would like a clarification. Also, please clarify this in text, e.g. in 4.1.3, as it is important for the reader to understand whether all samples have same dimension.\n\nCan you state explicitly in the text whether you are using any weighting of the SSL loss VS the reconstruction loss? I think not, but please state it explicitly. If not, please add a short discussion of whether you think the gradients of the different losses are at comparable scale, and whether such weight is needed or not.\n\nCan you explain how were configuration parameters for the methods been performed? Since there is no weighting for each loss, I am particularly interested in things like the learning rate. Was it configured on the CAE? On the CAE+SSL loss? On each one separately? On which set, since there is no validation fold? Please add to text.\n\nSec. 4.2.2 “ Since autoencoder updates… for comparison”: Can you please clarify/rephrase what this means? Why at each epoch only? Do you mean specifically due to the SSL loss? Because in previous sentences you say that the autoencoder *still* gets updated at each batch via the reconstruction loss.\n\n#############################################\n\nMinors, or additional feedback for improving the work in the future (not subject to rebuttal):\n\nSec. 4.2.1. “This also contributes to the poor KNN performance on this dataset.”. I have not noticed any mention of KNN here (at least not until later in Fig.4). Where is this comment based on? If it refers to experiments later on (Fig 4), please clarify/refer to them in text.\n\nSec. 4.2.2. “The findings here also support … Prototype loss model.”: I think these results do not “support” previous results, as the experiments are made on the database the same database that showed the trend in the first place. If they were made on ECG5000, they would “support” another trend. I would remove this argument. What these results support is that the filter-size is not important on this database.\n\nThe authors could try to derive metrics that summarize performance across multiple settings. (e.g. “average” performance across all settings in each database, or average performance across databases on a particular setting (e.g. 4 labels)). Summarized in a table? It could help define clearly which method does better.\n\nIt would be interesting to explore how the amount of unlabelled data affects results of training.\n\nIt would be interesting to have 2 sets of unlabelled data, one for training and one for evaluation. To test whether the embedding generalizes and provides good clustering on unseen data, not just the ones it was trained on, and how the 2 cases (seen unlabelled vs on unseen unlabelled) compare.\n\nSec. 4.2.1: The authors provide a description of how training evolves (“During training, the clusters… ground-truth data”) in this case of sparse labels. Perhaps in future work it would be interesting to actually provide an empirical analysis of this behaviour (e.g. showing how clustering changes between epochs? Perhaps even on a toy-dataset, in a low-dimensional embedding, to observe how the losses behave).\n\nIt would be nice to have statistical significance tests between model results in Fig 2, especially about the main claims. (E.g. to prove that Silhouete is significantly better than others)\n\nTypos and other minors:\n\nSec.1: “and is not always segmented”: Unclear to me what this means, perhaps rephrase a bit.\n\nSome typos: \n\nSec. 1 “have been show” => shown\n\nSec.1 “an semi-supervised” => a\n\nSec. 1 “model semi-supervised model”\n\nSec. 1 “a unsupervised” => an\n\nSec 3.2.2 “the a partitioning”\n\n\n**=========== Update to Review, after Updates to Manuscript during Rebuttal period ==========**\n\nSummary of improvements during rebuttal and remaining concerns:\n\n- Extended the literature review, with references to an unsupervised clustering method (k-shapes),  and methods that use CAE, LSTM and VAE. This is a significant improvement. The literature review still does not discuss previous losses that lead to clustering within the latent space, which is what the proposed methods here perform, and are hence closely related (see my initial comment).\n\n- Rephrased/corrected claims about performance of the method. The corresponding claims about performance is now that the methods “can usually improve k-Means clustering performance”, improving the baseline CAE. This is now accurate. Yet, I do not think the performance is a sufficiently strong argument unfortunately (especially given the limited evaluation, only against few and basic baselines).\n\n- Evaluation has been expanded by employing unsupervised clustering via k-Means, k-Means+PCA, and k-Shape in Table 1. However, there is still no comparison with any other more advanced method, such as SSL based, DL based, etc (as per my initial comment). I think there is a lot of improvement here, before the paper can display improvement over the current state of the art.\n\n- No improvement on the ablation study, which is currently not particularly useful, e.g. by investigating aspects of the losses and their behaviour to provide more insights into the method, or performing empirical sensitivity-analysis to meta-parameter values.\n\n- Added clarifications about some values hyper-parameters used (Adam optim with default TF2.1 learning rate for all). There seems there was no attempt to find optimal hyper-parameter configuration for each method, in order to perform fair comparisons (e.g. each method may require different weighting/learningrate). The authors seem to acknowledge the issue and defer it to future work. However, I think that without such investigation (e.g. via an empirical study and sensitivity to these hyper-parameters), we cannot be strongly confident about conclusions on the relative performance of methods, as they can be sensitive to configuration.\n\n- Provided the code of the work as supplementary material, hence reproducibility is greatly increased. Thank you.\n\n- Improved clarity of the paper by improving Fig 1 and some of the math notation problems. \n\n\nOverall, the work has been improved during the revision, hence I will raise my rating (from 3 to 4). However, I believe the remaining problems of limited literature review, evaluation, no sensitivity analysis to hyper-parameters (or effort thereafter to configure them for each method) or other sort of empirical analysis that would give insights to the behaviour of the losses, still keep my score relatively low.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses an important problem of semi-supervised learning of time-series data.  Their approach is based on a convolution autoencoder for learning a time-series latent space.  To guide learning an appropriate embedding, they explore three alternative internal clustering metrics (prototype loss, Silhouette loss, and DB index loss) coupled with the autoencoder reconstruction loss. 

The approach is reasonable and interesting, however as pointed out by the reviewers, the current submitted version needs major revision for it to be accepted.  Key weaknesses are:
1.	It lacks an extensive literature review.  The reviewers have made several suggestions for improving this.
2.	The experiments are weak.  First, state-of-the-art baselines are missing.  Second, additional alternative evaluation metrics will strengthen the evaluation of unsupervised methods (e.g., adding NMI, accuracy, cross-validation of internal metrics).  Providing evaluation of when which loss would work better on what types of data would provide insight to the various losses proposed.

Decision: Reject