Learning Interpretable Neural Discrete Representation for Time Series Classification
Authors: Etienne Le Naour
Ghislain Agoua
Nicolas Baskiotis
Vincent Guigue
Venue: Submitted to ICLR 2023 (2023)
Keywords: Time series classification
discrete neural representation
interpretability
deep learning

Abstract: Time series classification is a challenging research field with many real-life applications. Recent advances in deep learning have significantly improved the state of the art: recurrent or convolutional architectures allow automatic extraction of complex discriminating patterns that improve performance. Those approaches suffer from a lack of interpretability: the patterns are mapped into a high dimensional latent vector space, they are not representable in the time domain, and are often even not localizable. In this paper, we present a novel neural convolutional architecture that aims to provide a trade-off between interpretability and effectiveness based on the learning of a dictionary of discrete representations. The proposed model guarantees (1) that a small number of patterns are learned, and they are visualizable and interpretable (2) a shift equivariance property of the model associated with a time-consistency of the representation (3) a linear classifier over a limited number of patterns leading to an explainable decision. To ensure the robustness of the discrete representation, they are learned in an unsupervised process independently of the classification task. This allows further great performances in transfer learning. We present extensive experiments on the UCR benchmark wrt usual baselines. The interpretability of the model is illustrated empirically. The chosen trade-off results obviously in a decrease in performance compared to the state of the art. The performance drop is however limited and very dependent on the application domain. The experiments highlight the efficiency of the model for the transfer learning task, showing the robustness of the representations.

Under review as a conference paper at ICLR 2023
LEARNING INTERPRETABLE NEURAL DISCRETE REP- RESENTATION FOR TIME SERIES CLASSIFICATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Time series classification is challenging research field with many real-life appli- cations. Recent advances in deep learning have significantly improved the state of the art: recurrent or convolutional architectures allow automatic extraction of complex discriminating patterns that improve performance. Those approaches suffer from a lack of interpretability: the patterns are mapped into high dimen- sional latent vector space, they are not representable in the time domain, and are often even not localizable. In this paper, we present a novel neural convolutional architecture that aims provide a trade-off between interpretability and effective- ness based on the learning of dictionary of discrete representations. The pro- posed model guarantees (1) that small number of patterns are learned, and they are visualizable and interpretable (2) a shift equivariance property of the model associated with time-consistency of the representation (3) linear classifier over a limited number of patterns leading to an explainable decision. To ensure the ro- bustness of the discrete representation, they are learned in an unsupervised process independently of the classification task. This allows further great performances in transfer learning. We present extensive experiments on the UCR benchmark wrt usual baselines. The interpretability of the model is illustrated empirically. The chosen trade-off results obviously in a decrease in performance compared the state of the art. The performance drop is however limited and very dependent on the application domain. The experiments highlight the efficiency of the model for the transfer learning task, showing the robustness of the representations.
INTRODUCTION
Over the recent years, Deep Neural Networks (DNNs) have become efficient for Time Series Clas- sification (TSC) tasks (Fawaz et al., 2019 Tang et al. 2021 These models are highly capable of extracting complex discriminative Teatures different frequencies. In the meantime, representa- tion learning approaches demonstrate good performances for clustering (Ma et al., 2019). few-shot learning classification (Franceschi et al.) 2019 Malhotra et al. 2017). missing values imputation forecasting (Zerveas et al.) 2022). However, they lack interpretability. Indeed mapping signals into an abstract continuous space of high dimension which weights have no meaning prevents any justification of the decision resulting from these architectures. In their review of representation- based learning, [engio et al. (2013) emphasize the ability to extract Explanatory Factors and retain Temporal Coherence build good representations. Current approaches do not meet these criteria. Moreover, most of the time, these opaque models are highly specialized in a single task and cannot be transferred to other problems.
In this paper, we propose to discuss the fundamental properties required to build transparent, explainable, and potentially transferable time series classification process between tasks. First, con- cerning temporal aspects, we assume that important information has limited support. Therefore, we try to preserve both the shift equivariance of the extracted patterns and the temporal consistency of the latent representation (Cohen & Welling 2016 Bengio et al.) 2013). Second, to achieve transparent model interpretable by an expert, we impose that the extracted patterns belong to a dic- tionary of limited size and that their pre- pre-images in the time domain can be computed (Araujo al.) 2019). Third, we restrict ourselves to a strongly regularized linear classifier to explain the deci- sion at the model-fevel for each task (in addition to instance-wist wise interpretation) (James al.)

---

Under review as a conference paper at ICLR 2023
2013). Finally, want model that is ready for multi-tasking and transfer, i... model where the representation learned in an unsupervised way, independently from the decision module.
We present new convolutional architecture relying neural discrete representation inspired by Van Den Oord et al. (2017) that meets these criteria. Unsupervised representation learning is based discrete hierarchical autoencoder. The discretization is simple vector quantization, which efficiently removes noise in the manner of matrix factorization approaches (Gray 1984). The hier- archical aspect represents the best compromise to keep good expressiveness with atoms modeling phenomena at different frequencies while limiting the size of the pattern dictionary (Razavi et al., 2019). For the classification part, a logistic regression is used on the n-grams formed by the suc- cessive detections of the dictionary patterns, The model is penalized by an elasticnet regularization (Zou & Hastie 2005). We demonstrate the competitiveness of this processing chain on UCR data and compare the results to the state of the art while providing qualitative interpretations of the deci- sions (Dau et al. 2019). We propose an analysis of the origin of performance losses in some classes of applications to better identify the strengths and weaknesses of the different properties related to the interpretability.
1 Encoding
2
Discretization phase Pattem dictionnary
3
Decoding phase Additive reconstruction
Classification phase Penalized logistic regression
>>
Top elements decoded
Top level
N Cimmei
vQ
Fi
Codebook top
Counts
Histogram of n-grams
Upscaling
Stacking convolution
Support
EncoderT
Vq
level
Logistic regression
wWW
Nh.
Codebook bottom
Decodery
Bottom elements decoded
C1 (C,.C,..... C+1
Encoder
Decoder
Initial Time Series
X
Reconstruction
Unsupervised Loss MSE(X. 8) VQ Losses
Overview of the whole process: unsupervised representation learning (1+2+3) and logistic regression on extracted features (4). One should notice that the training of the architec- ture (1+2+3) is independent of the classification task. For more details on encoder's is and decoder's structure see Appendix A.T
Figure 1:
Our main contributions can be summarized as follows:
Formalization of the properties required for more transparent signal classification archi- tecture.
A model derived from Van Den Oord et al. (017)) taking into account the previous con- straints for more interpretability and allowing transfer between tasks.
A series of experiments demonstrating both the efficiency of the implemented architecture and the qualitative and interpretative interest of the previous constraints.
2 INTERPRETABILITY OBJECTIVES FOR TIME SERIES CLASSIFICATION AND GENERALIZATION
In the Time-Series Classification (TSC) field, interpretability can be instance-wise or/and class- wise. Instance-wise interpretability allows weighting the responsible areas of the instance for the
2

---

Under review as a conference paper at ICLR 2023
classification decision. But, it does not enable retrieval of class-specific features. Class-wise inter- pretability characterizes a category by a set of features extracted from the representations. Ideally, in the time series framework, these features should be expressive and localizable in time.
K-NN classifiers paired with Euclidean distance (ED) or Dynamical Time Warping (DTW) distance (Rakthanmanon et al.] 2012) are popular for time series classification tasks. Although these models are sometimes presented as instance-wise interpretable in the sense of reasoning by analogy, they do not allow discriminative feature retrieval for the classification tasks (for both instance and class levels). Most famous instance-wice interpetability methods are gradient-based methods. We can mention ResNet Fawaz et al.] (2019) that benefits from the class activation methods (Zhou et al.) 2016) because of the Global Average Pooling layer before the softmax classifier.
Some methods fall between class-wise interpretability and instance-wis interpretabildyy The dic- tionary method developed by [e Nguyen et al. (2019) relies on a linear classifier over a symbolic representation. The extracted symbolic representation is composed of local statistics that could be in the time-domain (Lin et al., 2003) as well as in the trequency-domate (Schäter & Högqvist, 2012). However, identical elements in the representation can be representative of different patterns. Thus, even if the authors can highlight the areas of interest for each time series, they cannot extract class- specific features.
In this paper, the proposed model aims to build class-wise interpretable model based on neural representation of patterns. In the following paragraphs, we list the associated required criteria.
A temporally consistent representation based on patterns. One of our main objectives is to build a representation with a meaningful temporal dimension, the elements of which are related to patterns the initial time series. The time consistency guarantees that the pre-images of two successive elements and b in the representation are two successive regions (R, and R₂) in the initial time series. These two regions might have a non-empty intersection and their borders are easily computable. Furthermore, an element of the representation must be decodable in the time domain independently of its neighbors. Generally, unsupervised neural models for time series do not allow the interpretation of the extracted features according to the temporal dimension. For example, Franceschi et al. (2019) builds a time scries representation using an encoder and a contrastive loss, triplet loss like Mikolov et al. (013) where the representation is squeezed along the temporal axis. Secondly, a desirable property for the encoded elements of the representation is the shift equivariance property (Cohen & Welling, 2016). It means that the proposed model should encode two identical patterns with the same value. Moreover, since our model is an autoencoder, we intend two identical elements of the representation to be decoded as the same pattern regardless of their position. Theoretically. it is not easy to satisfy these two properties in an efficient architecture but empirically, we can enforce our model to tend toward this aim.
A model that can be interpreted by an expert. By taking all of the above factors into account, we can obtain a time-consistent representation whose elements are identifiable, decodable, and lo- catable. However, we need additional requirements to make this representation interpretable for a human expert. First, to distinguish the elements of the representation and to make them mean- ingful, the elements must have discrete and limited support. Van Den Oord et al. (2017); Fortuin et al. (2019) demonstrate that image, audio, or video data can be accurately characterized by dis- crete factors derived from autoencoder based on the vector quantization mechanism (Gray) 1984). Secondly, in the time series framework, the temporal dimension of the initial time series has to be compressed through the encoder so that unique element of the representation must correspond to a (more or less global) pattern that is robust to noise. It is well known that compression is the key to noise robustness, by aggregating similar phenomena (Gillis & Vavasis 2013). For the compres- sion to be effective and meaningful, several hicrarchical levels of representations must be allowed to characterize the different frequencies of the time series (Razavi et al., 2019). The main difference between our proposal and the state of the art lies in the use of simple additive reconstruction: this approach limits the expressiveness of our model but allows us much greater interpretability of the discrete elements of the representation.
Multi-tasks Transfer The two previous constraints concerned the representation of time series. We impose in our architecture that the classification phase is decoupled from the representation learning, to be able to transfer the encoder-decoder from one application to another. The represen- tation learning is therefore unsupervised, the model being trained exclusively on a reconstruction criterion, in the manner of language models (Devlin et al., 2018). Any classifier can then exploit the time series representation. Franceschi et al. (2019) demonstrated that their trained unsupervised
3

---

Under review as a conference paper at ICLR 2023
model could be reused for new time series and that their classifier (SVM with RBF kernel) still performed well on the transferred representation. In contrast, for deep end-to-end models, the gains from transfer seem limited (Fawaz et al. 2018). Our experiments demonstrate the interest of our approach in a transfer setting.
An explainable decision by sparse linear model. The classification decision must preserve interpretability. Linear models such as logistic regression enable the interpretation of the relationship between the extracted features and the class. However, when the logistic regression support is too large, the linear decision is hard to understand. Thus, it is relevant to select the features used for the decision by using & penalty.
MODEL
As explained above, our global objective is 2 folds: build meaningful, robust & transferable rep- resentation of the input time series which respects several fundamental properties on the temporal level on the one hand (localization, translation, ...) and the transparency toward the expert on the other hand. We then train an efficient linear classifier. Our model is depicted in Figure] [
3.1 UNSUPERVISED REPRESENTATION LEARNING
To fulfill the first objective, we choose to use a convolutional auto-encoder architecture with a dis- cretization mechanism. As stated in (Bengio et al. 2013), one of the fundamental principles of representation learning is that the space of hidden factors characterizing the underlying distribution of the data is of a much lower dimension than the space of observations. Besides, dimension reduc- tion using neural networks (Hinton & Salakhutdinov 2006) allows focusing on important features of the signal while being less sensitive noise (Van Den Oord et al. (017). This architecture is built on the neural discrete model of Van Den Oord et al. (217)) Razavi et al. (2019) but we do not learn probabilistic distribution the representation space and
it for interpretability (not for
generation purposes). The mechanisms of this architecture allow us to respect the conditions stated in Section 2
First, at the encoder level, a sequence of convolutions interspersed with non-linearities allows one learn patterns underlying the time series while respecting the temporal consistency. Furthermore, Cohen & Welling (2016) proves that convolutions with strides of one respect the shift equivari- ance. If the stride greater than one, the theoretical shift equivariance is lost (Azulay Weiss, 2018) because the Ayquist-Shannon sampling theorem (Nyquist. 1928) is not respected anymore. However, using a stride of one does not adequately reduce the dimension. practice, we prefer to use strides of two and we observe that there is some empirical shift equivariance in that very similar patterns are encoded in the same way despite the dimension reduction and their localiza- tion. We have conducted series of experiments on the ShapeletSim dataset (Bagnall et al. 2017aa) where we quantify this phenomenon empirically (see Appendix (A.2). In the literature, MMouton iitrrature,MMouton et al. (202)) has demonstrated that the quality of the empirical shift equivariance depends the stride (the larger it is the more property lost) but also on the variance between intermediary elements in given window. They call this concept, local homogeneity. In practice, in our architecture, an encoder is a succession of convolutions that divides the size of the signal by two (stride of two) at each new convolution. These convolutions are interspersed with LearkyRelu activation functions. Once the desired dimension reduction is reached, we use residual blocks (which do not downscalc) for better training stability. For more details on an encoder block see Appendix A..
Once the time series has been projected into a vector space of smaller temporal dimension, we use vector quantization (VQ). describe the vector quantization mechanism in the context of this autoencoder we use the notations of Van Den Oord et al. (2017). Given a set of K centroids (ca e Rdd E 1, ..... K), vector quantization consists in assigning an output point of the encoder z, Rd to the nearest centroid:
Zq f ex where k = arg min = -
The benefit of the VQ mechanism, also known as the self orgf-orggnizing map Kohonen 1990), is that it enables the centroids to move as iterations go on. The number of centroids K is fixed before the learning process and is hyperparameter of the model. Then, 2g the vector of the point which
4

---

Under review as a conference paper at ICLR 2023
Ze has been assigned is given as input to the decoder. In nutshell, similar patterss are enforced to be encoded as very similar vectors thanks to empirical shift equivariance. And vector quantization mechanism assigns nearby vectors to the same centroid while respecting temporal coherence.
After having performed the VQ, we pass the quantized vectors to the decoder by ensuring that we keep the original temporality. For the choice of the decoder, parameters are symmetrical with those of the encoder. The transposed convolutions are chosen because they allow upscaling of the encoded signal while keeping temporal coherence. Moreover, the transposed convolutions are shift equivariant in translations if the element in the representation decode bounded by vectors of Oge. This property was not true in the instance- -wine interpretable classifier of Le Nguyen et al. (2019). However, in practice, there are always edge effects, but the reconstruction between the element and the same shifted element very close (see Appendix (A.2). In practice, decoder block is succession of transposed convolutions with stride of 2. The transposed convolutions are interspersed with LeakyRelu activation functions. We note that the upscaling parameters of the decoder are perfectly symmetrical with the downscaling parameters of the encoder. Moreover, we do not use any bias in our transposed convolutions to avoid the creation of possible artifacts. For more details on a decoder block see Appendix A.I
transposed convolutions whose
(A.2)
Finally, similarly to the architecture of Razavi et al. (2019). we set up two levels of representation: a level that characterizes the geggffregeency features and level that characterizes the low- now-frequency features (see Figure D. Thus, for the downscaling phase, there will be two successive encoders. A first encoder whose output will be used to characterize the high frequencies (bottom level) and a second one that takes as input the output of the first encoder and whose output will be used to characterize the low frequencies (top-level). As seen in Figure [ there will be a quantization for each frequency level. One note that before quantizing the high-frequency information, the output of the bottom encoder must be cleansed of the www-freeeeccy information. Finally, there will be two decoders for the two levels of representation. Each decoder will reconstruct with its information the signal in the initial space (the two decoders are not sequential but parallel). The final reconstruction will be the sum of these two decoded signals.
The unsupervised architecture is trained using the loss below, introduced by Van Den Oord et al. (017). To simplify the notations present the loss for single level model and an instance T; of a time series with T time steps. The transition to the hierarchical framework is not complicated, it is simply matter of summing up the different losses linked to the vector quantization for the different levels of representations. Let's ' introduce some notations. The operator sg stands for the stop gradient operator that blocks gradients during the backward computation time, D is the number of time steps in the representation space, C is the number of channels, and K is the number of available centroids for quantization. The encoder is defined de and the decoder as ther- Finally, the matrices involved in the model are E E KxC the set of K vectors that form the embedding space and Z E DxC' the temporal vectors sequence after quantization for instance i.
arg min || 1/0 (0,()))) [|sgióo 0,0
2 +110b0(zz)) sgl =g(E]] E|
The first term of the loss refers to the reconstruction ability of the model. In practice, we consider the mean square error pointwise between the real-time series and the decoder output. This part of the loss optimizes the decoder and the encoder. Although the argmin operator is not differentiable (at the vector at the VQ level in eq Van Den Oord al. (2017) advise passing the gradients of the embedding vectors to the encoder output vectors that have been assigned to them. Thus the reconstruction loss can optimize the encoder.
The second term of the loss refers to the codebook loss, it moves the embedding vectors according to the vectors that have been assigned to them. However, practice, we use an exponential moving average optimizer instead. That trick is employed by Van Den Oord et al. (2017) and reproduced in several articles (Razavi et al. 2019). It turns out to be empirically stable to train and allows the choose the solver independently for the rest of the loss. In the model implementation, the solver for the other terms of the loss is Adam (Kingma & Ba. 2014).
The last part of the loss is the commitment loss and ensures that the encoder outputs do not land too far from the embedding vectors. It guarantees better training stability. In practice we choose
= 0.25 Van Den Oord et al. (2017) and Razavi et al. (2099)
5

---

Under review as a conference paper at ICLR 2023
3.2 CLASSIFIER INSIGHTS: LOGISTIC REGRESSION ON HISTOGRAMS OF N-GRAMS
Similar to dictionary methods for TSC (Lin et al. 2003; Schafer & Högqvist 2012), the proposed representation learning process leads numerical representation of the time series: rather than the actual embedding vectors, the series are represented by the sequence of the indices of the vectors in the codebook. To perform the classification task in this context, n-grams are generally extracted and used as input to K-NN classifier or to linear classifier. For instance, Le Nguyen et al. (2017) proposes an efficient algorithm to find the most relevant n- n-grams for a logistic regression classifier.
In the following, only unigrams and bigrams are used for the two levels of representations. We show in Section f that with these simple statistics on our representations the results are already close to SOTA for unsupervised models. Bigrams provide a great indication of the local dynamics while remaining a relatively low dimensional space.
To obtain the final representation of time series i, the histogram h; of unigrams and bigrams is constructed by concatenating the two histograms of the levels of the representation. Logistic regression is used to solve the binary classification problem:
1 P arg min w w + p||0l|1 + wie
Σ log (exp (-ys (hi w + b)) + 1),
(3)
with C' the regularization parameter, p the trade-off between the & penalty and the lg penalty and b the bias. Sparsity in interpretable classification models is desirable to more easily understand the decision (Wu al.] 2018 Li al. 2019). In the case of multiclass classification, the usual One vs All method (Rifkin & Klautau 2004) is applied to provide better interpretability.
4 EXPERIMENTS AND RESULTS
4.1 QUANTITATIVE RESULTS: CLASSIFICATION TASK ON UCR TIME SERIES ARCHIVE
4.1.1 PROTOCOL AND HYPERPAAAAETEER SEARCH
In the following, for all considered datasets the number of centroids for vector quantization is fixed 32 for each scale. Experimentally, this hyperparameter is not very sensitive, but too few vectors lead worse performances as the representation is unable to catch important features, when too many are useless- numerous atoms correspond to very few representations and can lead to overfitting.
The main hyperparameters for our unsupervised architecture the number of convolutional lay- ers of the encoder providing the bottom time-scale reduction and the top time-scale reduction. We explore with grid search a reduction of {4, 8, 16, 32, 64, 128) for the bottom scale and of (2, 4, 8, 16, 32} for the top scale. Regarding the classification part, we explore various values for the hyperparameters p and C' of the logistic regression.
The protocol for the choice of the hyperparameters is the following: the unsupervised architecture trained for the different possible time-scale reductions on the whole dataset; the training dataset is divided into two parts, 70% used to train the classifier and 30% used for validation. The choice of time-scale reduction made through the results on the validation dataset. The optimal C and p hyperparameters are finally selected by cross-validating
the whole training [e
Unfortunately, many of the training data sets in the UCR archive are too small (resulting in a vali- dation set of 4 to 10 instances). Validating on such small data sets does not make sense and gives very variable results (Xu & Goodacre, 2018). Thus, we choose to consider only the datasets with validation set larger than 30 instances. Furthermore, we restrict the considered datasets to time series with enough time steps (> 80) and on supervised problems where the number of classes to predict is between 2 and 6. In summary, 39 datasets meeting these conditions considered in the following experiments. The presented results report the accuracy of the best model on the test set of each dataset.
To evaluate our classification performance, we compare ourselves to several efficient supervised and unsupervised methods. For supervised models, the considered baselines are: the ensemble method
'Training for a fixed hyperparameter set can range from 2 minutes to 2 hours depending on the dataset and the dimension reduction. For fixed dataset, we only use a single Nvidia Titan V for the unsupervised training

---

Under review as a conference paper at ICLR 2023
HIVE-COTE (Lines et al. 2018). the dictionary method BOSS Schäfer 2013). the random volutional kemnels method ROCKET (Dempster et al. 2020). the deep learning methods ResNet (Fawaz et al. 2019) and InceptionTime (Fawaz et al. 2020). For unsupervised models, compare our model to the results of Franceschi et al. (2019) (Triplet Loss + SVM), and I-NN's on Fuclidian distance and DTW distance. Finally, the random method consists in predicting each time the major- ity class. The reported results of the baselines are from the 2021 updated results of the survey by Bagnall al. (2017b) and the results of Franceschi et (2019)
4.1.2 QUANTITATIVE RESULTS
We present the results boxplot (see Figure to see how our method compares existing models.
Accurss
0.6
0.4
Random kemals
Unsupervised representation
Ensemble Dictionnary
Deep Learning
0.2
Durs
1-NN
1-NN DTW
Triplet
Cote
Recbet
Inception
(K-1)
Figure 2: Boxplot of accuracies. The results the 39 datasets respecting the conditions of the protocol. Triangles represent the arithmetic mean.
The aggregated results presented in the boxplot hide wide variety of applications: we detail these categories Table [ to show that the hierarchy of approaches is variable. Some tasks (like Chlo- rineConcentration or Yoga) require patterns learned on discriminative criteria but many data are very well handled by representations from auto-encoders. Other performance drops (on Image and Simulated) are due to the form of the classifier: ResNet with equal representations. For more details on the results see Appendix A.3
the gap between logistic regression and
Table Average Accuracies for unsupervised representation + classifier methods on the 39 datasets of interest by type. The last column shows the results of the best supervised end-to-end model. The best results are in bold and the second best results are underlined (if they exist).
Triplet
I-NN
Best supervised Hive Cote
Ours + LR
Ours + ResNet
I-NN ED
Loss
Type
Count
Random
DTW
+ SVM
(w=100)
(K=1)
Device
0.70
0.72
0.44
0.57
0.67
0.70
0.81
Image
13
0.70
0.75
0.49
0.73
0.72
0.78
0.82
Motion
0.91
0.91
0.53
0.86
0.84
0.93
0.95
Sensor
8
0.84
0.86
0.60
0.78
0.78
0.88
0.92
Simu
0.78
1.00
0.26
0.91
1.00
1.00
1.00
Spectro
0.69
0.63
0.47
0.61
0.56
0.67
0.89
Spectrum
3
0.73
0.80
0.34
0.51
0.70
0.77
0.94
Global
39
0.76
0.79
0.49
0.70
0.73
0.80
0.87
accuracy
In terms of unsupervised methods, our interpretable model (unsupervised representation + logistic regression) is slightly less accurate than Franceschi et al. (2019). However, when using powerful

---

Under review as a conference paper at ICLR 2023
classifier but less interpretable to see the structure of the ResNet in these experiments), the performances are improved. Even in this case, there is still small gap in accuracy compared to the best supervised models. We believe that this gap is bridged by the following two points. Firstly, our representation can extract features that will subsequently lead to interpretable and understandable classification (see Section 4.2) Besides, our feature extraction model is reusable for time series of the same size (see Section 4.3
representation such as ResNet (refer to the Appendix A.4
4.2 VISUALIZATION OF INTERPRETABILITY (USECASE: FRBEZERREGULARTER DATASET)
To illustrate the inter- pretability of our model, focus in this section on the analysis of the re- sults the FegezerReg- ularTrain dataset, where performs well (97% of accuracy). There are two classes, one represent- ing the power demand of the fridge freezer in the kitchen, and the other representing the power demand of the less fre- quently used freezer in the garage. After applying our procedure, book length is 19 (downscaling by 2d) and the top codebook length is 3 (downscaling by 27). The reconstruction is very accurate with an MSE loss of less than 0.002. Figure shows the weight of each feature of the learned classifier. Only a few features are used as a consequence of the regular- ization and those features belong mainly the bottom-scat scale representation.
Figure 3: Coefficients for the logistic regression. Here are the efficients of the logistic regression for the FreezorRegularTrainn dataset classification problem. The intercept is 0.15. The dotted line repre- sents the boundary between the features from the codebook bottom and features from the codebook top.
obtain representations where the bottom code-
Class 1
resonstructioo
-1
Bottom codebooks
50
300
Class 2
b*
Centroid reconstruciio
-1
The two patterns represent the peak power demand class class devices respectively, they differ only in power.
-2
Figure 4: Visualization of the decoded most important feature for each class according to the logistic regression coefficients. On the left, is a visualization of a few power curves for each class. On the right, the most discriminative features for each class. Patterns are rebuilt according the decoder and individually (in the representation space, neighhor embedding vectors are set to Oge) Moreover, when we look at the position of the centroid of interest and its receptive field, we notice that both patterns are related to the power demand peak.
Let us note a" and b* are the two most discriminative features with the higher coefficients in absolute. Their values respectively 4.23 and -1.86 (for intercept of 0.15). Since the logistic regression
8

---

Under review as a conference paper at ICLR 2023
acts histograms of unigrams and bigrams coding for centroid vectors, it is straightforward to retrieve the meaning of a* and b" Here, a" and b* correspond to unigrams appearing each time at most once in the bottom representation sequence. By reconstructing these centroids through the decoder, we see that these patterns represent the power demand of respectively the class device and the class 2 device (see 4 Thus, without any prior knowledge, it can be inferred from the model that the level of peak power is an important feature to tell apart the two devices. For another interpretable use-case see Appendix A.5
4.3 TRANSFER LEARNING
The other main ad vantage of our model is to reuse the learned representation for one time series for another time series of the same length to extract a representation. Then, we only have to learn the logistic regression for the specific classification problem and cross-validate hyperparameters (p and C). This process is fast and requires only one single CPU for the training phase (for the logistic regression). Table shows the transfer accuracy for datasets where the time series are composed of 80-time steps. For more results, see Appendix [A.6] In addition to being fast, the transferred representation enables good accuracy. The results sometimes the logistic regression on the representation learned on the time series itself. It demonstrates that our unsupervised model can learn and extract key features for different time series without being re-trained. Table 2 shows the transfer accuracy for datasets where the time series are composed of 80-time steps. For more results, see Appendix A.66
obtain the adapted
better than
Table 2: Transfer learning on time series of length 80. Columns indicate the source datasets whose unsupervised architectures are re-used. Rows indicate the target datasets. For each target dataset, the target time series is encoded thanks to the unsupervised model learned on the source dataset. The logistic regression is next trained on the obtained representations. Bold accuracies stand for the best results for each target dataset. Underlined accuracies stand for the results when the source and target datasets the same (without transfer). Dataset names shortened for the sake of space.
Target
MTW
MOC
MAG
DAG
DTW
DOC
PPAG
PPTW
PPOC
POC
MTW
0.49
0.53
0.52
0.47
0.49
0.47
0.57
0.47
0.54
0.52
MOC
0.70
0.70
0.74
0.65
0.69
0.70
0.70
0.66
0.70
0.74
MAG
0.55
0.56
0.49
0.57
0.57
0.55
0.62
0.53
0.55
0.55
DAG
0.63
0.71
0.74
0.68
0.73
0.70
0.70
0.71
0.65
0.77
DTW
0.58
0.58
0.63
0.57
0.57
0.53
0.61
0.58
0.53
0.58
DOC
0.71
0.74
0.75
0.69
0.69
0.75
0.75
0.67
0.67
0.75
PPAG
0.83
0.82
0.85
0.80
0.81
0.83
0.83
0.84
0.81
0.86
PPTW
0.73
0.76
0.76
0.79
0.77
0.74
0.80
0.76
0.71
0.76
PPOC
0.82
0.78
0.80
0.78
0.80
0.79
0.77
0.74
0.82
0.80
POC
0.71
0.73
0.73
0.70
0.71
0.73
0.72
0.67
0.68
0.79
For these classification problems which are relatively close, we observe that transferring the unsu- pervised model and training a logistic regression on top of it allows us to improve the accuracy.
CONCLUSION AND PERSPECTIVES
The proposed process enables building interpretable representations of time series, classifying ac- curately for wide variety of problems, and explaining the classification decision over the represen- tation. The reusable unsupervised model a step forward toward general semantics of time series as in image text. For future work, it seems interesting consider the choice of the reconstruction loss in the unsupervised architecture. Indeed, the mean squared error (MSE) tends to smooth the time series when the model reconstructs them, which weaken some important features. One can consider using the DILATE Loss (Le Guen & Thome (019)) instead of the MSE. Another lim- itation of our process is that we only consider unigrams and bigrams. By reusing the coordinate descent of lfrrim & Wiuf (011)), we can find longer sequences of centroids that allow us to discrim- inate classes. Finally, the last track to consider is use autoML methods to automatically find the number of hierarchical levels to use in the unsupervised model.

---

Under review as a conference paper at ICLR 2023
REFERENCES
André Araujo, Wade Norris, and Jack Sim. Computing receptive fields of convolutional neural networks. Distill, 2019.
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize poorly small image transformations? Journal of Machine Learning Research, 2018.
Anthony Bagnall, Aaron Bostrom, James Large, and Jason Lines. Simulated data experiments for time series classification part 1: accuracy comparison with default settings. arXiv preprint arXiv:1703.09988) 2017a.
Anthony Bagnall, Jason Lines, Aaron Bostrom, James Large, and Eamonn Keogh. The great time se- ries classification bake off: a review and experimental evaluation of recent algorithmic advances. Data mining and knowledge discovery, 2017b.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 2013.
Taco Cohen and Max Welling. Group equivariant convolutional networks. International conference on machine learning. 2016.
Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Ych, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The uer time series archive. IEEE/CAA Journal of Automatica Sinica, 2019.
Angus Dempster, François Petitjean, and Geoffrey Webb. Rocket: exceptionally fast and accu- rate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 2020.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.810.0000 140004805, 2018.
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and fierre-Alain Muller. Transfer learning for time series classification. IEEE international conference big data (Big Data), 2018
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. Deep learning for time series classification: review. Data mining and knowledge discovery, 2019.
Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey I Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petit- jean. Inceptiontime: Finding alexnet for time series classification. Data Mining and Knowledge Discovery, 2020.
Vincent Fortuin, Matthias Hüser, Francesco Locatello, Heiko Strathmann, and Gunnar Rätsch. Som- vae: Interpretable discrete representation learning on time series. International Conference on Learning Representations, 2019.
Jean-Yves Franceschi, Aymeric Diculeveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems, 2019.
Nicolas Gillis and Stephen A Vavasis. Fast and robust recursive algorithms for separable nonnegative matrix factorization. IEEE transactions on pattern analysis and machine intelligence, 36(4):698- 714, 2013.
Robert Gray. Vector quantization. JEEE Assp Magazine, 1984.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 2006.
Georgiana Ifrim and Carsten Wiuf. Bounded coordinate-descent for biological sequence classifica- tion in high dimensional predictor space. ACM SIGKDD international conference on knowledge discovery and data mining, 2011.
10

---

Under review as a conference paper at ICLR 2023
Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. An introduction statistical learning. 2013.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2014.
Teuvo Kohonen. The self-organizing map. Proceedings of the IEEE, 1990.
Vincent Le Guen and Nicolas Thome Shape and time distortion loss for training deep time series forecasting models. Advances in Neural Information Processing Systems, 2019.
Thach Le Nguyen, Severin Gsponer, and Georgiana Ifrim. Time series classification by sequence learning in all-subsequence space. IEEE international conference on data engineering (ICDE), 2017.
Thach Le Nguyen, Severin Gsponer, Iulia Ilie, Martin O' Reilly, and Georgiana Ifrim. Interpretable time series classification using linear models and multi-resolution multi-domain symbolic repre- sentations. Data mining and knowledge discovery, 2019.
Yuchao Li, Shaohui Lin, Baochang Zhang, Jianzhuang Liu, David Doermann, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Exploiting kernel sparsity and entropy for interpretable enn compres- sion. Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, 2019.
Jessica Lin, Eamonn Keogh, Stefano Lonardi, and Bill Chiu. A symbolic representation of time series, with implications for streaming algorithms. ACM SIGMOD workshop on Research issues in data mining and knowledge discovery, 2003.
Jason Lines, Sarah Taylor, and Anthony Bagnall. Time series classification with hive-cote: The hierarchical vote collective of transformation-basse ensembles. ACM Transactions on Knowledge Discovery from Duta, 2018.
Qianli Ma, Jiawei Zheng, Sen Li, and Gary W Cottrell. Learning representations for time series clustering. Advances in neural information processing systems, 2019.
Pankaj Malhotra, Vishnu TV, Lovekesh Vig, Puneet Agarwal, and Gautam Shroff. Timenet: Pre- trained deep recurrent neural network for time series classification. European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, 2017.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. International Conference on Learning Representatiinss ICLR, 2013.
Coenraad Mouton, Johannes C Myburgh, and Marelie H Davel. Stride and translation invariance in cnns. Southern African Conference for Artificial Intelligence Research, 2021.
Harry Nyquist. Certain topics in telegraph transmission theory. Transactions of the American Insti- tute of Electrical Engineers, 1928.
Thanawin Rakthanmanon, Bilson Campana, Abdullah Mueen, Gustavo Batista, Brandon Westover, Qiang Zhu, Jesin Zakaria, and Eamonn Keogh. Searching and mining trillions of time series subsequences under dynamic time warping. Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. 2012.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq- Advances in neural information processing systems, 2019.
Ryan Rifkin and Aldebaro Klautau. In defense of one-vs-all classification. The Journal of Machine Learning Research, 2004.
Patrick Schäfer. The boss is concerned with time series classification in the presence of noise. Data Mining and Knowledge Discovery, 2015.
Patrick Schäfer and Mikael Högqvist. Sfa: a symbolic fourier approximation and index for similarity search in high dimensional datasets. International conference on extending database technology, 2012.

---

Under review as a conference paper at ICLR 2023
Wensi Tang, Guodong Long, Lu Liu, Tianyi Zhou, Michael Blumenstein, and Jing Jiang. Omni-scale cnns: a simple and effective kernel size configuration for time series classification. International Conference on Learning Representations, 2021.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 2017.
Mike Wu, Michael Hughes, Sonali Parbhoo, Maurizio Zazzi, Volker Roth, and Finale Doshi-Velez. Beyond sparsity: Tree regularization of deep models for interpretability. AAAI Conference on Artificial Intelligence, 2018.
Yun Xu and Royston Goodacre. On splitting training and validation set: A comparative study of cross-validation bootstrap and systematic sampling for estimating the generalization performance of supervised learning. Journal of analysis and testing, 2018.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eick- hoff. A transformer-based framework for multivariate time series representation learning. ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2022.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. IEEE Conference Computer Vision and Pattern Recog- nition (CVPR), 2016.
Hui Zou and Trevor Hastie. Regularivation and variable selection via the elastic net. Journal of the royal stafistical society: series B (statisfical methodology), 67(2):001-320, 2005.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1
INSIDE AN ENCODER BLOCK OR A DECODER BLOCK
ENCODER STRUCTURE
INPUT (Batch size, Nb channels longth in)
Conv1D Kernel size = Stride
REDUCE OUTPUT
Conv1D Kernel size 3
ConvID Kernel size
FINAL OUTPUT (Batch size, Nb channels pul, seq.
Leaky
(Batch size, ous, seq. longth ous)
Leaky Relu
Paddligg Diiaiion
Paddlngg Dilatin =1
Padding 1. Dilation
Downscaling blocks
Residual blocks
DECODER STRUCTURE
INCREASED OUTPUT (Batch size, Nb channels seq. length in)
INPUT (Batch sire, Nb
Con/TransposeDD
Com'TransposeDD
Kerne size or
Kernel size 3 or Stride =
Stride 2
Leaky Relu
seq. length
Padding Dilatin 1,
Padding
Dilation
blas False
bias False
Upscaling blocks
Figure 5: Encoders and decoders architectures. The convolution kernels with strides of two allow to contract the temporal dimension and thus to aggregate local information. The dotted line means that several convolution blocks with stride of two can be added. At the end of the encoding, residual blocks added speed up the learning process. In the decoder, the choice of the parameters of the transposed convolutions is perfectly symmetrical with the parameters of the convolutions in the downscaling blocks. The dotted line means that several transposed convolution blocks with stride of two can be added. There is no bias in these transposed convolutions to avoid potential artifacts.
13

---

Under review as a conference paper at ICLR 2023
A.2 EMPIRICAL SHIFT EQUIVARIANCE, SHAPELETSIM DATASET EXPERIENCE
The theoretical shift equivariance of the non-strided convolutions guarantees that a pattern in the signal which occurs at time t or at time & + is encoded in the same way with a shift T More for- mally this can be written as follows. Let S be an element of the signal (for instance a subsequence), and let 7' be the group of discrete translations along the temporal axis. If we take to be any dis- crete translation in T and to be a function equivariant by discrete translation for then there exists r' E T such that
f(r(S)) -
In the proposed model, theoretical shift equivariance does not stand because of the strided convo- lutions (stride of 2). However, observe that empirically, our model has some empirical shift equivariance in the way patterns are encoded. The experiment below attempts to show this.
In the ShapeletSim dataset, there are 100 instances generated with white noise (class 0) and 100 instances generated with white noise as well as a randomly inserted triangle (class 1). The purpose of this experiment is to show that the triangle is encoded (most the time) in the same way in the codebook regardless of its position in the time series.
After applying our procedure 4.1.1 we obtain two representations where the bottom codebook length is 125 (downscaling by 29) and the top codebook length is 16 (downscaling by 29). The reconstruction loss O. 12 pointwise. In Figure 6 we see examples of reconstructions for instances of class 1.
4.5
+1.
Ncc stuuttie
-1.0
1.0
Figure 6: Visualization few time series for ShapeletSim dataset where triangle is encoded at random position (class 1). The blue curve stands for the true time series and the red one for its reconstruction.
In Figure 7 we see how an instance of class is decomposed between the codebook bottom and the codebook top.
14

---

Under review as a conference paper at ICLR 2023
Codebook Bottom
13 7.7778
Decoder
Reconamustion
Reccnsination Mestlc 122
1.0
10
Figure 7: Visualization of how the triangle is encoded for given position in time according the bottom/top decomposition.
We notice that for the 100 instances of class the triangles enconded almost always in the same way. However, there are some edge effects to take into account. Table shows for class the percentage of instances where the triangle is encoded with given sequence of centroids. We can also see the percentage of instances of class O where this given sequence of centroids is present.
Table 3: Pattern of the sub-sequence that encodes the triangle.
Centroides sequences '77'
White Noise (class 0)
White noise + triangle (class 1)
23 %
100 %
'777'
1 %
91 %
'13777'
%
88 %
'1377113
() %
75 %
The Table be interpreted as follows. In 91 % of the cases of class the triangle is encoded in the bottom codebook by the uub-sequence of centroids 7'. This sub-sequence of centroids is found in 1 % of the cases of class
To conclude with this experiment, distinguish class () from class 1. We from ResNet's performance that this task is not simple.
interesting to see how simple it is for other algorithms to
Table 4: Test accuracy comparison for few methods the ShapeletSim dataset
1-NN DTW
Triplet Loss (K=1)
1-NN ED
Ours + LR
ResNet
Hive Cote
(w=100)
+ SVM)
Test Accuracy
0.95
0.65
0.54
0.89
0.73
1.0
15

---

Under review as a conference paper at ICLR 2023
A.3 ADDITIONAL QUANTITATIVE RESULTS
Table 5: Results and comparisons for unsupervised methods on the 39 datasets of interest. The best results are bold and the second best results are underlined (if they exist). Sometimes dataset names are shortened for the sake of space.
Triplet Loss
Ours + LR
Ours + ResNet
Ran- dom
1-NN ED
1-NN DTW
Type
Dataset
(K=1)
SVM
Device
Computers
0.77
0.73
0.50
0.58
0.70
0.69
Device
SmallKitchenApp
0.70
0.69
0.33
0.34
0.64
0.68
Device
ScreenTypp
0.48
0.45
0.33
0.36
0.40
0.41
Device
RefrigerationDevices
0.47
0.55
0.33
0.39
0.46
0.53
Device
ECG200
0.77
0.84
0.64
0.88
0.77
0.87
Device
ECG5000
0.92
0.93
0.58
0.92
0.92
0.94
Device
LargeKitchenApp
0.77
0.83
0.33
0.49
0.79
0.80
mage
MiddlePhalanxOCC
0.70
0.78
0.57
0.77
0.70
0.75
Image
ProximaIPhalanxTWW
0.75
0.81
0.35
0.71
0.76
0.82
Image
ProximalPhalanxOCC
0.82
0.85
0.68
0.81
0.78
0.87
Image
ProximalPhalanXOAGG
0.83
0.84
0.49
0.79
0.80
0.85
Image
PhalangesOC
0.79
0.75
0.61
0.76
0.73
0.81
Image
MixedShapesRegulrrraii
0.67
0.86
0.27
0.90
0.84
0.92
Image
MiddlePhalanxTTW
0.50
0.56
0.29
0.51
0.51
0.60
Image
MiddIePhalanxOAG
0.49
0.63
0.57
0.52
0.50
0.62
Image
HandOutlines
0.88
0.89
0.64
0.86
0.88
0.92
Image
Yoga
0.70
0.73
0.54
0.83
0.84
0.82
Image
DistalPhalanxTWW
0.57
0.68
0.30
0.63
0.59
0.70
Image
DistalPhalanxOAG
0.68
0.71
0.47
0.63
0.77
0.72
Image
DistalPhalanxOC
0.75
0.72
0.58
0.72
0.72
0.76
Motion
GunPointOOv
1.00
0.99
0.52
0.95
0.84
1.00
Motion
GunPointMvV
0.97
0.99
0.53
0.97
1.00
1.00
Motion
GunPointAgeSppa
0.97
0.91
0.51
0.90
0.92
0.98
Motion
WormsTwoClass
0.70
0.75
0.57
0.61
0.62
0.75
Sensor
PreezerRegularThia
0.97
0.98
0.50
0.80
0.90
0.99
Sensor
Wafer
0.99
1.00
0.89
1.00
0.98
0.99
Sensor
Starl StarLiigtCCurves
0.96
0.96
0.58
0.85
0.91
0.96
Sensor
ChlorineConcentratiin
0.55
0.56
0.53
0.65
0.65
0.72
Sensor
Earthquakes
0.73
0.75
0.75
0.71
0.72
0.75
Sensor
FordA
0.89
0.93
0.52
0.67
0.55
0.92
Sensor
PowerCons
0.91
0.94
0.50
0.93
0.88
0.96
Sensor
FordB
0.75
0.79
0.50
0.61
0.62
0.79
Simu
TwoPatterns
0.78
1.00
0.26
0.91
1.00
1.00
Spectro
EthanolLevel
0.55
0.27
0.25
0.27
0.28
0.42
Spectro
Strawberry
0.93
0.90
0.64
0.95
0.94
0.95
Spectro
Ham
0.57
0.71
0.51
0.60
0.47
0.65
Spectrum
SemgHannSSubjccrhh2
0.72
0.88
0.20
0.40
0.73
0.77
Spectrum
SemgHandGenderChi
0.89
0.91
0.65
0.76
0.80
0.84
Spectrum
SemggaandMovemenchh
0.57
0.61
0.17
0.37
0.58
0.71
16

---

Under review as a conference paper at ICLR 2023
Table 6: Results and comparisons for supervised methods on the 39 datasets of interest. The best results are in bold and the second best results are underlined (if they exist). Sometimes dataset names are shortened for the sake of space.
Ours Res- Net
Incep- tion
Ours + LR
Hive Cote
Rock-
Res- Net
Type
Dataset
Boss
Time
Device
Computers
0.77
0.73
0.81
0.84
0.80
0.86
0.87
Device
SmallKitchenApp
0.70
0.69
0.83
0.82
0.75
0.80
0.77
Device
ScreenType
0.48
0.45
0.72
0.61
0.58
0.76
0.71
Device
RefrigerationDeviccs
0.47
0.55
0.56
0.54
0.44
0.52
0.51
Device
ECG200
0.77
0.84
0.86
0.90
0.88
0.88
0.90
Device
ECG5000
0.92
0.93
0.95
0.95
0.94
0.94
0.94
Device
LargeKitchen.App
0.77
0.83
0.92
0.93
0.84
0.95
0.95
Image
MiddlePhalanxOCC
0.70
0.78
0.81
0.83
0.81
0.82
0.83
Image
ProximalPhalTW
0.75
0.81
0.82
0.80
0.77
0.79
0.78
Image
PhalO
0.82
0.85
0.89
0.90
0.87
0.91
0.91
Image
PrG
0.83
0.84
0.86
0.85
0.83
0.82
0.82
Image
PhalangesOC
0.79
0.75
0.83
0.84
0.82
0.85
0.86
Image
MixedShapesReeTT
0.67
0.86
0.97
0.97
0.92
0.97
0.97
Image
MiddlePhalanee T
0.50
0.56
0.58
0.59
0.53
0.53
0.53
Image
MiddlePhalanx
0.49
0.63
0.70
0.71
0.66
0.60
0.59
Image
HandOutlines
0.88
0.89
0.92
0.94
0.90
0.94
0.95
Image
Yoga
0.70
0.73
0.91
0.91
0.91
0.88
0.91
Image
DistalPhalanxTTW
0.57
0.68
0.70
0.70
0.67
0.67
0.67
Image
DistalPhalanxOAG
0.68
0.71
0.82
0.81
0.82
0.78
0.77
Image
DistalPhalanxOC
0.75
0.72
0.82
0.82
0.81
0.81
0.82
Motion
GunPointOvv
1.00
0.99
1.00
0.99
1.00
1.00
1.00
Motion
GunPointMVv
0.97
0.99
1.00
1.00
1.00
0.99
1.00
Motion
GunPointAgeSpan
0.97
0.91
1.00
0.99
0.99
0.99
0.98
Motion
WornsTwoClass
0.70
0.75
0.79
0.79
0.81
0.77
0.80
Sensor
FreezerRegu Train
0.97
0.98
1.00
0.99
0.99
1.00
1.00
Sensor
Wafer
0.99
1.00
1.00
1.00
1.00
1.00
1.00
Sensor
StarLightCurves
0.96
0.96
0.98
0.98
0.98
0.97
0.98
Sensor
ChloConcentration
0.55
0.56
0.73
0.80
0.66
0.84
0.86
Sensor
Earthquakes
0.73
0.75
0.75
0.75
0.75
0.72
0.73
Sensor
FordA
0.89
0.93
0.94
0.94
0.92
0.93
0.96
Sensor
PowerCons
0.91
0.94
0.99
0.96
0.89
0.89
0.99
Sensor
FordB
0.75
0.79
0.93
0.81
0.91
0.82
0.86
Simu
TwoPatterns
0.78
1.00
1.00
1.00
0.99
1.00
1.00
Spectro
EthanolLevel
0.55
0.27
0.85
0.63
0.51
0.85
0.88
Spectro
Strawberry
0,93
0.90
0.98
0.98
0.97
0.97
0.98
Spectro
Ham
0.57
0.71
0.84
0.86
0.84
0.81
0.85
Spectrum
Semg
0.72
0.88
0.95
0.91
0.84
0.55
0.76
Spectrum
SemgHandGCh2
0.89
0.91
0.97
0.92
0.89
0.81
0.88
Spectrum
SemgHandMCh2
0.57
0.61
0.89
0.65
0.66
0.42
0.55
17

---

Under review as a conference paper at ICLR 2023
Table 7: Results for supervised methods the 39 datasets of interest by type. The best results
are in bold and the second best results are underlined (if they exist). Sometimes dataset names are shortened for the sake of space.
Ours +
Ours + ResNet
Hive Cote
Inception Time
Type
Count
Rocket
Boss
ResNet
LR
Device
7
0.70
0.72
0.81
0.80
0.75
0.82
0.81
Image
13
0.70
0.75
0.82
0.82
0.79
0.80
0.80
Motion
4
0.91
0.91
0.95
0.94
0.95
0.94
0.95
Sensor
0.84
0.86
0.92
0.90
0.89
0.90
0.92
Simulated
0.78
1.00
1.00
1.00
0.99
1.00
1.00
Spectro
0.69
0.63
0.89
0.82
0.77
0.88
0.90
Spectrum
0.73
0.80
0.94
0.83
0.80
0.59
0.73
18

---

Under review as a conference paper at ICLR 2023
A.4 RESNET ARCHITECTURE REPRESENTATION AND TRAINING PROCEDURE
Output (Batch size,
Nb classes)
Block linear: Linear layer LeakyRelu (3 times)
Concatenate output (Batch size, 2*Nb channels)
Flatten bottom output (Batch size, Nb channels)
Flatten top output (Batch size, Nb channels)
Global average pooling along temporal axis
Global average pooling along temporal axis
Block 2 ConviD LeakyRelu
Block 2 ConviD LeakyRelu (3 times) residual connection
(3 times) + residual connection
Block 1 Conv1D LeakyRelu (3 times) + residual connection
Block 1 Conv1D LeakyRelu (3 times) residual connection
Bottom sequential embedding (Batch size, Nb channels, seq. length
Top sequential embedding (Batch size, Nb channels, seq. length top)
bottom)
Figure 8: ResNet architecture on our unsupervised representation. It is important to note that in this case, the Resnet model takes as input the vectors of the representation and not the indices. This allows the model to take into account the distance between the centroids.
Procedure to train the ResNet representation. For the choice of the representation, take the same representation as for the logistic regression. Then, we split the initial training data into training (70%) and validation (30%). For this training, choose the following two hyperparameters:
weight decay (research in grid of 15 values between 10 and 10-1)
Number of epochs (research in grid of 10 values between 50 and 500).
The validation dataset allows us to select the right hyperparameters. One notice that no hyperpa- rameters research is done the architecture itself. The ResNet model aims show that it is easy to improve the accuracy by using a non-linear model on the representation.
19

---

Under review as a conference paper at ICLR 2023
A.5 ADDITIONAL RESULTS FOR INTERPRETABILITY POWERCONS DATASET
This dataset contains the individual household electric power consumption for one day. The sam- pling is 10 minutes, so the time series is 144 long. Time series are divided into two classes: class stands for the warm season and class 2 for the cold season.
Class
Class
-2
50
100
150
Warm season
Cold season
Figure 9: Overview of an instance for each class.
After applying our procedure, we obtain two representations where the bottom codebook length is 18 (downscaling by 29) and the top codebook length is (downscaling by 2*). The reconstruction loss is around 0.06 pointwise. In Figure 10 see the decomposition between the different frequency levels.
Rocor www.rriteee botion:
3.74
150
26
120
Reonsracian (HEEE44 41
Notrondeer
Figure 10: Bottom/Top decomposition for an instance. In the bottom plot, the blue curve stands for the initial time series, and the red curve stands for the reconstructed time series.
Once the unigrams and bigrams of the representations are extracted, we train a penalized logistic regression. The Figure II shows the coefficients of this logistic regression. Unlike the use-case 4.2 the top-level features seem to be discriminating for the classification problem.
20

---

Under review as a conference paper at ICLR 2023
Figure 11: Logistic regression coefficients. Intercept is 2.27. Top three features are bigram '9 7', unigram '26' and unigram "I' for the top level, their coefficient are respectively -2.17, 1.91, -1.76. The dotted line represents the boundary between the features from the codebook bottom and features from the codebook top.
Now let's look at the two most discriminating unigrams in detail. They are both extracted from the top codebook.
Figure 12: Decoded unigram '26' at third position (which is the most of the time in setting)
In Figure that the pattern favorably characterizing the 'cold season' class shows high power consumption in the evening.
Furi veom
Figure 13: Decoded unigram '1' second position (which the case most of the time in our setting)
In Figure 13 see that the pattern favorably characterizing the 'warm season' class shows a low power consumption during the middle of the day.
21

---

Under review as a conference paper at ICLR 2023
A.6 ADDITIONAL RESULTS FOR TRANSFER LEARNING
For the transfer Section 4.3 the transfer allows us to greatly improve our results where we were not very efficient before. Table 8 shows how the transfer allows to improve on these datasets.
Table 8: Results for the above datasets benefiting from the transfer and comparison with 1-NN
methods. Bold accuracies stand for best accuracies.
I-NN
Ours LR
Ours (transfer) + -LR
1-NN ED
Datasets
Random
DTW
(w=100)
MiddlcPhalanxTW
0.49
0.57
0.29
0.51
0.51
MiddlaPPhllanxxuttnneCCrrr
0.70
0.74
0.57
0.77
0.70
0.49
0.62
0.57
0.52
0.50
DistalPhalanxxOuuiineeegeee
0.68
0.77
0.47
0.63
0.77
DistalPhalanxTW
0.57
0.63
0.30
0.63
0.59
DisaaPhalannKOuttinCCorrrl
0.75
0.75
0.58
0.72
0.72
Proximd/PhalmmOOOttilekgggggug
0.83
0.86
0.49
0.79
0.80
ProximalPhalanxTW
0.76
0.80
0.35
0.71
0.76
Proximal/PallanxOuutiiCCCorr
0.82
0.82
0.68
0.81
0.78
ThalangesOutinesSCreee
0.79
0.79
0.61
0.76
0.73
Table 9 presents transfer results for other time series of length 720.
Table 9: Transfer time series 720 time steps long, Columns indicate the source datasets whose unsupervised architectures are re-used. Rows indicate the target datasets. Bold accuracies stand for the best results for each target dataset. Underlined accuracies stand for the result without transfer.
Target datasets
Computers
RefriDevices
ScreenType
SmalIKA
LargeKA
Computers Refrrievvces
0.77
0.63
0.71
0.60
0.67
0.47
0.47
0.48
0.46
0.48
ScreenType
0.53
0,46
0.49
0.38
0.42
SmallKA
0.69
0.67
0.61
0.70
0.71
LargeKA
0.78
0.73
0.71
0.67
0.77
22

---

Under review as a conference paper at ICLR 2023
A.7 UNSUPERVISED ARCHITECTURE VISUALIZATION
Vector Quantization
Top sequential embeddling
N Bomantls
encoding
AM
Convolutionnal
Encoder Top
Uptaampiigg
Decoder Top
Botoo encoding output
Nistomett
Convolutionaal
Mean Operator
wWW
Vector Quantization
Decoder Bottom
Recanstruction
Convolutionnal Encoder Bottom
det
Codebooo bottom
embodding
Figure 14: Unsupervised architecture details
A.8 COMPUTE RECEPTIVES FIELDS REGIONS
With our model, it is very useful to able to calculate the receptive fields and in particular the receptive field regions relative an element (or a region) of the representation. The formulas have been computed by Araujo et al. (2019):
40 = VT.
8;
+
Pt
ki)
8;
i=1
1=1
II
Ug
11.
8
1=1
Where
Up stands for the left-most coordinates of the receptive field in the intial time series
Up stands for the right-most coordinates of the receptive field in the intial time series
stands for the left-most coordinates in the representation
Tp stands for the left-right coordinates in the representation
ki stands for the kernel size at depth
st stands for the stride depth I
pt stands for the padding at depth
L stands for the depth of the network
23


Reviews:
[{"review_id": "L5XfEQq6KnG", "reviewer": "Reviewer_znQo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a model for Time Series Classification (TSC).\nThe model is made of two parts: first an encoder is trained to extract interpretable features using a VQ-VAE strategy and second, a classification head builds n-grams (in practice unigrams and bigrams) and performs regularized logistic regression on the histograms of n-grams.\n\nThe model is evaluated in terms of accuracy, transferability and interpretability.", "summary_of_the_review": "Though this paper tackles an interesting problem, my opinion is that the model is not especially novel (made of two standard blocks). \nThis would not be a problem if the experiments proved the superiority of the method wrt baselines but there are limitations in the conducted experiments that make the overall contribution too narrow.", "strength_and_weaknesses": "* Strengths:\n    * The paper deals with an important topic (interpretability in Time Series Classification)\n    * The overall idea to rely on unsupervised feature extraction to improve on transferability is interesting\n* Weaknesses\n    * The presentation of the state-of-the-art misses important work on the topic of interpretable time series classification\n    * The conducted experiments are not sufficient to convince the reader that the method is a good pick for transfer learning or interpretable TSC\n\nThese items are discussed in more details in the following.\n\n1. Literature review\n\nThe overall claim of the paper is that the proposed model leads to improved performance on interpretability and transfer learning.\nIn order to support this claim, a more thorough study of the literature on the topics would be necessary, in order to position the proposition wrt previous work.\n\n* One could cite:\n    * on interpretable TSC:\n\n        1. Fang et al. Efficient Learning Interpretable Shapelets for Accurate Time Series Classification (ICDE 2018)\n        2. Wang et al. Adversarial Regularization for Explainable-by-Design Time Series Classification (ICTAI 2020)\n\n    * on transfer learning for time series:\n    \n        3. Wilson et al. Multi-Source Deep Domain Adaptation with Weak Supervision for Time-Series Sensor Data (KDD 2020)\n\nAnd it would be helpful to use these works as baselines whenever possible on the interpretability / transferability experiments to showcase the potential superiority of the proposed method.\n\n2. Experiments\n\n    1. Evaluation of interpretability\n        * Here, interpretability is evaluated on a single example with no baseline, which makes it hard to evaluate the performance of the method in general.\n\n    2. Problems with the experimental setting:\n\n        > the unsupervised architecture is trained for the different possible time-scale reductions on the whole dataset (p.6)\n\n        * This will result in unsupervised representations that might overfit the test data, hence impair the overall evaluation\n\n        * Also, I do not understand why the related hyper-parameter $K$ is not discussed in Section 4.1.1, as it seems to be an important hyper-parameter for the method. In Figure 3, it seems that $K$ is chosen smaller for the top-scale codebook, why is that? (or did I misunderstand something?)\n\nSome additional minor comments:\n\n* Eq3 : the argmin should be over $\\mathbf{w}, b$ I guess\n* The following text appears twice in the same paragraph\n\n    > Table 2 shows the transfer accuracy for datasets where the time series are composed of 80-time steps. For more results, see Appendix A.6\n* In Section 4.3, by \"codebook length\", I guess the authors mean \"codeword length\". \"codebook length\" is misleading since it could correspond (or, at least, in my mind it does) to the number of atoms (codewords) in the codebook", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "COxSb_x6vJ", "reviewer": "Reviewer_icKJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an interpretable representation learning method to extract features from time series and use them for the classification task. The authors borrow the idea from unsupervised autoencoder, vector quantization, multi-level encoding and interpretable probing tasks. Joint learning with decoding recovery loss plus losses associated with the codebook, lead to a feasible and stable training framework for representation learning in time series. The author claims such scheme offers interpretable features from encoding, and can be used in time series classification and transfer learning tasks.", "summary_of_the_review": "Overall, given the fact that the proposed method is not competitive to other existing methods, and the important claim of interpretability is handwaving and vague, I feel there is lots of rooms to improve this paper.", "strength_and_weaknesses": "Pros:\n1. The interpretability in time series feature extraction, is largely of interests to the community.\n2. The paper is well written and easy to follow. The description of the complete system including the probing logistic regression task, is clear to understand. The Figure 1 helps a lot.\n3. The details of practical tricks including stop-gradient, the regularizer to stabilize training, etc, are provided.\n\nCons:\n1. The property \"we intend two identical elements of the representation to be decoded as the same pattern regardless of their position\", is not equivariance, but invariance. Equivariance requires a transformation in the input leads to a similar transformation in the output, e.g. x -> f(x), x+d -> f(x)+f(d). What described here is time-invariance or stationary process.\n2. The interpretability claim is very handwaving. From the example showed in Figure 4, one can easily detect the peak value difference from the original time series (left two figures). And shapes are also similar in the original series. The feature only verified this. To provide a more conniving example, the author should use some examples that is very hard to differentiate from the original time series, but easy to detect in the feature space, and the features (at least part of them) have a clear physical meaning for human to understand.\n3. The final performance is not as good as other TSC baselines.\n4. The transfer learning task does not have baselines to compare, thus hard to judge the value.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0XpqaesS2Wo", "reviewer": "Reviewer_5jZC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a neural network based approach for managing the so called tradeoff between interpretability and accuracy in timeseries data by learning a dictionary of discrete representations. They guarantee that 1) only a small number of patterns that can be visualised easily are learnt 2) training a linear classifier over a limited number of patterns provides a more explainable decision 3) a shift equivalence property of the model that is associated with a time consistency of the representation.", "summary_of_the_review": "The overall approach has promise but there are too many unanswered questions and parts that are either poorly explained or lack clarity or adequate comparisons. For this reason I am rejecting the paper.", "strength_and_weaknesses": "The authors try to tackle an important problem and provide some examples of how it works in multiple experimental domains.\n\nThe paper has several weaknesses. \n\n1) The authors never make it clear what they mean by interpretability in this particular context. Yet interpretability is inherently tied to the downstream task so its difficult to understand what any guarantees really mean here. Do the authors mean sparsity? Are sparse models all interpretable? What happens if the features themselves are not interpretable eg pixels in an image?\n\n2) The authors are missing several important contributions to understanding timeseries data. For example, Wu et al 2022 (see thesis https://dash.harvard.edu/handle/1/37371748) present a way of understanding and summarizing the crucial aspects of a time series using concepts. How does the proposed approach of learning a discrete dictionary compare to the idea of using concept bottleneck models to summarize important parts of the time series\n\n3) The biggest weakness is the lack of convincing application in the experiments section. The authors use the UCR dataset and explicitly state that validation is an issue because of lack of sample size so they instead focus on the sets of reasonable size yet not many details are provided about this. How long are the time series? How does performance vary over varying length of time series? If you learn a poor yet interpretable representation of the time series at a particular point, does it compound throughout the time series?\n\n4) The authors talk about the accuracy vs interpretability tradeoffs. I would like the authors to rethink that statement in light of this fantastic paper https://www.nature.com/articles/s42256-019-0048-x\n\n5) It is unclear how the approach generalizes across different applications and modalities of data.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject