Title: Abstracted Shapes as Tokens - A Generalizable and Interpretable Model for Time-series Classification
Venue: NeurIPS 2024
Accepted: yes
Keywords: Time-series

Interpretability

Self-supervised Learning

Pre-trained Model
Abstract: In time-series analysis, many recent works seek to provide a unified view and representation for time-series across multiple domains, leading to the development of foundation models for time-series data. Despite diverse modeling techniques, existing models are black boxes and fail to provide insights and explanations about their representations. In this paper, we present VQShape, a pre-trained, generalizable, and interpretable model for time-series representation learning and classification. By introducing a novel representation for time-series data, we forge a connection between the latent space of VQShape and shape-level features. Using vector quantization, we show that time-series from different domains can be described using a unified set of low-dimensional codes, where each code can be represented as an abstracted shape in the time domain. On classification tasks, we show that the representations of VQShape can be utilized to build interpretable classifiers, achieving comparable performance to specialist models. Additionally, in zero-shot learning, VQShape and its codebook can generalize to previously unseen datasets and domains that are not included in the pre-training process. The code and pre-trained weights are available at https://github.com/YunshiWen/VQShape.

Abstracted Shapes as Tokens - A Generalizable and Interpretable Model for Time-series Classification

Abstract

In time-series analysis, many recent works seek to provide a unified view and representation for time-series across multiple domains, leading to the development of foundation models for time-series data. Despite diverse modeling techniques, existing models are black boxes and fail to provide insights and explanations about their representations. In this paper, we present VQShape, a pre-trained, generalizable, and interpretable model for time-series representation learning and classification. By introducing a novel representation for time-series data, we forge a connection between the latent space of VQShape and shape-level features. Using vector quantization, we show that time-series from different domains can be described using a unified set of low-dimensional codes, where each code can be represented as an abstracted shape in the time domain. On classification tasks, we show that the representations of VQShape can be utilized to build interpretable classifiers, achieving comparable performance to specialist models. Additionally, in zero-shot learning, VQShape and its codebook can generalize to previously unseen datasets and domains that are not included in the pre-training process. The code and pre-trained weights are available at https://github.com/YunshiWen/ VQShape .

1 Introduction

As one of the fundamental forms of data, time-series (TS) exist in a wide range of domains and applications, including healthcare, weather, traffic, motions, human activities, sensors, etc. Modeling TS data across multiple domains has been a challenging task since TS data can have diverse sampling rates, lengths, magnitudes, frequencies, and noise levels. Due to this heterogeneity, most of the existing machine learning methods for TS modeling focus only on a single dataset or a single domain.

Recently, motivated by the success of large pre-trained models in natural language processing and computer vision, various approaches adopted from these two fields have been proposed to build a unified view and feature space for TS data from different domains [Liang et al., 2024]. Most of the models use a transformer as the backbone and pre-train it on a diverse range of datasets [Zerveas et al., 2021, Nie et al., 2023, Goswami et al., 2024]. These methods have achieved great success in TS representation learning, benefiting various downstream tasks and demonstrating their

generalizability. Despite their success, most of them remain black boxes since they cannot provide human-understandable representations. While tokenizers have played increasingly important roles in pre-trained models for language and vision, in TS, pre-training is often conducted by predicting the next or masked timestamp, time window, or patch, lacking the concept of discrete tokens as in LLMs. Very recently, Talukder et al. [2024] developed TOTEM, which utilizes VQ-VAE [van den Oord et al., 2017] to obtain the codebook and reconstruct the TS. Nevertheless, like all other VQ-VAE models, the tokens from the codebook are just latent vector representations and lack physical meaning.

Alternatively, in interpretable TS modeling, shapelets have been recognized as interpretable and expressive features for TS data. Initially defined as TS subsequences that discriminate different categories in classification [Ye and Keogh, 2011], they were later generalized to representative patterns [Grabocka et al., 2014]. Specifically, shapelets can transform TS data into low-dimensional representations either in the form of the distance between a shapelet and a TS, or as a logical predicate that measures the probability of a shapelet existing in a TS [Lines et al., 2012]. However, despite their effectiveness in classification tasks, this shape-level feature lacks flexibility since shapelets with pre-defined lengths are optimized for capturing discriminative features for making dataset-specific predictions. For example, when measuring human motion with accelerometers, an adult and a child performing the same gesture may record TS with different offsets, scales, and durations. Although they share the same shape-level concept, multiple shapelets are required to describe them separately. Additionally, shapelet-based interpretable models are specialized to a single dataset, and the learned shapelets fail to transfer to different domains.

In this paper, motivated by the limitations of existing pre-trained models and interpretable models in TS, we propose VQShape, a self-supervised pre-trained model that provides abstracted shapes as interpretable and generalizable tokens for TS modeling. Firstly, we decompose a TS subsequence into a set of attributes, including abstracted shape, offset, scale, start time, and duration. By incorporating vector quantization, VQShape learns a codebook of abstracted shapes that are generalizable and descriptive, representing TS from various domains. Evaluated on various classification tasks, and without fine-tuning, VQShape achieves comparable performance to black-box pre-trained models while additionally providing interpretable latent-space tokens and representations to describe TS data. Our contributions are summarized below:

We present an interpretable representation composed of abstracted shapes and attributes to describe TS data based on shape-level features, which enables the learning of datasetagnostic interpretable features.

We introduce VQShape, to the best of our knowledge, the first self-supervised pre-trained model that extracts interpretable representations from any TS data. VQShape also learns a codebook containing abstracted shapes that generalize to multiple datasets.

Pre-trained on diverse datasets and without fine-tuning, VQShape achieves comparable performance to existing black-box models on benchmark classification datasets. We explicitly demonstrate that the representations and VQShape are interpretable and generalizable for unseen datasets and domains.

2 Related Work

Deep learning methods for TS analysis. Deep learning methods are increasingly applied to TS analysis. Existing methods can be categorized into two groups depending on whether they use a Transformer structure as the backbone. For non-Transformer-based models, classical deep learning models such as MLP, CNN, and ResNet demonstrate decent performance on various tasks [Wang et al., 2017]. Recent methods have developed various feature engineering techniques to model explicit features of TS data. TimesNet [Wu et al., 2023] transforms TS into 2D space to capture multi-period features in a modularized way, achieving state-of-the-art performance on various tasks. TS2Vec [Yue et al., 2022] employs hierarchical contrastive learning for unsupervised representation learning of TS data. T-Rep [Fraikin et al., 2024] introduces a self-supervised representation learning approach by augmenting the TS with time embeddings, providing additional temporal structure to the latent space.

Transformers have been increasingly applied to TS analysis, but usually with some modifications to the original structure. For example, Autoformer [Wu et al., 2021] modifies the attention mechanism by incorporating an Auto-Correlation mechanism to capture temporal dependencies. When applying Transformers to real-valued data, transforming the inputs into patches has been recognized as an

effective approach for images [Dosovitskiy et al., 2021] since the tokens could contain more semantic meaning, like a 'word' in language. Similarly, PatchTST [Nie et al., 2023] shows that TS analysis also benefits from combining patched inputs with Transformers, viewing a TS as a sequence of 64 'words'.

Pre-trained Models for TS data. The success of large pre-trained models in language and vision motivates the development of foundation models for TS analysis. Existing approaches aim to find a unified view for TS data from different perspectives. For example, TST [Zerveas et al., 2021] uses the Transformer model [Vaswani et al., 2017] and is pre-trained using masked reconstruction, while TimeGPT-1 [Garza et al., 2023] is pre-trained by generating a forecasting window. MOMENT [Goswami et al., 2024] extends a patch-based Transformer [Nie et al., 2023] to multiple datasets by unifying the lengths of TS data using padding and sub-sampling. The model is also pre-trained to reconstruct the masked patches. TOTEM [Talukder et al., 2024] applies a convolutional neural network (CNN) encoder to raw TS data and uses vector quantization (VQ) on the encoder outputs, providing a discrete and domain-invariant codebook for TS data. TOTEM is pre-trained as a VQVAE [van den Oord et al., 2017] to reconstruct the whole TS, viewing the latent-space codes from convolutions as a unified representation. UniTS [Gao et al., 2024] introduces a prompt-based method to unify predictive and generative tasks within a single model and pre-training process. Although these methods learn representations that benefit various downstream tasks and demonstrate generalizability, these pre-trained models remain black boxes since they cannot provide human-understandable representations.

3 Proposed Method

Towards interpretable TS modeling, we first present the formulations of shape-level representations, describing univariate TS data using a set of abstracted shapes and attributes. Then, we introduce the architecture of VQShape and its components with detailed workflow and products from each step.

Notations. Let ( X , Y ) = { ( x i , y i ) | i = 1 , . . . , N } denote a TS classification dataset with N samples, where x i ∈ R M × T is a multivariate TS sample and y i ∈ { 1 , . . . , C } is the class label. Here, M is the number of variables, T is the length in timestamp, and C is the number of categories. Each multivariate TS sample x i can be viewed as a set of univariate TS samples where x i m ∈ R T denotes the TS at the m th variable. For simplicity in notations, in this paper, x m i,t 1 : t 2 denotes a subsequence of x i m between timestamp ⌊ Tt 1 ⌋ and ⌊ Tt 2 ⌋ , where t 1 , t 2 ∈ [0 , 1] are relative positions.

3.1 Shape-level representation

For a univariate TS x , a subsequence s k can be represented by an attribute tuple τ k = ( z k , µ k , σ k , t k , l k ) where

z k ∈ R d code is the code for abstracted shape of s k ,

µ k ∈ R 1 is the offset of s k ,

σ k ∈ R 1 is the scale (standard deviation) of s k and σ k > 0 ,

t k ∈ R 1 is the relative starting position of s k in x and 0 ≤ t k ≤ 1 -l min,

l k ∈ R 1 is the relative length of s k w.r.t. the length of x and l min ≤ l k ≤ 1 -t k .

Here, l min is the hyperparameter that defines the minimum length of a shape. We set l min = 1 / 64 as it is the length of a patch. In this work, we develop a pre-trained transformer model to produce a set of attribute tuples T = { τ k | k = 1 , . . . , K } given a univariate TS x . Additionally, the model learns a codebook of abstracted shape z that is reusable and generalizable for datasets from different domains.

3.2 VQShape Architecture

The VQShape model contains a TS encoder E , a TS decoder D , a latent-space codebook Z , a shape decoder S , an attribute encoder A enc, and an attribute decoder A dec. An overview of VQShape is presented in Figure 1. We then present a detailed formulation for each component.

Attribute Decoder

Quantization

Codebook

0

1

Shape

Decoder

(CNN)

Interpolate

Latent-space Operations

Attribute Encoder

Reconstruction

Time-series Decoder

(Transformer)

Forward Pass

Input Univariate Time-series

Instance Norm. + Patching

Linear Encoding + Positional Embedding

Time-series Encoder

(Transformer)

TS Encoding. VQShape contains a patch-based transformer encoder [Nie et al., 2023, Goswami et al., 2024] which first transforms a univariate TS x into K non-overlapping fixed-length patches with dimension d patch . Then, the patches are encoded by learnable linear projection and additive position embedding, forming patch embeddings that serve as inputs to a transformer model. The transformer outputs K latent embeddings ˆ h ∈ R d embed . Formally, the TS encoder is denoted by { ˆ h k ∈ R d embed | k = 1 , . . . , K } = E ( x ) . Note that ˆ h k could contain information from all patches instead of only the k th patch.

Attribute Decoding. The attribute decoder A dec takes a latent embedding h k and extracts an attribute tuple ˆ τ k = (ˆ z k , µ k , σ k , t k , l k ) . Formally, A dec performs

Each decoding function in { f z , f µ , f σ , f t , f l } is implemented using a multi-layer perceptron (MLP) with one hidden layer and ReLU activation. Following a common notation [Esser et al., 2021], ˆ τ denotes the attribute tuple before quantization.

Codebook and Vector-Quantization. The latent-space codebook is denoted by Z = { z q ∈ R d code | q = 1 , . . . , N code } . To learn a generalizable codebook that contains only the abstracted shape-level features, we use low-dimensional codes with d code = 8 . This configuration also creates a bottleneck for reconstruction, minimizing additional information that can be inferred besides the abstracted shapes. The quantization follows VQ-VAE [van den Oord et al., 2017] that selects the discrete code based on Euclidean distance where

Shape Decoding. The abstracted shape of a TS subsequence is a sequence with its length, offset, and scale information removed through normalizations. Given τ k = ( z k , µ k , σ k , t k , l k ) , we first extract the target subsequence from x specified by t k and l k denoted by x t k : t k + l k . Then, x t : t + l is interpolated to a fixed length of d s to remove the length information. The shape decoder S takes z k and outputs another sequence with the same length. Formally, for τ k , this step produces two sequences

Note that the output of S is normalized such that S ( z k ) has the offset and scale information removed.

Attribute encoding and reconstruction. The attribute tuple after quantization τ k = ( z k , µ k , σ k , t k , l k ) is transformed by a learnable linear projection denoted by h k ∈ R d embed = Linear ( τ k ) . Then, the TS decoder D takes { h k | k = 1 , . . . , K } and outputs the reconstructed TS ˆ x .

4 Pre-training

VQShape is pre-trained on diverse datasets to learn dataset-agnostic features and tokens. In this section, we introduce the self-supervised training strategies and objectives of VQShape. Then, we discuss the representations the model could provide to down-stream tasks.

4.1 Objectives

The optimization objectives of VQShape during the pre-training stage are summarized below.

Reconstructions. Analogous to most of the VQ-VAE approaches, VQShape is trained to accurately reconstruct the input TS to learn essential latent-space representations for modeling TS data. Additionally, to provide interpretable representations, the decoded shapes should be similar to the actual subsequences. Therefore, the reconstruction minimizes two objectives:

Time-series reconstruction:

Vector Quantization. We follow VQ-VAE [van den Oord et al., 2017] to define the vectorquantization objective which trains the encoder E and codebook Z . Additionally, inspired by Yu et al. [2024], we add additional entropy terms to encourage codebook usage. We find these terms could improve pre-training stability and avoid collapse of codebook usage. The objective for learning the codebook is defined by

where sg ( · ) is the stop-gradient operator and H ( · ) is the entropy function for discrete variables. q ( z, Z ) = softmax ([ ∥ ˆ z -z k ∥ 2 2 | z k ∈ Z ]) measures the distance between ˆ z and all codes in Z as a categorical distribution.

Disentanglement of shapes. In Equation 5, the attributes ( z k , µ l , σ k ) are optimized towards accurate subsequence reconstructions. It is important to note that, since ( t k , l k ) defines s target k , they are essential for learning the abstracted shapes and the codebook. However, it is challenging to use gradients from reconstruction in Equation 4 solely to learn ( t k , l k ) for making informative subsequence selection. Therefore, we introduce an additional regularization that encourages the latent-space tokens (attributes) to capture shape-level information with diverse positions and scales. This regularization is defined as

̸

In Equation 7, κ ( t k , l k ) defines a coordinate transformation which maps ( t k , l k ) into a space where (1) small l k values become more diverse and (2) large l k values from different t k become more concentrated. By making different ( t k , l k ) diverse in this space, L div encourages the model to capture disentangled shape-level information while increasing the use of short sequences to capture local details. Figure 8 visualizes an example of transformation κ . ϵ is a hyperparameter that defines a threshold distance in the transformed coordinate where two ( t k , l k ) samples are considered sufficiently diverse.

The overall pre-training objective is to minimize

where λ x , λ s , λ vq , λ div are hyperparameters that define the weighting between the components. During pre-training of VQShape, we set λ x = λ s = λ vq = 1 , λ div = 0 . 8 , and λ commit = 0 . 25 .

Design Analysis. Overall, the encoding process in VQShape (Transformer encoder and attribute decoder) introduces an inductive bias by representing and summarizing univariate TS using a set of abstracted shapes along with their position, length, offset, and scale. The pre-training objectives guide the components toward learning interpretable representations (via subsequence reconstruction in Equation 5) and disentangled representations (via regularization in Equation 7), while preserving the information necessary to describe the TS (via reconstruction in Equation 4). These objectives introduce interpretability to the conventional deep autoencoder structure. By pre-training on diverse datasets with a universal codebook, VQShape further leverages this inductive bias to produce discrete and dataset-agnostic representations, resulting in a vocabulary of abstracted shapes that can be used as primitives to describe TS data.

Model Configurations. The settings of VQShape related to the model size correspond to those of the MOMENT-Small [Goswami et al., 2024] model. Specifically, we interpolate all the input univariate TS x to have length T = 512 , which is broken into K = 64 patches with d patch = 8 . The Transformer layers in the encoder E and decoder D have 8 heads, an embedding dimension d embed = 512 , and a feed-forward layer of size 2048 . We employ an asymmetric structure with an 8-layer encoder E and a 2-layer decoder D [He et al., 2022]. The codebook Z contains N code = 512 codes, each with dimension d code = 8 . The subsequences s target k and decoded sequences s k have length d s = 128 . We set the minimum shape length l min = 1 / 64 . With these settings, VQShape has 37.1 million parameters.

In the pre-training stage, we train VQShape with the AdamW optimizer, using weight decay λ = 0 . 01 , β 1 = 0 . 9 , β 2 = 0 . 999 , gradient clipping of 1 . 0 , and an effective batch size of 2048. We employ a cosine learning rate schedule with an initial learning rate of 1 e -4 , a final learning rate of 1 e -5 , and 1 epoch of linear warm-up. The pre-training dataset contains univariate TS extracted from the training split of 29 datasets from the UEA Multivariate TS Classification Archive [Bagnall et al., 2018], excluding the InsectWingbeat dataset, resulting in 1,387,642 univariate TS. We train VQShape for 50 epochs on this dataset using bfloat-16 mixed precision.

4.2 Representations for down-stream tasks

VQShape provides two types of representations: Latent-space Tokens and Code Histogram .

Tokens. Similar to the latent-space feature map of typical VQ approaches such as VQ-VAE [van den Oord et al., 2017] and VQ-GAN [Esser et al., 2021], VQShape also provides a set of tokens as representations. For an input univariate TS x , the token representations are composed as T ∈ R K × ( d code +4 ) = { τ k = ( z k , µ k , σ k , t k , l k ) | k = 1 , . . . , K } . The token representations can be useful for general down-stream tasks but are less interpretable than the code histogram representations in classification tasks.

Code Histogram. Inspired by Concept Bottleneck Models (CBMs) [Koh et al., 2020] developed in computer vision, we can also view each z q ∈ Z as a concept for TS data. As CBMs have concept scores as representations, VQShape provides a similar representation in the form of a histogram of codes. Based on Equation 2, we can also have a vector of code indices

Then, the code histogram representation is defined as r ∈ R N code = histogram ( q ) where each element in r is the frequency of index q in q . Intuitively, the code histogram representation is analogous to BOSS [Schäfer, 2015] but with non-deterministic window size and dataset-agnostic symbols. In classification tasks, this type of representation can be more interpretable since classifiers based on these features are able to produce rule-like predictions that are straightforward to interpret and understand.

5 Experiments

Datasets. We evaluate the pre-trained VQShape on multivariate TS classification tasks to demonstrate the effectiveness of learned shape-level representations on down-stream tasks. The evaluations are performed on the test split of 29 datasets from the UEA multivariate TS classification archive [Bagnall et al., 2018, Ruiz et al., 2021], with the InsectWingbeat dataset excluded. Table 4 summarizes statistics of those datasets. Details on experiment setups are included in Appendix A.

Baselines. We benchmark VQShape against baselines from four categories, including (1) classical methods: DTW [Chen et al., 2013] and Shapelet Transform with Random Forest classifier (STRF) [Bostrom and Bagnall, 2017], (2) supervised learning methods: DLinear [Zeng et al., 2023], Autoformer [Wu et al., 2021], FEDformer [Zhou et al., 2022], PatchTST [Nie et al., 2023], and TimesNet [Wu et al., 2023], (3) unsupervised representation learning methods: TS-TCC [Eldele et al., 2021], TST [Zerveas et al., 2021], TS2Vec [Yue et al., 2022], and T-Rep [Fraikin et al., 2024], (4) models pre-trained on multiple datasets: MOMENT [Goswami et al., 2024] and UniTS [Gao et al., 2024]. We compute the classification accuracy as metrics and compare the methods based on the statistics of accuracy and ranking. Details on benchmarking setups are included in Appendix A.

Table 1: Statistic and comparisons of the baselines and VQShape. The best case is marked with bold, the second best is marked with italic, and the third best is marked with underline. Some baselines fail are incompatible with some datasets which result in 'N/A'. For fair comparison, we report the statistics with and without 'N/A'. Complete results are presented in Table 5.

Results. On multivariate TS classification tasks, for each TS sample x i ∈ X , we extract the token representations using VQShape and apply a learnable linear layer on this feature vector to predict the class label. Table 1 summarizes the performance of VQShape and baseline methods on the 29 UEA datasets. Complete results are presented in Table 5. From the results, we observe that the best-performing methods, including TimesNet, T-Rep, TS2Vec, MOMENT, and VQShape, have similar performance, and none of the methods has performance that is statistically significant compared to the others. Therefore, we conclude that VQShape achieves comparable performance to the state-of-the-art baselines while additionally providing interpretable pre-trained features.

Frozen Pre-trained Representations. Next, we compare the frozen pre-trained representations from three existing pre-trained models: MOMENT, UniTS, and VQShape. Since the pre-training dataset could have a dominant effect on the models, for fair comparisons, we reproduce MOMENTsmall and UniTS by training them on the same datasets as VQShape. Table 2 summarizes the statistics of the pre-trained models on the 29 UEA datasets. Complete results are presented in Table 6. From the results, we observe that VQShape outperforms MOMENT and UniTS on most of the datasets and in overall statistics.

5.1 Generalizability

The experimental results presented above evaluate models pre-trained on the training splits of the UEA datasets and tested on the test splits of the same datasets, demonstrating the models' ability to generalize to unseen samples from the same domains. Besides in-domain generalizability, VQShape

and its codebook can also generalize to datasets and domains not observed during pre-training. To demonstrate cross-domain generalizability, we train another model using 9 datasets from the UEA archive that are commonly selected to train and evaluate deep learning models [Zerveas et al., 2021, Wu et al., 2023], and then evaluate it on all 29 datasets. The right half of Table 2 summarizes the performance of this model, compared with MOMENT and UniTS trained with the same setup. We observe that VQShape and MOMENT trained on fewer datasets result in similar but slightly worse performance, indicating that the representations learned by the models can generalize to unseen domains.

5.2 Interpretability

Universal Codebook of Abstracted Shapes.

One of the most essential components of VQShape is the dataset-agnostic codebook that contains abstracted shapes. In Figure 6 of Appendix C.1, we decode all 512 codes in the codebook of VQShape to visualize their corresponding abstracted shapes. We observe that a large number of codes are decoded into similar shapes, which suggests that the codebook size can be further reduced. We then visualize the distribution of codes learned from pre-training (see Figure 7) which contains about 60 clusters. Inspired by this observation, we train a variant named VQShape-64 with codebook size N code = 64 . Figure 2 presents the decoded codebook of VQShape-64.

Interpretable Representations. Overall, the encoding of VQShape can be interpreted as 'TS x is decomposed into (shape z 1 with offset µ 1 and scale σ 1 , at t 1 with length l 1 ), . . . ', and the decoding can be interpreted as 'The composi-

s

0

s

1

s

2

s

3

s

4

s

5

s

6

s

7

s

8

s

9

s

10

s

11

s

12

s

13

s

14

s

15

s

16

s

17

s

18

s

19

s

20

s

21

s

22

s

23

s

24

s

25

s

26

s

27

s

28

s

29

s

30

s

31

s

32

s

33

s

34

s

35

s

36

s

37

s

38

s

39

s

40

s

41

s

42

s

43

s

44

s

45

s

46

s

47

s

48

s

49

s

50

s

51

s

52

s

53

s

54

s

55

s

56

s

57

s

58

s

59

s

60

s

61

s

62

s

63

tion of (shape z 1 with offset µ 1 and scale σ 1 , at t 1 with length l 1 ), . . . becomes ˆ x '. Figure 3 includes an example of interpretable representations learned by VQShape. From visualizations, we can confirm that VQShape can learn abstracted shapes that capture shape-level information with various positions and scales.

Discriminative Representations for Classification. We further show that the interpretable representations produced by VQShape also capture discriminative patterns that distinguish different categories in classification tasks. Figure 4 visualizes the average code histogram for samples from two categories. From the feature maps, it is obvious that several codes have significant differences in frequency between the two categories; these serve as discriminative features in classification tasks. We decode and visualize their corresponding abstracted shapes. The intuition provided by the histogram features can be interpreted as: 'Samples from the CW circle category usually contain shape s 61 in variate 1, and samples from the CCW circle category contain shape s 33 in variate 3, etc.'

VQShape

Encode

VQShape

Decode

Class: CW circle

Class: CCW circle

Variate 1

Variate 2

Variate 3

4

2

Q

b

4

2

0

m

2

7

Q

3

2

1

6

4

2

Q

6

口

4

2

0

5.3 Ablation Study

Representations and Codebook Scaling. We compare the performance of classifiers trained with token or histogram representations from the same pre-trained VQShape models. We also study how the size of the codebook affects the quality of these representations. Based on the results presented in Figure 5, we conclude that token classifiers always outperform histogram classifiers since token representations are more expressive than histogram representations from the same pre-trained model.

Additionally, the performance of token classifiers increases steadily as codebook size increases, since tokens from larger codebooks could contain more details. However, for histogram representations, choosing the appropriate codebook size is essential; we observe that a codebook size of 64 results in the best performance. These results match our previous observation in Section 5.2 where the model with N code = 512 results in only approximately 60 clusters of codes. A small codebook would learn codes that are too abstract to provide detailed representations, while a large codebook would learn similar codes that could lead to misleading features in the histograms. From

32

64

128

512

Codebook Size

0.7

0.71

0.72

Token

Histogram

the experimental results, we conclude that N code = 64 strikes the best balance between abstraction and expressiveness for pre-training on the UEA datasets, producing the best histogram representations.

Note that within the scope of this paper, the number of code clusters remains a post-hoc discovery that can only be determined after pre-training. One could start with a large codebook size such as N code = 512 , visualize the distribution of codes, and re-train with the appropriate codebook size. Developing a mechanism to adjust the codebook size dynamically during training or inference would be a valuable direction for future work.

Other Ablation Studies. We additionally conduct two ablation studies: (1) setting d code = 32 to compare high-dimensional and lowdimensional codes, and (2) setting λ s = 0 to assess the value of introducing shape-level information to the VQ-VAE structure. Table 3 summarizes the statistics of these two cases and the default setting. The complete results are presented in Table 7 and Table 8. From the overall statistics, pre-training with λ s = 0 leads to degraded performance, which indicates that learning shape-level abstractions through sub-

sequence reconstruction introduces useful information for classification tasks. Using codes with d code = 32 produces token classifiers with similar performance and better histogram classifiers. However, as stated in Section 3.2, we use low-dimensional code to create a bottleneck where the code should mainly contain the shape-level information. Using high-dimensional code may introduce additional information beyond the decoded shapes, which reduces interpretability. Therefore, the effect of using high-dimensional code would require extensive study.

6 Conclusion and Limitations

This paper introduces VQShape, a self-supervised pre-trained, interpretable, and generalizable model for TS analysis. Taking advantage of large pre-trained Transformers and ideas from shapelets, VQShape extracts a set of interpretable representations from TS data, composed of abstracted shape, offset, scale, position, and duration. Pre-trained and evaluated on multivariate classification datasets from the UEA archive, VQShape achieves comparable or even better performance than blackbox models pre-trained on large datasets and with significantly more parameters, while providing additional interpretable representations. Furthermore, using VQShape, we present a codebook containing a set of abstracted shapes that generalize to various TS datasets, including unseen datasets.

In this paper, we do not term VQShape a foundation model for TS data, since the amount of pretraining data is still limited compared to other large pre-trained models such as MOMENT, and we are focusing only on classification tasks because the extracted shape tokens are mostly interpretable for classification. As presented in Table 6, based on the comparison between VQShape pre-trained on 29 and 9 datasets, simply adding more pre-training data may degrade performance on some datasets. This suggests that learning shape-level representations may not benefit from some types of data. For example, the BasicMotion datasets mostly contain signals with high-frequency sinusoidal components that do not contain meaningful shape-level information in short subsequences. Such datasets may "pollute" the pre-training of VQShape since the model will try to capture unnecessary high-frequency features. Therefore, if pre-trained on datasets at scale, additional pre-processing and input engineering should be included; however, we do not conduct explicit pre-processing in this paper. It would also be an important future step to develop interpretable frameworks for other TS analysis tasks, such as forecasting, imputation, and anomaly detection, using the interpretable tokens extracted by VQShape.

Acknowledgments

This work was supported in part by IBM through the IBM-Rensselaer Future of Computing Research Collaboration. We thank Dr. Eamonn Keogh for his valuable comments and suggestions for revision. We also appreciate the UEA team's effort in creating and publicly sharing the benchmark datasets.

References

Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The UEA multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075 , 2018.

Aaron Bostrom and Anthony Bagnall. A shapelet transform for multivariate time series classification. arXiv preprint arXiv:1712.06428 , 2017.

Yanping Chen, Bing Hu, Eamonn Keogh, and Gustavo EAPA Batista. DTW-D: time series semisupervised learning from a single example. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 383-391, 2013.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2021.

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 2352-2359, 2021.

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12873-12883, 2021.

Archibald Felix Fraikin, Adrien Bennetot, and Stephanie Allassonniere. T-rep: Representation learning for time series using time-embeddings. In The Twelfth International Conference on Learning Representations , 2024.

Shanghua Gao, Teddy Koker, Owen Queen, Thomas Hartvigsen, Theodoros Tsiligkaridis, and Marinka Zitnik. UniTS: Building a unified time series model. arXiv preprint arXiv:2403.00131 , 2024.

Azul Garza, Cristian Challu, and Max Mergenthaler-Canseco. TimeGPT-1. arXiv preprint arXiv:2310.03589 , 2023.

Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. In International Conference on Machine Learning , 2024.

Josif Grabocka, Nicolas Schilling, Martin Wistuba, and Lars Schmidt-Thieme. Learning time-series shapelets. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 392-401, 2014.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022.

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning , pages 5338-5348. PMLR, 2020.

Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: A tutorial and survey. arXiv preprint arXiv:2403.14735 , 2024.

Jason Lines, Luke M Davis, Jon Hills, and Anthony Bagnall. A shapelet transform for time series classification. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 289-297, 2012.

Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations , 2023.

Alejandro Pasos Ruiz, Michael Flynn, James Large, Matthew Middlehurst, and Anthony Bagnall. The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Mining and Knowledge Discovery , 35(2):401-449, 2021.

Patrick Schäfer. The boss is concerned with time series classification in the presence of noise. Data Mining and Knowledge Discovery , 29:1505-1530, 2015.

Sabera J Talukder, Yisong Yue, and Georgia Gkioxari. TOTEM: Tokenized time series embeddings for general time series analysis. In ICLR 2024 Workshop on Learning from Time Series For Health , 2024.

Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems , 2017.

Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. In Journal of Machine Learning Research , volume 9, pages 2579-2605, 2008.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , volume 30, 2017.

Zhiguang Wang, Weizhong Yan, and Tim Oates. Time series classification from scratch with deep neural networks: A strong baseline. In 2017 International joint conference on neural networks (IJCNN) , pages 1578-1585. IEEE, 2017.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In Advances in Neural Information Processing Systems , 2021.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations , 2023.

Lexiang Ye and Eamonn Keogh. Time series shapelets: a novel technique that allows accurate, interpretable and fast classification. Data mining and knowledge discovery , 2011.

Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A Ross, and Lu Jiang. Language model beats diffusion - tokenizer is key to visual generation. In The Twelfth International Conference on Learning Representations , 2024.

Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. TS2Vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8980-8987, 2022.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 11121-11128, 2023.

George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. Atransformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining , pages 2114-2124, 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning , pages 27268-27286. PMLR, 2022.

A Experiment Setup

Environment. VQShape is implemented using Python 3.11.8 and PyTorch 2.2.2. The pre-training and evaluations are conducted on a machine with Intel Core i7-11700k CPU, 32GB of RAM, and a Nvidia RTX 4090 24GB GPU.

Classification tasks. Taking frozen pre-trained representations from VQShape, we learn a linear classifier to make predictions. When training the linear classifier, we found that token representations work better with regularization of L2 on classifier weights, while histogram representations are more compatible with dropout on features. Therefore, to avoid overfitting and obtain the optimal performance, we tune both the L2 regularization weight (or weight decay) and the dropout rate.

When learning the linear classifier, we repeat the experiment with five random seeds and report the average accuracy in Table 5. The standard deviations of the five runs are included in Table 7. Note that we exclude the InsectWingbeat dataset since the dataset contains inconsistent and very short TS samples such as T = 1 . Considering that the dataset has significantly more samples and channels than other datasets, the high volume of such short samples may have a negative effect on our method since the short TS do not contain any meaningful shape-level features.

Baseline Results. For baseline results presented in Table 1 and Table 5, we reproduce STRF with the Aeon-Tookit 2 and Scikit-learn packages. We reproduce DLinear, Autoformer, FEDformer, PatchTST, and TimesNet using implementation from Wu et al. [2023] 3 . Results of DTW, TS-TCC, TST, and TS2Vec are obtained from the TS2Vec paper [Yue et al., 2022]. Results of MOMENT

[Goswami et al., 2024] and T-Rep Fraikin et al. [2024] are obtained from the original papers. For results of UniTS [Gao et al., 2024], since the full results on the UEA datasets are not reported in the original paper, we follow the official implementation to reproduce the results using a pre-trained checkpoint 4 and prompt learning.

Benchmarking Frozen Pre-trained Models. To obtain the results in Table 2 and Table 6, we train MOMENT and UniTS from scratch using the official implementations 5 . Note that the way MOMENT, UniTS, and VQShape produce predictions can be different. MOMENT and VQShape provide frozen representations and learn a separate classifier, whereas UniTS learns additional prompt tokens and classification heads. Therefore, we follow the default procedure and the official implementation to evaluate UniTS. We evaluate MOMENT and VQShape under the exact same procedure of learning linear classifiers with regularization tuning.

Note that the key difference between results in Table 5 and Table 6 is that MOMENT and UniTS in Table 5 are pre-trained on a significantly larger volume of data (which includes the forecasting datasets), while the three models are only pre-trained on the UEA datasets in Table 6.

B Full Experiment Results

C Additional Visualizations

C.1 Visualization of abstracted shapes in the codebook

s

0

s

1

s

2

s

3

s

4

s

5

s

6

s

7

s

8

s

9

s

10

s

11

s

12

s

13

s

14

s

15

s

16

s

17

s

18

s

19

s

20

s

21

s

22

s

23

s

24

s

25

s

26

s

27

s

28

s

29

s

30

s

31

s

32

s

33

s

34

s

35

s

36

s

37

s

38

s

39

s

40

s

41

s

42

s

43

s

44

s

45

s

46

s

47

s

48

s

49

s

50

s

51

s

52

s

53

s

54

s

55

s

56

s

57

s

58

s

59

s

60

s

61

s

62

s

63

s

64

s

65

s

66

s

67

s

68

s

69

s

70

s

71

s

72

s

73

s

74

s

75

s

76

s

77

s

78

s

79

s

80

s

81

s

82

s

83

s

84

s

85

s

86

s

87

s

88

s

89

s

90

s

91

s

92

s

93

s

94

s

95

s

96

s

97

s

98

s

99

s

100

s

101

s

102

s

103

s

104

s

105

s

106

s

107

s

108

s

109

s

110

s

111

s

112

s

113

s

114

s

115

s

116

s

117

s

118

s

119

s

120

s

121

s

122

s

123

s

124

s

125

s

126

s

127

s

128

s

129

s

130

s

131

s

132

s

133

s

134

s

135

s

136

s

137

s

138

s

139

s

140

s

141

s

142

s

143

s

144

s

145

s

146

s

147

s

148

s

149

s

150

s

151

s

152

s

153

s

154

s

155

s

156

s

157

s

158

s

159

s

160

s

161

s

162

s

163

s

164

s

165

s

166

s

167

s

168

s

169

s

170

s

171

s

172

s

173

s

174

s

175

s

176

s

177

s

178

s

179

s

180

s

181

s

182

s

183

s

184

s

185

s

186

s

187

s

188

s

189

s

190

s

191

s

192

s

193

s

194

s

195

s

196

s

197

s

198

s

199

s

200

s

201

s

202

s

203

s

204

s

205

s

206

s

207

s

208

s

209

s

210

s

211

s

212

s

213

s

214

s

215

s

216

s

217

s

218

s

219

s

220

s

221

s

222

s

223

s

224

s

225

s

226

s

227

s

228

s

229

s

230

s

231

s

232

s

233

s

234

s

235

s

236

s

237

s

238

s

239

s

240

s

241

s

242

s

243

s

244

s

245

s

246

s

247

s

248

s

249

s

250

s

251

s

252

s

253

s

254

s

255

s

256

s

257

s

258

s

259

s

260

s

261

s

262

s

263

s

264

s

265

s

266

s

267

s

268

s

269

s

270

s

271

s

272

s

273

s

274

s

275

s

276

s

277

s

278

s

279

s

280

s

281

s

282

s

283

s

284

s

285

s

286

s

287

s

288

s

289

s

290

s

291

s

292

s

293

s

294

s

295

s

296

s

297

s

298

s

299

s

300

s

301

s

302

s

303

s

304

s

305

s

306

s

307

s

308

s

309

s

310

s

311

s

312

s

313

s

314

s

315

s

316

s

317

s

318

s

319

s

320

s

321

s

322

s

323

s

324

s

325

s

326

s

327

s

328

s

329

s

330

s

331

s

332

s

333

s

334

s

335

s

336

s

337

s

338

s

339

s

340

s

341

s

342

s

343

s

344

s

345

s

346

s

347

s

348

s

349

s

350

s

351

s

352

s

353

s

354

s

355

s

356

s

357

s

358

s

359

s

360

s

361

s

362

s

363

s

364

s

365

s

366

s

367

s

368

s

369

s

370

s

371

s

372

s

373

s

374

s

375

s

376

s

377

s

378

s

379

s

380

s

381

s

382

s

383

s

384

s

385

s

386

s

387

s

388

s

389

s

390

s

391

s

392

s

393

s

394

s

395

s

396

s

397

s

398

s

399

s

400

s

401

s

402

s

403

s

404

s

405

s

406

s

407

s

408

s

409

s

410

s

411

s

412

s

413

s

414

s

415

s

416

s

417

s

418

s

419

s

420

s

421

s

422

s

423

s

424

s

425

s

426

s

427

s

428

s

429

s

430

s

431

s

432

s

433

s

434

s

435

s

436

s

437

s

438

s

439

s

440

s

441

s

442

s

443

s

444

s

445

s

446

s

447

s

448

s

449

s

450

s

451

s

452

s

453

s

454

s

455

s

456

s

457

s

458

s

459

s

460

s

461

s

462

s

463

s

464

s

465

s

466

s

467

s

468

s

469

s

470

s

471

s

472

s

473

s

474

s

475

s

476

s

477

s

478

s

479

s

480

s

481

s

482

s

483

s

484

s

485

s

486

s

487

s

488

s

489

s

490

s

491

s

492

s

493

s

494

s

495

s

496

s

497

s

498

s

499

s

500

s

501

s

502

s

503

s

504

s

505

s

506

s

507

s

508

s

509

s

510

s

511

C.2 Code Distribution

In Figure 7, we plot the distribution of 512 codes in the codebook transformed to 2D space using tSNE [van der Maaten and Hinton, 2008]. The codes can be roughly cluster into 60 groups, suggesting that there may only exists 60 diversed abstracted shapes in the codebook.

80

60

40

20

0

20

40

60

t-SNE(1)

60

40

20

0

20

40

60

t-SNE(2)

C.3 Visualization of κ transform

We provide a visualization κ ( t, l ) transform discussed in Section 4.1 in Figure 8. The left figure shows ( t, l ) samples uniformly sampled from the original coordinate. According to Section 3.1, ( t, l ) samples can only appear in the lower-triangular plane. After the κ ( t, l ) transform, the corresponding samples are plotted in the transformed coordinate. Samples with small l in the original coordinate becomes more separated in the new coordinate, which encourage the model to capture local details in short subsequences. Samples with large l becomes more concentrated and are less sensitive to their t value since they are likely to capture redundant information.

Original Coordinate

Transformed Coordinate

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We claimed in the abstract and introduction that we propose a generalizable and interpretable representation for TS data, as well as a pre-trained model VQShape to extract the representations. We detailed the formulation and training procedure of VQShape, and validate with experiments on classification tasks that the model and its representations are indeed interpetable and generalizable.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We briefly discuss limitations in Section 5 and have present additional limitations in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not contain theoretical result that requires proof. But we demonstrate the effectiveness of our formulations using experiment results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We detailed the formulations and training objective of the model, as well as the experiment setups. These information is sufficient to reproduce our results without code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We open-source our implementations and pre-trained checkpoints. All the datasets used in this paper are publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We include detailed information about model architecture and hyperparamers in Section 4.1. Details on experiment setups are provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the standard deviation of classification accuracy in Table 7 and Table 8.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include detailed software and hardware environment in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Experiments in this paper is conducted only using publicly available datasets, without involving any human subjects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This paper focuses on the development of interpretable models for time-series data, which could potentially improve the transparency of deep leanring methdos.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not contains data or models that have a high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the datasets used in this paper are publicly available and properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: This paper introduce a new model and its implementation, which is documented and provided in the supplementary materials.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve any human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve any experiment that requires IRB approval or equivalent.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to provide an interpretable and generalizable model, called VQshape, for learning on time-series. The key idea of the work is to learn representation based on shapelets, or shape-level features of timeseries as quantized vectors. By learning such a codebook, VQshape can learn in a dataset-agnostic and interpretable way. The authors pre-trained the model on diverse datasets and showed that the proposed model achieves comparable performance as other models on UEA classification benchmark. They also studied the interpretablity and generalizability of the proposed method.\n\nSTRENGTHS:\n- Interesting idea. Using the concept of shapelet to learn time series may provide more interpretable models.\n\nWEAKNESSES:\n1. While the high-level idea is interesting, I am not convinced that the model worked.\n- Method 3.1 & 3.2: You use a transformer model to produce attribute tuples, why and how could you guarantee the produced attributes are accurate? PatchTST type of architectures fundamentally have this architetcure-level bias that breaks the shape of timeseries during the tokenization stage, how did you mitigate this side-effect and learn real shapes inside data?\n- In equation 5, is your subsequence a latent subsequence? If the answer is yes, how did you ensure the target latent subsequence has sufficient fidelity? If the answer is no, isn't your final reconstruction loss just a multiplication of the first term?\n- Line 188, what is the point of using shapelets if your model needs to interpolate all the input univariate? Did you consider that even in the same dataset, the length of the input may vary, and naively interpolating them would further mess up the sampling rate, even within the same dataset?\n- Figure 4 -- The learned codes are highly identical to each other. There lack high-frequency components in your constructed codebook.\n\n2. Experimental results are not convincing. \n- The proposed method has a lot of parameters, a lot of pre-trainings, yet it only performs similarly as TimesNet (which is much smaller and does not need pretraining).\n- Lack of other models as baselines.\n- Lack of experimental results as a whole. Can the proposed model be used for other tasks, e.g. regression, imputation, anomaly detection?\n\n3. The writing has serious clarity issues. Two examples below:\n- grammar error in Line 17\n- \"token\" in transformer has a very specific meaning. Yet the paper use the same word token for quantized vectors from the codebook.\n\n4. Lack of related works about interpretability.\n- There exist other ways to build interpretability into time series models. Common methods include: (1) Visualization of attention map across multi-scale transformer features [1]; (2) Use added tokens to \"prompt\" transformers for specific purpose [2]. When benchmarking the interpretability of the proposed approach, the authors should consider these methods. \n[1] Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., & Sutskever, I. (2020). Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341.\n[2] Xiao, J., Liu, R., & Dyer, E. L. GAFormer: Enhancing Timeseries Transformers Through Group-Aware Embeddings. In The Twelfth International Conference on Learning Representations.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nYes, they are mentioned in section 6", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces VQShape, an interpretable pretaining method for time series (TS) classification. VQShape uses transformer-based TS encoding combined with a VQ-VAE style codebook representation. The latter enables the representation of a TS as a set of shapelets. The VQShape encodes shapelets in a generalized form with additional information on length, position, and offset to overcome the typical limitations of shapelets, such as dataset specificity. The experimental evaluation highlights that VQShape performs on par with other pretraining methods while being interpretable and smaller.\n\nSTRENGTHS:\n- The paper introduces the novel method VQShape, which brings interpretability to time series pretraining.\n- To do this, VQShape introduces a modular and general way of utilizing shapelets over multiple datasets.\n- The paper is well-motivated and follows a clear structure. The writing is understandable and relatively easy to follow.\n\nWEAKNESSES:\n- The mathematical notation is not always precise and could need some clarification (specific details in the questions below)\n- The figures overall, but specifically Fig. 1, need more extensive captions to improve their comprehensibility.\n- The results in Tab. 1 do not include standard deviation information, which makes it difficult to judge the significance of the results and compare the different methods.\n\nQUESTIONS:\nQuestions and improvements to the mathematical notation:\n\n- Introducing $l_\\text{min}$ in 3.1 would make the representation easier to understand.\n- Similarly, providing formulas for $t_k$ and $l_k$ would be beneficial right there.\n- Further, for $l_\\text{min}$, $t_k$ and $l_k$ It is unclear if these should be integers or real numbers. While the text (and the later usage of them) points to them being relative, i.e., in \\[0, 1\\], the initial definition in lines 113 and 114, in combination with $T$ does not seem fitting to that. Also, when used to specify a time-series subsequence (e.g., line 142), they should be timestamps instead of being relative. The mathematical notation in this regard should be checked and aligned.\n- Relating $\\tau$ to $\\hat\\tau$ near l. 129 would make the section more digestible.\n- Overall, more intuition for the mathematics could make the paper even easier to understand.\n\nGeneral remarks and questions:\n\n- For me, the claim that VQShape outperforms MOMENT based on the results in Tab. 1 does not seem justified without further information. While the mean accuracy and mean rank of VQShape are a bit better than those of MOMENT, the differences are quite small compared to the large variations between the datasets overall. Claiming that VQShape outperforms moment would require some significance test, similar to comparing the mean rank to the other methods. (See \\[1\\] for more details on aggregation of results; see \\[2\\] for significance testing).\n- Regarding the results in Tab. 2, which show that VQShape can perform comparably even when trained on substantially fewer datasets: Do the authors have an intuition of how the other pretraining methods would do when pretrained on just a subset of the datasets?\n- Overall, all figures could benefit from better captions to give context to the figure. Each figure + caption should ideally be understandable on its own, or at least to a reasonable degree. In this paper, the figure captions do not provide the necessary context to the reader. Further, In the caption of Fig. 3: the \"presnetation\" should be \"presentation\"\n- In Figure 3: What are the different channels? Are these variates from the dataset? Further, the top and bottom parts each represent one sample, or did I misunderstand the histograms?\n- In the appendix, line 391 states that the standard deviations for Tab. 1 should be somewhere, but the reference is to the wrong table (presumably), and the standard deviations are nowhere in the appendix.\n- The clarity and the structure of the references could be improved. They are rather inconsistent (e.g., ICLR is noted differently among several entries) and contain a lot of unnecessary parts (e.g., URLs for published papers)\n- The visualization of the entire codebook in Appendix B.2 and B.3 is super insightful. However, many shapelets seem very similar, and their count should be easily reducible, as hinted in lines 264-268. Is it only visible in hindsight and necessary to learn a large codebook first, or could one start out with a smaller codebook right away?\n- The method introduces a possibly helpful indictive bias that would be difficult to enforce in deep auto-encoder architectures: It assumes that a TS can be summarized entirely by shapelets with position, scale, etc. This very implicit influence on the encoder could be discussed better.\n\nReferences:\n- \\[1\\] Fleming, Philip J., and John J. Wallace. \"How not to lie with statistics: the correct way to summarize benchmark results.\" _Communications of the ACM_ 29.3 (1986): 218-221.\n- \\[2\\] Demšar, J. Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research (JMLR), 2006.\n\nLIMITATIONS:\nWhile the main limitations are discussed in the paper, a short discussion of the trade-off between the codebook size could be beneficial: A larger size means potentially better learning, while a smaller codebook would be better for interpretability.\n\nThe main paper or the appendix should briefly discuss the societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors propose a self-supervised model which can be used for classification. Their method learns abstracted shapes which serves as interpretable tokens and an information bottleneck in their modeling architecture. They compare with state-of-the-art methods on standard classification datasets and show promising performance.\n\nSTRENGTHS:\n1. The paper is well written. \n2. The idea is conceptually interesting yet simple, and the results are promising. The authors also compare with some state-of-the-art methods. \n3. The paper provides interesting insights such as impact of the quality of pre-training data on downstream classification performance\n\nWEAKNESSES:\nI think my biggest concern in the paper is on the benchmarking aspect: (1) the authors use UEA repository, but I would also encourage them to use the UCR time series classification repository. As a strech goal, real world and harder time series classification datasets such as PTB-XL or MIT-BIH can also be used for benchmarking, (2) the authors compare with a very limited number of baselines. The MOMENT paper for example compares with a wide range of deep learning and statistical techniques such as TS2Vec, k-NN, etc. and I believe it is important to have these larger scale comparisons in the paper.\n\nQUESTIONS:\n1. Why did you exclude the InsectWingBeats dataset from your analysis? Why not use all the UCR and UEA datasets for classification? \n2. You found that a lot of decoded shapes are similar (increasing or decreasing lines), and found ~60 clusters. What was the impact of reducing the size of the code book on the performance of your model? \n3. How is the size of the codebook determined, is it always an afterthought? \n4. How can you encourage the decoded shapes to be as diverse as possible?\n\nLIMITATIONS:\nThe authors discuss some limitations of their approach in sections 5 and 6.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces VQShape, a model designed for time-series (TS) data representation learning and classification. VQShape provides interpretable and generalizable representations by utilizing vector quantization to create a codebook of abstracted shapes. These shapes represent low-dimensional codes that describe time-series data across various domains. VQShape represents time-series data by decomposing TS subsequences into attributes like abstracted shape, offset, scale, start time, and duration. This method allows for the creation of interpretable and dataset-agnostic features. The representations derived from VQShape can be employed to construct interpretable classifiers that achieve performance on par with specialized models tailored for specific tasks. VQShape also demonstrates robust generalization capabilities in zero-shot learning scenarios, where it can effectively handle datasets and domains that were not encountered during the pre-training phase.\n\nSTRENGTHS:\n- The paper introduces VQShape, a novel approach to time-series data representation that leverages vector quantization to create interpretable and generalizable representations. The concept of using abstracted shapes as tokens for time-series modeling is innovative, particularly in how these shapes are linked to the latent space features. \n- The paper is well-structured and articulates the motivations, methodology, and findings clearly. \n- Experimental results show the effectiveness of the approach when compared to baselines and other recent approaches.\n- The paper focuses on the task of learning represention for time series classification which is important in many domains.\n- The paper performs clear ablations to show the effectiveness of the proposed model components.\n\nWEAKNESSES:\n- The paper shows some interesting results on generalization towards unseen datasets but its not immediately clear how it compares to other approaches like MOMENT or TST. I would recommend performing this evaluation for other approaches too.\n- The paper can provide more ablations particularly on subsequence reconstruction loss as its one of the important component of the model.\n- How does the performance change when dimension of code size is increased?\n- Other clarification questions: how to define l_min? how to get h_k from tau_k?\n\nQUESTIONS:\n- Performance comparison on zero-shot evaluation with other approaches.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}