Abstracted Shapes as Tokens - A Generalizable and Interpretable Model for Time-series Classification
Authors: Yunshi Wen
Tengfei Ma
Tsui-Wei Weng
Lam M. Nguyen
Anak Agung Julius
Venue: NeurIPS 2024 poster (2024)
Keywords: Time-series
Interpretability
Self-supervised Learning
Pre-trained Model
Primary area: interpretability_and_explainability
TL;DR: A generalizable, interpretable, pre-trained model for time-series modeling and classification.

Abstract: In time-series analysis, many recent works seek to provide a unified view and representation for time-series across multiple domains, leading to the development of foundation models for time-series data. Despite diverse modeling techniques, existing models are black boxes and fail to provide insights and explanations about their representations. In this paper, we present VQShape, a pre-trained, generalizable, and interpretable model for time-series representation learning and classification. By introducing a novel representation for time-series data, we forge a connection between the latent space of VQShape and shape-level features. Using vector quantization, we show that time-series from different domains can be described using a unified set of low-dimensional codes, where each code can be represented as an abstracted shape in the time domain. On classification tasks, we show that the representations of VQShape can be utilized to build interpretable classifiers, achieving comparable performance to specialist models. Additionally, in zero-shot learning, VQShape and its codebook can generalize to previously unseen datasets and domains that are not included in the pre-training process. The code and pre-trained weights are available at https://github.com/YunshiWen/VQShape.

Abstracted Shapes as Tokens A Generalizable and Interpretable Model for Time-series Classification
Yunshi Wen
Tengfei Ma"
Rensselaer Polytechnic Institute weny2orpi.edi edu
Stony Brook University
tengfei www.mmamyooooooooooo...o edu
Tsui-Wa Weng
Lam M. Nguyen
Anak Agung Julius Rensselacr Polytechnic Institute
University of California,
IBM Research
San Diego lweng@ucsd.c edu
lamnguyen. ml mwt.dibm.ccmm
agungdecse rpi rpt.ee.ecerrrieuuuu edu
Abstract
In time-series analysis, many recent works seek to provide a unified view and representation for time-series across multiple domains, leading to the development foundation models for time-series data. Despite diverse modeling techniques, existing models black boxes and fail provide insights and explanations about their representations. generalizable, and interpretable model for time-series representation learning and classification. By introducing novel representation for time-series data, we forge connection between the latent space of VQShape and shape-level features. Using vector quantization, we show that time-series from different domains can be described using a unified of low-dimensional codes, where each code can be represented as an abstracted shape in the time domain. On classification tasks, we show that the representations of VQShape can utilized build interpretable classifiers, achieving comparable performance to specialist models. Additionally, in zero-shot learning, VQShape and its codebook can generalize to previously unseen datasets and domains that are not included in the pre- pre-training process. The code and pre-trained weights are available at athttps: : //githubb www./omm/////// homm/unsshiWe/// VQShape
this paper, we present VQShape, a pre-trained,
Introduction
As of the fundamental forms of data, time-series (TS) exist in wide range of domains and applications, including healthcare, weather, traffic, motions, human activities, sensors, etc. Modeling TS data across multiple domains has been challenging task since TS data can have diverse sampling rates, lengths, magnitudes, frequencies, and noise levels. Due to this heterogeneity, most of the existing machine learning methods for TS modeling focus only on single dataset or a single domain.
Recently, motivated by the success of large pre-trained models in natural language processing and computer vision, various approaches adopted from these two fields have been proposed build a unified view and feature space for TS data from different domains [Liang et al.] 2024]. Most of the models
transformer the backbone and pre-train it on diverse range of datasets [Zerveas et al. 2021 Nie et al. 2023 Goswami et al. 2024]. These methods have achieved great success in TS representation learning. benefiting various downstream tasks and demonstrating their
"Corresponding Yunshi Wen (veny20rpi.c edu) and Tengfei Ma (tengfei www.stoyybrook.coommmmmmm edu)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

generalizability. Despite their success, most of them remain black boxes since they cannot provide human- uunaan-nnddsttaaaalee representations. While tokemizers have played increasingly important roles in pre-trained models for language and vision, in TS, pre-training is often conducted by predicting the next or masked timestamp, time window, or patch, lacking the concept of discrete tokens as in L.S.s. Very recently. Talukder et al. (2024) developed TOTEM, which utilizes VQ-VAE [van den Oord et al., 2017] to obtain the codebook and reconstruct the TS. Nevertheless, like all other VQ-VAE models, the tokens from the codebook are just latent vector representations and lack physical meaning.
Alternatively, in interpretable TS modeling, shapelets have been recognized as interpretable and expressive features for TS data. Initially defined as TS subsequences that discriminate different categories in classification Ye and Keogh| [011]] they were later generalized to representative patterns Grabookk et al. 2014]. Specifically, shapelets can transform TS data into low-dimensional representations either in the form of the distance between a shapelet and a TS, or as a logical predicate that measures the probability of a shapelet existing in TS [Lines et al., 2012). However, despite their effectiveness in classification tasks, this shape-level feature lacks flexibility since shapelets with pre-defined lengths are optimized for capturing discriminative features for making -ataset-specific predictions. For example, when measuring human motion with accelerometers, an adult and child performing the same gesture may record TS with different offsets, scales, and durations. Although they share the same shape-level concept, multiple shapelets are required to describe them separately, Additionally, shapelet- thapelet-based interpretable models specialized to single dataset, and the learned shapelets fail to transfer to different domains.
In this paper, motivated by the limitations of existing pre-trained models and interpretable models in TS, we propose VQShape, a self-supervised pre-trained model that provides abstracted shapes as interpretable and generalizable tokens for TS modeling. Firstly, we decompose a TS subsequence into a set of attributes, including abstracted shape, offset, scale, start time, and duration. By incorporating vector quantization, VQShape learns a codebook of abstracted shapes that are generalizable and descriptive, representing TS from various domains. Evaluated on various classification tasks, and without fine-tuning, VQShape achieves comparable performance to black-box pre-trained models while additionally providing interpretable latent-space tokens and representations to describe TS data. Our contributions are summarized below:
We present an interpretable representation composed of abstracted shapes and attributes to describe TS data based on shape-level features, which enables the learning of dataset- agnostic interpretable features.
We introduce VQShape, to the best of our knowledge, the first self-supervised pre-trained model that extracts interpretable representations from any TS data. VQShape also learns a codebook containing abstracted shapes that generalize to multiple datasets.
Pre-trained on diverse datasets and without fine-tuning, VQShape achieves comparable per- formance to existing black-box models benchmark classification datasets. We explicitly demonstrate that the representations and VQShape are interpretable and generalizable for unseen datasets and domains.
2 Related Work
Deep learning methods for TS analysis. Deep learning methods are increasingly applied TS analysis. Existing methods can be categorized into two groups depending on whether they use Transformer structure as the backbone. For non-Trranforrmer-bass models, classical deep learning models such as MIP, CNN, and ResNet demonstrate decent performance on various tasks (Wang] ct al. 2017 Recent methods have developed various feature engineering techniques to model explicit features of TS data. TimesNet [Wu et al., 2023 transforms TS into 2D space capture multi-period features in a modularized way, achieving state-of-the-art performance on various tasks. TS2Vec Yuu сt al. 20 employs hierarchical contrastive learning for unsupervised representation learning of TS data. R-Rep [Fraikin et al.) 2024 introduces a supervised representation learning approach by augmenting the TS with time embeddings, providing additional temporal structure to the latent space.
Transformers have been increasingly applied to TS analysis, but usually with some modifications to the original structure. For example, Autoformer [Wu et al. 2021] modifies the attention mechanism by incorporating an Coro-Correlation mechanism to capture temporal dependencies. When applying Transformers to real-valued data, transforming the inputs into patches has been recognized as an

---

effective approach for images Dossoviskkiy et al. 2021] since the tokens could contain more semantic meaning, like "word" in language. Similarly, PatchTST (Nie al. 2023] shows that TS analysis also benefits from combining patched inputs with Transformers, viewing a TS as sequence of 64 "words".
Pre-trained Models for TS data. The success of large pre-trained models in language and vision motivates the development of foundation models for TS analysis. Existing approaches aim to find unified view for TS data from different perspectives. For example, TST [Zerveas al. 2021]
uses the Transformer model [Vaswani et al. 2017]] and is pre-trained using masked reconstruction.
while TimeGPT-1 [Garza al. [023] pre-trained by generating a forecasting window. MOMENT
Goswami et al. 2024 extends patch-based Transformer (Nie et al. 2023] to multiple datasets by unifying the lengths of TS data using padding and sub-sampling. The model is also pre-trained to reconstruct the masked patches. TOTEM (Talukder et al. [024] applies a convolutional neural network (CNN) encoder to raw TS data and uses vector quantization (VQ) on the encoder outputs, providing discrete and domain- invariiiv codebook for TS data. TOTEM is pre-trained as VQ- VAE [van den Oord et al., 2017 to reconstruct the whole TS, viewing the latent-space codes from convolutions as unified representation. UniTS [Gao et al. (024) introduces prompt-hasee hhaept method to unify predictive and generative tasks within single model and pre-training process. Although these methods learn representations that benefit various downstream tasks and demonstrate generalizability, these pre-trained models remain black boxes since they cannot provide human- audrensddnndsst representations.
Proposed Method
Towards interpretable TS modeling, first present the formulations of shape-level representations, describing univariate TS data using a set abstracted shapes and attributes. Then, we introduce the architecture of VQShape and its components with detailed workflow and products from each step.
Notations. Let (X. = {2 9:)li = I...., N} denote TS classification dataset with N samples, where 2; M a multivariate TS sample and y; fl,....CC C} the class label. Here, M is the number of variables, T is the length in timestamp, and C' is the number of categories. Each multivariate TS sample v; can viewed as a set of univariate TS samples where x;" R denotes the TS at the mh variable. For simplicity in notations, in this paper, .m i2 denotes subsequence of ar"h between timestamp [Tt,] and (Tt2), where 1, t2 E [0, 1] are relative positions.
3.1 Shape-level representation
For univariate TS subsequence 8% be represented by an attribute tuple The (2k, Mk, kk, tk. 1k) where
E Rook is the code for abstracted shape of sk.
E R= is the offset of 8k.
R is the scale (standard deviation) of and 0x > 0,
* ty E R is the relative starting position of sy in and 0 < tx S Imin-
. la E IR3 is the relative length of 85 w.r.t. the length of and mmin S lg < L2-
Here, 4min is the hyperparameter that defines the minimum length of shape. We set Imin = 1/64 as the length of a patch. In this work, we develop pre-trained transformer model to produce set of attribute tuples T= = [7x k = 1,..., K) given a univariate TS x. Additionally, the model learns a codebook of abstracted shape that reusable and generalizable for datasets from different domains.
3.2 VQShape Architecture
The VQShape model contains TS encoder E, a TS decoder D, latent-space codebook 2, shape decoder S, an attribute encoder Aenes and an attribute decoder Aite.. An overview of VQShape presented in Figure We then present detailed formulation for each component.
3

---

Lines Encoding Postoonal Embeeding
Lalientsiazee Operations
Iepart Unfuaciate Time- Ail.
Instanee Nom. Peachin
Timesse Encooeer
Afriuu Deccnder
(Taundfrmee)
Shace (CNN)
fs
Rxoma
Figure 1: Overview of VQShape
TS Encoding. VQShape contains a patch-based transformer encoder (Nee et al. 2023 Goswami et al. 2024) which first transforms univariate TS x into K non-overlapping fixed-length patches with dimension drutch Then, the patches are encoded by learnable linear projection and additive position embedding, forming patch embeddings that serve inputs a transformer model. The transformer outputs K latent embeddings h IR Formally, the TS encoder denoted by (he E R Rd k = I,...,,,, K} Exx). Note that hu could contain information from all patches instead of only the hth patch.
Attribute Decoding, The attribute decoder Adec takes latent embedding h& and extracts attribute tuple f (&x, Me, ok tu, 4x). Formally, Adee performs
P fz(hx),
Mix = fu(h)),
c. = (āks MA The (x) Adco(hu), where
(fft
ts sigmoid (hx)) (1 (min),
4. sigmoid(fi(ha) (1 (x) Amin-
Each decoding function in {fz, fur fi} is implemented using multi-layer perceptron (MLP) with one hidden layer and ReLU activation. Following a common notation [Esser et al. 202 denotes the attribute tuple before quantization.
The latent-space codebook is denoted by 2 (zy E R 1= Noode). ). To learn a generalizable codebook that contains only the abstracted shape-level features, we use low-dimensional codes with donde = 8. This configuration also creates bottleneck for reconstruction, minimizing additional information that can be inferred besides the abstracted shapes. The quantization follows VQ-VAE [van den Oord et al., 2017 that selects the discrete code based on Euclidean distance where
Codebook and Vector-Quantization
Z4 = arg min 1 zgll: E,E2
Shape Decoding. The abstracted shape of a TS subsequence is sequence with its length, offset, and scale information removed through normalizations. Given Tk (2k, Hkx ok tg. 14), we first extract the target subsequence from æ specified by tę and 4g denoted by Zeest, +4.- Then, Trast is interpolated to a fixed length of d' to remove the length information. The shape decoder takes zę and outputs another sequence with the same length. Formally, for Tk, this step produces two sequences
Iarget
IRdd
interpolate(
Sk E d
S(2*) ak + pk-
Note that the output of S is normalized such that S(zx) has the offset and scale information removed.

---

Attribute encoding and reconstruction. The attribute tuple after quantization
(2k, 1kk, kk, lk. (x) is transformed by learnable linear projection denoted by h rdd Linear(Tk). Then, the TS decoder DD takes {h k l...., K and outputs the reconstructed TS f.
4 Pre-training
VQShape is pre-trained on diverse datasets to learn dataset-agnostic features and tokens. In this section, introduce the self-supervised training strategies and objectives of VQShape. Then, discuss the representations the model could provide to down-stream tasks.
4.1 Objectives
The optimization objectives of VQShape during the pre-training stage are summarized below.
Reconstructions. Analogous to most of the VQ-VAE approaches, VQShape is trained to accurately reconstruct the input TS to learn essential latent-space representations for modeling TS data. Addi- tionally, to provide interpretable representations, the decoded shapes should similar to the actual subsequences. Therefore, the reconstruction minimizes two objectives:
Time-series reconstruction:
L = 1
(4)
K
Subsequence reconstruction:
L.
|| target
8k 1x112
(5)
K
Vector Quantization. We follow VQ-VAE vvan den Oord al. 2017] define the vector- quantization objective which trains the encoder and codebook Z. Additionally, inspired by Yu al. 2024), add additional entropy terms to encourage codebook usage. We find these terms could improve pre-training stability and avoid collapse of codebook usage. The objective for learning the codebook defined by
L = sg(z))1² Acomii |sg(3)
=13 + (H((2, - (( 2)))
quantization
codebook usage
where sg(-) is the stop-gradient operator and H(-) is the entropy function for discrete variables. (2, 2) softmax ([1 zx a categorical distribution.
2x1 zy Z)) measures the distance between 2 and all codes in Z as
Disentanglement of shapes. In Equation the attributes (zk, (k) optimized towards accurate subsequence reconstructions. is important to note that, since (4a, (x) defines 8% Jurget they are essential for learning the abstracted shapes and the codebook. However, it is challenging to gradients from reconstruction in Equation 4 solely learn (4e,4e) for making informative subsequence selection. Therefore, we introduce an additional regularization that encourages the latent-space tokens (attributes) to capture shape-level information with diverse positions and scales. This regularization is defined as
K
Las
1k1 + 122) relu (c -1rk((t 1e,) - ra k(tke 11))
K2
R
12=1
-I((4))/ In(1min) sin(l -I((4)/ In(4min)
cos(LAT)
where n((t, 1x)
In Equation] 7 ni(t, 46) defines coordinate transformation which maps (f2, 12) into a space where (1) small 4 values become more diverse and (2) large l values from different tą become more concentrated. By making different (tx. l&) diverse in this space, Lai encourages the model to capture disentangled shape-level information while increasing the use of short sequences to capture local details. Figure X visualizes an example of transformation threshold distance in the transformed coordinate where two (4x: lx) samples are considered sufficiently diverse.
is hyperparameter that defines
5

---

The overall pre-training objective is to minimize
Cpeetrii = Xx + AgGs + Avehin + Aav
(8)
where Azi Ass Ang Any are hyperparameters that define the weighting between the components. During pre- vre-training of VQShape, we set X₂ = X2 = Xug = 1, Aui = 0.8, and Acommit 0.25.
Design Analysis. Overall, the encoding process in VQShape (Transformer encoder and attribute decoder) introduces an inductive bias by representing and summarizing univariate TS using a set of abstracted shapes along with their position, length, offset, and scale. The pre-training objectives guide the components toward learning interpretable representations (via subsequence reconstruction in Equation 5) and disentangled representations (via regularization in Equation 7 while preserving the information necessary to describe the TS (via reconstruction in Equation 4 These objectives introduce interpretability
the conventional deep autoencoder structure. By pre-training on diverse datasets with a universal codebook, VQShape further leverages this inductive bias to produce discrete and dataset-aa -agnostic representations, resulting in vocabulary of abstracted shapes that can be used as primitives to describe TS data.
Model Configurations. The settings of VQShape related the model size correspond to those of the MOMENT-Sman [Goswami et al. 2024 model. Specifically, we interpolate all the input univariate TS have length T = 512, which broken into K = 64 patches with daatch 8. The Transformer layers in the encoder & and decoder ID have 8 heads, an embedding dimension dambed = 512, and feed-forward layer of size 2048. We employ asymmetric structure with 8-layer encoder & and a 2-layer decoder 'D [e et al. 2022]. The codebook Z contains Neode = 512 codes, each with dimension dd = 8. The subsequences $ target and decoded sequences 8ę have length d = 128. We set the minimum shape length Imin 1/64. With these setrings, VQShape has 37.1 million parameters.
In the pre-tnainin stage, we train VQShape with the AdamW optimizer, using weight decay X = 0.01, 31 = 0.9, 32 0.999, gradient clipping of 1.0, and an effective batch size 2048. We employ a cosine learning rate schedule with an initial learning rate of le final learning rate of 1e-5, and I epoch of linear warm-up. The pre-training dataset contains univariate TS extracted from the training split of 29 datasets from the UEA Multivariate TS Classification Archive (Bagnall et al. 2018]. excluding the InsectWingbeat dataset, resulting in 1,387,642 univariate TS. We train VQShape for 50 epochs on this dataset using bfloat-11 -16 mixed precision.
4.2 Representations for down-stream tasks
VQShape provides two types of representations: Latent-space Tokens and Code Histogram.
Tokens, Similar to the latent-space feature map of typical VQ approaches such as VQ-VAE van den Oord et al. 2017] and VQ-GAN (Esser et al. 2021]. VQShape also provides a set of tokens as representations. For an input univariate TS x, the token representations are composed as
T E I. (1
{ (Zk, Mil Tk, tx, 4)) k .......K. The token representations can be useful for general down- tasks but are less interpretable than the code histogram representations in classification tasks.
Code Histogram. Inspired by Concept Bottleneck Models (CBMs) Koh al. 2020 developed in computer vision, we can also view each 2y E 2 as a concept for TS data. As CBMs have concept scores as representations, VQShape provides similar representation in the form of histogram of codes. Based on Equation we can also have a vector of code indices
q = =
arg min m.......
|| - zgll
K
Then, the code histogram representation is defined as r E R Nook element in is the frequency of index q in q. Intuitively, the code histogram representation is analogous to BOSS [Schäfer 2015) but with non-deterministic window size and dataset-agnostic symbols. In classification tasks, this type of representation can be more interpretable since classifiers based on these features are able to produce rule-like predictions that are straightforward interpret and understand.
histogram(q) where each

---

Experiments
Datasets. We evaluate the pre-trained VQShape on multivariate TS classification tasks to demon- strate the effectiveness of learned shape-level representations on down-stream tasks. The evaluations are performed on the test split of 29 datasets from the UEA multivariate TS classification archive [Bagnall et al. 2018 Ruiz et al.] 2021], with the InsectWingbeat dataset excluded. Table 4 summarizes statistics of those datasets. Details on experiment setups are included Appendix A
Baselines. We benchmark VQShape against baselines from four categories, including (1) classical
methods: DTW [Chen et al. 2013] and Shapelet Transform with Random Forest classifier (STRF)
Bostrom and Bagnall| 2017]. I, (2) supervised learning methods: DLinear [Zeng et al., 2023) Auto-
former [Wu et al.
2021 FEDformer [Zhou et al. 2022]. PatchTST (Nie et al. 2023], J, and TimesNet
(Wu et al. 2023 (3) unsupervised representation learning methods: TS-TCC [Eldele et al. 2021]
TST Zerveas et al.
202T Ve2 (Yue et al. 2022], and T-Rep [Fraikin et al. [024]. (4) models
pre-trained multiple datasets: MOMENT Goswami et al., [024] and UnITS [Gao et al. 2024].
We compute the classification accuracy as metrics and compare the methods based on the statistics of accuracy and ranking. Details on benchmarking setups are included in Appendix
Table I: Statistic and comparisons of the baselines and VQShape. The best case is marked with bold, the second best marked with italie, and the third best marked with underline. Some baselines fail are incompatible with some datasets which result in "N/A". For fair comparison, we report the statistics with and without "N/A". Complete results are presented in Table
Clasical
Sepervised
Unsopervised Represerraiin
Pre Stliee
DTW
STRE
BLines
Autofociner
FED
PuchiST
TanesNet
TS-TCC
15T
ReR
T52V
MOMENT
Un:TS
VigShuge
0.635
0.682
0.630
0.686
0.629
0.733
Median Accuncc
0:711
0.679
(1.77
0.555
0.586
0.756
0.797
0.755
0.520
11.994
11312
0.259
0.684
0,870
Moun Rask
7,138
7,828
3,090
9.448
7,750
9,296
5,243
7,172
5,445
5.207
4.897
5929
9.838
5.621
Medion Rank
7.0
5.0
9.0
10.0
3.0
3.0
3.0
8.0
9.01
3.8
5.5
10.0
5.0
Num. Top
5
0
Eop-3
Whi'lee
Wilcowen p-oaloo
0.206
0,025
0,002
0.000
0,733
0.000
Statistics without NIA
Mean Accunacy
0.642
0,658
B.85
0.561
06001
0.657
0.208
0.660
0.823
0,210
0.70s
0.888
0,688
0.720
Ausuayy
0.712
0.730
0.759
0.679
0,812
5,992
53885
Rack
10.5
Nom Top-1
Num. Tip
10
10
C.
Nam WhrTie
13
18
20
70
15
17
19
14
12
13
23
Num. Lose
13
12
14
3
Wilcowon posluz
0.187
0.036
0.001
0.001
0.111
0.001
0.094
0.005
0.666
1.000
0.000
Results. On multivariate TS classification tasks, for each TS sample x, E X, we extract the token representations using VQShape and apply a learnable linear layer on this feature vector to predict the class label. Table [ summarizes the performance of VQShape and baseline methods on the 29 UEA datasets. Complete results are presented in Table 5 From the results, we observe that the best-performing methods, including TimesNet, T-Rep. TS2Vec, MOMENT, and VQShape, have similar performance, and none of the methods has performance that is statistically significant compared to the others. Therefore, we conclude that VQShape achieves comparable performance to the state-of-the- thett-rrrtttt baselines while additionally providing interpretable pre-trained features.
Frozen Pre-trained Representations. Next, we compare the frozen pre-trained representations from three existing pre-trained models: MOMENT, UniTS, and VQShape. Since the pre-training dataset could have a dominant effect on the models, for fair comparisons, we reproduce MOMENT- small and UniTS by training them on the same datasets VQShape. Table summarizzes the statistics of the pre-trained models on the 29 UEA datasets. Complete results are presented in Table/6] From the results, we observe that VQShape outperforms MOMENT and UniTS on most of the datasets and in overall statistics.
5.1 Generalizability
The experimental results presented above evaluate models pre-trained on the training splits of the UEA datasets and tested on the test splits of the same datasets, demonstrating the models' ability to generalize to unseen samples from the same domains. Besides in-domain generalizability, VQShape
7

---

Table 2: Comparison between three models pre-trained on all or a subset of the UEA datasets. The best cases are marked with bold. Complete results are presented in Table
Pre-trained on:
29 datasets
datasets
MOMENT
UniTS
VQShape
MOMENT
UniTS
VQShape
Mean Accuracy
0.697
0.581
0.723
0.697
0.559
0.723
Median Accuracy
0.736
0.649
0.810
0.733
0.649
0.792
Mean Rank
1.655
2.862
1.483
1.655
2.966
1.310
Num. Top-1
13
0
16
11
0
20
and its codebook can also generalize to datasets and domains not observed during pre-training. To demonstrate cross-domain generalizability, archive that are commonly selected to train and evaluate deep learning models [eerves et al. 2021 Wu et al. 2023], and then evaluate it all 29 datasets. The right half of Table summarizes the performance of this model, compared with MOMENT and UniTS trained with the same setup. We observe that VQShape and MOMENT trained on fewer datasets result in similar but slightly worse performance, indicating that the representations learned by the models can generalize to unseen domains.
train another model using 9 datasets from the UEA
5.2 Interpretability
Universal Codebook of Abstracted Shapes. One of the most essential components of VQShape is the dataset-agnostic codebook that contains abstracted shapes. In Figure 6 of Ap- pendix C.I we decode all 512 codes the codebook of VQShape to visualize their cor- responding abstracted shapes. We observe that large number of codes are decoded into similar shapes, which suggests that the codebook size can be further reduced. We then visualize the distribution of codes learned from pre-training (see Figure 7 which contains about 60 clus- ters. Inspired by this observation, we train variant named VQShape-64 with codebook size Noode = 64. Figure 2 presents the decoded codebook of VQShape-64.
Sie
Sis
Si
St1
Syy
Sxt
Say
Say
Sa
Zn
5c
Interpretable Representations. Overall, the encoding of VQShape be interpreted as "TS x is decomposed into (shape 21 with offset μl and scale aı, at i with length ......" ..." and the decoding can be interpreted as "The composi- tion of (shape zi with offset ln and scale aı, l with length 4)). ... becomes d". Figure includes an example of interpretable representations learned by VQShape. From visualizations, we can confirm that VQShape can learn abstracted shapes that capture shape-level information with various positions and scales.
Figure 2: Visualization of the decoded codebook from VQShape-64.
Discriminative Representations for Classification. We further show that the interpretable rep- resentations produced by VQShape also capture discriminative patterns that distinguish different categories in classification tasks. Figure 4 visualizes the average code histogram for samples from two categories. From the feature maps, it is obvious that several codes have significant differences in frequency between the two categories; these serve as discriminative features in classification tasks. We decode and visualize their corresponding abstracted shapes. The intuition provided by the histogram features can interpreted as: "Samples from the CW circle category usually contain shape 361 in variate 1, and samples from the CCW circle category contain shape 533 in variate 3, etc."

---

Pa 16,199
foe OoGAZ2 D2 63 6O!
VigShupe Encode
VQShape Dacade
&b 18,32238)
Figure 3: An example of abstracted shapes and their attributes (i.e., token representations) extracted by VQShape. For better presentation, we visualize 6 of the 64 shapes.
Class: CW circle
861
Variate
Variate
Variate 3
87
833
Class: circle
Figure 4: Example of how the code histogram representations provide discriminative features for classification. Histogram representations are obtained from VQShape-64. The histograms are averaged over samples of the two classes from the test split of the U'WaveGesturel.imaa dataset. The top and bottom rows represent samples labeled as "CW circle" and "CCW circle", respectively. Each column represent variate (channel).
5.3 Ablation Study
Representations and Codebook Scaling, We com- pare the performance of classifiers trained with token or histogram representations from the same pre-trained
0.72
Mean Accuracy
VQShape models. We also study how the size of the code- book affects the quality of these representations. Based on the results presented in Figure we conclude that token
classifiers always outperform histogram classifiers since token representations are more expressive than histogram representations from the same pre-trained model.
Histogram
32
64
128
512
Additionally, the performance of token classifiers increases steadily as codebook size increases, since tokens from larger codebooks could contain more details. However, for histogram representations, choosing the appropriate codebook size essential; we observe that codebook size of 64 results in the best performance. These results match our previous observation Section 5.2 where the model with Noode = 512 results in only approximately 60 clusters of codes. A small codebook would learn codes that are too abstract to provide detailed representations, while large codebook would learn similar codes that could lead to misleading features in the histograms. From
Codebook Size
Figure 5: Mean accuracy of classifiers trained with token and histogram rep- resentations across different codebook sizes. Performance of the token clas- sifiers improve with larger codebook, while performance of the histogram clas- sifiers peak at codebook size 64 and de- cline with larger codebook.

---

the experimental results, we conclude that Nasde 64 strikes the best balance between abstraction and expressiveness for pre-training on the UEA datasets, producing the best histogram representations.
Note that within the scope of this paper, the number of code clusters remains a post-hoe discovery that only be determined after pre-training. One could start with large codebook size such as Nudee = 512, visualize the distribution of codes, and re-train with the appropriate codebook size. Developing mechanism adjust the codebook size dynamically during training or inference would be a valuable direction for future work.
Other Ablation Studies. We additionally con- duct two ablation studies: (1) setting d 32 to compare high-dimensional and low- dimensional codes, and (2) setting A = 0 to assess the value of introducing shape-level in- formation to the VQ-VAE structure. Table 3 summarizes the statistics of these two cases and the default setting. The complete results are presented in Table and Table 8 From the overall statistics, pre-training with X = 0 leads to degraded performance, which indicates that learning shape-level abstractions through sub-
Table 3:
Statistics of ablation cases on code dimen- sion and shape reconstruction loss.
Hyperparameter
Value
Code dim
drnde
8
32
8
Codehook size
Apcode
512
512
$12
Shape loss
A
1
1
0
Mean Accuracy
Token
0.723
0.721
0.708
Median Accuracy
Token
0.810
0,800
0.761
Mean Accuracy
Histogram
0.709
0.717
0,707
Median Accuracy
Histogram
0.762
0.810
0.747
sequence reconstruction introduces useful information for classification tasks. Using codes with
duode = 32 produces token classifiers with similar performance and better histogram classifiers. However, as stated in Section 3.2 we use low-dimensional code to create a bottleneck where the code should mainly contain the shape-level information. Using high-dimensional code may introduce additional information beyond the decoded shapes, which reduces interpretability. Therefore, the effect of using high- sigh-imensional code would require extensive study.
6 Conclusion and Limitations
This paper introduces VQShape, a self-supervised pre-trained, interpretable, and generalizable model for 'S analysis. Taking advantage of large pre-trained Transformers and ideas from shapelets, VQShape extracts set of interpretable representations from TS data, composed of abstracted shape, offset, scale, position, and duration. Pre-trained and evaluated multivariate classification datasets from the UEA archive, VQShape achieves comparable or even better performance than black- box models pre-trained on large datasets and with significantly more parameters, while providing additional interpretable representations. Furthermore, using VQShape, we present a codebook containing of abstracted shapes that generalize to various TS datasets, including unseen datasets.
In this paper, we do not term VQShape a foundation model for TS data, since the amount of pre- training data is still limited compared to other large pre- mre-trained models such as MOMENT, and we are focusing only on classification tasks because the extracted shape tokens are mostly interpretable for classification. As presented in Table 6 based on the comparison between VQShape pre-trained on 29 and 9 datasets, simply adding more pre-training data may degrade performance on some datasets. This suggests that learning shape-level representations may not benefit from some types of data. For example, the DasicMotion inn datasets mostly contain signals with high-frequency sinusoidal components that do not contain mcaningful shape-level information in short subsequences. Such datasets may "pollute" the pre-training of VQShape since the model will try to capture unnecessary gigh-frequency features. Therefore, if pre-trained on datasets at scale, additional pre-processing and input engineering should be included; however, we do not conduct explicit tre-processing in this paper. would also be an important future step to develop interpretable frameworks for other TS analysis tasks, such as forecasting, imputation, and anomaly detection, using the interpretable tokens extracted by VQShape.
Acknowledgments
This work was supported in part by IBM through the IBM-Rensselaer Future of Computing Research Collaboration. We thank Dr. Eamonn Keogh for his valuable comments and suggestions for revision. We also appreciate the UEA team's 's effort in creating and publicly sharing the benchmark datasets.
10

---

References
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The UEA multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.000 00005,, 2018.
Aaron Bostrom and Anthony Bagnall, A shapelet transform for multivariate time series classification. arXiv preprint arXiv:1712.06422 2017.
Yanping Chen, Bing Hu, Eamonn Keogh, and Gustavo EAPA Batista. DTW-D: time series semi- supervised learning from single example. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 383-391, 2013.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2352-2359. 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pultern recognition, pages 12873-12883, 2021.
Archibald Felix Fraikin, Adrien Bennetot, and Stephanie Allassonniere. T-rep: Representation learning for time series using time-embeddings. In The Twelfth International Conference on Learning Reprexentations, 2024.
Shanghua Gao, Teddy Koker, Owen Queen, Thomas Hartvigsen, Theodoros Tsiligkaridis, and Marinka Zitnik. UniTS: Building a unified time series model. arXiv preprint 2430002403.00131, 2024.
Azul Garza, Cristian Challu, and Max Mergenthaler-Cansseo. TimeGPT-1. arXiv preprint arXiv:2310.03589, 2023.
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. In International Conference Machine Learning, 2024.
Josif Grabocka, Nicolas Schilling, Martin Wistuba, and Lars Schmidt-Thieme. Learning time-series shapelets. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 392-401, 2014.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang. Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang- Concept bottleneck models. In International conference on machine learning, pages $338-5348. PMLR, 2020.
Yuxuan Liang, Haomin Wen, Yuqi Nie, Yushan Jiang, Ming Jin, Dongjin Song, Shirui Pan, and Qingsong Wen. Foundation models for time series analysis: tutorial and survey. arXiv preprint arXiv:2403.1 14735, 2024.
Jason Lines, Luke M Davis, Jon Hills, and Anthony Bagnall. A shapelet transform for time series classification. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 289-297, 2012.
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, 2023,

---

Alejandro Pasos Ruiz. Michael Flynn, James Large, Matthew Middlehurst, and Anthony Bagnall. The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Mining and Knowledge Discovery, 35(2):401-445 2021.
Patrick Schäfer. The boss is concerned with time series classification in the presence of noise. Data Mining and Knowledge Discovery, 29:1505-15533 1530, 2015.
Sabera Talukder, Yisong Yue, and Georgia Gkioxari. TOTEM: Tokenized time series embeddings for general time series analysis. In ICLR 2024 Workshop Learning from Time Series For Health, 2024.
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In Advances Neural Information Processing Systems, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. In Journal of Machine Learning Research, volume 9, pages 2579- 200922605522 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, volume 30, 2017.
Zhiguang Wang. Weizhong Yan, and Tim Oates. Time series classification from seratch with deep neural networks: A strong baseline. In 2017 International joint conference on neural networks (IJCNN), pages 1578-1585. IEEE, 2017.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transform- ers with auto-correlation for long-term series forecasting. In Advances in Neural Information Processing Systems, 2021.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference Learning Representations, 2023.
Lexiang Ye and Eamonn Keogh. Time series shapelets: a novel technique that allows accurate, interpretable and fast classification. Data mining and knowledge discovery, 2011.
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A Ross, and Lu Jiang. Language model beats diffusion tokenizer is key to visual generation. In The Twelfth International Conference on Learning Representations, 2024.
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. TS2Vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8980-89877 8987, 2022.
Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121-11128, 2023.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 21 14-2124, 24, 2021.
Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on PMLR, 2022.
machine learning, pages
12

---

A Experiment Setup
Environment. VQShape implemented using Python 3.11.8 and PyTorch 2.2.2. The pre-training and evaluations are conducted on a machine with Intel Core 17-1 CPU, 32GB of RAM, and Nvidia RTX 4090 24GB GPU.
Table 4: Information of the 30 benchmark datasets from the UEA archive.
Dataset
Train Size
Test Size
Variables M
Length
Categories
Type
Nuain
Nis
T
C
Articulary WordRecognition
275
300
9
144
25
MOTION
AtrialFibrilationn
15
15
2
640
3
ECG
BasicMotions
40
40
6
100
4
HAR
ChajeoterTrajectories
1422
1436
119
20
MOTION
Cricket
108
72
6
1197
12
HAR
DuceSeckGeese
50
50
1345
270
5
AUDIO
ERing
270
65
6
HAR
Eigen Worms
128
131
17948
5
MOTION
Epilepsy
137
138
3
206
4
HAR
ElhanolConnentratiion
261
263
3
1751
4
OTHER
FaceDetection
5890
3524
2
EEG
FingerMovements
316
100
28
50
2
EEG
HandMovementDDircttin
160
74
10
400
4
EEG
Handwriting
150
580
3
26
HAR
Heartbeat
204
205
61
405
2
AUDIO
Innghettngheat
25000
25000
200
22
10
AUDIO
Japanese
270
370
12
29
9
AUDIO
L.SST
2459
14
OTHER
Libras
180
180
2
45
15
HAR
Motorlmagery
278
100
64
3000
2
EEG
NATOPS
24
6
HAR
PEMS-SF
267
173
263
144
7
MISC
PenDigits
7494
3498
2
8
10
MOTION
PhonemeSpectra
3315
3353
11
217
39
SOUND
RacketSports
151
152
4
HAR
Sel/RegulltionSSCII
268
293
6
896
2
EEG
SelfRegglationSSCP2
200
180
7
1152
2
EEG
SpokenArrabiDDiggn
6599
2199
10
SPEECH
StandWalkJump
15
4
2500
3
ECG
UWaveGestureLibrary
120
320
3
315
8
HAR
Classification tasks. Taking frozen pre-trained representations from VQShape, we learn linear classifier to make predictions. When training the linear classifier, we found that token representations work better with regularization more compatible with dropout on features. Therefore, to avoid overfitting and obtain the optimal performance, we tune both the L.2 regularization weight (or weight decay) and the dropout rate.
L2 on classifier weights, while histogram representations are
When learning the linear classifier, we repeat the experiment with five random seeds and report the average accuracy in Table The standard deviations of the five runs included Table 7 Note that we exclude the InsectWingbeat dataset since the dataset contains inconsistent and very short TS samples such as T Considering that the dataset has significantly more samples and channels than other datasets, the high volume of such short samples may have a negative effect our method since the short TS do contain any meaningful shape-level features.
Baseline Results. For baseline results presented in Table [ and Table 5 we reproduce STRF with the Peon-Tookip and Scikit-learn packages. reproduce DL Dinnerr, Autoformer, FEDformer, PSt and TimesNet using implementation from Wu et al. [2023 Results of DTW, TS-TCC. TST, and TS2Vec are obtained from the TS2Vec paper Yue al. 2022 Results of MOMENT
2The Acon-Tookit available at www. aeon teonntoookii..or.
Thhe implementations available at httttt:://ggthhii http:////oo//o//taoooorsseeeeeer
13

---

[Goswami et al. 2024 and T-Rep Fraikin et al. [2024] are obtained from the original papers. For results of UniTS Gao et al. [024], since the full results on the UEA datasets are not reported in the original paper, we follow the official implementation to reproduce the results using a pre-trained checkpoint and prompt learning.
Benchmarking Frozen Pre-trained Models. To obtain the results in Table 2 and Table 6 we train MOMENT and UniTS from scratch using the official implementations 5 MOMENT, UniTS, and VQShape produce predictions can be different. MOMENT and VQShape provide frozen representations and learn a separate classifier, whereas UniTS learns additional prompt tokens and classification heads. Therefore, we follow the default procedure and the official implementation to evaluate UniTS. We evaluate MOMENT and VQShape under the exact same procedure of learning linear classifiers with regularization tuning.
Note that the way
Note that the key difference between results in Table 5 and Table that MOMENT and UniTS in Table are pre-trained on significantly larger volume of data (which includes the forecasting
datasets), while the three models are only pre-trained on the UEA datasets in Table
B Full Experiment Results
Table 5: Full benchmark results on the 29 UEA datasets. For each dataset, the best case is marked with bold, the second best is marked with italic, and the third best is marked with underline.
Claskal
Sipervieed
Unsppevited Represertzton
Pre-miined
ITWW
STRF
D1 Jasier
Astofurner
FilDiomer
PricTTT
TimosNei
1S1
T-llpp
MOMENT
UniTS
Vi28tape
Darawci
Artosstry WeedReerggionn
0
01961
0.991
01975
0.955
0.917
0.965
01207
6,990
0197
BasiMMrioss
0.925
0.550
1,800
0.975
0,910
Chanader ecrectocis
0
0.849
0.966
0.894
0.914
0.976
0.987
0:95
0.975
0.989
0.995
N5A
0.335
0.909
Chickt
1800
0.944
0.811
(1194
0.189
0.03
0417
1,800
0195
0.012
1.00
0956
0.975
0.00
0.330
1.500
0.780
0.80
0.180
0.63
8
0.0
0.16)
ERing
0333
0.60
Epilepes
0.064
6.928
0.565
0.766
0.804
0.913
0.71
0.857
0.949
0900
0964
6,98
0.12
(.99
01302
01677
0.270
0.255
0.261
0.559
0.565
0.355
0.263
0331
0326
0177
0259
0.32
RoceDeteeso
0125
0.567
0.073
0.585
0.688
0.677
0.56
0.514
0.61
030
0453
0559
060
FinerrMeemenss
0.500
0.570
0.540
0.500
0.460
0.560
0950
6400
6642
Handurtiing
01280
01104
0189
0.185
0311
0140
0.315
0111
0314
0.30
133
(35
Hearheat
0.77
6.745
0.207
0.337
0.341
0.723
0.732
0.751
0.746
0.725
0683
0,722
0.73
056
Tapaazie Vowcels
0549
0.676
0.49
0.511
0.66
BA05
0.554
0930
0:578
0.02
0354
0.716
0322
0965
EST
0.481
0:11
8161
055
0:114
0.336
0.50%
0337
6411
0511
0.510
0.000
0.500
NIA
0.679
0.10
0.500
0510
0
NATORS
0.88
0.754
0.977
0.764
0.825
0.258
0831
0.312
0.550
0.004
0,328
0.55
0110
PEMSS-A SF
8711
0,925
0.809
0.33
0.867
0.00
0.844
0.734
6.240
0,00
0.82
0.00
0544
0.66
grs
8977
0385
0351
0.987
71965
11774
0.74
0.50
0.71
0,269
0372
0284
092
0.352
0.730
0.555
0355
SeirRegdeniesSCPP
0.775
03546
0181
0577
0.20
0335
0.751
0312
0.00
0,24
Ser Ror
0.380
0.449
0.5%
0.61
0353
0.530
03335
0.550
0.5)
0ATHR
8.5%
Spokem/sahhciKggss
0.63
0.679
09655
0.85
0988
0.07
0.89
0.70
0.913
0,994
0.98
0991
0024
0.76
Wuluuu
0.467
0.33
0,677
0.467
0.537
0.333
0.257
9.667
0.787
L/WooGrearcl Ihory
'Implementation and checkpoints of UniTS available
UniTS
'Implementation and pre-training for MOMENT are available at
https://github.com
moment- hrm.s//eeeee-tooomttmooooonndeee research
14

---

Table 6: Full results of three models pre-trained on the UEA datasets, The best case for models pre-trained on 29 datasets is marked with bold, and the best case for models pre- pre-trained datasets is marked with underline. The 9 datasets are marked with f.
Pre-trained on:
29 datasets
datasets
MOMENT
UniTS
VQShape
MOMENT
UniTS
VQShape
Dataset
Articulary WordRecognition
0.990
0.860
0.987
0.987
0.880
0.990
AtrialFibrillation
0.533
0,400
0.520
0.480
0,400
0,480
BasicMotions
0.760
0.775
0.910
0.790
0.775
0.950
Character Trajectories
0.982
0.723
0.969
0.982
0.735
0.966
Cricket
0.986
0.944
0.978
0.989
0.819
0.989
DeekDeckGeese
0.464
0.400
0.360
0.384
0.240
0,396
ERing
0,907
0.700
0.960
0.937
0.756
0,976
Figen Worms
0.663
0,466
0.603
0.667
0.389
0.583
Epilepsy
0.987
0.826
0.893
0.983
0.819
0.877
EthanolCooncentaation f
0.414
0.213
0.325
0.424
0.247
0.303
FaceDetection f
0.597
0.500
0.653
0.601
0.529
0.649
FingerMovements
0.630
0.510
0.642
0.630
0.490
0.638
HandMovementDirretion
0.381
0.257
0,546
0.408
0.216
0,508
Handwriting f
0.225
0.087
0.270
0.245
0.153
0.280
Heartbeat
0.744
0.649
0.663
0.733
0.649
0.661
Japanese Vowels f
0.706
0.843
0.945
0.709
0.824
0.952
LSST
0.429
0.361
0.511
0.423
0,415
0.510
Libras
0.908
0.633
0.814
0.879
0.556
0.813
MotorImagery
0.642
0.510
0.680
0.682
0.460
0.672
NATOPS
0.860
0.711
0.810
0.862
0.661
0.792
PEMS-SF
0.875
0.821
0.865
0.891
0.861
0.889
PenDigits
0,965
0.819
0.973
0.965
0.669
0.970
PhonemeSpectra
0.090
0,070
0.087
0.071
0.069
0.085
RacketSports
0.736
0.671
0.851
0.763
0.678
0.882
SelfRegulationSCCPI
0,829
0.652
0.904
0.821
0.717
0,898
SelffkegulationSCPP2
0.576
0.506
0.596
0.588
0,417
0.624
SpaienAraaiilDigiss
0.971
0.751
0.976
0.963
0.769
0.975
StumdWalkJump
0.493
0.533
0.787
0.507
0.333
0.747
UWaveGestureLibrary f
0.871
0.656
0.888
0.846
0.688
0.902
15

---

Table 7: Full results of VQShape variants with token classifiers. For a frozen pre-trained model, the mean and standard deviation of accuracies over five randomly initialized linear classifiers are reported.
Hyperparameter
Variants of VQShape
Codebook Size
Nusll
32
64
128
512 8
512
512
Code Dim.
d
8
Shape Loss Weight
Aai
Representatio:
Token
Token
Token
Token
Token
Token
Articulary WoudReoognition
0.979 + 0.032
0.988
0.021
0.990
0.021
0.987 = 0.018
0.981 = 0.030
0.991 0.025
AtrialFibrillation
0.547 + 0.073
0.520 1: 0.191
0.573
0.068
0.520 :: 0.068
0.547 4 0.090
0.653 1: 0.107
BasicMotions
0.900 +: 0.048
0.925
0.037
0.940
0.029
0.910 : 0.044
0.950 4: 0.053
0.960 0.046
ChuracterTruie Triectoriis
0.960 + 0.019
0.965 +.065-00.02.
0.963
0.027
0.969 0.030
0.958 + 0.085
0.958 0.037
Cricket
0.973 = 0.022
0,981 2 0.019
0.010 0.986
0.978 = 0.016
0.986 0.002200003
0.986 + 0.032
DuekDuckGeese
0.384
0.372
ERing
0.966 1: 0.016
0.979 1: 0.006
0.010 0.984
0.960 1 0.011
0.761 - 0.022
0.973 1 0.016
Eigen Worms
0.034
0,569
0.031
0.603 0.035
0.035
Epilepsy
0.730 0.023 0.023
0,800 0.031
0.811
0.020
0.893 = 0.031
0.803
0.817 + 0.030
EthanooConneettation
0,313 0.020
0.329 0.018
0.317 0.019
0.325 = 0.020
0.335 0.018
0.313 0.018
FaceDetection
0.640 1: 0.006
0.657 1 0.00T
0,639 0.013
0.653 1 0.011
0.672 0.007
0.651 0.008
FingerMovements
0.640 + 0.029
0.646 0.039
0.630 0.029
0.642 + 0.025
0.644 0.032
0.630 0.025
DindMooveneetDrrcctioo
0.438
0,478 + 0.031
0,459 0.048
0.546 0.049
0.565
0.457 + 0.041
Handwriting
0,231 0.009
0.278 2 0.016
O. 281 = 0.014
0.270 = 0.012
0.267 - 0.010
0.255 + 0.013
Heartbcat
0.063
Viwwess
0.950 + 0.022
0.950 0.019
0.955 0.014
0.945 + 0.014
0.937 + 0.017
0.941 + 0.029
LSST
0.476
0.012
0.511 + 0.011
0.502 +
0.495 0.015
Librus
0.809 = 0.016
0,801 0.020
O. 801 0.013
0.814 = 0.029
0.769 0.013
0,801 = 0.019
MotorImagery
0.678 + 0.046
0.670
0.037
0.656 0.029
0,680 = 0.039
0.688 1 0.024
0,600 0.046
NATOPS
0.781 1: 0.020
0.820 0.041
O. 0,800 0.023
0.810 0.021
0.790 0.025
0.800 0.022
PEMS-SF
0.842 + 0.042
0.876 + 0.038
O. 872 0.059
O. 865 + 0.044
0.837 0.041
PenDigits
0.967 0.025
0.973
0.919
0.014
PhonemeSPectra
0,083 + 0.003
0.077 1 0.003
0.085 0.003
0.087 = 0.003
0.107 = 0.003
0.092 1 0.003
RacketSports
0.851 0.019
0.841
SelfRegulationSCPPI
0.881 + 0.023
0.904 + 0.026
O. 887 0.025
0.904 + 0.039
0.904
0.880 + 0.020
SelfRegulationSCP2
0.597 + 0.024
0,593 0.027
0. 0.031
0.596 = 0.018
0.587 +.00000000
0.600 0.029
SpokenArabicDigiis
0.971 + 0.020
0.974 1 0.020
0.977 0.019
0.976 = 0.021
0.977 - 0.028
0.976 0.023
SlandWalkJump
0.760 1 0.108
0.813 1 0.103
0.733 - 0.100
0.787 1 0.131
0.760 1 0.112
0.787 0.126
UWWveGestturll.eee ahrary
0.893 0.017
0.898 0.009
0,912 0.019
0.887 + 0.012
0.846 + 0.010
0.891 0.013
16

---

Table 8: Full results of VQShape variants with histogram classifiers. For frozen pre-trained model, the mean and standard deviation of accuracies over five randomly initialized linear classifiers are reported.
Hyperparameter
Variants of VQShape
Codebook Size
32
64
128
512 8
512
512
Code Dim.
d
8
Shape Loss Weight
Representatio:
Histogram
Histogram
Histogram
Histogram
Histogram
Histogram
Articulary WordReoognition
0.983 + 0.056
0.996
0.050
0.993
0.038
0.991 = 0.031
0.986 = 0.043
0.988
0.030
AtrialFibrillation
0.560 +: 0.084
0.600
0.080
0.533
0.144
0.573 4: 0.100
0.520 0.103
0.507 0.112
BasicMotions
1.000 +: 0.041
1.000
0.019
0.985
0.012
0.955 :: 0.029
0.930 4: 0.025
1.000 0.016
ChuracterTruie Trajecoriis
0.885 + 0.033
0.908 + 0.030
0.931
0.034
0.942 0.040
0.928 + 0.007
0.938 0.036
Cricket
0.967 + 0.026
0.986 0.033
1.000 = 0.022
0.986 = 0.021
0.973 0.015
0.978 + 0.031
DuekDuckGeese
0.428 0.045
0.042
ERing
0.929 1: 0.064
0.019 0.956
0.856 0.018
0.927 1 0.013
0.935 - 0.017
0.928 1 0.016
Eigen Worms
0.027
0.553 0.017
0.669 0.018
0.016
0.018
Epilepsy
0.929 + 0.0-10
0,939
0.027
O. 96964
0.033
0.970 = 0.026
+888&
1.000 0.039
EthanolConneettation
0.301 = 0.012
0.313 0.018
0.308
0.014
0.319 = 0.017
0.319 0.020
0.310 0.015
FaceDetection
0.567 1: 0.009
0.567
0.010
0.008 0.578
0.584 1 0.006
0.653 1 0.013
0.601 0.009
FingerMovements
0.598 + 0.028
0.033 0.508
0.600
0.036
0.600 + 0.024
0.600 0.026
0.600 0.028
HindMooveneetDrrcctioo
0.441 + 0.037
0,438
0.030
0.432 0.045
0.427 0.046
Handwriting
0.156 = 0.009
0.212 - 0.010
0.007 0.196
0.148 = 0.008
0.233 0.008
0,206 + 0.010
Heartbcat
0.739
Viwwnss
0.939 + 0.048
0.948 0.043
0.050 0.945
0.926 + 0.048
0.936 + 0.044
0.942 + 0.039
LSST
0.515 0.034
0.521 0.035
0.029
0.515 + 0.025
0.516 0.025
0.556 0.018
Librus
0.768 + 0.020
0,801 = 0.029
0,811 0.023
0.818 = 0.020
0.730 0.021
0,850 0.021
MotorImagery
0.628 + 0.024
0.672 0.022
0.630
0.028
0.082 : 0.030
0.626 1 0.042
0.678 0.029
NATOPS
0.847 1: 0.020
0.844 0.022
0,886 0.020
0.808 0.016
0.790 1 0.022
0.864 0.020
PEMS-SF
0.839 + 0.050
0.964 + 0.036
0. 853 0.036
0. 0.846 + 0. 0.066
+
0.824 0.062
PenDigits
0,898 + 0.063
0,9
0.969 0.038
0.958 0.018
PhonemeSPectra
0,116 = 0.003
0,134 1 0.006
0.132 = 0.005
0.143 = 0.004
0.104 0.005
0,150 1 0.004
RacketSports
0.774
0.838
SelfR SelfRegulationSCPI
0.731 + 0.027
0.754 + 0.027
O. 788 0.022
0.762 + 0.030
0.881 0.015
0.810 + 0.028
SelfRegulationSCP2
0.576 - 0.027
0.588 0.026
0.577 0.023
0.572 = 0.022
0.617
0.583 0.028
SpokenArabicDigiis
0.912 : 0.032
0.959 0.044
0,961 0.048
0.969 = 0.061
0.973 0.045
0,969 0.051
SlandWalkJump
0.680 1 0.116
0.560 1 0.060
0.613 - 0.094
0.613 1 0.068
0.653 1 0.073
0.640 0.073
UWWveGestturel.eee Shrary
0.841 + 0.033
0.864 + 0.036
0,904 0.016
0.899 0.030
0.821 + 0.014
0.923 0.026
17

---

Additional Visualizations
C.1 Visualization of abstracted shapes in the codebook
Six
Sox
Sr
S0
2113
Sizi
Sua
Sue
Sub
Stc
Pat
Aum
FrM
Aus
Pas
Say
Sau
Ezo
Suss
Fry
Fie
Sue
Sim
Suxx
Suza
Sim
Sou
Stee
Syn
SyM
Plas
Sya
Pat
Ajst
PaHt
Fuli
Ajs
Ear
Stae
Agas
Ass
Past
Ryzt
Pgar
Rys
Pger
Sye
Sym
Syci
Sus
Sau
Scoe
Rust
Pau
Figure 6: Visualizations of the abstracted shapes decoded from the codebook of VQShape
18

---

Spu
Syz
Plal
fus
Portt
Sav
Sup
Fax
Sual
Ayg
2:4
Apst
Pyn
Agrz
Patt
Ave
Pys
Fest
Sis
Sys
Siss
Siee
Fyyt
P1o
Sies
Plas
Aui
Pyns
Frns
Arra
Pyrs
Apm
Arn
Pytt
Arr
Pass
Plat
Rar
Ajo
Ryn
Kai
Ayp
Jyst
Jys
Ayp
Ray
Pymt
San
Se0
Aet
Sei
Sel
Figure 6: (Continue) Visualizations of the abstracted shapes decoded from the codebook of VQShape
19

---

C.2 Code Distribution
In Figure 7 we plot the distribution of 512 codes in the codebook transformed 2D space using SNE Ivan der Maaten and Hinton 2008). The codes can be roughly cluster into 60 groups, suggesting that there may only exists 60 diversed abstracted shapes in the codebook.
60
40
20
(-NN62)
20
-00
600
-80
-60
-40
-20
0
20
40
60
[-SNE(1]
Figure 7: I-SNE plot of the codes.
C.3 Visualization of transform
We provide a visualization s(t, !) transform discussed in Section 4.1 in Figure 8 The left figure shows (f, samples uniformly sampled from the original coordinate. According Section 3.1 (t, samples can only appear in the lower-triangular plane. After the <tt, l) transform, the corresponding samples are plotted in the transformed coordinate. Samples with small / in the original coordinate becomes more separated in the new coordinate, which encourage the model to capture local details in short subsequences. Samples with large becomes more concentrated and are less sensitive their value since they are likely to capture redundant information.
Original Coordinate
Transformed Coordinate
3.0
42
x<t, Z)
92
00
Figure 8: Visualization of transformation x(t,, l).
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We claimed in the abstract and introduction that we propose generalizable and interpretable representation for TS data, as well as a pre-trained model VQShape to extract the representations. We detailed the formulation and training procedure of VQShape, and validate with experiments on classification tasks that the model and its representations are indeed interpetable and generalizable.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We briefly discuss limitations in Section 5 and have present additional limita- tions in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not contain theoretical result that requires proof. But we demonstrate the effectiveness of our formulations using experiment results.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We detailed the formulations and training objective of the model, as well as the experiment setups. These information sufficient reproduce our results without code.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
22

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We open- our implementations and pre-trained checkpoints. All the datasets used in this paper publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wttpccppcccccpcccccooooooop httpi//gggudeee/eeeSuuubbiiiii semiisssonni oiionicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include detailed information about model architecture and hyperparamers in Section Details on experiment setups provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the standard deviation of classification accuracy in Table 7 and Table
8
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include detailed software and hardware environment in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp:://eeut http://////eeipppcpppppb/eeeeeee wccppulicc/thicssuieeeee
Answer: [Yes]
Justification: Experiments this paper is conducted only using publicly available datasets, without involving any human subjects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper focuses the development of interpretable models for time-series data, which could potentially improve the transparency of deep leanring methdos.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not contains data or models that have a high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All the datasets used in this paper are publicly available and properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
latasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: This paper introduce new model and its implementation, which is documented and provided in the supplementary materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve any human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve any experiment that requires IRB approval or equivalent.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "bATPANYGLf", "reviewer": "Reviewer_8xHF", "summary": "The paper introduces VQShape, a model designed for time-series (TS) data representation learning and classification. VQShape provides interpretable and generalizable representations by utilizing vector quantization to create a codebook of abstracted shapes. These shapes represent low-dimensional codes that describe time-series data across various domains. VQShape represents time-series data by decomposing TS subsequences into attributes like abstracted shape, offset, scale, start time, and duration. This method allows for the creation of interpretable and dataset-agnostic features. The representations derived from VQShape can be employed to construct interpretable classifiers that achieve performance on par with specialized models tailored for specific tasks. VQShape also demonstrates robust generalization capabilities in zero-shot learning scenarios, where it can effectively handle datasets and domains that were not encountered during the pre-training phase.", "questions": "- Performance comparison on zero-shot evaluation with other approaches.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces VQShape, a novel approach to time-series data representation that leverages vector quantization to create interpretable and generalizable representations. The concept of using abstracted shapes as tokens for time-series modeling is innovative, particularly in how these shapes are linked to the latent space features. \n- The paper is well-structured and articulates the motivations, methodology, and findings clearly. \n- Experimental results show the effectiveness of the approach when compared to baselines and other recent approaches.\n- The paper focuses on the task of learning represention for time series classification which is important in many domains.\n- The paper performs clear ablations to show the effectiveness of the proposed model components.", "weaknesses": "- The paper shows some interesting results on generalization towards unseen datasets but its not immediately clear how it compares to other approaches like MOMENT or TST. I would recommend performing this evaluation for other approaches too.\n- The paper can provide more ablations particularly on subsequence reconstruction loss as its one of the important component of the model.\n- How does the performance change when dimension of code size is increased?\n- Other clarification questions: how to define l_min? how to get h_k from tau_k?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to carefully read our paper, recognizing our contributions and giving valuable feedback. Below, we address your questions.\n\n> Regarding \"Generalization experiment of MOMENT and TST\"\n\nThe authors thank the reviewer for this suggestion. However, as the pre-training and implementation of MOMENT are not fully open-sourced, we cannot complete this experiment within the tight rebuttal window. We will try to include this experiment in the revised version if accepted.\n\n> Regarding \"Ablation on subsequence reconstruction loss\"\n\nIf removed, the codes will not have a connection with shapes in the time domain, and the model will be similar to a regular VQVAE, losing the ability to provide interpretable features. We conducted this ablation study and reported the new results in Table 2 of the Rebuttal attachment (see the column with $\\lambda_s=0$). With representations produced by the model pre-trained with $\\lambda_s = 0$, the classifiers result in lower accuracy, indicating that the shape reconstructions actually provide additional useful information. We believe this serves as evidence of our advantage.\n\n> Regarding \"increase code size\"\n\nOur motivation for using low-dimensional codes is stated in Lines 135 to 138. Increasing the code dimension will enlarge the bottleneck and possibly create a defect that the codes contain hidden information beyond the decoded shapes. We conducted this experiment and reported it in Table 2 of the Rebuttal attachment (see the column with $d^{\\text{code}}=32$). Increasing the code dimension does not significantly improve the performance of classifiers. We acknowledge that the scaling pattern for code dimension may require more extensive study, but we are unable to complete additional experiments within the Rebuttal period.\n\n> Regarding \"$l_{\\text{min}}, h_k, \\tau_k$\"\n\n$l_\\min$ can be viewed as \"the minimum length of a meaningful shape.\" We chose $l_\\min = 1/64$ since the input series are transformed into 64 patches, indicating that a series with a length of $1/64$ is the basic unit for expressing a time-series. As mentioned in Line 148, $h_k = \\texttt{Linear}(\\tau_k)$, which is a linear embedding layer to convert $\\tau_k$ into embedding $h_k$.\n\n> Regarding \"Performance comparison on zero-shot evaluation with other approaches\"\n\nThere are a limited number of pre-trained time-series models applied to classification tasks, as many of them are designed for forecasting (e.g., Google-TimesFM, IBM-TTM, etc.). Due to limitations in published results and open-sourced implementation (MOMENT released a checkpoint but not the pre-training code) we are unable to reproduce the results within the Rebuttal period. We will additionally include a comparison with GPT4TS [Zhou et al., NeurIPS 2023] in the revised version if accepted.\n\nWe thank the reviewer again for their feedback and hope that we have addressed their questions. If so, we hope they will consider increasing their score. Please let us know if our clarifications require further discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "itpS0v2bOj", "reviewer": "Reviewer_F3fg", "summary": "Authors propose a self-supervised model which can be used for classification. Their method learns abstracted shapes which serves as interpretable tokens and an information bottleneck in their modeling architecture. They compare with state-of-the-art methods on standard classification datasets and show promising performance.", "questions": "1. Why did you exclude the InsectWingBeats dataset from your analysis? Why not use all the UCR and UEA datasets for classification? \n2. You found that a lot of decoded shapes are similar (increasing or decreasing lines), and found ~60 clusters. What was the impact of reducing the size of the code book on the performance of your model? \n3. How is the size of the codebook determined, is it always an afterthought? \n4. How can you encourage the decoded shapes to be as diverse as possible?", "limitations": "The authors discuss some limitations of their approach in sections 5 and 6.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written. \n2. The idea is conceptually interesting yet simple, and the results are promising. The authors also compare with some state-of-the-art methods. \n3. The paper provides interesting insights such as impact of the quality of pre-training data on downstream classification performance", "weaknesses": "I think my biggest concern in the paper is on the benchmarking aspect: (1) the authors use UEA repository, but I would also encourage them to use the UCR time series classification repository. As a strech goal, real world and harder time series classification datasets such as PTB-XL or MIT-BIH can also be used for benchmarking, (2) the authors compare with a very limited number of baselines. The MOMENT paper for example compares with a wide range of deep learning and statistical techniques such as TS2Vec, k-NN, etc. and I believe it is important to have these larger scale comparisons in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to carefully read our paper. We are glad you found our method provides interesting insights. Below, we address your main concerns and your questions. \n\n> Regarding \"why excluding the InsectWingBeats dataset\"\n\nWe discussed the reason at Line 393 in Appendix A. The InsectWingBeats dataset contains very short time-series, such as 1 or 2 timestamps, which do not contain any meaningful shape-level features. Considering the number of samples and channels of this dataset is significantly higher than other datasets (refer to Table 3 in Appendix A), we believe this dataset may pollute the pre-training of our model and therefore excluded it from the experiments.\n\n> Regarding \"choice of benchmarking datasets\"\n\nWe chose to perform quantitative evaluation on the UEA datasets because they are a default choice in most recent works [Zuo et al., AAAI 2023; Wu et al., ICLR 2023; Zhou et al., NeurIPS 2023] on time-series analysis and suggested to benchmark deep learning methods (Deep Time Series Models: A Comprehensive Survey and Benchmark, Zhou et al., 2024). We agree that adding UCR can make the experiments more comprehensive. However, the results for all the 128 UCR datasets are not usually available for the baseline methods (which also means including UCR results is not generally a mandatory requirement); obtaining the results will require additional time and we could not finish them within the Rebuttal period. We will include the results of our methods and baselines in the revised version.\n\n> Regarding \"number of baselines to compare\"\n\nWe chose TimesNet, T-Rep, and MOMENT as our baselines since they are the best-performing methods among supervised, representation learning, and pre-trained models. We agree that it is good to include more baselines, and we have included detailed results and comparisons with 14 baseline methods in Table 1 of the Rebuttal attachment (please also refer to the global response). Compared with more baselines, our method still achieves comparable performance as the SOTA methods in time-series classification and outperform other methods in some metrics, while additionally provides the advantage of producing interpretable features.\n\n> Regarding \"the impact of reducing the size of codebook\"\n\nWe studied the effect of reducing the codebook size on downstream classification tasks and summarized the results in the Rebuttal attachment. Please refer to the global response for details. As a summary, pre-trained with codebook size 64, the Histogram representations of VQShape achieve the best average classification accuracy of 0.715, which outperforms the SOTA baselines. This matches our hindsight discovery in Figure 5 of the paper where there are roughly 60 clusters of codes.\n\n> Regarding \"how is the size of the codebook determined\"\n\n We previously experimented with a heuristically chosen codebook size of 512. The number of \"distinct\" shapes could depend on the volume and distribution of pre-training data. The model can be pre-trained with a large codebook (such as 512) to guarantee expressiveness for fitting large datasets. However, the actual number of \"distinct\" shapes is a statistical pattern learned from pre-training data, where such a conclusion can only be made afterward.\n\n> Regarding \"how can you encourage the decoded shapes to be as diverse as possible?\"\n\nIn this paper, we use the entropy terms in Equation 6 to encourage the usage of all codes in the codebook, which also promotes the diversity of latent codes. To encourage the decoded shapes in the time domain to be diverse, we can apply additional regularization by adding \n$$\n\\mathcal{L}\\_{\\text{div}} = \\frac{1}{|\\mathcal{Z}|^2} \\sum_{z\\_{1} \\in \\mathcal{Z}}  \\sum\\_{z\\_2 \\in \\mathcal{Z}, z\\_2 \\neq z\\_1} e^{-|| \\mathcal{S}(z_1) - \\mathcal{S}(z_2) ||_2}\n$$\n\nto the overall loss function, as introduced by ADSN [Ma et al., AAAI 2020]. However, introducing an additional objective may make pre-training more challenging. We will leave it as future work as it is not the focus of the our current method. \n\nWe thank the reviewer again for their feedback and hope that we have addressed their questions. If so, we hope they will consider increasing their score. Please let us know if our clarifications require further discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A9a02ZGLjY", "reviewer": "Reviewer_8MWE", "summary": "The paper introduces VQShape, an interpretable pretaining method for time series (TS) classification. VQShape uses transformer-based TS encoding combined with a VQ-VAE style codebook representation. The latter enables the representation of a TS as a set of shapelets. The VQShape encodes shapelets in a generalized form with additional information on length, position, and offset to overcome the typical limitations of shapelets, such as dataset specificity. The experimental evaluation highlights that VQShape performs on par with other pretraining methods while being interpretable and smaller.", "questions": "Questions and improvements to the mathematical notation:\n\n- Introducing $l_\\text{min}$ in 3.1 would make the representation easier to understand.\n- Similarly, providing formulas for $t_k$ and $l_k$ would be beneficial right there.\n- Further, for $l_\\text{min}$, $t_k$ and $l_k$ It is unclear if these should be integers or real numbers. While the text (and the later usage of them) points to them being relative, i.e., in \\[0, 1\\], the initial definition in lines 113 and 114, in combination with $T$ does not seem fitting to that. Also, when used to specify a time-series subsequence (e.g., line 142), they should be timestamps instead of being relative. The mathematical notation in this regard should be checked and aligned.\n- Relating $\\tau$ to $\\hat\\tau$ near l. 129 would make the section more digestible.\n- Overall, more intuition for the mathematics could make the paper even easier to understand.\n\nGeneral remarks and questions:\n\n- For me, the claim that VQShape outperforms MOMENT based on the results in Tab. 1 does not seem justified without further information. While the mean accuracy and mean rank of VQShape are a bit better than those of MOMENT, the differences are quite small compared to the large variations between the datasets overall. Claiming that VQShape outperforms moment would require some significance test, similar to comparing the mean rank to the other methods. (See \\[1\\] for more details on aggregation of results; see \\[2\\] for significance testing).\n- Regarding the results in Tab. 2, which show that VQShape can perform comparably even when trained on substantially fewer datasets: Do the authors have an intuition of how the other pretraining methods would do when pretrained on just a subset of the datasets?\n- Overall, all figures could benefit from better captions to give context to the figure. Each figure + caption should ideally be understandable on its own, or at least to a reasonable degree. In this paper, the figure captions do not provide the necessary context to the reader. Further, In the caption of Fig. 3: the \"presnetation\" should be \"presentation\"\n- In Figure 3: What are the different channels? Are these variates from the dataset? Further, the top and bottom parts each represent one sample, or did I misunderstand the histograms?\n- In the appendix, line 391 states that the standard deviations for Tab. 1 should be somewhere, but the reference is to the wrong table (presumably), and the standard deviations are nowhere in the appendix.\n- The clarity and the structure of the references could be improved. They are rather inconsistent (e.g., ICLR is noted differently among several entries) and contain a lot of unnecessary parts (e.g., URLs for published papers)\n- The visualization of the entire codebook in Appendix B.2 and B.3 is super insightful. However, many shapelets seem very similar, and their count should be easily reducible, as hinted in lines 264-268. Is it only visible in hindsight and necessary to learn a large codebook first, or could one start out with a smaller codebook right away?\n- The method introduces a possibly helpful indictive bias that would be difficult to enforce in deep auto-encoder architectures: It assumes that a TS can be summarized entirely by shapelets with position, scale, etc. This very implicit influence on the encoder could be discussed better.\n\nReferences:\n- \\[1\\] Fleming, Philip J., and John J. Wallace. \"How not to lie with statistics: the correct way to summarize benchmark results.\" _Communications of the ACM_ 29.3 (1986): 218-221.\n- \\[2\\] Demšar, J. Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research (JMLR), 2006.", "limitations": "While the main limitations are discussed in the paper, a short discussion of the trade-off between the codebook size could be beneficial: A larger size means potentially better learning, while a smaller codebook would be better for interpretability.\n\nThe main paper or the appendix should briefly discuss the societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces the novel method VQShape, which brings interpretability to time series pretraining.\n- To do this, VQShape introduces a modular and general way of utilizing shapelets over multiple datasets.\n- The paper is well-motivated and follows a clear structure. The writing is understandable and relatively easy to follow.", "weaknesses": "- The mathematical notation is not always precise and could need some clarification (specific details in the questions below)\n- The figures overall, but specifically Fig. 1, need more extensive captions to improve their comprehensibility.\n- The results in Tab. 1 do not include standard deviation information, which makes it difficult to judge the significance of the results and compare the different methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to carefully read our paper and writing a detailed review. We appreciate your comments and suggestion on giving more comprehensive statistical comparisons with the baseline. Below, we address your main concerns and your questions.\n\n> Regarding \"standard deviation of experiment results\"\n\nWe report the standard deviation of classification accuracy across 5 runs in Table 4 of Appendix B.1. The authors apologize for the incorrect reference in Line 393 that points to Table 2 (a summary of Table 4) for these results. We will fix this in the revised version.\n\n> Regarding \"VQShape outperforms MOMENT without further information\"\n\nWe follow widely used metrics to compare our methods with the baseline methods, including average accuracy, average rank, and number of top-1. Many recent deep learning methods are compared solely based on average accuracy (refer to Wu et al., ICLR 2023, Zhou et al., NeurIPS 2023). The authors agree that the statistical comparisons pointed out by the reviewer are reasonable metrics. We provide additional statistics for comparison in Table 1 of the Rebuttal attachment (also refer to the Global response for details). Based on the results, our method still achieves comparable performance as the SOTA methods in time-series classification and outperform other methods in some metrics, while additionally provides the benefit of producing interpretable features.\n\n> Regarding \"performance of other models pre-trained on fewer datasets\"\n\nAs the pre-training and implementation of MOMENT are not fully open-sourced, we cannot complete this experiment within the tight rebuttal window. We will try to include it in the revised version if accepted.\n\n> Regarding \"clarification on Figure 3\"\n\nChannels are variates of multivariate time-series, and we will change the text in the figure accordingly. The top row consists of the histograms of the three variates averaged over all samples from the CW category, and the bottom row consists of the histograms of the three variates averaged over all samples from the CCW category. As a clarification, we will improve Figure 3 by replacing \"Channel\" with \"Variate\" and revising the caption by adding \"Histograms are averaged over all the test samples from the CW and CCW category, respectively.\"\n\n> Regarding \"codebook size\"\n\nWe previously experimented with a heuristically chosen codebook size of 512. The number of \"distinct\" shapes could depend on the volume and distribution of pre-training data. The model can be pre-trained with a large codebook (such as 512) to guarantee expressiveness for fitting large datasets. However, the actual number of \"distinct\" shapes is a statistical pattern learned from pre-training, where such a conclusion can only be made afterward. We studied the effect of reducing the codebook size on downstream classification tasks and summarized the results in the Rebuttal attachment (see Table 2 and the Figures). Please refer to the global response for details. These results indicate that one can start with a small codebook size but may sacrifice performance if the chosen size is too small to fit the data.\n\n> Regarding \"inductive bias that would be difficult to enforce in deep auto-encoder architectures\"\n\nBased on our understanding of this comment, we conducted an ablation by removing the subsequence reconstruction loss and reported it in Table 2 of the Rebuttal attachment (see the column with $\\lambda_s=0$). With representations produced by the model pre-trained with $\\lambda_s = 0$, the classifiers result in lower accuracy, indicating that the shape reconstructions actually provide additional useful information. We believe this serves as strong evidence of our advantage. However, as the shapes are abstracted subsequences, we observe that $\\mathcal{L}_s$ cannot be effectively minimized to a very small value during pre-training (stuck around 0.3), which could reflect the reviewer's point on \"difficult to enforce\".\n\nWe thank the reviewer again for their feedback and hope that we have addressed their questions. If so, we hope they will consider increasing their score. Please let us know if our clarifications require further discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g7tXcZp0YZ", "reviewer": "Reviewer_mtPy", "summary": "This work aims to provide an interpretable and generalizable model, called VQshape, for learning on time-series. The key idea of the work is to learn representation based on shapelets, or shape-level features of timeseries as quantized vectors. By learning such a codebook, VQshape can learn in a dataset-agnostic and interpretable way. The authors pre-trained the model on diverse datasets and showed that the proposed model achieves comparable performance as other models on UEA classification benchmark. They also studied the interpretablity and generalizability of the proposed method.", "questions": "NA", "limitations": "Yes, they are mentioned in section 6", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Interesting idea. Using the concept of shapelet to learn time series may provide more interpretable models.", "weaknesses": "1. While the high-level idea is interesting, I am not convinced that the model worked.\n- Method 3.1 & 3.2: You use a transformer model to produce attribute tuples, why and how could you guarantee the produced attributes are accurate? PatchTST type of architectures fundamentally have this architetcure-level bias that breaks the shape of timeseries during the tokenization stage, how did you mitigate this side-effect and learn real shapes inside data?\n- In equation 5, is your subsequence a latent subsequence? If the answer is yes, how did you ensure the target latent subsequence has sufficient fidelity? If the answer is no, isn't your final reconstruction loss just a multiplication of the first term?\n- Line 188, what is the point of using shapelets if your model needs to interpolate all the input univariate? Did you consider that even in the same dataset, the length of the input may vary, and naively interpolating them would further mess up the sampling rate, even within the same dataset?\n- Figure 4 -- The learned codes are highly identical to each other. There lack high-frequency components in your constructed codebook.\n\n2. Experimental results are not convincing. \n- The proposed method has a lot of parameters, a lot of pre-trainings, yet it only performs similarly as TimesNet (which is much smaller and does not need pretraining).\n- Lack of other models as baselines.\n- Lack of experimental results as a whole. Can the proposed model be used for other tasks, e.g. regression, imputation, anomaly detection?\n\n3. The writing has serious clarity issues. Two examples below:\n- grammar error in Line 17\n- \"token\" in transformer has a very specific meaning. Yet the paper use the same word token for quantized vectors from the codebook.\n\n4. Lack of related works about interpretability.\n- There exist other ways to build interpretability into time series models. Common methods include: (1) Visualization of attention map across multi-scale transformer features [1]; (2) Use added tokens to \"prompt\" transformers for specific purpose [2]. When benchmarking the interpretability of the proposed approach, the authors should consider these methods. \n[1] Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., & Sutskever, I. (2020). Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341.\n[2] Xiao, J., Liu, R., & Dyer, E. L. GAFormer: Enhancing Timeseries Transformers Through Group-Aware Embeddings. In The Twelfth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for taking the time to carefully read our paper. Below, we address your questions.\n\n> Regarding \"why and how could you guarantee the produced attributes are accurate?\"\n\nIt is important to note that our proposed method is a self-supervised pre-training model, and there are no labeled subsequences to fit. Therefore, \"accurate attributes\" is not a valid metric here. We can validate that the model produces \"good\" attributes since the decoded shapes are close to the time-series for samples not included in pre-training (see Figure 2 for an example).\n\n> Regarding \"architecture-level bias of PatchTST model\"\n\nPatchTST breaks time-series into patches and projects the patches into embeddings. However, as it is a Transformer with full attention, each embedding contains non-local information. Therefore, the authors do not think that this can be a side effect that affects our proposed idea and method. The subsequence reconstruction loss in Equation 5 minimizes the difference between predicted shapes and real subsequences, which encourages learning real shapes.\n\n> Regarding \"Equation 5 and loss function\"\n\nA latent code is decoded to a subsequence in the time domain. Equation 5 penalizes the difference between decoded shapes and real subsequences in the time domain. The authors are confused by the comment \"reconstruction loss just a multiplication of the first term.\" Equation 4 penalizes the difference between the predicted time-series reconstructed from a set of $\\tau_i$ and the input time-series, and Equation 5 penalizes the difference between a single predicted shape and a real subsequence. Additionally, they train different components in the model, where Equation 4 trains the time-series decoder $\\mathcal{D}$ and Equation 5 trains the shape decoder $\\mathcal{S}$; and gradients from them back-propagate to train the codebook and encoder.\n\n> Regarding \"interpolating the input time-series\"\n\nWe argue this comments with three points:\n- Related works: It is common in time-series models to interpolate the time-series. For example, MOMENT subsamples long sequences to 512 timestamps and pads the short ones, and TimesNet unifies the lengths of input series to perform convolutions. Most of the common baseline methods (summarized by Wu et al., ICLR 2023) do not explicitly consider the length and sampling rate of data. Therefore, we do not think interpolating the data will create a fundamental defect in time-series classification tasks.\n- For our method: Building generalizable models requires determining a generalizable way to model time-series data, such as predicting the missing patch in MOMENT and predicting the future in TimeGPT-1. In our case, we view the method of describing a time-series with 64 abstracted shapes as the generalizable way to model time-series data. Since the abstracted shapes and their attributes are defined in relative scales, we can interpolate the time-series into the same length to improve execution efficiency.\n- Difference with \"shapelets\": In this paper, we term our codes and decoded sequences as \"shapes\" instead of \"shapelets\" since \"shapelets\" are originally defined as exact subsequences from the original time-series data, while our \"shapes\" only represent certain trend patterns. The abstracted shapes are agnostics to sampling rate and sample length. We don't think using them after interpolating the data could be an issue.\n\n> Regarding \"redundant shapes in codebook and high-frequency codes\"\n\nOur current model is trained with a codebook size of 512. The codes in Figure 4 are learned from pre-training data. From the hindsight result, we see that many codes are decoded into similar shapes, which indicates that the codebook size can be reduced. We provide additional results on the model trained with smaller codebook sizes (see the Rebuttal attachment). Learning codes with high-frequency components is not guaranteed as classification datasets either do not contain high-frequency data or the high-frequency components cannot provide shape-level features. Additionally, modeling high-frequency components of time-series can be addressed by multiple tokens. However, we don't think this should be termed as a defect of the model as the codes are purely learned from data.\n\n> Regarding \"comparison with TimesNet\"\n\nTimesNet is a dataset-specific model trained with supervised learning. Our model is pre-trained on multiple datasets and the weights are frozen when adapted to specific datasets with linear probing, where the linear classifier can have much fewer parameters than TimesNet. Providing useful representations that can efficiently adapt to downstream tasks is the purpose of using pre-trained models such as VQShape and MOMENT. Additionally, our method provides interpretable and generalizable representations, while TimesNet is a black-box model with end-to-end predictions.\n\n> Regarding \"additional baselines and experiment results\"\n\nIn the submission, we chose the best methods among each category of methods and compared them with our method. We additionally provide extensive baselines and comparisons in the Rebuttal attachment (refer to global response and Table 1). We focused on classification tasks in this paper since shape-level features can be more significant and informative in classification tasks than in other time-series tasks. We additionally provide preliminary results on extending our method to forecasting and imputation (refer to global response and Table 2 in the Rebuttal attachment).\n\n> Regarding \"`the use of term token\"\n\nIt is not clear what \"very specific meaning\" refers to, and whether it is specific to NLP. As a reference, in computer vision, VQGAN also uses token to term their codes. The authors of Large Vision Model [Bai et al., 2023] explicitly term the VQGAN encoder as image tokenizer. Therefore, we believe using the term token in our case is appropriate.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
The authors would like to thank the reviewers for taking the time to review our manuscript and for providing constructive feedback. Here, we aim to address and clarify several key points raised by multiple reviewers, and explain added results and figures in the Rebuttal attachment.

> Regarding "comprehensive comparison with more baselines"

In the submission, we mainly benchmark our method against TimesNet, T-Rep, and MOMENT as they are reported by literature to be the SOTA of supervised learning, unsupervised representation learning, and pre-trained models respectively. We did not include more baselines and their results on the UEA datasets are not always available and we did not finishing reproducing them at the time of submission. We include a more extensive comparison with 10 additional baselines using more metrics in Table 1 of the Rebuttal attachment. We can conclude the best performing baselines on the UEA datasets are TS2Vec, T-Rep, TimesNet, Reformer, and MOMENT. While there is no dominant method that consistently outperforms others across all the metrics, our method achieves the best mean accuracy. Beyond comparable performance with the SOTA methods, our method additionally provides the benefit of producing interpretable and dataset-agnostic representations for time-series data. 

> Regarding "performance of model trained with different codebook size"

We conduct a more extensive study on the effect of these hyperparameters by evaluating the models pre-trained with different codebook size. We summarize the results in Table 2 of the Rebuttal attachment. Figure 1 in the Rebuttal attachment visualizes the relationship between codebook size and average classification accuracy over the 30 UEA datasets. From these results, we can conclude that using the histogram representation produced by the model trained with codebook size 64 and code dimension 8 results in the best performance on the UEA datasets. However, the performance of using token representations increases as codebook size increases, and the classifiers using histogram representations outperform the token representations when the codebook is small. This indicate that the linear classifier cannot effectively manage the token representations since they are more expressive then histogram representations. Additionally, we visualizes the shapes decoded from codebook in Figure 2 and latent code distribution in Figure 3 of the Rebuttal attachment. We can see that the decoded shapes have lower redundancy compared Figure 4 in the paper.

> Regarding "extension to other time-series tasks"

As an extension to the original submission, we have obtained preliminary results of applying VQShape in both zero-shot and fine-tuning to other time-series tasks including imputation and forecasting. Table 3 and Table 4 in the Rebuttal attachment benchmark the performance of VQShape against MOMENT and TimesNet on imputation tasks and forecasting tasks, respectively. On imputation tasks, VQShape significantly outperforms MOMENT in both zero-shot and fine-tuning settings, and achieves comparable but slightly worse performance than TimesNet. On forecasting tasks, VQShape is able to provide predictions in zero-shot settings but the error can be high. By fine-tuning the model, VQShape achieves comparable performance as MOMENT and TimesNet. These preliminary results demonstrate our statement in Line 302 of the paper where we foresee that VQShape can be extended to general time-series tasks. However, we acknowledge that the results obtained by fine-tuning VQShape is not a fair comparison with results reported by MOMENT using linear-probing, where we aim to conduct more comprehensive and appropriate benchmarking in future works.

Meta-review: The paper proposes a novel vector quantization approach to learn interpretable shapes in time series that not only enhance interpretability but also generalize to other data sets. Reviewers appreciated the novelty of the approach and the clear writing of the paper. Given the complexity of the notation and the experimental evaluations, the reviewers also raised numerous technical concerns and questions. However, these could be addressed during an extensive rebuttal and discussion period, such that multiple reviewers raised their scores. Overall, the consensus after rebuttal appears to be acceptance.
Decision: Accept (poster)