VISION TRANSFORMER FOR MULTIVARIATE TIME- SERIES CLASSIFICATION (VITMTSC)
Authors: Prem Shankar Kumar
Ashutosh Joshi
Srinivas Adavi
Venue: Submitted to ICLR 2023 (2023)
Keywords: time-series classification
vision-transformer
transformer

Abstract: Multivariate Time-Series Classification (MTSC) is an important issue in many disciplines because of the proliferation of disparate data sources and sensors (economics, retail, health, etc.). Nonetheless, it remains difficult due to the high-dimensionality and richness of data that is regularly updated. We present a Vision Transformer for Multivariate Time-Series Classification (VitMTSC) model that learns latent features from raw time-series data for classification tasks and is applicable to large-scale time-series data with millions of data samples of variable lengths. According to our knowledge, this is the first implementation of the Vision Transformer (ViT) for MTSC. We demonstrate that our approach works on datasets ranging from a few thousand to millions of samples and achieves close to the state-of-the-art (SOTA) results on open datasets. Using click-stream data from a major retail website, we demonstrate that our model can scale to millions of samples and vastly outperform previous neural net-based MTSC models in real-world applications. Our source code is publicly accessible at https://github.com/mtsc-research/vitmtsc to facilitate further research.


Under review as a conference paper at ICLR 2023
VISION TRANSFORMER FOR MULTIVARIATE TIME- SERIES CLASSIFICATION (VITMTSC)
Anonymous authors
Paper under double-blind review
ABSTRACT
Multivariate Time-Series Classification (MTSC) is an important issue in many disciplines because of the proliferation of disparate data sources and sensors (economics, retail, health, etc.). Nonetheless, it remains difficult due to the high- dimensionality and richness of data that is regularly updated. We present a Vision Transformer for Multivariate Time-Serie Sene-Series Classification (VitMTSC) model that learns latent features from time-series data for classification tasks and is applicable to large-scale time-series data with millions of data samples variable lengths. According to our knowledge, this is the first implementation of the Vision Transformer (ViT) for MTSC. We demonstrate that our approach works datasets ranging from few thousand to millions of samples and achieves close to the state-of-the ti--hhe-rtt art (SOTA) results on open datasets. Using click-stream data from a major retail website, we demonstrate that our model can scale millions of samples and vastly outperform previous neural net-based MTSC models in real-world applications. Our source code is publicly accessible at hhttps //github www. /tt///tsccc//t/ resssssrereearrrcciittttttte to facilitate further research.
INTRODUCTION
Deep neural networks (DNNs) have shown significant effectiveness with both text (Lin al. 2021 and images (Li et al. 202T). The availability of standard DNN architectures that effectively encode raw data into meaningful representations, leading in good performance new datasets and associated tasks with little effort, is a crucial factor enabling advancement. In image understanding, for instance, variations of residual convolutional networks, such ResNet (He et al. 2015). exhibit relatively outstanding performance on new image datasets or somewhat different visual recognition problems, such as segmentation. Despite being the most prevalent sort of data in the actual world, tabular data has only recently been the focus of deep learning research, such as TabNet (Arik & Pfister (219)) and Tab TabTransformer (Huang et al. 2020). Time-Series data is a special case tabular data. An intrinsic characteristic of time-Series is that observations within time-series are statistically dependent on assumed generative processes (Löning et al.) 2019). For example, the likelihood a user streaming whether they have streamed movie X before well as latent
particular movie X, is dependent factors like, the time duration since the last streaming of movie X, number of movies from the same genne/direetor/rccor
X that the user has streamed, cost of streaming, promotions movie X, etc. Due to this reliance, time-series data does not readily fit inside the traditional machine learning paradigm for tabular data, which implicitly implies that observations are independent and identically distributed (i.i.d.).
We consider time-series, imput to the model, to composed of (a) time- time-points which they are observed, and (b) observations (or time-series datapoints or TS datapoints) at those time-points. We denote, time-series object that contains N samples as X [X,, X,,, ..., X where elements of the sequence are observation(s) at time-points 1,, 12. .... iN respectively. The time-series data may be (a) Univariate time-series (UTS), in which a single variable is observed over time, or (b) Multivariate time-series (MTS), in which two more variables
recorded over time. We denote ix XM)" for 1, 2, .... M
an individual multivariate time-series datapoint as X, |X X² distinct variables/feattres observed at each time-point. Please note in large commercial datasets, an individual X categorical variable can have millions of distinct values (refer to Table 2
In this paper, we will focus Multivariate Time-Series Classification (MTSC) (Ruiz et al. 2021 which is an area of machine learning interested in assigning labels to MTS data. In non-DNN based MTSC methods, the data is first converted to i.i.d. via feature transformation, and then traditional

---

Under review as a conference paper at ICLR 2023
classifiers are applied (Ruiz al. 2021). Only a few available approaches for MTSC consider DNNs for this task (Fawaz et al. 2018) (refer to Section 2 One advantage of using DNN for MTSC is predicted efficiency benefits, particularly for large datasets (Hestness et al.] 2017 In addition, DNNs offer gradient descent (GD) based end-to-end learning for time-series data, which has numerous advantages: (a) quickly encoding multiple data kinds, including images, tabular, and time-series data; (b) eliminating the requirement for feature engineering, which is a fundamental part of non-DNN MTSC approaches: (c) learning from streaming data; and perhaps most crucially (d) end-to-end models enable representation learning, which enables many valuable applications including including data-efficient domain adaptation (Goodfellow et al. 2016), generative modeling (Radford al. 2015), semi-supervised learning (Dai et al. 2017), and unsupervised learning (Karhunen et al.) 2015).
We present Vision Transformer for Multivariate Time-Series Classification (VitMTSC), a ViT- based pure transformer approach to MTSC, to model long-range contextual relationships in MTS data. The VitMTSC model accepts raw time-series data as input and is trained with GD based optimization to facilitate flexible end-to-end learning. We evaluate VitMTSC using both publicly available and proprietary datasets. The majority of algorithms in the MTSC literature have been evaluated on open-source UEA datasets (Ruiz et al., 202I). We evaluate VitMTSC on five UEA datasets (Table I and demonstrate that VitMTSC achieves comparable performance to the current SOTA methods (Table 3.. Using click-stree fercam data (Table [ from a popular e-commerce site, we show that VitMTSC can meet the needs of modern datasets and performs much better than current DNN methods (Table on both small and large real-world datasets (refer to Section 4 for experiment details).
2 RELATED WORK
Since AlexNet (Krizhevsky et al., 2012). deep convolutional neural networks (CNNs) have advanced the state-of-the-un art across many standard datasets for vision problems. At the same time, the most prominent architecture of choice in
modeling is the Transformer (Vaswani al. 2017), which does not convolutions, but is based on multi-headed self-attention (MSA). The MSA operation is particularly effective at modelling long-range dependencies and allows the model to attend over all elements in the input sequence. This is in stark contrast convolutions where the corresponding "receptive field" is limited, and grows linearly with the depth of the network. The success of attention-based models in Natural Language Processing (NLP) has inspired approaches in computer vision to integrate transformers into CNNs (Wang et al. 2017: Carion et al. 2020). as well as some attempts to replace convolutions completely (Parmar et al. 2018 Bello et al. 2019 Ramachandran et al. (2019). With the Vision Transformer (VTT) (Dosovitskiy et al. 2020). however, architecture has recently outperformed its convolutional counterparts in image classification, Since then, the ViT model has also been applied to other domains, for example, video classification (Arnb et al. 2021 and many more variants have been proposed
a pure-transformer-
In the last two decades, time-series data has proliferated plethora of fields including eco- nomics (Wan & Si (017]) health and medicine (Gharchbaghi et al. 2015). scene classification (Nwe) al. (017). activity recognition (Chen al.) (2020), traffic analysis (Yang et al., 20144,, click-stream analysis (Jo et al.
2018) and more. DNN is used in modeling time-series tasks, such as forecast- 2020 Lim & Zohren 2021
ing (Benidis et al.
/ Torres et al.) 202T), classification (Wang et al., 2016)
Fawaz et al. 2018 8 Zou et al. [019 Fawaz et al. 2019), anomaly detection (Blazquez-Garcia Garcia et al. 2020: ChoiiCChheee et al.] 2021), and data augmentation (Wen et al. 2021). Similarly, Transformers (Wen)
et al.
2022) have been used in Time-Series forecasting (Li et al. 2019/Zhou Zhou et al., 2020 2022).
anomaly detection (Xu et al. [021 Tuli et al., 2022). classification (RuBwurm et al. 2019 Zerveas
cet al.
2021 Yang et al. 2021).
Historically, the problem of MTSC was addressed by non-linear transforms of the time-series on which standard classification algorithms are used. [Bostrom & Bagnall (2017) used modification of the shapelet transform to quickly find i shapelets of length n to represent the time-series. They then use the kxn feature vector in a standard classification model. Similarly, BagOfPatterns (Lin) et al. (012)) use the frequency of each word (token/feature) in the series to represent the series as histogram of words. Bag of Symbolic Fourier Approximation Symbols (BOSS) (Targe et al. 2018 uses Symbolic Fourier Approximation on frequencies of words in the series. Word ExtrAction for time SEries CLassification (WEASEL) (Schäfer Leser, 2017) extracts words with multiple sliding windows of different sizes and selects the most discriminating words according to the chi-squared
2

---

Under review as a conference paper at ICLR 2023
test. RandOm Convolutional KErnel Transform (ROCKET) (Dempster al. 2019) generates large number (10k) random convolutional kernels and extracts two features from the convolutions: the maximum and the proportion of positive values. It's originally designed for univariate time-series classification (UTSC) and later adapted to MTSC. Though ROCKET shows great performance on smaller MTS benchmark datasets, using for datasets with hundreds of millions of MTS data is impractical (refer to 4.3).
DNN approaches for MTSC have borrowed heavily from image processing models mainly Convolutional Neural Nets (CNN). Wang et al. (20T6); Lin et al. (2020) used attention based CNNs.
while Fawaz et al. (2019) used an ensemble of CNNs and Zou et al. (201); Fawaz et al. integrated residual networks with CNNs. Similarly. Transformers
220
used by Yang et al. 202
for UTSC. Rußbwurm et al. (2019); Zerveas et al. (2021) used transformer for the MTSC, and we
have compared and contrasted their approaches with our proposed approach (refer to 4.6)
METHOD
3.1
VISION TRANSFORMER FOR MULTIVARIATE TIME-SERIES CLASSIFICATION (VITMTSC)
Vision Transformer for Multivariate Time-Series Classification (VitMTSC)
Transformer Encoder
Class
MIP Head
Class 2
Transformer Encoder
MLP
Norm
Patch Position
Embedeing
Exto loumobre
(etss]] amaudting
Linear Projection Encoded/Embeddedd features
Multi-Head Attention
(vartable length)
User
Norm
Tolks
Nakur
1g
Time
Lineer Projection Eeeiiiedd
Ersccled/E Seecdle///-bbeiieed
Figure 1: Vision Transformer for Multivariate Time-Series Classification (VitMTSC) adapted from ViT (Dosovitskiy et al. 2020).
Figure gives an overview of the VitMTSC architecture. We follow the original ViT design as closely possible to take advantage of its scalable architecture, efficient implementations and future innovations (e.g., [ao et al. (2022)). Many variants ViT has been proposed but we will describe changes made the original ViT model to handle MTSC.
3.1.1 TOKENIZATION: How TIME-SERIES DATA ENCODED FOR TRANSFORMER ENCODER?
The original Transformer architecture
1D sequence of token-ids input, where token-id is index of a word or sub-word in the vocabulary (V), and number of elements in this ID sequence is the input sequence length. In the VIT, 2D images are first divided into multiple smaller 16x 2D patches, and then each patch is flattened as ID vector. This ID vector is the input to the transformer, and total number of image patches serve the effective input sequence length. Similar the standard ViT, we start with MTS data, X, and the number of time-steps in the MTS gives us input sequence length, N for the Transformer. However, though input to ViT is strictly numerical (pixel values), an MTS datapoint can contain combination of continuous and categorical features (2). There are multiple ways to encode continuous and categorical features. Each individual feature of an MTS datapoint is separately encoded to produce corresponding ID vector. Then these encoded ID vectors

---

Under review as a conference paper at ICLR 2023
aggregated (concatenated) and presented in ID format input to the VitMTSC transformer encoder.
Feature Encoding. There are different ways of encoding features. An individual continuous numerical feature of an MTS datapoint can be represented as real-valued scalar, as linear projection from scalar, or as Soft One-Hot Encoding (SOHE) (Li et al., 2018; de Souza Pereira Moreira et al., 2021). Categorical features in real-world datasets can take on millions of distinct values. Thus, encoding like one-hot encoding that use vectors proportional to the cardinality the feature values, do not work for commercial datasets. Instead we use target encoding (Rodríguez et al. 2018). For each categorical feature, the mean a continuous target column is calculated, and the group-specifie mean of cach row is used to create new feature (column). To prevent overfitting, the mean is computed via k-fold cross- cross- validation and smoothed with the overall mean of the target variable. This transformation results in single feature for each categorical variable; not increasing the dimensionality of the dataset. In our dataset, we normalized continuous numerical scalar features and target encoded categorical features (NVIDIA 2022). This helped keep model complexity low, in comparison to using corresponding embeddings (e.g., Patel & Bhattacharyya (2017) shows that embedding dimension lower than 19 are not performant and our corresponding commercial embedding size is at least 256).
Padding and Truncation. Since time-serics data can have variable length, we post-pad the in- put array with Os to generate sequences of pre-defined length N. Depending on the application, longer sequences are truncated to either the first or last N datapoints or use NLP style chunk- ing (Briggs. 2021). We choose N to be the 99th percentile length in the dataset in our real-world application Table 2
3.1.2 TOKEN EMBEDDING
The original Transformer receives as input a ID sequence of NLP tokens, map it to D dimensions with trainable linear projection, and uses this constant latent vector size D through all of its layers. To handle 2D images, ViT, reshapes the original image in multiple smaller patches. They then flatten each patch and embed it D-dimensional vector with a trainable linear projection. Similar ViT, embed the input sequence with a trainable linear projection (E in Eq. D. The token embedding learns to represent each MTS datapoint as a vector in R. Similar to BERT's [CLS] and ViT's [class] token, VitMTSC prepends a learnable embedding to the sequence of embedded patches (time-steps) (zf Xalass), whose state the output of the Transformer encoder serves as the representation of the time-series. In addition, to patch embeddings we also use positional embeddings (Epos) We discuss positional embeddings in Section 3.1.3
Zo Xclass: X E; X, E...; X,x E + Epos
(I)
3.1.3 POSITIONAL EMBEDDING
Time-series data is ordered but Transformers are permutation invariant without positional information. The original Transformer (Vaswani al.) 2017) uses hand-crafted absolute positional encoding. Since the relative time between the data-points can be important for MTSC, we add learnable 1D position embeddings (Eposi see Algorithm 2 of PPhuog & Hutter (2022) to the time-step (patch) embeddings to retain this positional information before feeding input to Transformer encoder layer. The positional embedding is added to the token embedding form token's 's initial embedding for the transformer encoder (Eq. D..
Positional Encoding as Timestamp Features Some datasets can have a set of (unordered) data points with the same timestamp. For example, for time-ordered set of products purchased by a customer, a single customer cart (same timestamp) may include multiple products. In such a case, absolute or learnable positional encodings will impose an arbitrary order on these datapoints (products). To avoid such an arbitrary order we do not use positional encoding on such datasets. Zhou et al. (2020) added both learnable and timestamp positional encoding to token embedding. Instead, we include the timestamp information (month, day, hour, etc of the cart purchase) of the datapoints as categorical features in the time-series. We found that the VitMTSC able to this information better than an arbitrary order imposed by the learnable positional encoding (refer T))le
3.1.4 ATTENTION
Attention is the main architectural component of transformers; see Algorithm of Phuong & Hutter (002)). In practice, transformers use multi-headed self-attention (MSA) multiple attention heads
4

---

Under review as a conference paper at ICLR 2023
(with separate learnable parameters) run in parallel and combine their outputs; Algorithm 5 of Phuong Hutter (2022). VitMTSC also uses padding mask - it first converts Os to 1s, and non-O values to Us, and then multiplies the padding mask with large negative value. In VitMTSC's encoder MSA, the padding mask added to the scaled score so that the original empty locations are now replaced with large negative values which will be eventually ignored by softmax activation function; see Algorithm 4 of [Phuog & Hutter (2022), called with Z = X and Mask.
3.1.5 LAYER NORMALIZATION
Vie3
Layer Noam
adoidon
FWN
Layer Norm
Layor Noamm
eddricn
MaliHiaa
Ansriion
Nerm
(a)
(b)
Figure 2: (a) Post-LN Transformer layer; (b) Pre-LN Transformer layer. adapted from (002)).
Xiong et al.
Layer Normalization (Ba et al. 2016) explicitly controls the mean and variance individual neural network activations; see Algorithm 4 of Phuong & Hutter (2022). The original Transformer uses Layer Normalization after "MSA/FFN block + Residual connection" in the transformer encoder called POSTNORM. Recently, GPT-2 (Radford al., (2019), ViT (Dosovitskiy et al. 2020) and the ViTMTSC, use Layer Normalization just before "MSA/FFN block" " in the transformer encoder called PRENORM (Figure 2).
3.1.6 ENCODER-ONLY TRANSFORMER AND MODEL PREDICTION
Like VIT, the VitMTSC transformer encoder consists of alternating layers of multiheaded self- attention (MSA) + residual connection and multilayer perceptron (MLP) + residual connection. The MLP contains two layers with a Gaussian Error Linear Unit (GELU) (Hendrycks & Gimpel 2016) non-linearity. Layernorm (LN) is applied just before MSA/MLP block and then residual connections applied (Figure 2). VitMTSC's [class] token state the output of the Transformer encoder (zL) serves as the time-series representation y (Eq. 2 The classification head implemented by multi-layer perceptron (MLP) with hidden layer and is attached to y the [class] token output of the transformer encoder; see detailed Algorithm | which has been adapted from Algorithm 8 of
Phuong Hutter 2022
y LN(2.)
(2)
EXPERIMENTS
We evaluate VitMTSC model along with current SOTA MTSC models on both publicly available and proprietary datasets.
2 All current transformer models are some variation of the original Transformer. To set new standard in DL publishing, [Phuong & Hutter (2022) authors explicitly encourage the reader to copy and adapt them to their needs and cite the original "adapted from [PH22]".
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1: P ETransformerrxx () adapted Algorithm 8 from Phuong Hutter (0022)
Input: x E V. an MTS sequence. Let V denote a set, such that Xix V, often ıdentified with [Ny] {1, [yy. Let X =X x(1 = x[]]x[2]...x[(]] E V".
Output: p E ((C),a probability distribution over the classes in task.
Hyperparameters: kmaz: L, H, de. dmiip: d E NN
Parameters: 0 includes all of the following parameters:
W. E Rd.xNv Wp Rd, Xxmax, the token embedding and positional embedding matrices.
For L E [L]
Wi, multi-head attention parameters for the layer I. 7t Bt, of BB E RRd, two sets of layer-norm parameters.
W mlpl Ramip xd, b'mipt E RAmde, W mipa R xde ,,"" miig2 E RAmte, MLP parameters.
Wouel mip3 E Rd. (ddey, mnpps RRdmy Sonee, E Iddoxdmip, mipa E Rdmer, the final linear projection and layer-norm parameters.
mlpt
1
+ length(x)
2
for / [ ee Wol:x0[U] W
3
X [e1,e2,..ee)
4
for / 1.2, ..... do
5
for t E [4] X|: + layer [[: t). 7i ;i))
6
X X+ MHAttention (X W, Mask 1)
7
for l E [$] X(; layer_norm( [[; 71, 3g)
X X+ w mlp2 GELU(W mip1 X b" b'mipi 1
b² min21
9
end
10
X
X[: 1]. discard everything and keep only [class] token.
11
fort E [ X[;,t] +- layer_none boyer_norm([[:: 7,,
12
X x+ Wout GELU (Woorr + b 1") + b out?
mlpp
1T
13
return p softmax(X)
Table 1: Open-source UEA datasets used for evaluation. has only continuous (CONT) features and time-series datapoints are combined across both train and test datasets.
DATASET LENGTH CONT CAT CLASSES TRAIN TEST TS DATAPOINTS
PenDigits
8
2
N/A
10
7,494
3,498
87,936
Arabi
4-93
13
N/A
10
6,599
2,199
350,253
CharacterTrajectoris
60-182
3
N/A
20
1,422
1,436
342,939
FaceDetection 62
N/A
6
5,890
3,524
583,668
Wingbeeingbeat 2-22
200
N/A
10
25,000
25,000
335,534
4.1 DATA
The majority of algorithms in the MTSC literature have been evaluated on open-source UEA datasets (Ruiz et al. 2021). The individual datasets (Table [ in the UEA datasets very small (the largest dataset has only 50k time-series data). Modern commercial datasets on the other hand contain hundreds of millions of time-series datapoints (refer Table 2) Also, UEA datasets only contain continuous variables, whereas real-world datasets can include a mix of continuous, discrete, and categorical variables. Though evaluate VitMTSC on UEA datasets, the real power of the method is seen in real-world data.
4.1.1 OPEN-SOURCE UEA DATASETS
The UEA MTSC archive is repository of benchmark datasets for time-seme meressries classification (Bagnail et al. 2018). It includes 30 multivariate datasets, four of which have variable lengths. We test our model on the five datasets (refer T) available in the UEA repository, which include three datasets of variable length, condition frequently observed in real-world data.
4.1.2 REAL- WORLD COMMERCIAL DATASETS
We used VitMTSC and other DNN MTSC models to analyze clickstream data on different stores of popular e-retailer. The aim of our study was identify the set of customers that would be responsive

---

Under review as a conference paper at ICLR 2023
Table 2: Commercial datasets with both continuous (CONT) and categorical (CAT) features
DATASET LENGTH CONT CAT CLASSES TRAIN VALID TEST TS DATAPOINTS
Small
5-300
8
20
2
595K
198K
198K
94M
Large
10-300
8
20
2
4.2M
0.9M
0.9M 922M
to specific marketing campaigns. Using click-stream data as input, we build a binary classifier with the target variable being respond/not respond to marketing campaign. We use response data from similar historical marketing campaigns for each retail store and train a separate model for each store. One of our smallest store has 992k time-series while the largest one has 6m. Each data point in the time-series consists of 20 features - 8 continuous and 12 categorical that include products that the customer has clicked/purchased. product category. interaction type (click/purchase), time features (month, day, hour, etc), product attributes (price, discount, etc), customer attributes and more. Also, the space of values that categorical variable take can be quite large (e.g.: product ID can be one of hundreds of thousands to hundreds of millions of products depending on store).
4.2 RESULTS
4.3
BENCHMARK DATASETS
For MTSC, ROCKET (Dempster et al. 2019), and CMFMTS (Baldan & Benítez 2020)
current
SOTA Non-DNN methods and TST (erveas et al.) 2021) current SOTA DNN method. On the datasets of interest us, they outperform all other models. Since UEA datasets specify both train and
test datasets, we are able to use performance numbers from Baldan & Benítez (2020) and andZZereaa al. (2021) as CMFMTS and TST performances respectively. Unlike current practice in MTSC research(Section 6.6 of Fawaz et al. (2019), and Section 4 of [eereas et al. (202I)), we further divided train data in 80-20% split and use this new validation dataset to tune both typee-parameters and guide training to avoid overfitting on test dataset. We use provided test dataset only in the end check model generalization capabilities, like we do in commercial applications. Table gives the average accuracy of CMFMTS, TST, ROCKET and VitMTSC on these five open-source MTSC datasets.
ROCKET is one of the most prominent time-series classification algorithm. Unlike Zerveas et al 2021), we were able to train ROCKET on all open-source datasets using sktime implementatiori However, a p3dn.24xlarge (768 GB RAM and 96 CPUs) machine, ROCKET took 2 hours 40 minutes to complete training/inference on the biggest open-source dataset, InsectWingbeatt (Table that has only 50K time-series datapoints. This huge run-time renders unusable commercial datasets (Table 2 with millions of time-series datapoints and we won't 't discuss it further.
Table 3: Average accuracy of VitMTSC and other models UEA datasets. A dash indicates that the corresponding dataset was not evaluated in the respective paper. The VitMTSC(val) column shows model overfit, since was also used for hyper-parameter tuning.
DATASET
CMFMTS TST
ROCKET
VitMTSC(val)
VitMTSC((tst)
PenDigits
95.9
97.8
99.4
97.2
Digi
97.5
99.8
99.8
99.0
96.9
CharacterTraeectoriiss
Trajectories
97.1
99.3
97.1
98.5
FaceDetection
57.9
68.9
64.0
72.7
67.0
InseeeWingbeat
67.7
68.7
53.6
60.4
60.3
As we can see, model overfits validation set since it was used for both hyper-parameter tuning and
guide training but generalized poorly on 3 out of unseen test datasets especially on smaller datasets. We hope that future MTSC papers will follow our approach and won't 't use provided test dataset for
https:// /tt///www.w... skt http://www..tt..eo.o..eee/////// http:///wwwwttttreooo///////////

---

Under review as a conference paper at ICLR 2023
either training or hyper-parameter tuning. This will help benchmark generalization capabiliitess of MTSC models correctly. Since Zerveas et al. (2021) has already explored impact of pre-training on UEA datasets, didn't. t. Rather, future, would like explore pre-training methods that works well on ViT e.g., BEiT (Bao et al. 2022).
4.4 OFFLINE EXPERIMENTS COMMERCIAL DATASETS
We compared our model against several DNN time-series classifiers that learns, like VitMTSC, end-to-end directly from raw MTS data without feature engineering. For comparative analysis, we implemented and evaluated six published MTSC DL models FCN (Wang al. 2016).
Inception Time (Fawaz et al.] 2019)), ResCNN (7ou et al.) 2019), ResNer (Fawaz et al. 2018), CropTransformer (RuBwurm et al. 2 and TST (Zerveas al. (021) - and the VitMTSC model on our smallest dataset (refer Table 2 We divided this dataset into 595K (336K +ve and 259K -ve) training, 198K (112K +ve and 86K -ve) validation and 198K (112K +ve and 86K -ve) test sets. Our time-series sample consisted of 10-300 datapoints depending user activities. Since our data unbalanced, we use class weights to account for the imbalance. Also, as mentioned in Section B.1.3] do not use absolute or learnable positional encodings and instead include the temporal information in our multi-variate data timestamp encoding. Table shows the PRAUC for the different models. Based on these results, we trained only our VitMTSC model for our larger datasets with 6M datapoints.
Table 4: PRAUC of VitMTSC different DL models data from of smaller stores. The model predicts whether a customer will respond/not respond to a marketing campaign.
MODEL
RESPOND NOT RESPOND
FCN
0.57
0.47
Inception Time
0.56
0.42
ResCNN
0.55
0.41
ResNet
0.54
0.44
CropTTrasffrrma
0.55
0.45
TST
0.56
0.45
ViGMTSC
0.96
0.95
4.5 ONLINE EXPERIMENT ON COMMERCIAL DATASETS
In online A/B test between VitMTSC and existing heuristic-based baseline method for finding potential customers who will respond to a marketing campaign, the conversion rate (the percentage of customers who responded to the campaign out of those who were targeted) went up by an average of 352% (p-value < 0.001).
4.5.1 ABLATION STUDY ON COMMERCIAL DATASETS
We performed four ablations to evaluate the importance of placement of layer- noyer-normalization, padding masks, class weights and positional encoding (refer Table 5). Though performed the ablations on nine datasets, for simplicity, we present results from only two datasets (refer Table 2 with the understanding that other datasets followed similar trends. Section 3.5.5 and Section 4 provides detailed information on Layer-Normalization and its impact. As also noticed by Nguyen & Salazar (2019). even with learning rate warmup. the VitMTSC PostNorm version failed converge on our datasets. Adding explicit positional encoding deteriorates performance. This is because positional encoding adds an arbitrary order multiple activities that occurred the same time. Instead using the categorical time features (month, day, hour, etc) improves the performance of the model. The importance of clags-weight depends on the amount data. Only in the Large dataset where the data size is an order of magnitude more than other datasets, class blasswwiigtting was not required. All other datasets (eight stores though T report results on only one of the eight stores and the largest store) showed improved performance with class weights. Similarly. masking allowed the classifier to ignore irrelevant (padded) data- mata-points and improved performance. Table gives the average PRAUC of the VitMTSC with different configurations these real-worle commercial datasets.
8

---

Under review as a conference paper at ICLR 2023
Table 5: VitMTSC ablation study: average PRAUC commercial datasets. We out (a) Pre and Post noyer-normalization, (b) using learnable positional encoding vs timestamp positional encoding (including the temporal information as features in our multi-variate data), (c) using class weights and (d) padding masks. Each ablation uses the best values from the other three combinations.
DATASET LAYER-NORM PRE POST
POSITION- POSITION-ENCODING CLASS-WEIGHT NO
MASK YES NO
LEARNABLE TIMESTAMP YES
Small
0.961 0.574
0.943
0.961
0.961 0.955
0.961 0.957 0.971 0.969
Large
0.975 0.528
0.949
0.971
0.971 0.975
4.6
COMPARISON OF OTHER MTSC TRANSFORMER MODELS TO VITMTSC
CropTransformer (RuBwurm et al. 2019) and TST Zerveas et al., 2021 are other two Trans- former models for MTSC. The CropTTranton Transformer official implementation uses Rectified Linear Unit (RELU) Nair & Hinton 2010) activation function, POSTNORM mayer-normalization (Ba et al.
2016 and hand-crafted absolute positional encoding like the original Transformer (Vaswani et al.
2017). TST uses POSTNORM like the original transformer but replaces noryaaroomaiizoioon with batch-normalization (loffe & Szegedy 2015), learnable positional encoding and GELU activation function. VitMTSC on the other hand, uses PRENORM layer-normalization GELU activation func- tion and learnable positional and timestamp encoding (refer to 3.1.3 Also, similar BERT's [CLS] and ViT's [class] token, VitMTSC uses only [class] token at output layer for MLP and discards rest of transformer encoder output. In contrast, both CropTransformer and TST don't 't use [CLS] token and feed full output of transformer encoder, in the end, to the MLP layer for classification. This [class] token provides an aggregate representation
MTS sequence to final MLP Head for classification. Table 4 and 5 show impact of these choices in model performance on commercial datasets (..122 Both official implementations of TST and Crop'Tranfformet are in PyTorch and Py PoTcch offical transformer encoder implementation added support for PreNorm in mid-202 [ For this reason, we decided to provide the Algorithm for VitMTSC (3.1.6).
The main reason why VitMTSC significantly outperform CropTransformer (Rubwurm et al. 2019) and TST (Zerveas et al. 202T) on real-world datasets is: VitMTSC, similar to ViT. uses PRENORM
layer-nornalizationrreee to 3.1.5p. p. Nguyen & Salazar [2019)[Xiong al. (2020) provide detailed
insights of impact of PRENORM vs POSTNORM in Transformer. In our ablation study on using PRENORM and POSTNORM (refer Table 5). we found that PRENORM is the key learning and
larger commercial datasets (refer Table As mentioned in Section 4.5. lnn also noticed by Nguyen & Salazar 2019), even with learning rate warmup, the VitMTSC PostNorm version failed to converge on our datasets.
CONCLUSION AND FUTURE WORK
We introduce Vision Transformer for Multivariate Time-Series Classification (VitMTSC), novel deep learning architecture for multivariate time-series classification (MTSC) based on ViT (Dosovitskiy al. 2020). We demonstrate, on open-source datasets, VitMTSC's performance matches the current SOTA methods, the model scales to large datasets seen commercial retail sites and provides far superior performance than other neural net based MTSC models. Since, the VitMTSC model can learn directly from raw multivariate time-series data, we can avoid the manual feature engineering, which is fundamental part of non-DNN MTSC algorithms.
Finally, we emphasize the significance of PRENORM Layer normalization in transformer networks. Attempts have been made comprehend VIT's success (Raghu al.) 2021: Trockman & Kolter 2022) and enhance its performance further (Chen et al. 2021). As part of further research, we will investigate effect PRENORM/POSTNORT Layer normalization on ViT, since we observed significant impact in our commercial MTS datasets.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Fevzi Alimoglu. Combining multiple classifiers for pen-based handwritten digit recognition. 1996. URL https::
ist. psu. hctp://ieewdlc/oooocccaalddddda. 1. 1.25. 62996 wtw.9errppeeellttteeppoo
Sercan Ömer Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. CoRR, 00s//198.07442, 2019. URL
ltargg/bs/19900 07442
Anurag Arnab, Mostafa Dehghami, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. Vivit: A video vision transfomner. CoRR, abs/2103.15691, 2021. URL
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL
aattss://rrxiie wwtp///ggg/////oo/eoo/o//eeessst http//bss//o.0.. 06450
Anthony J. Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn J. Keogh. The UEA multivariate time series classification archive, 2018. CoRR, abs/181 1.00075, 2018. URL org. abs /1111. 00075
Francisco Javier Baldan and José Manuel Benítez. Multivariable times series classificatton through interpretable representation. CoRR, abs/2009.03614, 2020. URL /tL//tttps///rx..v.orgg
abs/ abs/2009.030 03614
Hangbo Bao, Li Dong. Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers.
In International Conference on Learning Representations, 2022. URL /tpp:tttp///oeereeeeeee net
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V. Le. Attention augmented convolutional networks. CoRR, abs/1904.09925, 2019. URL
1904. 09925
Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Bernie Wang, Danielle C. Maddix, Ali Caner Türkmen, Jan Gasthaus, Michael Stella, Laurent Callot, and Tim Januschowski. Neural forecasting: Introduction and literature overview. CoRR, abs/2004. 10240, 2020. htt.tttss//xxxi..or///// http:///axxxx...//////////////// 2004. 10240
David Salinas, Lorenzo
Ane Blázquez-Gareía, Angel Conde, Usue Mori, and José Antonio Lozano. A review on out- lier/anomaly detection in time series data. CoRR, abs/2002.04236, 2020.
URL https
/arxii. /azxiv..rc/abb//0222000 0422332366
Aaron Bostrom and Anthony J. Bagnall. A shapelet transform for multivariate time series classifica- tion. CoRR, abs/1712.06428,, 2017. htR.hhttpp//axxiv.... httt:::/arxiv..org/ab///// 1//21//.111222
James Text, 2021.
Briggs.
How
to
Apply
Transformers
to Any Length of
URL
bttp::/towardddrrscciieee. com/
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagonuyko. End-to-end object detection with transformers. CoRR, abs/2005. 12872, 2020. URL
https::/aaxiz:: org/ abs 2005.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretabiiiityy beyon attention visualization. CoRR, abs/2012.09838,, 2020.
09838
Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, and Yunhao Liu. Deep learning for sensor-based human activity recognition: Overview, challenges and opportunities. CORR, Abs/2001.07414 2020. URL UBL.Ettp..//aar...cc//////....1 67416
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training sew.eunrriiieeeeeefuruerrreeddde vision trans- formers. CoRR, 0bbs2214400257, 2021. URL/:Epps////exxv.orr//////22 02057
Yanping Chen and et al. Adena Why. Uer computational entomology, 2017. URL https: /ww
ucr edv/ aedu=/eamonn/c//
10

---

Under review as a conference paper at ICLR 2023
Yanping Chen, Adena Why, Gustavo E. A. P. A. Batista, Agenor Mafra-Neto, and Eamonn J. Keogh. Flying insect classification with inexpensive sensors. CoRR, abs/1403.2654, 2014. URL http://arxiv org. h/tp///b////a www..ab.....2.. 2654
Kukjin Choi, Jihun Yi, Changhwa Park, and Sungroh Yoon. Deep learning for anomaly detection in time-series data: Review, analysis, and guidelines. IEEE Access, 9: -20006550 2021. doi:
N0.1109/ACCESS.202....
Conda. Conda: open source package management system and environment management system, 2020.
Zihang Dai, Zhilin Yang, Fan Yang, William W. Cohen, and Ruslan Salakhutdinov. Good semi- supervised learning that requires a bad GAN. CoRR, abs/1705.099883, 2017. h http://
arxiv.org abs 1000 09783
Gabriel de Souza Pereira Moreira, Sara Rabhi, Jeong Min Lee, Ronay Ak, and Even Oldridge. TransformersArec: Bridging the gap between nip and sequential / session- recommendation. In Proceedings of the 15th ACM Conference on Recommender Systems, RecSys '21, pp. 143-153, New York. NY, USA, 2021. Association for Computing Machinery, ISBN 9781 9781450384582 doi: 3474225
10.1145/346023 .3474255. /t
Angus Dempster, François Petitjean, and Geoffre I. Webb. ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels. CoRR, 0bbs/1910.130511 2019, URL
http://arxiy> org.orgg........ aaaaabas1990 1910 13051
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. CoRR, abs/2010. 19999, 2020. U
http://axxivv.....
2010 11929
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL 1cs. www.uciieeuuua..
fast.ai. nbdev create delightful python projects using jupyter notebooks. Github, 2020. URL
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Ale Alain Muller. Deep learning for time series classification: review. CoRR, abs/1809.04356, 2018. URL
org/
halgg//bba absp/bass///// 04356
Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F. Schmidt, Jonathan Weber, Geoffrey I. Webb, Lhassane Idoumghar, Picrre-Alain Muller, and François Petitjean. Inceptiontime: Finding alexnet for time series classification. CoRR, abs/ hets/1999.000 909999 2019. BRLLattpp//arxxv.rrrr
909. C4939
Arash Gharchbaghi, Per Ask, and Ankica Babic. A pattern recognition framework for detecting dynamic changes cyclic time series. Pattern Recognition, 4863).696-78,, 2015. ISSN 0031- 3203. doi: 10. 10.10006/jpatcog220444...0.111 octienee//rrileee/////1013332332
wttp://wwwwciieeeeereettttt com/
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
7 7/www. www.deeeplearn.ngbboo.. org
Nacereddine Hammami and Mokhtar Sellam. Tree distribution classifier for automatic spoken arabic digit recognition. 2009 International Conference for Internet Technology and Secured Transactions, (ICITTT), pp. 1-4, Nov 2009. doi: 10. 00.1109/CT1ST.220095550000
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.00388 2015. URRLhttp: /tatt:://xxiv....// www.iiiixooo wrr. /abs/ / 03385
Dan Hendryeks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415, 2016. URL http:///xxxvv.../aabss.ss /blg/bbb/1506
08415
11

---

Under review as a conference paper at ICLR 2023
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory F. Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang. and Yanqi Zhou. Deep learning scaling is predictable, empirically. CoRR, abs/ 2017. URL
abs/ 011.12..000000
Xin Huang, Ashish Khetan, Milan Cvitkovic, and Zohar S. Karnin. Tabtransformer: Tabular
data modeling using contextual embeddings. CoRR, abs/2012.06678, 2020. URL https
/ www.iv..rggaas///...
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.03167, 2015. URL
org/
abs/1502 031677
Yohan Jo, Keith Maki, and Gaurav Tomar. Time series analysis of clickstream logs from online courses. CoRR, abs/1809.04477. 2018. UKLhttps///rrawv..orr/////////
04177
Kaggle. Decmeg2014 decoding the human brain - predict visual stimuli from meg record-
ings of human brain activity, 2017. URL httpt::tt////tt http://wwwwggglll... www.:kaggllll.....o... wwt.co//mmmmmmoiionn.....
Juha Karhunen, Tapani Raiko, and KyungHyun Cho. Chapter 7 unsupervised deep learn- ing: A short review. In Ella Bingham, Samuel Kaski, Jorma Laaksonen, and Jouko Lampinen (eds.), Advances in Independent Component Analysis and Learning Machines, pp. 125-142. Academic Press, 2015. ISBN 978-0-12-802806-33 B978-0- 8978-0-12-800280-330007777 atticcee//11///77112228888888883
doi: https://doi.org/10.100 URL https:: www.nceceeediiii.c.c....c...... com..//cccccccc....oococ.c.cccc cow/sciencc/////cccccccccm
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Wein- berger (eds.), Advances Neural Information Processing Systems, volume 25, Curran Asso- ciates, Inc., 2012. URL httpt::t//////t /ttp:///ccccceeeeeeeeeeeee www.:ediiinnnnnnnnnneeeomee http:///w...nnoooos.ooo///ee//// pcpps//aaapee//.2//...e..e c399862d3b9d6bb (bteee66666666636226666666653560
pdf
James Large, Anthony J. Bagnall, Simon Malinowski, and Romain Tavenard. From BOP to BOSS and beyond: Time series classification with dictionary based classifiers. CoRR, abs/1809.0675)
2018. URL
/ttl://pp//rrrx..oorrr/a/////// /1809 URtpttp:://sxxiv org.ass
06751
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. CoRR, 0e//1907000235 2019. htt.:ttp://axxxiv.o
00235
Yang Li, Nan Du, and Samy Bengio. Time-dependent representation for neural event sequence prediction. 2018.
Zewen Li, Fan Liu, Wenjie Yang, Shouheng Peng, and Jun Zhou. A survey of convolutional neural networks: Analysis, applications, and prospects. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-21, 2021. ISSN 2162-2388. doi: 10.11099NNNLS
PyTorch Lightning. PyTorch Lighining, 2021. URL lightning!
Bryan Lim and Stefan Zohren. Time- forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A: Mathematical. Physical and Engineering Sciences, 379(2194):
20200209, feb 2021. doi: 1.1188//sta.222000222 URL
2F 2020 0209
Huiwei Lin, Yunming Ye, Ka-Cheong Leung, and Bowen Zhang. A multivariate time series classifi- cation method based on self-attention. In Jeng- Sann-Shyang Pan, Jerry Chun- Wei Lin, Yongquan Liang, and Shu-Chuan Chu (eds.), Genetic and Evolutionary Computing, pp. 491 491-499, Singapore, 2020.
Springer Singapore. ISBN
Jessica Lin, Rohan Khade, and Yuan Li. Rotation invariant similarity in time series using bag-of- patterns representation. J. Intell. Inf. Syst., 39()) oct 2012. ISSN 0925-9902, doi: 10. URL /tt
12

---

Under review as a conference paper at ICLR 2023
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. CoRR, abs/2106.00554, 2021. URL httpt::/aaxiiiiiiie wwwpaa////......//////// 04554
Markus Löning, Anthony J. Bagnall, Sajaysurya Ganesh, Viktor Kazakov, Jason Lines, and Franz J. Király. sktime: A unified interface for machine learning with time series. CoRR, abs/ 1909.0787.. 2019.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference Machine Learning, ICML' 10, pp. 807-814, Madison, WI, USA, 2010. Omnipress. ISBN 9781605589077.
Toan Q. Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. CoRR, abs/1910.05895, 2019. URL http:::/aaxw..or///aaaaat 1900. 05895
NVIDIA. RAPIDSAI: Collection of Libraries for End to End GPU Data Science, 2018. URL
NVIDIA. NVTabular: GPU Accelerated ETL for Recommender Systems, 2021, URL https
//github. Ctt.u...c//VVDDD.. A--MAA--MM--er MMrriin in aar
NVIDIA. NVTabular: GPU Accelerated ETL for Recommender Systems Target Encoding,
2022. http://psss//iiiiimmeeeeeee www.inggig............ io io/NVTabu wwwaaauwaarrrrr ar/maini generated/ / nvtabular ops TargetEncoding .html
Tin Lay Nwe, Tran Huy Dat, and Bin Ma. Convolutional neural network with multi-task learning scheme for acoustic scene classification. In 2017 Asia-Paciffe Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 1347-1350, 2017. doi: 10.1109/
ASSSPA.2017.828222.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, and Alexander Ku. Image transformer. CoRR, abs/1802.05751, 2018. URL http:://axxw..rgg/aabb...
05751
Kevin Patel and Pushpak Bhattacharyya. Towards lower bounds on number of dimensions for word embeddings. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume Short Papers). pp. 31 31-36, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing. URL htwph:tttt...m fttttps://aclantholooyy. org
Uber Petastorm. Introducing Petastorm: Uber ATG's Data Access Library for Deep Learning, 2021. htt.hu..comm//ii hww.bb/pwpptttttttoreoeo
Mary Phuong and Marcus Hutter. Formal algorithms for transformers. 2022. doi: 10.48550/ARXIV. 2207.09238. URL.tttpps.//aaxx..orrr///aaabba
09238
Ploomber. Ploomber- we develop tools to streamline the development to production data science workflow. Ploomber, 2021. URL http:ttp.://ggihbb..mm//o/oooott
Alec Radford, Luke Metz, and Soumith Chintala. resupperrisedd learning with deep convolutional generative adversarial networks, 2015. URI/nttpe::/axxxv.orrgg
06434
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language Models are Unsupervised Multitask Learners. 2019. URL
better--angiagg-mmdeel
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers like convolutional neural networks? CoRR, abs/2108.088 000100,0888......000.00000000111 2021. URL
https::/arxizs wwg. 2210 2
Prajit Ramachandran, Niki Panmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alon attenntoo in vision models. CoRR, abb//190.05599, 2019. URL http?
7/ raxiiv.commm org/abs 906. 05909
13

---

Under review as a conference paper at ICLR 2023
Sebastian Raschka, Joshua Patterson, and Corey Nolet. Machine learning in python: Main develop- ments and technology trends in data science, machine learning, and artificial intelligence. CoRR, abs/2002.04803,, 2020. URL https: pattpss:/arxiv..rrt abs 2002. 04803
Pau Rodríguez, Miguel A. Bautista, Jordi Gonzålez, and Sergio Escalera. Beyond one-hot encoding: Lower dimensional target embedding. Image and Vision Computing, 75:21-31, 2018. ISSN 0262-8856. doi: https://doi.org/ 1ttp:/00111.1......
1tt.:////....111...i.iiiiii.aa./ imavis 2018 0..00004. URL https
www. scweecedieddiii rect. com wtt.tttww.....c..eeeeeeeeeeeete http://www..............o..aao/a
Alejandro Pasos Ruiz, Michael Flymn, James Large, Matthew Middlchurst, and Anthony Bagnall. The great multivariate time series classification bake off: A review and experimental evalu- ation of recent algorithmic advances. Data Min. Knowl. Discov,, 35(2):401 mar 2021. ISSN 1384-5810. doi: 10. s10618 020-0027-
URL /RLtttp:::/ool.orr///000
Marc Rußwurm, Sébastien Lefévre, and Marco Körnr. Breizhcrops: A satellite time series dataset for crop type identification. CoRR, abs/1905.1 1893, 2019. URL http::://axxivooo/ abs/
1905. 11893
Patrick Schäfer and Ulf Leser. Fast and accurate time series classification with WEASEL CoRR, 1bs/1701.07681, 2017. http:://axx.v..ogg/a//////
José F. Torres, Dalil Hadjout, Abderrazak Sebaa, Francisco Martinez-Alvarez, and Alicia Troncoso. Deep learning for time series forecasting: A survey. Big Data, 9(1):3-2,, 2021. doi: 10. 1089/big. 2020.0159.
/ttp::/:///o..oo0000.10000000 2tt.:///iiii............ 2020. 0159 PMID: 33275484.
Asher Trockman and J Zico Kolter. Patches are you need?, 2022. URL httpps://penreeieew
net fort//orum?11 dew.ddd=vvsSSddddw
Shreshth Tuli, Giuliano Casale, and Nicholas R. Jennings. Tranad: Deep transformer networks for anomaly detection in multivariate time series data. CoRR, abs/2201.07284, 2022. URL
https://arxiin wttp://////.....////// 2201.07284
abs/
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, 7bb.1776.,,762,, 2017. URL
http://arxiv.org/330
03762
Yuqing Wan and Yain-Whar Si. A formal approach to chart patterns classification in financial time series. Information Sciences, 411:151-17: 2017. ISSN 0020-0255. doi: 10.1016/j.ims 2017.05.028.
SDC2002551631636X
Xiaolong Wang, Ross B. Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. CoRR, abs/1711.07971, 2017. / 1771l 07971
Zhiguang Wang, Weizhong Yan, and Tim Oates. Time series classification from scratch with deep neural networks: A strong baseline. CoRR, 1111111116455... 2016. Utt.http://aaxxx
abs/1611. 06455
Qingsong Wen, Liang Sun, Fan Yang, Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. Time series data augmentation for deep learning: A survey. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 4653-4660.
International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 10.24963/ 2021/ Survey Track.
10c/i.20 URL /RLLattpp///o....r/11.....
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen. Ziging Ma, Junchi Yan, and Liang Sun. Trans- formers time series: survey, 2022.
07125
Ruibin Xiong, Yunchang Yang. Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie- Yan Liu. On layer normalization in the transformer architecture. CoRR, abs/2002.04745.. 2020. URL hRI.https:
2200 04745
14

---

Under review as a conference paper at ICLR 2023
Jiehui Xu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Anomaly transformer: Time series anomaly detection with association discrepancy. CoRR, abs/2110.02642,, 2021. URL https
//arxiv. abs 21100 02642
Chao-Han Huck Yang, Yun-Yun Tsai, and Pin-Yu Chen. Voice2series: Reprogramming acoustic
models for time series classification. CoRR, abs/2106.09296, 2021. URL httthhtt///aaa/
org/abs/ 2106. 09296
Zhaosheng Yang, Qichun Bing, Ciyun Lin, Nan Yang, and Duo Mei. Research on Short-Term Traffic Flow Prediction Method Based on Similarity Search of Time Series. Mathematical Problems in Engineering, 2014:1-8, August 2014. doi: 10.1155/2014/184533 URL httthhttt//idee
sepec..og///hiiee http///nnnnnmmm////.eeaaa 184632.. 1tmml..mtml
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eick- hoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery amp; Data Mining, KDD '21, pp. 21 2124, New York, NY, USA. 2021. Association for Computing Machinery. doi: 10.1145/3445548.447777
ISBN 4503833333
/tts:ts:::///// //doi org/10.1145/
3447548 0.1.8.467740 34674011
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence sene-series forecasting, CoRR, abs/2012.07436, 2020. URL httpp::/arxii..ooo wttp:////.bbaaa....///// 0743311220670
Tian Zhou, Ziqing Ma. Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. CoRR, abs/2201. 112240, 2022.
URL httttt:///aaiittii org. /tt..//22...1000
Xiaowu Zou, Zidong Wang, Qi Li, and Weiguo Sheng. Integration of residual network and convolu- tional neural network along with various activation functions and global pooling for time series classification. Neurucomputiing, 367:39-45 45, 2019. ISSN 0925-2312. doi: http://doi.org/10.0016/
neucom. 2ttcom.2019.08.022. URL http::ttt//.o www. scwencedi recte com science/article/
pii/ D21/8092523121931166
15

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 EXPERIMENT DETAILS
Amazon EC2 Deep Learning GPU Machine
Processing using
Model Training Inference
NVIDIA RAPIDS and NVTABLUAR libraries
Model Training
Feature Engineering
Pytorch
Pytorch Lightning
Analysis
$3
Transformation
Filtering
Petestorm
Training
CSW/TVV
Imputation
Data
Model
to Parquet
Feature Creation
MultiGPUU
Data loader
6
Feature Perprocessing
Multi-PUU Data Sharding
Model Inferance
Continous Normaliztiin,
Load Model
DatawareHouse Tables
Categorical features: Encode
100% Inference
Post procossing
Pro shufle and store
to Time-sorien data transform
Figure 3: VitMTSC deep learning training and inference pipeline
A.1.1 TOOLS
To efficiently process terabytes (TB) of commercial MTSC data, we use Graphical Processing Units (GPUs) for end-to enddto-end to accelerated data processing as well model training/inference (Raschka) al. 2020). We use combination of open-source libraries: NVIDIA RAPIDS and NVTabu- lar (NVIDIA 2018 2021) GPU accelerated libraries for data-processing and feature engineering: b) Uber Petastorm (Petastorm 2021) based llate-s for multi- gultii-puuee trainin/iinferenee Pytorch- Lightning (Lightning 2021 for model training//mference; d) nbdev (fast.ai 2020) to enable code sharing across Jupyter notebooks; and Conda (Conda 2020) and Ploomber (Ploomber) 2021) for workflow automation. With help of these tools, we are able to write end-to-end ML pipeline in Jupyter and test it locally deploy it to Cloud. To reproduce experiment results open-source UEA datasets, we provide the fully self-contained code in supplemental materia
A.1.2 ENVIRONMENT
We run all experiments locally on Amazon Elastic Compute Cloud (EC2) deployed full workflow remotely on Amazon Web Service (AWS) Batch service on 94dn.4xlarge machines. All machines have AWS Deep Learning Amazon Machine Images (DLAMI) and Ubuntu operating system. The p3dn.22 24xlare machine has 96 CPUs, 768GB RAM and 256GB of total GPU memory divided across eight GPU cores. Similarly, Axlaree machine has 16 CPUs, 64GB RAM and 16GB of total GPU memory on a single GPU core.
machine or
A.1.3 TECHNIQUES
The open-source UEA datasets (refer Table used a single NVIDIA Tesla V100 GPU core whereas commercial datasets (refer Table used all eight NVIDIA Tesla V100 GPU cores for both training and inference. We use the Extract step Extract-Transie Transform-Load (ETL) to pull data from Dataware House (DW) to Amazon Simple Storage Service (Amazon S3) and use GPU workflow (Figure 3 for the remaining steps of the ML modeling.
STo facilitate further research, release code pttpss:/ggtthen wttp:h/..coottttteeeeeeeeeeeeeee otteeeereooo/oooooo////oe/ooeeer
vitmtac
16

---

Under review as a conference paper at ICLR 2023
350
Figure 4: Top 3 products all business-relate bulk purchases identified by VitMTSC model indicate that the user will be responsive to a marketing campaign targeted towards small businesses.
keep data in tabular format until last stage of Step for data-processing and feature engineering. And then we convert tabular data to time-series format, such that, all data related a single time-series are single row in parquet file. This helps Uber Petastorm (Petas torm. 2021) based data-loaders to load all data for time-series in one read during multi-guu trainingginference.
As shown in Figure 3
A.2 INSPECTING VITMTSC
Transformer self-attention layers assign an attention value between each pair of tokens. In NLP, a token is typically a word a word-part. For images, each token represents image patch, while in the VitMTSC model, each token represents single MTSC datapoint in the time series. Each token is linked the [class] token via the self-attention mechanism. The strength of the attention link can intuitively indicate each token's 's contribution to the classification. However, the attention values only represent a single facet of the Transformer network. Additionally, non-positive activation functions and skip connections impede accurate interpretation. Chefer et al. (2020) proposed attributing relevance to features in Transformer models by introducing a relevancy propagation rule that is applicable for both positive and negative transformations, normalizing relevance for non- non-parametric layers such as "add" (e.g., skip- connectio)))) and matrix multiplication, integrating the attention and relevancy scores, and combining the integrated results for multiple attention blocks.
We adapted the VIT implementation of the Chefer al. (2020) to VitMTSC and demonstrated interpretable model outcomes. We focus on the k time-series datapoints that contributed the most to the prediction. This method identifies the top time-series data points in sample that contribute to specific classification. Figure 4 for instance, depicts the top three product purchases that were significant for classifying whether a user would be receptive to a small business-orient oriented marketing campaign. Note that each of the three purchases consists of bulk business-related products.
A.3 UEA DATASETS
The UEA MTSC archive is repository of benchmark datasets for time-series classification (Bagnall et al. 2018). It includes 30 multivariate datasets, four of which have variable lengths. We test our model on the following five datasets (Table available in the UEA repository, which include three datasets of variable length, condition frequently observed in real-world data.
A.3.1 PENDIGITS
UCI Archive dataset (Dua & Grafi 2017 Alimoglu 1996). This is handwritten digit classification task. 44 writers were asked to draw the digits [0..9], where instances are made up of the x and y coordinates of the pen traced across a digital screen. The data was originally recording these coordinates at a 500x500 pixel resolution, however was normalised and sampled to 100x 100. Then, on expert knowledge from the dataset's authors, the data was spatially resampled such that instead of each consecutive attribute having constant time step (100ms) but variable spatial step, they instead had constant spatial step and variable time step. From experimentation by the authors, the data was resampled to 8 spatial points, such that each instance is dimenions of 8 points, with a single class label [0...9] being the digit drawn.
A.3.2 SPOKENARABICDIGITN
This data set is taken from the UCI repository (Dua & Grafi 2017 Hammami & Sellam 2009). It is derived from sound. Dataset from 8800 (10 digits x 10 repetitions x 88 speakers) time series of 13 Frequency Cepstral Coefficients (MFCCs) had taken from 44 males and 44 females Arabic native
17

---

Under review as a conference paper at ICLR 2023
speakers between the ages 18 and 40 to represent ten spoken Arabic digit. Each line the data base represents 13 MFCCs coefficients in the increasing order separated by spaces. This corresponds to one analysis frame.
A.3.3 CHARACTERTRAJECTONN
From the UCI archive (Dua & Grafi, 2017). The characters here were used for a PhD study on primitive extraction using HMM based models. The data consists of 2858 character samples. The data was captured using a WACOM tablet. Dimensions were kept x, y, and pen tip force. The data has been numerically differentiated and Gaussian smoothed, with sigma value of 2. Data was captured at 200Hz. The data was normalised. Only characters with a single 'PEN-DOWN' segment considered. Character segmentation was performed using pen tip force cut-off point. The characters have also been shifted so that their velocity profiles best match the mean of the set. Each instance is 3-dimensional pen tip velocity trajectory. The original data has different length cases. The class label is one of 20 characters 'a', "b', 'c', 'd', 'e", 'g', 'h', 'I', 'm', 'n', 'o', "p", 'q', 'r', 's', 'u', 'v', 'w', 'y', 'z'. To conform with the repository, we have truncated all series the length of the shortest, which is 182, which will no doubt make classification harder.
A.3.4 FACEDETECTION
This data comes from Kaggle competition in 2014 (Kaggle 2017). The problem is to determine whether the subject has been shown the picture of face scrambled image based on MEG, independent of the subject. This data set is just the training data from the competition, split by patient into 10 train subjects (subject01 to subject10) and 6 test subjects (subjectl) to 16). This data cannot sensibly be randomly resampled. A leave one patient out approach is the most suitable. There are about 580-590 trials per patient, giving 5890 train trials and 3524 test subjects. Each trial consists of 1.5 seconds of MEG recording (starting 0.5sec before the stimulus starts) and the related class label, Face (class 1) Scramble (class 0). The data were down-sampled to 250Hz and high-pass filtered at IHz, giving 62 observations per channel. 306 timeseries were recorded, one for each of the 306 channels, for each trial.
A.3.5 ASSECWINGGBES
The WingbeeW data was generated by the UCR computational entomology group (Chen & Adena Why 2017) and used in the paper "Flying Insect Classification with Inexpensive Sensors. Journal of Insect Behaviour 2014" (Chen et al., 2014). The original data is reconstruction of the sound of insects passing through sensor. The data in the archive is the power spectrum of the sound. The 10 classes are male and female mosquitos (Ae. aegypti, Cx. tarsalis, Cx. quinquefasciantss Cx. stigmatosoma), two types of flies (Musca domestica and Drosophila simulans) and other insects. spectorgram of each second sound segment was created with window length of 0.061 seconds and an overlap of 70%. Each instance in this multivariate dataset is arranged such that each dimension is frequency band from the spectrogram. Each of the 10 classes in this dataset consist of 5,000 instances.
A.4 PYTORCH/PYTORCH LIGHTNING CODE FOR VITMTSC
A.4.1 IMPORTS
import torch
import fiforrhhliggtning as pl from torch import nn from torch. nn import functional as F from torchmetrics import functional as FM from petastorm import make_batch_readee from petastorm.
pytorch pystorch from einops import rearrange, repeat
import DataLoader
A.4.2 RESIDUAL CONNECTION
class Residual (nn Module)
def iiit _init__n________________________ iself, fn) super () ()__initt.... init
self. fn fn
18

---

Under review as a conference paper at ICLR 2023
def forward **kwargs) return self. fn (x, **kwargs) + x
A.4.3 PRENORM: LAYER-NORMALZATION
class PreNorm (nn. Module)
def __ini__(sell (self, dim, fn)
super (). (). init__)
self.norm = nn. Lww.rayyrNooocoo (dim)
self.fn fn
def forward x, **kwargs); return self. fn (self.nomm (x), * **kwargs)
A.4.4 FEED FORWARD NETWORK SUB-LAYER
class FeedForward (nn Module) def
(self, dim, hidden_dim, dropout=0,00 : super () init_()
self.net - mn. Sequential
nn.Linear (dim, sidden_dim) nn. GELUELU))
nn. nn.Dropout (dropout), nn Linear (hidden_dise dim, dim), nn.Dropout (dropout),
def forward (self, return self. net (x)
A.4.5 MULTI-HEAD ATTENTION SUB-LAYER
class Attention (on.oonnnnno.nou.n.. Module)
def _niit___eee____ (self, dim, heads=10, dim head=32, dropout=0.0):
super ()._(()_nn))) inii__n))))))
inner_dim dim_head heads self. heads heads
self.scale = dim head ** -0.5
self.to_gkv = nn. Lineir (dim, inner_dim 3, bias-False)
self.to_out nn. mn.equuntial (nn, Linear (inner_dim, dim), nn. Dropott (dropout))
def forward
Sogister_hook-Falsee
b, n, _ h *x. shape, self, qkv self.to_oke (x) (3unk dim=-1)
k, v = map (lambda t: rearrange "b n (h d) -> bbr d", h=h), gkv)
dots frrcheeinsum ( (bbhid, bbjd->bhij", a, k)
Lask_valuss = -torch. finfo (dots. dtype) max
if mask is not None:
mask F.pad (latt. mask mask. unsquezze (1) .unsqueeze (2)
flatten (1), (1, 0), value=True)
assert mask. shapelh dots Nots.shape [-1], "mask has 4 incorrect dimensions" misssmmakke__F11 (mask == 0.0, mask_val lue)
19

---

Under review as a conference paper at ICLR 2023
del mask
attn softssa (dim=-1)
out = sarch.einsum ("bhij, bhjd->bhid", attn, v)
out out rearrange (out, "b h d -> b (h d) ")
(out)
return out
A.4.6 TRANSFORMER ENCODER
class Transformer (nn Module)
def _eeltt_seeef, dim, depth, heads, dim dim_head, dim,
4 dropout): super iniit_
self.layezs nn Mod::leList (()) for in range (depth)
self layers aaperd (nn. Modulelisst
Residual (PreNorm (dim, Attention (dim, heads
- heads, dim_head dim_head, dropout ))
Residual (PreNorm (dim, FeedForward mlp_dim, dim, dropout dropout)))
11)
def (selfd x, mask None, register_hook False) for attn, if in self.layers:
x attn (x, mask mask, register_hook register_hook) = (x)
return
A.4.7 VITMTSC MODEL.
class VitMTSCModel (p1. LightningModuue)
def -init _init_ntseeee (self, config, c_in NUMBER_OT_FEATURES, clout
NUM_TARGET,
seq_len SEQUENCE LENDENNEELENNTH,.OOOO class weight
torch. F Tensor (CLASS_WEIGHT] )
super (VitMTSCModel, . init_ ()
self.d_model configl"d_modell
self.depth config["depth"]]
self.heads = config["heads")
self.mip_dim config["nip_din"]
self.din_hhed = config["din_head"]
self.dropout_p config["dropout"]
selff.emb_dropout__ config(""mb_
self.1r config["1r"]
self self.weight_dedee config ("weight_decayy
self.patience configl"patiences
silf.pos_emeddding nn. Parameee (torch. randn (1, leq_ +
1, self. self.d_model))
self.patch_to. tt__tt_______e__ to_embedding = nn. Lineir (c_in, self. self.d_mode modeel)
nn Parameter (torch. randn (1, 1, 4 self.d melf.d_model)
sel f nn. Dropout (self emb emb_dropout_p)
20

---

def validation_stedt (self, batch, batch_ide idx) 8, Yr mask batch Under review as a conference paper at ICLR 2023
helf.transforeee Transformer (self. (self. model, self.
- self.heads, self.dim head, self.mlp_dee _im, self.dropout_p)
self.to_cll__toen = mn. Identityy ()
self.mlp_head = Sequentiia
nn Ln.LayeNNorm d
Ln.ein seeef.d_modde nn.GELU(,
self self.mlp_dee mip_dim),
nn.Dropout (self. dropout_p),
Lnneir (self weseef.mlp_diiiie c_out)
self.c_out c_out www.rreggieerrufeeeeeeee cclsss_weight', class_weight)
def forward x, mask None, register_hook book False)
x = helf:ppacct_tooemmediiii (x) # bs len x nvars -> bs seg x d_mode
b, n, - shape $ bs, seq ien
cls _os_to = repeat lewelf.cls_toooeeeeeeeee '() n d -> b d', b =
- b) bs x 1 x d_model
x =
cat ((cls_token))) x), dim=1) bs x (seg len 1) + x d_odel
+= self. Los_eeeeddding in + 1]] += 1 (seg len +
4 1) x d_model -> # bs x (seq len 1) x m_ded
self.dropout (x) bs (seg len
1) d model
trafftrrasformer (x, mask mask, register_hook register_hook) # bs x (seq len 1) x m_ddel
x self.to_cls_teeeeeeeeeeee tot.ttosssttttt(iiitt 01) # bs modeel return self.mlp_head(x) bs x num classes
def confiigree.ggtiinzzersstsllll
optimizer = torch.optin wwpppti..aaamm....mmmm..m AdamW (self. (self..aramettess.. (),
- 1r=self.lr, weightt_decay=selF wtwighh....ttttteedeaa...
scheduler
+ patience=self, patiencea
return ("optimizer"; optimizer, "lr_scheduler"; scheduler,
+ "monitor": "train_loss")
def training_step (self, batch, batch_ y, - mask = batch
y_hat self (x, mask)
y (
train_loss F.cross_entrrppy (y_hat, y, weight
train_auc accuracy (F (o.softmax (y hat, dim=l), Y,
num num__clasess self.c_our
selff_oog('taann_ioss, train_loss, on_stee-False,
on_epoch=True, prog_bar-True, logger-True)
train_auc, on_step=False,
on_epoch-True, prog_bar=True, logger=True) return trrii_llsss
21

---

Under review as a conference paper at ICLR 2023
y_hat self (x, mask)
y = y.long() ()
val vas_ loss F.cross entropy hat, y, weight
y
4 welffclass_weige...
val_auc FM. (Fo.oaccaaacsss (y_ (y_hat, dim=l), Yr
num_ciasses self.c_out:
self.log('val_lcs val_loss, on_step-False,
on_epoch-True, prog_ Srog_bar=True, logger=True,
sync_dist-True
self. self.log('val_auc) val_auc, onlstep-False, on_epoch-True, 4 prou_bar=True, logger=True) return va val_loss
def test_ (self, batch, batch_idx)) x, y, _ mask = batch
y_hat self (x, mask)
= y.long() ()
test_loss F.cross_entropy (y_hat, y, weight
colffclass..m (alaas_weight)
test_
aM. (F softmax hat, dim-l), yr
num classes = self.c_out)
test_loss, on_step=False,
on_epoch=True, prog_bar=True, logger-True,
+ sync_dist-Trus
test_auc, on_step-False,
4 on_epoch=True, prog_bar-True logger=True) return test_loss
A.4.8 VITMTSC: PETASTORM DATAMODULE
def petastorm_collatee (rows) (rows):
data pd. Dataaaaaaaraeeemm (rows)
_ase_id_de data df. iloc[:,
NUMBER OF FEATURES SEQUENCE LENGTH-1
:NUMBER_OP OF_FFAFURRES _FFFAATTRSSSSSOOERRELLLLECCCE
@ase_id_tensor
torch.tenser (case_id id values. aatpe (np. float64)
target_de df data df. iloci;
NUMBER_OR OF FEATURES
LENGTH-0
target_tensor
torch.tenson (target_dd df va ues asteepesstppeeee. (np. float322)
tata_tensor_df - df
df ilo. 0 OEEUUUENCE LENGTE
tensor
corch.tensee
waluorrsddvvvluuu..oommm values. www..s......e.sss.ssp.....e (ip.fll))32)
data_tensor rearrange (data_tensor, 't (b h) b',
DEQUENCE_LEEGTTH
mask df i L-SEQUENCE_LENNE
[1=SEQUENCE_LEKCE
mask_tensor torch. tensor (mask_df.valuus ast (np. float32))
return data_tensor, cas__i@_tenso..squeez mask_tensor s wtw.teeeoorrrsueeeeeeeeeeee squeeze ()
(),
class V (pl. http://pe.....ttl.o........aaaaa
def init__
22

---

Under review as a conference paper at ICLR 2023
self, config, train files, valid_files,
test_files,
Mun_workers-NNM_WW
cransform_spec-None,
shard_count-NNUMGGGN
nmn_spchss-AX__PPCCH,,
(mperr))_iiit__)
sellftraiinfiles = train_files
self.valid_files valid_files
self.test_files test_files
self.batch_size = confiig"batch_size")
self.num_workers num_workers
self.transform_spe transform_spec
self.shard_count shard_count
self self.num_epocha. = num_epochs
def (2ai_Gataloaderrsee) (self):
self.traing = make_batch_reader (
self.train_fiiest
cur
("LOCAL RANK"]), ["NOCAL
shardd_conttssllf.aaar__oooomm num_epochs=sllff..unneppooo.
return DataLoader (ssf..rrih.ss
fn)
def val val sel_datalooderrsee) (self))
print (
wal__aataloddrr: local rank :",
int (os. aeviironn"LOoa RANK"]), "shard count: ", self. self.shard_count,
make_batch_reader self.valid_filee workers_ sonk:rss_conttsoll..nunwooooooe
transfoom_pporsselff.tttfffor spec,
cur_
RANK"))
sherdd_couttseellF..srr...oomo nmm.epchhs=sell..unneppcott
return [sta sie oollatee.fappctsttommmccolleeee
batch_size=sel bat.hfffattchsiieeeeee
def test
print
"test_dataloader: local rank :" int www.eeeiiionnnoooo RANK"]),, "shard count: " self.shard_coonf
23

---

Under review as a conference paper at ICLR 2023
ds make
(
self. self.test_fiies, sunkerr__oonttssei..n.m......e w
workers_counee
sharrdLoontt-sefff.heee..o.tttt mmm.epochs==elff.nmmeppott
return Dataloader (seff
batch_size=sel..
+ collatte_fmpptatttmmooollllt
24


Reviews:
[{"review_id": "qYlL1mk2n-J", "reviewer": "Reviewer_fdnF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed the `VitMTSC` model, a vision Transformer model that learns latent features from raw time-series data for classification tasks, and could be applied on large-scale time series data with variable lengths. The model can reach comparable performance on UEA datasets against previous state-of-the-art methods and outperform them by a large margin on real-world dataset.\n\nThe contributions of this paper are from proposing a VitTransformer based approach to model long-range contextual relationship in MTS data, and demonstrate its superior performance on real-world application data.\n", "summary_of_the_review": "To summarize, this paper introduced a Vision Transformer based model called `VitMTSC` to improve multivariate time series classification. However, the novelty in this paper seems to be limited and the results have a huge gap between open-source UEA dataset and real-world application dataset. Even though the performance on real-world dataset looks great and the model can handle large scale dataset, the engineering part seems out of the scope of this paper and there's a lot more to explain on the experiment results. Therefore, I recommend this paper to be rejected.\n\n", "strength_and_weaknesses": "This paper provides an interesting idea of using a Vision Transformer for a Multivariate Time-Series Classification problem. The strengths of the paper are:\n1. The figures are well-made and improve the clarity of the model description.\n2. Details on tokenization, feature encoding, padding and truncation are discussed in the paper, improving the paper's reproducibility.\n3. This paper did some interesting ablation study on layer normalization that change its position from after `MSA/FNN block + Residual connection` which is also called `POSTNORM` to before `MSA/FFN block` which is also called `PRENORM`. Experiments show it can improve applications' performance greatly on real-world dataset.\n4. The experiments section includes some real world dataset, including offline and online tests, which is interesting to see.\n\nOn the other hand, the weaknesses of the paper are:\n\n1. Even though the author(s) claimed this paper is based on Vision Transformer, there's no concept of \"patch\" applied in this feature extraction stage. The input was naturally given in different time stamps and features were extracted in a one-by-one fashion. After tokenization and feature encoding, there's only a linear projection applied onto each of the feature vectors. I tend to consider this process as some kind of feature engineering for the input of a Transformer. Can you please further explain how this is related to vision transformer's patch concept and whether the novelty comes from this process?\n2. Covariates (time features in this paper) like month, day, hour have proven to be very useful in time series, while using them as position encoding could help, have you tried combining it into the original feature vector before linear projection and combine with learnable position encoding?\n3. In the UEA dataset, the performance of the proposed model looks a bit worse than state-of-the-art models, but performance a lot better then these models on real world dataset. There seems to be not enough case study and analysis to further understand where the difference comes from and how we can mitigate the gap. I suggest the author(s) to provide more insights on the cause of this difference.\n4. The bolded numbers in tables are a bit confusing, for example in Table 3, VitMTSC's validation set results were bolded but other results are from the test set. In Table 5, some of the results in the same ablation study groups are bolded but some are not, e.g., the `padding mask` group where the dataset is `Large`. Can you please add some explanation on the bolded number meaning?\n5. The author(s) claimed to have `experimented with different aggregation strategies` but there were no conclusions or results mentioned. Can you please show the conclusion or performance with different aggregation strategies' impact?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XNcDlaS13o", "reviewer": "Reviewer_YY3h", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the application of ViT models for the task of timeseries classification. ", "summary_of_the_review": "It's unclear whether this model improves against other sota models in the literature based on Table 3. It seems that baselines perform equally well or even better here. The proposed model seems to perform better only on the internal dataset (private?) which makes the comparison with public benchmarks really difficult. I would urge the authors to expand more on the public benchmark experimentation and discuss (and expand) Table 3.", "strength_and_weaknesses": "S\n\n- extensive experiments with multiple datasets and setups\n- straightforward application of ViT to timeseries\n\nW\n\n- lack of discussion wrt better results in baselines (TST, Rocket)\n- limited novelty compared to existing models (ViT)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MJ-kOuPF1H", "reviewer": "Reviewer_ivET", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper applies VisionTransformer for Multivariate Time-Series Classification (VitMTSC) model that learns latent features from raw time-series data for classification tasks and is applicable to large-scale time-series data with millions of data samples of variable lengths. Their results show that it's can obtain near state-of-the-art results on public benchmark and obtain significant improvements on real-world commercial datasets.", "summary_of_the_review": "This paper applies vision transformer in Multivariate Time-Series Classification problems. Results show that it can outperform state-of-the-art on commercial datasets while slightly left behind on public benchmark datasets. It's unclear why this performance gap exist, making this paper less valuable. ", "strength_and_weaknesses": "Strength\n* The idea is quite simple and cast time series as an \"Image\" and apply ViT to time series classification tasks.\n* Results on real-word commercial datasets are significantly better than baselines.\n\nWeaknesses\n* The idea is hardly novel and it's more like a pure application of Vision Transformer without any deep insight.\n* It's unclear why VitMTSC is significantly better than state-of-the-art baselines on commercial datasets. Authors don't discuss these insights in their papers and it's unclear when practitioners should utilize vision transformer for time series classification.\n* The accuracy is pretty high on PenDigits, SpokenArabicDigits and CharacterTrajectories. Is it due to data imbalance? Could authors provide other metric, e.g. F1?\n* This paper mainly explores binary classifications and multiple-class classification, however, is not fully considered. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject