NuTime: Numerically Multi-Scaled Embedding for Large- Scale Time-Series Pretraining
Authors: Chenguo Lin
Xumeng Wen
Wei Cao
Congrui Huang
Jiang Bian
Stephen Lin
Zhirong Wu
Venue: Accepted by TMLR ()

Abstract: Recent research on time-series self-supervised models shows great promise in learning semantic representations. However, it has been limited to small-scale datasets, e.g., thousands of temporal sequences. In this work, we make key technical contributions that are tailored
to the numerical properties of time-series data and allow the model to scale to large datasets, e.g., millions of temporal sequences. We adopt the Transformer architecture by first partitioning the input into non-overlapping windows. Each window is then characterized by its
normalized shape and two scalar values denoting the mean and standard deviation within each window. To embed scalar values that may possess arbitrary numerical amplitudes to high-dimensional vectors, we propose a numerically multi-scaled embedding module enumerating all possible numerical scales for the scalars. The model undergoes pretraining with a simple contrastive objective on a large-scale dataset over a million sequences collected by merging existing public data. We study its transfer performance on a number of
univariate and multivariate classification tasks, few shot learning, unsupervised clustering and anomaly detection benchmarks. Our method exhibits remarkable improvement against previous pretraining approaches and establishes the new state of the art, even compared with domain-specific non-learning-based methods.

Published in Transactions on Machine Learning Research (07/2024)
Nime: Numerically Multi-Scaled Embedding for Large- Scale Time-Series Pretraining
Chenguo Lin Peking University
chenguolin@stu..ku.cc edu.cn
Xumeng Wen
zumenguen&mmicrroofftoomm
Microsoft Corporation
Wei Cao Microsoft Corporation
weicao@@microsoft.comm
Congrui Huang Microsoft Corporation
comhua@@microsoft.comm
Jiang Bian Microsoft Corporation
jiang. Baang.iian@miirrooff.ccom
Stephen Lin Microsoft Corporation
stevelin.@micrrooff.comm
Zhirong Wu Microsoft Corporation
wwchirron.mmiccrooot..oom
Reviewed OpenReview:
Abstract
Recent research on time-series self-supervised models shows great promise in learning mantic representations. However, it has been limited to small-scale datasets, e.g., thou- sands of temporal sequences. In this work, make key technical contributions that are tailored to the numerical properties of time-serics data and allow the model to scale to large datasets, e.g,, millions of temporal sequences. We adopt the Transformer architec- ture by first partitioning the input into non-overlapping windows. Each window then characterized by normalized shape and two scalar values denoting the mean and stan- dard deviation within each window. To embed scalar values that may possess arbitrary numerical amplitudes in a high-dimensional space, we propose numerically multi-scaled embedding module enumerating all possible numerical scales for the scalars. The model undergoes pretraining with a simple contrastive objective on a large-scale dataset over million sequences collected by merging existing public data. We study its transfer perfor- mance on a number of univariate and multivariate classification tasks, few shot learning, unsupervised clustering and anomaly detection benchmarks. Our method exhibits remark- able improvement against previous pretraining approaches and establishes the new state of the art, even compared with domain-specific non-learning-base methods. Code is available at: httpps://githb..e com/ hhegm/ooinngn///uiiinee httppp///gtttiioooo////aaaaaaiaa
This work done during an internship Microsft.. indicates equal first. author contribution.

---

Published in Transactions on Machine Learning Research (07/2024)
(a)
(b)
10"
107
10"
107
Value
107
10"1
5240
10-4
52375
10-1
UEA LSST
10
MebournPedesstan
HAR
UEA LSST
20
10
20
Timesteps
Timesteps
Figure 1: (a) Numerical scales of three temporal sequences from three datasets differ significantly. (b) Even a single sequence may contain multiple scales of numerical variations. The zoom-in view shows the local structure of small variations. Note that sequences are shifted above the x-axis and presented in logarithric scale for better visualizations.
1 Introduction
Despite the phenomenal achievement of large-scale representation learning on various data modalities (Brown et al. 2020 Radford et al. 2021 Caron et al. 2021), research for time-series representation learning is mostly
limited to small-scale datasets without attaining generalization capabilities (Eldele et al. 2021b Yue et al,
2022 Zhang et al. 2022). Since time-scries data may cover a diverse range of domains, such as medicine, weather and Traffic, large-scale training across domains brings special challenges for transfer learning.
We notice a unique characteristic of time-series data and its representation. While RGB images are repre- sented by fixed and discretized numerical values from 0 to 255, and natural languages tokenized to fixed dictionary, time-series data exhibit numerical values that are continuous and of drastically different amplitudes. For instance, temperature usually varies from -30 to 30 degrees Celsius, but altitude is on the scale of 10x meters. The scale of numerical variations generally depends on the physical properties of the time-series data. As illustrated in Figure ! sequences from certain time-series categorics and even a single time-series sequence may exhibit structures at multiple scales due to a change in physical properties.
Deep neural networks trained with gradient descent need proper normalization for optimization to good local minimum (Toffe & Szegedy 2015 Ba et al.] 2016). However, encoding time-series data to normalized vector space is non-trivial problem. Z-score normalization is popular technique that assumes a single dominant numerical scale in the dataset. Instance normalization preprocesses tormalizatiofpppppooossss data by per-instance statistics, and it thus removes information that is critical for representalion ion learning. As a result, both of the conven- tional data encoding methods fail to effectively encode time-series data with a high variation of numerical scales (i.c., amplitudc). The dilemma between normalization for effective network optimization and high variation of data scales poses challenge for time-series representation learning, especially in the large-scale scenario.
We introduce NuTime, Transformer-based architecture for timo-scries data with novel embedding module to effectively embed data of arbitrary numerical amplitude. For a time-series sequence, we first divide into non-overlapping small windows, so that data within each window has a simple structure that can easily modeled at a single scale. A window is characterized by three factors: its mean and standard deviation (std), and the normalized shape. Embedding vectors of the three factors are combined and fed an input token general-purpose Transformer for representation learning. The normalized shape across both windows and samples are of similar numerical scales, and thus be embedded by simple lincar layer. The challenge of embedding the entire sequence reduces to the embedding of a set of means and standard deviations, whose numerical amplitudes may vary arbitrarily.
Instance normalization in this paper refers to normalizing each sample individually, which is non- parametric and different. from the process proposed Ulyanov all (2016).
2

---

Published in Transactions on Machine Learning Research (07/2024)
To encode these scalars to high-dimensional vector space, we propose Numerically Multi-scaled Embed- ding (NME) module. Since encouding through network modules may need to assume the numerical scale of inputs (loffe & Szegedy 2015 Ba et al. 2016 Wu & He 2018 our idea is to simply enumerate all possible scales for arbitrary input scalar and later fuse these emboddings across mumerical scales. We use a basic building block of linear layer followed by a layer normalization (Ba el al. 2016) to map a scalar a normalized vector. Such basic building block is sensitive the input range, which we find is controlled by multiplier on the bias term in the linear layer. We thus use paralleled building blocks with different bias multipliers set for each numerical scale. The output embeddings are aggregated by weighting mechanism to derive the final scalar embedding.
To conduct large-scale representation learning, we collect pretraining data by combining existing datasets from various sources, yielding a dataset with over one million time-series sequences. We pretrain our NuTime model using a straightforward Bootstrap Your Own Latent (BYOL) Grill al. 2020 self-supervised learn- ing objective and study the transfer performance on popular classification benchmarks. NuTime achieves the new state-of-the-arls results across univariate and multivariate time series classification lasks, and is the first time deep learning models beats traditional approaches (Middlehurst et al. 2021b). We also demon- strate that NuTime outperforms recent approaches on few-shot learning without being designed for this task. Moreover, it can easily transfer to other downstream tasks, such as clustering and anomaly detection.
In summary, this work makes three key contributions:
We propose mumerically multi-scaled embedding module for encoding scalar values in a wide range into normalized vector space, which allows time series models to scale to large datasets.
We design a general-purpose purpose Transformer- based solution for time series representation learning with each input token representing its shape embedding, mean embedding, and std embodding.
We conduct large-scale self-supervised pretraining for time-series data and demonstrate that trans- ferable reprosentations could be learned from the vast set of disparate data. It is the first time that deep learning models outperform traditional methods on the UCR and UEA classification archives.
2
Related Work
2.1 Deep Learning for Time-Series Analysis
The tasks of interest for time-series analysis include classification, anomaly detection, and forecasting. While many efforts have applied deep learning time series, a consensus has not been reached on what is the most suitable neural architecture, and in what cireumstances can deep learning oulperform a traditional approach. For classification, some of the leading methods are based on convolutional inception networks [Ismail Fawaz
ct al. 2020) and recurrent neural networks (Che et al.) 2018). For forecasting, LSTMs (Shih et al. 2019)
graph neural networks [Wu et al.] 2020), Transformers Zhou et al. 2021 Wu et al. 2021 Nie et al. 2023
and even pure MLP-based networks (Oreshkin et al. 2019) all achieve competitive performance. Our work does not deal with designing the backbone architecture for time series, but rather it focuses on encoding numerical values of different scales of variation.
2.2 Unsupervised Representation Learning for Time Series
Several studies have successfully applied unsupervised representation learning to time-series data. T- Loss (Franceschi et al. 2019 is leading effort that combines dilated causal architecture and triplet loss. TNC (Tonekaboni al. 2021 TS-TCC (Eldele et al. 2021a] and TS2Vec (Yue et al. 2022 further
incorporate dedicated learning objectives, c.g., contextual and hicrarchical losses, and handcrafted augmen-
tation functions. TST (Zerveas el al. 2021). Ti-MAE (Li al. 2023) and SimMTM (Dong el al. 2023
formulate masked modeling frameworks for time series representation learning. BTSF (Yang & Hong
2022
and TF-C (Zhang et al., (022) introduce a complementary frequency domain with consistency between the temporal and the frequency domain as the supervision signal. All of these works show that the unsupervised pretrained model can offer a substantial improvement against the fully supervised counterpart.
3

---

Published in Transactions on Machine Learning Research (07/2024)
Albeit encouraging, most of these works limit pretraining to small datasets and focus "one-to-one" scenario, i.e,, pretrain on single dataset and finetune the same or similar domain. Zhang et al. (0022) take a step further and investigate a "one-to-many" setting that finetunes an EEG-pretrained model for either hand-gesture recognition or mechanical fault prodiction. They also discuss a "many-to-one" sctting where the model pretrained on a mixture of multiple datasels and subsequently finetuned on a single pure dataset. However, the finetuning performance decreases with increasing heterogeneity of pretraining datasets. The essence of unsupervised pretraining, which is capable of taking advantage of large amounts of data, remains to be explored.
2.3 Numerical Data Normalization
Data encoding and normalization play a key role in machine learning systems. Z-score and instance normal- ization are two popular methods commonly used in time-series analysis. Z-score assumes single numerical data scale and normalizes the data according to datasct statistics. Instance normalization standardizes cach sample to zero mean and unit standard deviation, but removes part of the information recover the raw data. To address this issue, reversible instance normalization (Kim et al. 2021) is proposed to add back the mean and std statistics at the network predictions for forecasting problems. Gorishniy et al. (2022) explores embedding numerical features using pieco-wise linear and periodic functions for tabular data. Morel et al.
(002)) attempls to formulate multi-scaled invariant features along the temporal dimension. Our work is similar in proposing an effective numerical embedding for scalar values. However, the goal of this work is to enable large-scale pretraining for time-scries data.
3
NuTime
3.1
Problem Statement
We aim to conduct self-supervised pretraining for time series large-scale dataset, covering various domains with very different signal characteristics. Due to the nature of the physical properties, time-series data may have different numerical scales of variations. For example, sequences belonging to a certain category may exhibit variations on a numerical scale of 0.01, whereas those from another category may vary on a numerical scale of 10000. Variation scales may even change within a single time-series sequence. Joint training on such diverse large-scale datasets introduces new challenges.
Normalization for data preprocessing is a viable technique for mitigating the aforementioned issue. Popular normalization methods include (1) Z-score and (2) instance normalization. 7-score involves computing the mean and standard deviation statistics for the entire dataset, and normalizing each sample by subtracting the mean and dividing by the std. However, it does not address the numerical challenge for the following reasons:
Z-score assumes a single domain scale of the dataset. Samples of this scale may be properly normal- ized, while samples out the scale may be badly normalized.
During transfer learning, the target domain may not share the same statistics as those from the training dataset, and thus suffer from this domain mismatch.
Instance normalization operates by slandardizing each sample using its respective per-sample mean and standard deviation. After processing, each sample is guaranteed to have zero mean and unit standard deviation. The caveats for this method include:
Essential information about the statistics of samples is removed, which could be detrimental to representation learning.
Instance normalization assumes a single scale of variation within single sample. It will be ineffective if the sequence is long and contains multiple scales of variations.
Based on these observations, we propose our approach for modeling large-scale time-series data.
4

---

Published in Transactions on Machine Learning Research (07 (274))
MLP head
Transformer Encoder
Data Embedding for One Window
14
Window wise Data Embedding
Mcan Scalars
Concat.
FC
NME
Std Scalars
LN
Normalized
Shapes
Patchify
Raw Input
Figure 2: Architecture Overview. The proposed model first patchifies the input sequence into non- overlapping windows. Each window is represented by its normalized shape, the window mean and the window std. Embeddings for the three components are concatenated and transformed to be fed as input tokens into a Transformer encoder. Details about the numerically multi-scaled embedding (NME) for scalar values are explained in Section 3.3
3.2
Architecture Overview
To build the pretraining model for Lime-series analysis, exploit the general-purpose Transformer Vaswani et al. 2017 Nie et al., 2023). as it has been successfully adopted in natural language, vision and time-series. The idea is to convert the input sequence into a set of tokens and then feed the tokens into the Transformer architecture An extra [CLS] token is inserted at position in the input, and an MLP head is appended after the Transformer encoder for classification. In the following, we will assume the time-series data to be univariate for simplicity. The extension to the multivariate case is explained in Section 3.4
The overall architecture for NuTime is depicted in Figure 2 Each time-series sequence is split into non- overlapping windows in a similar fashion to non-overlapping patches used in Vision Transformers Dosovitskiy et al. 20200. Given the numerical challenges described earlier, it is not feasible to embed each window using a simple linear layer. Instead, we may assume that data within each window has a single scale of variation given the window size is sufficiently small. We normalize each window by its mean and std, then represent the window by three factors: the normalized shape, the mcan scalar, and the std scalar. We concatenate the normalized shape embedding and the two scalar embeddings, and further project the concatenated one to the feature dimension of the Transformer. The resultant embedding is treated being added with a positional encoding, it is fed to the Transformer encoder.
imput token. After
The normalized shape can be easily embedded with a linear layer and a layer normalization. However, embedding scalar values of unknown scales of variations is less obvious.
3.3
Numerically Multi-scaled Embedding
A Case Study of Linear + LayerNorm for Encoding Scalars We begin the study by analyzing and understanding the encoding behavior of a simple linear layer followed by layer normalization (LayerNorm) (Ba)
et al. 2016 LayerNorm is crucial here as the linear layer maintains the magnitude of the input scalar, which would cause unstable optimization for neural network training. Denoting the input scalar as z, we can express this simple encoding block follows:
z = FC(x)= = x w wowkk k b
(k = 1 by default), E((z]
(1)
2 w
= LN = Var(z)
+ =
Vo + Kog + 3,
(2)
where w and b are parameters of the linear layer, randomly initialized from the Gaussian distribution NN(0, o2) and (0, ag) respectively. 7 and B are learnable affine parameters for the layer normalization
5

---

Published in Transactions on Machine Learning Research (07 (07/2024)
random weights
pretrained
C-001
100
(c)
Numerically Multi-scaled Embedding (NME)
Layer Norm
Wi
k
b
Zi
Yi
(x(2)
Layer Norm
(z(xx)
+
kg
b2
Z2
Y2
e(x)
input scalar
scalar value embedding
axxx)
Layer
Wn
kn
*
b,
Z
Norm
Yn
Linear
(b)
Figure 3: (a) Output of Basic Building Block. The input and the output response for the basic building block of linear layer and a LayerNorm with different multipliers k set for the bias term. Only a single channel for the output is visualized. The function would saturate when the input is out of scale related to k. (b) Numerically Multi-scaled Embedding. To encode an input scalar in arbitrary values, the proposed numerically multi-scaled embedding (NME) module ensembles outputs multiple basic building blocks with different multipliers & by weighted average. (c) Input and Output Response Curves of the NME Module. The two figures show channels of the output e(z), of randomly initialized module and pretrained module under log scale. The embedding module models complex function which reflects multiple scales of variations.
and are assumed to be constant 1 and 0 here for simplicity. Note that we add multiplier to the bias parameter. This would help us understand the behavior of the embedding module.
We first notice that the output y with respect x no longer a linear function. In Figure a), we plot one channel of the output y as function of for different k values. The parameters and randomly initialized. When |æc|
> |&l, y will converge to constants =w/aw; and when |a|| < [k], y will converge to constants =b/ob- This means that y would fail to encode anything about when is signéficantly larger smaller than scale defined by k.
Numerically Multi-scaled Embedding Given the above analysis for the basic building block, we would nced to set a proper & at a scale similar to the input x. However, one cannot simply set A: == œ, because the overall function will cancel out æ. We choose enumerate all possible scales and ensemble the embeddings across scales. Let (z) denote the embedding of input scalar x scale k, The numerically multi-scaled embedding (NME) c(x) is defined as
|log ([zl//k + +e)
c(x) = Σ (x(x)) (2),, where ax(x)= =
(3)
Σj= log (|////k e)

---

Published in Transactions on Machine Learning Research (07/2024)
a; is weighting term based on the proportion between and k;s is the number of ensembled embeddings, and 6 is small number to avoid computation error. Ablation on the weighted average is presented in Appendix C We densely set the value of Yę as 10-4,10 ,..,......,,.................... 1..,,,.....,., 10", so that the range of k; is wide enough to cover the mcan and std statistics across the whole training dataset while the scale factor of 10 is determined empirically. The computation complexity of the proposed embedding module negligible compared with the Transformer model, as it only involves an extra few paralleled linear layers.
With the proposed mumerically multi-scaled embedding, normalized vector space, which ensures that gradients for learning could back- propagae smoothly.
represent arbitrary scalar values into
3.4 Extension to Multivariate Data
For multivariate time-series, we encode each window for each time-series channel independently using the aforementioned method. Parameters for encoding each window are shared across multivariate channels. Then, embeddings for each window are concatenated across channels and an additional linear layer (which is not shown Figure 3 follows transform them to the Transformer feature dimension. The resultant embeddings fed the Transformer encoder.
4 Experiments
4.1 Experimental Settings
Existing datasets for time-series analysis are relatively small individually. To ad- dress this limitation and facilitate large-scale pretraining, we propose to merge several existing datasets into unified dataset. We consider three main sources: (1) UCR time series archive (Dau et al. 2019), (2) UEA time series archive (Bagnall et al. papers (Eldele et al. 2021b Zhang et al., 2022). The original training and testing splits of these datasets are retained, and only the training portions are merged. The merged dataset consists of approximately 1.89 million univariate sequences for training. With a crop size of 512 and window size of 16, the training set amounts to about 60 million tokens. Details of the three data sources are provided below.
Pretraining Dataset
2018) and (3) eight additional datasets used in recent technical
(1) The UCR time series archive (Dau et al. 2019 contains 128 univariate time series datasets from various sources, including sensor, motion, trajectory, etc. In total, there are 60,555 in the training set of these 128 sub-datasets by their official split.
(2) The UEA benchmark (Bagnall et al. 2018 contains 30 datasets with a wide range cases, dimen- sions and series lengths for multivariate time scrics classification. Multivariate data in the UEA archive is partitioned into univariate sequences for joint self-supervised pretraining. This finally leads to 1,386,874 sequences for training.
(3) Other commonly used datasots in recent technical papers EEdele et al. 2021b Zhang et al. 2022
include: EPILEPSY (Andrzejak et al.) 2001h, SLEEPEEG (Kemp et al. 2000). HAR (Anguita et al. 2013 GESTURE [Lu et al. 2009), FD-A (Lessmeier et al. 2016), FD-B (Lessmeier et al., 2016), ECG (Clifford et al. 2017 and EMG Goldberger et al. 2000). These datasets in total contain 441,757 training sequences. More information about these datasets included in Appendix A
Pretraining Objective For self-supervised pretraining, we adopt the Bootstrap Your Own Latent (BYOL) (Grill et al. 2020) objective for its simplicity and effectiveness. In BYOL, two views of the in- put after data augmentation are fed to a Siamese network, where the base encoder trained to predict the roprosentation of the momentum encoder. The base encoder is followed by an additional predictor to avoid representation collapse. Please refer to the original paper for more details.
Implementation Details We adopt a 6-layer and 8-head standard Transformer encoder with fixed sinu- soidal positional encoding (Vaswani et al. 2017) as the backbone for our experiments. It uses 128-dimensional latent vectors through all of its layers, with 512 dimensions for the MLP hidden layer size. The window size for input patches is 16. For the numerically multi-scaled embedding, we choose use scales, which range from 10-1 to 10a by factors of 10.

---

Published in Transactions on Machine Learning Research (07 2224)
(a) Critical Difference on UCR
(b) Critical Difference on UEA
11
10
9
8
7
5
4
10
9
6
5
2
STC 2.482
9.91
Tume HC2
DTW-D 2.7115 2.711
3,1938
NuTime
TDE L6118
4. 2148
STCS.9611
3,4233
HC2
InceptionTime 6.161
MiniRocket 6.3527
Arsenal 6.3527
DrCIF 6.8973
4,9643
5,8830
MuitiRocket
TS- S--HIEEF
HC1 5.9615
TDE 0,384
CIF 5.738
5,0182
5.3269
5.3269
Arsenal
ROCKET
DrCIF
Figure 4: Test accuracy critical difference diagrams of NuT'ime's performance versus supervised state-of-the- art methods across (a) 112 datasets from the UCR archive and (b) 26 datasets from the UEA archive. We report the results of NuTime as an ensemble of five runs, each finetuned with different random seeds using the same pretrained model.
For pretraining, we simply choose the data augmentation of "random resized crop" for the BYOL objective. It randomly crops a sub-sequence from the original data between the range of 80% to 100%, and subsequently resizes the selected sub-sequence to a length of 512 using bilinear interpolation. The learning rate is 2e-3 for a batch size of 2048. The model is trained for a lotal of 100 epochs with a linear learning rate warm-up in the first 10 epochs of training and a cosine learning rate decay scheduler (Loshchilov & Hutter 2017 with an end rate of zero. For optimization, we use AdamW (Loshchilov Hutter 2018) with 31 = 0.9, B2 0.999 and a weight decay of 0.05. The pretraining takes 6 hours on 4 VI00 GPUs.
We transfer the pretrained model downstream classification tasks by full finetuning. The finetuning takes 100 epochs with a learning rate of 2c-4 by default. Since the model is pretrained on univariate data, an additional linear layer is added in the data embedding module for the mullivariate classification tasks. For all the experiments, we report the top-1 accuracy (%, "Acc." for short) and macro F1 score (%) on the test set using the best model on the validation set.
4.2 Univariate Time Series Classification
Compare with Supervised Baselines We first evaluate our model for univariate time series classification on 112 sub-datasets from the UCR archive. The 112 sub-datasets are chosen to exclude datasets containing series of uncqual length or missing values following the practice of HIVE-COTE2.0 (HC2) (Mddleeurst ct al. 2021b). The state-of-the-an method HC2 is a heavily engincered system that ensembles a distinct set of classifiers: the shapelet-based classifiers (Bostrom & Bagnall 2015). the ensemble of convolution-bassed classifiers (Dempster et al. 20200, the dictionary-based representation TDE (Middlehurst et al.) 2021a and the interval-based DrCIF [Middlchurst et al.) 2020). Moreover, HC2 takes 1,500 epochs to train the learning- based part in its ensemble system. Due lo great domain expertise being engineered into the stale-of-the-art methods, only one deep learning method InceptionTime Time (Tsmail Fawaz et al. 2020) can rank in the top 10 of the leaderboard. No prior self-supervised time series models perform close to HC2 and related methods. For fair comparisons, we ensemble 5 runs of results, each finetuned for 500 epochs with different random seeds using the same pretrained model. As shown in the critical difference diagram in Figure [a), NuTime achieves first place this challenging benchmark. This is the first time that a pretrained model with a transfer learning pipeline outperforms domain-specific features and classifiers. Detailed comparisons with other methods are shown in Figure 5 Full results for these 112 datasets are in Appendix D.I.
To compare with previous self-supervised representation learning methods, we consider the downstream classification tasks 125 UCR datasets, EPILEPSY, FD-B and EMG following Yue et al. (2022) and Zhang et al. (2022). The baseline methods conduct unsupervised learning on individual small datasets containing only thousands of sequences, and evaluate the Icarned representation on the lest. set via finetuning, Our approach is the first one that is able to pretrain across datasets with high diversity, and is also the simplest one in terms of minimum data augmentations and a simple BYOL learning objective. The results are summarized in Table D and full results for the 125 UCR sub-datasets are in Appendix [.I] The reported performance for our model is the average performance of 5 independent runs. Our Nu model outperforms the baselines on all the tested benchmarks.
Compare with Self-Supervised Baselines
8

---

Published in Transactions on Machine Learning Research (07/2024)
(a) NuTime vs HC2 62/12/38
(b) NuTime vs MuitiRocket 65/12/35
(c) NuTime vs InceptionTime 71/10/31
1.0
10
NuTime
NuTime better here
NuTime is better here
is better here
NUTime Accuracy
NuTime Accuracy
NuTTme Accuracy
02
HC2 better
MultiRacket here
Inception 0.4
0.0
0.0
0.A 0.5
10
6.2
0.4
0.9
0.0
0a
10
HC2 Accuracy
MultiRocket Accuracy
InceptionTime Accuracy
Figure 5: Accuracy comparison of NuTime and (a) HIVE-COTE2. (Middlchurst et al. 2021b), (b) Mul- tiRocket Tan et. al. 2022). (c) Inception Time [smail Fawaz et al. 2020 on 112 datasets from the UCR archive. Each subfigure's title displays a win/tie//oss comparison between NuTime and other methods. The two dotted lines indicate the 5% interval.
Table Performance comparisons with self-supervised models for univariate time series classification.
125 UCR Datasets
EPILEPSY
FD-B
EMG
Method
Avg. Acc.
Acc.
Macro-F1
Acc.
Macro-F1
Acc.
Macro-F1
TNC
76.1
T-Loss
80.6
TS-TCC
75.7
92.53
86.33
24.99
54.18
78.89
59.0419.52
TS2Vec
83.0
93.95+0.41
90.45=0.67
47.90
43.89+1.07
78.54+3.18
67 67.66+5.01
TF-C
94.95
91.49+5.34
69.38+2.31
74.87
81.71+2.87
76.83+3.11
Ti-MEE
82.32
89.71
68.55
70.88
66.56
69.99
70.89
SimMTM
95.49
92.81
69.40
75.11
97.56
98.14
Ours
86.9
95.73=0.10
93.11 10.16
92.86+2.04
93.64+1.99
100.0:0.0
100.0+0.0
4.3
Multivariate Time Series Classification
Compare with Supervised Baselines We transfer the same pretrained model on univariate time-series data to multivariate classification benchmarks by an extension to the data embedding module described in Section 3.4 We evaluate its performance on the UEA archive and compare it with state-of thottfttt--rttt techniques, which domain-specified supervised methods. The critical difference diagram and detailed comparisons to previous methods are shown in Figure b) and Figure B Detailed results on 26 datasets are in Appendix D.2 NuTime achieves first place on this challenging benchmark against heavily engincered competitors, demonstrating that the pretrained model successfully learns a transferable representation from single-dimensional data to multi-dimensional data.
Compare with Self-supervised Baselines We also compare NuTime with strong self-supervised repre- sentation learning models. The results are summarized Table and full results are shown in Appendix D.2. Our model outperforms others by scaling the pretraining data effectively, even with a simple data augmen- tation and simple contrastive learning objective.
4.4 Few-Shot Learning
One critical capability for a large-scale representation learning model is the few-shot generalization. We follow Narwariya et al. (2020) for a few-shot time-series benchmark using 41 datasets from the UCR archive. We consider the 5-shot learning scenario and 100 episodes are drawn from each dataset. By finctuning the pretrained model with few-shot data, NuTime outperforms baselines including 1NN based on Euclidean distance and dynamic time warping (DTW), BOSS (Bag-of-SFA-Symbol) (Schäler 2015 ResNet (He et al.
9

---

Published in Transactions on Machine Learning Research (07/2024)
(a) NuTime vs HC2 15/2/9
(b) NuTime vs ROCKET 20/3/3
(c) NuTime vs HC1 21/1/4
1.0
10
NuTime
NuTime better here
NuTime
is better here
IS better here
Accuracy
NuTime Accuracy
Accuracy
0.2
0.0
0.A 0.5 HC2 Accuracy
10
0.2
0.4
0.
0.0
0
10
ROCKET Accuracy
HC1 Accuracy
Figure 6: Accuracy comparison of NuTime and (a) HIVE-COTE2.0
Middlehurst et al 2021b), (b) ROCKET (Dempster et al. 2020 and (c) HIVE-COTEL.0 (Lines et al.
2016) on 26 datasets from the
UEA archive. Each subfigure's 's title displays a win/tic/loss comparison between NuTime and other meth- ods. The two dotted lines indicate the 5% interval.
Table 2: Performance comparisons with self-supervised models for multi-
Table 3: Pew-shot learn- ing results (5 shots) on the UCR archive.
variate time series classification.
29 UEA Datascts
GESTURE
Method
Method
Acc.
Avg. Acc.
Avg. Rank
Acc.
Macro-F1
INN
56.6
TS-TCC
68.2
4.5
71.8813.49
69.8413.60
DTW
61.8
TS2Vec
71.2
3.2
69.17+3.33
65.70+3.92
BOSS
62.5
TE-C
76.42+1.96
75.72+3.11
ResNet-Scratch
62.7
Ti-MAE
71.88
68.37
FS-1
65.3
SimMTM
80.00
78.67
FS-2
66.3
Ours
77.810.44
1.5
80.0011.36
78.9710.90
Ours
67.5
2016
trained from scratch and dedicate meta-learning methods designed for few-shot adaption, FS-1 and
FS-2
Narwariya et al. 2020). The results are summarized in Table D with details shown in Appendix D.3
4.5
Additional Downsteam Tasks
Besides univariate and multivariate classification, we also evaluate the proposed NuTime model on addi- tional downstream tasks including clustering and anomaly detection. Experimental results are provided Appendix NuTime consistently outperforms all other methods by significant margin in both these down- stream tasks. This suggests that our proposed method capable of learning distinetive and generalizable time-series representations from a large-scale dataset, even with simple and straightforward self-supervised approach.
4.6 Ablation Study
Ablation studies are conducted on 128 UCR datasets. We report the average performance by either finetuning the model from the pretrained checkpoint or training the model from scratch. All results are the average of runs.
Data Normalization and Encoding We first study various ways to preprocess and normalize the data by keeping the overall Transformer backbone. We consider Z-score, instance normalization (IN) and no preprocessing at all (iee,, identity) for the input sequence. A linear layer and a LayerNorm are used to encode windows to tokens. We also consider alternative methods proposed in PLE (Gorishniy et al.) 2022) encode the mean and the std scalars. In Table g our numerically multi-scaled embedding outperforms all the baselines. The PLE method relies on the quantiles of the training dataset and thus is difficult to scale properly when the data is complex and large.
10

---

Published in Transactions on Machine Learning Research (07/2024)
Table 4: Ablation study for various data encoding and normalization methods.
Table 5:
Ablation study for varying the number of scales in the numerical embedding module.
Encnding
Fine-tune
From Seratch
Num. Scales
Fino-tuned
From Scratch
Acc.
Macro-F1
Acc.
Macro-Fi
Ace.
Macro-F1
Acc.
Macro-F1
Z-score
80.88=0.13
76.75=0.46
76.59=a.55
71.26
0
80.48.00:5
77.4310.10
68. 08.14=0.40
61.43100.2
IN
79.97
75.47=0.84
75.38
69.73
1
86.45520:20
20.01
74.19=0.56
67. 17.64:11.2
Identity
82.0220.ar
78.1220.40
73.5314.30
67.13
PLE-Q
84.41=0.10
81,97=0.18
77. a7.86
=2.57=0.80
3
86.64.10.18
84.1110.29
78.32=.0.66
73.01
PLE-T
83.59
80,97=0.19
73.25=a..0
66.84=1.06
5
86.74.10.13
84.35110.3
79.30
74.0911.48
PLE-P
84.202o.a
81 37
69.34.2..00
62.43-100.70
7
86.79-0.114 84.57=0.124
78.89-0.70
73.72=0.78
Ours
86.87.0.11
84.7420.18
79.3020.84
74.09
9
86.87=0.11
84.7440.18
78.58=0.45
73. 13:0.85
Table 6: Ablation study on window size and mean/std embedding dimensions. The default window size and mean/std embedding dimension in the main paper are set to 16 and 32 respectively.
Window size
4
8
16
32
64
128
Avg. Accuracy
84.6610.31
85.24
86.87
15.1240.45
84.2210.14
82.69
Avg. Macro-F1
81.60
82.4
84.7410.18
82.5810.46
81.59
19.93
Emb. dimension
8
16
32
64
128
Avg. Accuracy
85.96=0.16
19.00.007
$6.08+0.09 09
86.87=0.11
85.91=0.20
86.00=0.15
Avg. Macro-F1
83.09=0.2
83.51
83.22:t0.21
84.74
83.00=0.17
83.10
Additionally, also consider an improved instance normalization baseline where extra tokens encoding the instance mean and std are appended to the Transformer network. Such approach improves the fine-tune results from 79.97 to 83.3 by accuracy and from 75.47 to 79.65 by Macro-Fl score. However, it still falls behind our approach by a wide margin. This demonstrates the benefits of window-wise normalization over inst normalization.
Number of Numerical Scales We vary the number of multipliers in our multi-scaled numerical data embedding module. In Table 5 the performance improves from a single scale 9 scales for the transfer setting. This shows that the capability to encode multi-scaled data is critical and our method provides an effective solution. Training from scratch attains its best results with 5 scales. Since the individual datascts are relatively small, this number of scales is sufficient in this case.
Window size and Embedding Dimensions
We provide the ablation study on window size and mean/ste /std embedding dimension in Table g The default window size and mean/ste embedding dimension used in the main paper are set to 16 and 32 respectively.
Pretraining Data As many works study self-supervised transfer learning within the same domain Eldele
et al. 2021b Yue et al. 2022 Zhang et al. 2022), we also pretrain our model on each individual dataset from the UCR archive and evaluate the performance on the same dataset. It achieves an accuracy of 79.7% and a Macro-Fi score of 74.8%, which is much lower than our large-scale pretraining results of 86.9% accuracy and 84.7% Macro-F1 score, which suggests that our model successfully learns transferable representation from large-scale data.
5 Conclusion
In this paper, we propose the NuTime model for large-scale time series pretraining. The model is based on the Transformer architecture, which takes input as a set of tokens from non-overlapping windows. Each window is represented by its normalized shape, the window mean and the window standard deviation. We develop multi-scaled numerical embedding method for representing' the scalar values of mean and std. The model can take raw values of time-scries data as input without any data normalization and transformation. To demonstrate that the proposed model can learn numerical structures with different scales of variations, we conduct the first large-scale pretraining on a dataset with great domain diversity. The pretrained model achieves state-of-the-art performance when transferred to downstream classification benchmarks. We hope that this work will pave the way to general-purper foundation models for time-series analysis.
11

---

Published in Transactions on Machine Learning Research (07/2024)
Limitations and Discussions The learned representation of the model may be subjective to the biases and inequalities from the training data. Also, the model might introduce unexpected behaviors data it never sees during training. The optimal lyper-parameters of the model may vary according to the training datasct. Thus, training the NuTime model on other dataset may require additional such as window size, and pre-defined numerical scales.
tuning,
The proposed method aims effectively encode time-series data from diverse domains. It is not yet able to decode the representation to numerical value at the original scale, and thus it is not directly applicable for forecasting problems. Straight-forwase decoding through linear layers assumes single mumerical scale of the output space. In order to decode the vector reprosentation to numerical values of arbitrary scales, we hypothesize that an iterative procedure needs to be developed. For example, diffusion models decodes RGB values in an iterative fashion and RAFT (Teed & Deng, 2020) predicts optical flow values with an recurrent network. However, decoding time series values may encounter even bigger challenges as the numerical variations are lot higher than RGB pixels and optical flows. We wish to tackle this problem in the future.
References
Ralph G Andrzejak, Klaus Lehnertz, Florian Mormann, Christoph Rieke, Peter David, and Christian E Elger. Indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: Dependence recording region and brain state. Physical Review E, 64(6):061907, 2001.
Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra Perez, and Jorge Luis Reyes Ortiz. A public domain dataset for human activity recognition using smartphones. In Proceedings of the 21th international European symposium on artificial neural networks, computational intelligence and machine learning, pp. 437 442, 2013.
Jimmy Lei Ba, Jamic Ryan Kiros, and Gcoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.066 06450, 2016.
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint NKiu:181100001 2018.
Aaron Bostrom and Anthony Bagnall. Binary shapelet transform for multiclass time series classification. In Big Data Analytics and Knowledge Discovery: 17th International Conference, WaK 2015, Valnciaa, Spain, September 1-4, 2015, Proceedings 17, pp. 257-269. Springer, 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 1901, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650 160009660,000 2021.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientifie reports, 8(1):6085, 2018.
Gari Clifford, Chengyu Liu, Benjamin Moody, H Lehman Li-wei, Ikaro Silva, Qiao Li, AE Johnson, and Roger G Mark. Af classification from a short single lead ecg recording: The pprsionet/computtin in cardiology challenge 2017. In 2017 Computing in Cardiology (CinC), pp. 1-4. IEEE, 2017.
Hoang Anh Dau, Authony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time scries archive. IEEE/CAA Journal of Automatica Sinica, 6(6):1293 1305, 2019.
Angus Dempster, François Petitjcan, and Gcoffrey I Webb. Rocket: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34(5): 1454- 2020.
12

---

Published in Transactions on Machine Learning Research (07/2024)
Jiaxiang Dong, Haixu Wu, Haoran Zhang, Li Zhang, Jianmin Wang, and Mingsheng Long. Simmtm: A simple pre-training framework for masked time-series modeling. In Advances in Newral Information Processing Systems (NeurIPS), 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Rep- resentations (ICLR), 2020.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cun- tai Guan. Time-scrics reprosentation learning via temporal and contextual contrasting. arXiv preprint 1:Xi::2106144112, 2021a.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 2352 2359, 2021b.
Jean- Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems, 32, 2019.
Ary Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E. Mietus, George B Moody, Chung-Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101 (23):e215-c220) 2000.
Yury Gorishniy, Ivan Rubachev, and Artem Babenko. On embeddings for numerical features in tabular deep learning. arxiv, 2203.05556, 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural informalion processing systems, 33:21271 11-21271-21284, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelorating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 048-456.77 pmlr, 2015.
Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey I Webb, Lhassane Idoumghar, Pierre- Alain Muller, and François Petitjean. Inceptiontime: Finding alexnct for time series classification. Data Mining and Knowledge Discovery, 34(():1936-119) 2020.
Bob Kemp, Aeilko H Zwinderman, Bert Tuk, Hilbert AC Kamphuisen, and Josefien JL Oberye. Analysis of a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the eeg, IEEE Transactions on Biomedical Engineering, 4795))1155-1194 2000.
Tacsung Kim, Jinhce Kim, Yunwon Tac, Cheonbok Park, Jang-Ho Choi, and Jacgul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Represendations, 2021.
Christian Lessmeier, James Kuria Kimotho, Detmar Zimmer, and Walter Sextro. Condition monitoring of bearing damage in electromechanical drive systems by using motor current signals of electric motors: A benchmark data set for data-driven classification. In PHM Society European Conference, volume 3, 2016.
Zhe Li, Zhongwen Rao, Lujia Pan, Pengyun Wang, and Zenglin Xu. Ti-mae: Self-supervised masked time series autoencoders. Xin preprint arXin:2301.0880 08871, 2023.
13

---

Published in Transactions on Machine Learning Research (07/2024)
Jason Lines, Sarah Taylor, and Anthony Bagnall. Hive-cote: The hierarchical vote collective of transformation--bssed ensembles for time series classification. In 2016 IEEE 16th international confer- ence on data mining (ICDM), pp. 1041-10110 1046. IEEE, 2016.
Jiayang Liu, Lin Zhong, Jehan Wickramasuriya, and Venu Vasudevan. uwave: Accelerometer-basee person- alized gesture recognition and its applications. Pervasive and Mobile Computing, 5(6):657-675, 2009.
llya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations (ICLR), 2017.
Πya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference
Learning Represendations (TCLR), 2018.
Matthew Middlehurst, James Large, and Authony Bagnall. The canonical interval forest (cif) classifier for time series classification. In 2020 IEEE international conference on big data (big data), pp. 188 195. IEEE, 2020.
Matthew Middlehurst, James Large, Gavin Cawley, and Anthony Bagnall. The temporal dictionary ensemble (tde) classifier for time series classification. In Machine Learning and Knowledge Discovery in Databoses: European Conference, ECML, PKDD 2020, Ghent, Belgium, Seplember 18, 2020, Proceedings, Part I, pp. 660 676. Springer, 2021a.
Matthew Middlehurst, James Large, Michael Flymn, Jason Lines, Aaron Bostrom, and Anthony Bagnall, Hive-cote 2.0: a new meta ensemble for time series classification. Machine Learning, 110(11-12):3211 3243, 2021b.
Rudy Morel, Gaspar Rochette, Roberto Leonarduzzi, Jean-Philippe Bouchaud, and Stéphane Mallat. Scale dependencies and self-similar models with wavelet scattering spectra. arXiv preprint or orXin:2204.1011 2022.
Jyoti Narwariya, Pankaj Malhotra, Lovekesh Vig, Gautam Shroff, and TV Vishmu. Mota-learning for few- shot time series classification. In Proceedings of the 7th ACM IKDD CoDS and 25th COMAD, pp. 28 36, 2020.
Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. time series is worth 64 words: Long-term forecasting with transformers, In International Conference on Learning Representations (ICLR), 2023.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprinl arXiv:1905, 10437, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748 8763. PMLR, 2021.
Patrick Schäfer. The boss concerned with time scries classification in the presence noisc Data Mining and Knowledge Discovery, 29:1505 1530, 2015.
Shun-Yas Shih, Fan-Keng Sun, and Hung-yi Lec. Temporal pattern attention for multivariate time series forecasting. Machine Learning, 108:1421-144)) 2019.
Chang Wei Tan, Angus Dempster, Christoph Bergmeir, and Geoffrey I Webb. Multirocket: multiple pooling operators and transformations for fast and effective time series classification. Data Mining and Knowledge Discovery, 36(5):1623-1644 2022.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 402 -19. Springer, 2020.
14

---

Published in Transactions on Machine Learning Research (07/2024)
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. arXiv preprint. orXiv:2106.00735 2021.
Dmitry Ulaanvv, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint
2016.
Laurens van der Maaten and Geoffrey F. Hinton. Visualizing data using I-sne. Journal of Machine Learning Research, 9:2579 2605, 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Newral Information Processing Systems (NeurIPS), 30, 2017.
Haixu Wu, Jichui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting, Advances in Newral Informalion Processing Systems, 34: 22419 22430,2230000 2021.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 753 753-763, 2020.
Ling Yang and Shenda Hong. Unsupervised time-series representation learning with iterative bilinear temporal-spectin fusion. In International Conference on Machine Learning, pp. 25038-250525 25054. PMLR, 2022.
Zhihan Yuc, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 1980-8987 2022.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Amuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2114 2124, 2021.
Xiang Zhang, Ziyuan Zhao, Theodoros Tsiligkaridis, and Marinka Zitnik. Self-sueevised contrastive pre- training for time series via time-froquency consistency. In Proceedings of Neural Information Processing Systems (NeurIPS), 2022.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. In- former: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 11106-11115, 2021.
15

---

Published in Transactions on Machine Learning Research (07/2024)
A More Information about Datasets
(1) EPILEPSY (Andrzejak al. 2001 is an epilepsy scizure recognition dataset composed of EEG recordings for 500 subjects labeled by 5 categories with 9,200 training and 2,300 test samples. 4 classes that do not have an epileptic seizure in this dataset are merged into one class. Thus, each time series sample has a binary label. To compare the proposed NuT'me model with other self-supervised methods, we use the dataset split by Zhang et al. 2022). having 60 samples for training, 20 samples for validation, and 11,420 samples for testing.
(2) SLEEPEEG (Kemp et al. 2000) contains 371,055 training and 90,315 test whole-night sleeping elec- troencephalograpayy (EEG) recordings that are classified into 5 patterns inchuding Wake (W), Non-rapid eye movement (N1, N2, N3) and Rapid Eye Movement (REM).
(3) HAR (Anguita et al. 2013 is a multivariate time series with 9 channels that are sensor readings for 30 subjectives when they are performing 6 actions like walking, sitting and lying down. There are 5,881 and 2,947 samples in the training and test dataset respectively.
(4) GESTURE (Luu et al., 2009) is a set of eight simple gestures generated from accelerators consisting of the XYZ coordinates of each motion, and it contains 320 and 120 training and test samples respectively.
(5) FD-A and (6) FD-B (Lessmeier et al. 2016 is real-world fault diagnosis dataset generated by an electromechanical drive system with 2 fault classes and health class in different conditions of rolling bearings. FD-A and (6) FD-B refer to the data collected under condition and condition B. For a fair comparison, we adopt the same training and test split as Zhang et al. (2022), and there are 60 and 13,559 samples in FD-B training and test dataset for classification benchmarking.
(7) ECG (Clifford et al. 2017) contains 8,528 clectrocardiogram (ECG) recordings into four categories that indicate different underlying conditions of cardiac arrhythmias. These long recordings are divided into short sequences of 5 seconds that are physiologically meaninglul, and each short sequence has 1500 observations and is regarded as a single sample.
(8) EMG (Goldberger et al., 2000) is a set of electromyography (EMG) recordings from the tibialis anterior muscle from subjectives, and each of them indicates a classífication ion category. EMG univariate and split into training and test set following Zhang et al. (0022) leading to 122 training and 41 test samples.
B Additional Downstream Tasks
Clustering For the clustering task on two-class dataset EPILEPSY, we employ K-means (K=2) the learned representations from NuTime and its sule-supe counterparts. These representations are ob- tained after independent fine-tuning on EPILEPSY, following the settings of Zhang ct al. (2022) We evaluate the clustering performance using three widely adopted metrics: Silhouette score, Adjusted Rand Index (ARI), and Normalized Mutual Information (NMI). In order to highlight the generalization the time-series rep- resentations learned by NuTime, we also perform clustering using the NuTTme representations without any fino-tuning on the downstream task data.
Anomaly Detection Following the approach presented by Zhang et al. 2022 we conduct anomaly detection to identify abnormal time series samples instead of outlier observations. Specifically, we adopt a highly imbalanced subset of FD-B with 1000 samples. Among these samples, 900 correspond to undamaged bearings, while the remaining 100 samples pertain bearings with either inner or outer damage. A one-class SVM is applied to the learned representations from NuTime and its self-supervised counterparts perform anomaly detection.
The results for clustering and anomaly detection are presented in Table i and Table M respectively. All experiments are repeated 5 times, and we report the mean and standard deviation of all metrics across these runs. Our NuTime model consistently outperforms all other models by a significant margin in both additional tasks. Remarkably, even without finctuning on the specific data of the clustering task and directly utilizing representations from the pretrained model, our NuTime model demonstrates superior performance compared to other self-supervised methods after finetuning across most metrics. This suggests
16

---

Published in Transactions on Machine Learning Research (07 (07/2024)
that proposed method is capable of learning distinctive and generalizable time-scries representations from large-scale dataset, even with simple and straightforward self-supervised approach.
To visually illustrate the distinet representations learned by NuTTme (i.e., the token at position of the Transformer output), we employ t-SNE (van der Maaten Hinton 2008 on the two datasets used for additional downstream tasks. The resulting visualizations are depicted in Figure
Table 7: Performance on EPILEPSY for clustering task. All the methods are pretrained in self-supervised manner. "Ours (w/o ft.)" denotes that the pretrained NuTime model representations are directly employed for the clustering task without undergoing finetuning, whereas the remaining methods are finetuned on the EPHLEPSY dataset.
Method
Silhouette Score
ARI
NMI
KNN
0. 0.1208:40.0271
0.0549 0.0059
0.00964 0.0014
TNC
0.2353=0.0015
0.0211#0.0061
0.0082+0.0018
CPC
0.2023=40.0011
0.0153+0.0196
0.0063
TS-TCC
0.5154+0.0168
0.6307=0.032r
0.5178
TF-C
0..039==0.0117
0.6583=0.0259
0.5567 00.017
Ours (w/o ft.)
0.4923=0.0011
0.6854+0.0000
0.5737 00000
Ours
0.7945=0.0016
0.7944+0.00146
0.6450#0.000
Table 8: Performance on an FD-B subset for sample-level anomaly detection task. All the methods are pretrained in self-supervised manner and finetuned on the FD-B anomaly detection subset.
Method
Precision
Recall
Fl Score
AUROC
KNN
0.4785-20.0356
0..055
0.506110.000
0. 0.7653+0.0153
TNC
0.835420.0494 0.7882+0.0157
0.795720.0108
0.8231+0.0379
CPC
0.5967+0.0776 0.4896=000866
0.5238=0.0792
0. 0.7859#0.065 03366
TS-TCC
0.6219+0.0183 00.451
0.4759=0.0503
0. 0.7966=0004411
TF-C
0.8526+0.0365 0.7823++0 0299
0.8312:4.01s0
0.8598+0.0288
Ours
1.0000++.0000
0.8962240.0778
0.9448:20.22m 0200
0.9481=000m
(a) Epilepsy Visualization
(b) FD-B Subset Visualization
100
cluster 0
normal outllers
20
-25
-20
-50
-75
100
-75
Figure Visualization of the learned representations from NuTime EPILEPSY and FD-B subset. It highlights the distinct separation of these representations, which is highly advantageous for downstream tasks, such as clustering and anomaly detection.
17

---

Published in Transactions on Machine Learning Research (07 (024)
C Weighted Average in Ensemble Module
We design a hcuristic weighting term within the ensemble module, which is based on the proportion between the imput scalar I and the provided bias multipliers k. It is motivated by an insight that the input scalar a should be embedded by the basic building block (i.e., linear + LayerNorm) which has a that shares a comparable numerical scale with 2, Otherwise, as the number of ensemble scales increases, the information encapsulated within x would be diluted by a simple average ensemble. Figure confirms this hypothesis.
(a) finetuned
(b) from scratch
87.0
weighted average average
weighted average
86.9
average
79
86.8
UCR Avg. AcC-
86.7
UCR Avg. ACC.
86.5
86.0
Num Scales
Num Scales
Figure 8: Average accuracy (%) on 128 UCR datasets by (a) finetuning from the pretrained checkpoint and (b) training from scratch with or without the weighting term in the ensemble module. "Num Scales" stands for the number of ensembled numerical scales.
D Full Results on UCR/UEA Datasets
D.1 Univariate Time Series Classification
The full results, in terms of top-1 accuracy, for both supervised and self-supervised pretrained methods on each sub-dataset in the UCR benchmark presented in Table and Table I0 respectively.
Table 9: Full results for 112 UCR datasets of NuTime and supervised counterparts.
Datasct
NuTime
HC2
TS-CHIEP
HC1
Inception Time
MiniRocket
MultiRocket
ACSF1
0.700
0.930
0,840
0.940
0.910
0.900
0.900
Adiac
0.854
0.813
0.798
0.811
0.844
0.770
0.839
ArrowHead
0.880
0.863
0.806
0.823
0.863
0.794
0.863
BMR
1.000
1.000
0.993
0.953
1.000
1.000
1.000
Beef
0.967
0.867
0.767
0.900
0.667
0.833
0.767
BeetlePly
0.950
0.950
1.000
0.950
0.850
0.900
0.900
CirdChicken
1.000
0.900
0,950
0.950
0.950
0.900
0,900
CBF
0.998
1.000
0.998
0.999
0.999
1.000
0.996
Car
0.933
0.917
0.867
0.833
0.900
0.850
0.917
Chinatown
0.980
0.983
0.968
0.980
0.985
0.974
ComortneConcentraationn
0.716
0.771
0.660
0.739
0.876
0.807
0.782
CinCECGTorrs
0.930
1.000
0.981
0.995
0.853
0.837
0.946
Coffee
1.000
1.000
1.000
1.000
1,000
1.000
1.000
Computers
0.844
0.764
0.700
0.776
0.808
0.752
0.784
CricketX
0,846
0.826
0,821
0,808
0.849
0.826
0.795
Cricket
0,833
0.851
0,800
0.823
0.844
0.856
0,838
CricketZ
0.882
0.864
0.831
0.823
0.849
0.859
0.831
Crop
0.772
0.765
0.764
0.773
0.799
0.751
0.775
DistomSSieeReduccton
0.993
0.958
0,964
0.935
0.948
0.967
0.961
0.777
0.748
0.748
0.755
0.734
0.748
0.770
0.812
0.775
0.757
0.772
0.786
0.761
0.808
0.727
0.705
0.676
0.676
0.655
0.719
0.683
BCG200
0.900
0.890
0.840
0.860
0.910
0.910
0.910
18

---

Published in Transactions on Machine Learning Research (07/2024)
ECG5000
0.940
0.947
0.946
0.947
0.942
0.947
0.947
ECGFiveDays
0.979
1.000
1.000
1.000
1.000
1.000
1.000
FOGHorizontalSggnal
0.743
0.657
0.655
0.610
0.630
0.652
0.644
ROGVerticalSignal
0.605
0.569
0.597
0.547
0.517
0.533
0.525
Earthquakes
0.748
0.748
0.748
0.748
0.712
0.748
0.748
ElectricDevvces
0.812
0.758
0.760
0.746
0.721
0.726
0.740
EthanolLevel
0.726
0.696
0.528
0.684
0.832
0.570
0.632
FaceAAl
0,839
0.868
0.842
0.779
0.793
0.947
0.804
FaceFour
0,966
1.000
1.000
0.989
0.955
0.977
0.943
FCcesUCR
0,933
0.963
0.967
0.959
0.972
0.961
0.961
FiftyWords
0.862
0.833
0,848
0.776
0.846
0.833
0.857
Fish
0.994
0.989
0.994
0.994
0.983
0.971
0,983
FordA
0.937
0.954
0.950
0.951
0.961
0.942
0.958
FordB
0,802
0,831
0.823
0.832
0.849
0.806
0.835
Rrenzen
0.999
1.000
0.998
0.998
0.997
0.996
1.000
FreszerSmallTrain
0.990
0.999
0.998
0.986
0l846
0.950
0.995
GunPoint
1.000
1.000
1.000
1.000
1.000
0.993
1 1.000
AgeSan
1.000
0.997
1.000
0.997
0.991
0.997
1.000
CumnPointMaaleeeeesseeeaaaa
1.000
1.000
1.000
1.000
1.000
0.997
1.000
CunPoonnOOlaVersusSSYooo
1.000
1.000
1.000
1.000
1.000
0.994
1.000
Ham
0.762
0.695
0.705
0.705
0.724
0.714
0.743
HandOutlines
0.962
0.938
0.938
0.932
0.959
0.943
0.946
Haptics
0.591
0.545
0.529
0.539
0.371
0.519
0.555
Herring
0.719
0.672
0.641
0.672
0.703
0.672
0.703
HouseTTwenty
0.983
0.966
0.975
0.983
0.916
0.966
0.992
InlineSkate
0.449
0.553
0.527
0.485
0.485
0.475
0.478
IPReellP Train
1.000
1.000
1.000
1.000
1.000
1.000
1.000
SsectEPCSmalllTaan
1.000
1.000
1.000
1.000
1.000
0.984
1.000
Insect inggeWaWnnna
0.626
0.663
0.644
0.660
0.637
0.661
0.675
SulyPowerDemanne
0.958
0.972
0.965
0.960
0.966
0.969
0,968
LargeKitchenAppii Aplliannes
0,891
0.907
0,768
0.853
0.904
0.904
0.888
Lightning2
0.803
0.787
0.836
0.770
0.836
0.754
0.689
Lightning7
0.849
0.822
0.767
0.726
0.795
0.822
0.836
Mallat
0.977
0.978
0.971
0.973
0.958
0.957
0.925
Meat
1.000
0.933
0.883
0.917
0.950
0.950
0.933
Medicallmages
0.830
0.808
0.796
0.732
0.796
0.793
0.803
0.623
0.597
0.571
0.591
0.552
0.591
0.643
MiddllePhalanncOntlinCCreret
0.866
0.852
0.825
0.835
0.835
0.838
0.863
MiddicPhalannXIVV
0.610
0.558
0.565
0.584
0.332
0.565
0.539
MixedShapesSRegglarlll
0,979
0.976
0.971
0.969
0.970
0.967
0.981
MixedShapesSmallIlree
0,954
0.959
0.949
0.954
0.911
0.938
0.965
MoteStrain
0.966
0.970
0.927
0.968
0.894
0.915
0.945
0.926
0.947
0.917
0.924
0.958
0.955
0.961
NonlnvasiveFeeaaBBCCC horrxxx
0.937
0.967
0.948
0.945
0.959
0.969
0,964
OSULeaf
0.963
0,963
0.996
0,983
0.921
0.942
0.967
OliveOil
0.800
0.933
0.900
0.867
0.867
0.933
0.933
PhalangeSOutlneesSCCe
0.848
0.831
0.818
0.821
0.852
0.834
0.843
Phoneme
0.379
0.355
0.361
0.381
0.337
0.283
0.351
PigAirwayPresaure
0.899
0.933
0.976
0.962
0.942
0.091
0.596
PigArtPressure
0.995
0.995
0.981
0.990
1.000
0.952
0.952
PigCVP
0.942
0.966
0.962
0.962
0.966
0.938
0.875
Plane
1.000
1.000
1.000
1.000
1.000
1.000
1 1.000
PowerCons
0.994
0.983
0.989
1.000
0.994
0.933
0,983
ProtimalPaalannOOutinnegggg
0.883
0.854
0.859
0.839
0.849
0.859
0.854
0.942
0.900
0.883
0.890
0.931
0.900
0.921
ProximalPhalanx//
0.839
0.829
0.824
0.824
0.800
0,820
0,820
RediigerationDevices
0.640
0.539
0.571
0.579
0.515
0.536
0.517
Rock
0.960
0.920
0.900
0.960
0.600
0.900
0.860
ScreenTType
0.603
0.579
0.499
0.557
0.595
0.477
0.563
HanngHandGenderCta
0.947
0.957
0.923
0.970
0.873
0.918
0.958
SemgHand MangHHHnnMowmmmenn Movement Ch2
0,876
0.856
0.878
0.871
0.584
0.638
0.776
ScngllaadSuujocthh2
0.933
0,902
0,924
0.953
0.733
0.889
0,924
ShapeletSim
1.000
1.000
1.000
1.000
0.994
1.000
1.000
ShapesAAl
0.933
0.922
0.925
0.932
0.925
0.907
0.925
MuullKiicchenAAplliine
0.853
0.837
0.824
0.835
0.779
0.813
0.827
StmoothSubspace
0.993
0.987
1.000
0,980
0.980
0.980
0,980
Sony BnonAAIIOoossuraeeee
0.955
0.908
0.832
0.767
0.879
0.923
0.889
SonyATBORobooSurre
0.930
0.930
0.896
0.942
0.952
0.915
0.938
19

---

Published in Transactions on Machine Learning Research (07/2024)
StarLightCurves
0.981
0.982
0.983
0.981
0.978
0.981
0.982
Strawberry
0.968
0.976
0.970
0.970
0.984
0.981
0.976
SwedishLeaf
0.973
0.965
0.965
0.954
0.970
0.963
0.978
Symhols
0.985
0.976
0.978
0.980
0.985
0.973
0.982
SyntheticControl
0.997
1.000
0.997
0.997
0.997
0.997
0.997
ToeSegmentation1
0.965
0.969
0.974
0.965
0.969
0.952
ToeSegmentation2
0.938
0.938
0.962
0.962
0.946
0.915
0.908
Trace
1.000
1.000
1.000
1.000
1.000
1.000
1.000
TwoLcaEECG
1.000
0.999
0.994
0.998
0.996
0.999
0.998
TwoPatterns
1.000
1.000
1.000
0.999
1.000
1.000
1.000
UMD
1.000
0.993
0.986
0.979
0.986
0.993
0.993
UWaveGesturel All
0.975
0.974
0.970
0.968
0.951
0.975
0.980
UWaveGestureLibran
0.877
0.855
0.843
0.829
0.822
0.858
0.870
UWaveGestureLibran
0.823
0.775
0.771
0.747
0.771
0.773
0.802
UWaveGesturrdLlbraay
0.845
0.797
0.786
0.777
0.769
0.794
0.816
Wafer
0.999
1.000
0.999
1,000
0.999
0.999
1,000
Wine
0.778
0.870
0.870
0.796
(.630
0.815
0.907
WordSynonyms
0.746
0.752
0.795
0.697
0.745
0.751
0.771
Worms
0,818
0.753
0.818
0.649
0.805
0.714
0.792
WwomsTwoClass
0.844
0.792
0.831
0.766
0.766
0.740
0.792
Yoga
0.918
0.934
0.854
0.912
0.905
0.917
0.920
Average
0.884
0.877
0.865
0.865
0.858
0.853
0.868
Table 10: Full results for 125 UCR datasets of NuTTme and self-supervised counterparts.
Dataset
NuTime 0.678
TS2Vec-1
TS2Vec-16 Te2
T-Loss
TNC
TS-TCC
TST
ACSFI
0.840
0.900
0.910
0.900
0.730
0.730
0.760
0.640
Adíac
0.830
0.775
0.762
0.765
0.675
0.726
0.767
0.550
0.604
AlLGesturWWimoke
0.751
0.763
AllGesture's Wiimote
0.781
0.764
0.793
0.774
0.726
0.699
0.741
0.423
0.729
AllGesture WilmoteZ
0.776
0.731
0.746
0.770
0.723
0.646
0.689
0.447
0.643
ArrowHead
0.873
0.823
0.857
0.817
0.766
0.703
0.737
0.771
0.703
BME
1.000
0.973
0.993
0.980
0.993
0.973
0.933
0.760
0.900
Beef
0.887
0.767
0.767
0.633
0.667
0.733
0.600
0.500
0.633
BeetleFly
0.930
0,850
0.900
0.900
0.800
0.850
0.800
1.000
0.700
BirdChicken
CBF
0.995
1.000
1.000
1.000
0.983
0.983
0.998
0.898
0.997
Car
0,913
0,883
U. 0.833
0.700
0.833
0.683
0.583
0.550
0.733
Chinatown
0.976
0.968
0.965
0.959
0.951
0.977
0.983
0.936
0.957
ChlorineConcentt
0.699
0.810
0.832
0.812
0.749
0.760
0.753
0.562
0.648
CiaCECGTors
0.919
0.812
0.827
0.825
0.713
0.669
0.671
0.508
0.651
Coffee
1,000
1.000
1.000
1.000
1.000
1.000
1.000
0.821
1.000
Computers
0.664
XricketXX
0.815
0.800
0.782
0.805
0.713
0.623
0.731
0.385
0.754
Cricket
0,815
0.756
0.749
0.769
0.728
U.397
0.718
0.467
0.744
CricketZ
0.847
0.785
0.792
0.790
0.708
0.682
0.713
0.403
0.754
Crop
0.754
0.753
0.756
0.753
0.722
0.738
0.742
0.710
0.665
DistomSiteReduuttion
0.994
0,980
0.981
0.987
0.984
0.977
0.961
0.967
DistalPhalanx wtwaaalannnnniiiieeeeeegoo
0.776
0.719
0.727
0.719
U.727
U.741
U.755
U.741
0.770
DistalPhalannCOmiinnCCCCrr
DistalPhalanxTWV
0.734
0.662
0.698
0.683
0.676
0.669
0.676
0.568
0.590
ECG200
0.885
0,890
0.920
0.880
0.940
0.830
0.880
0.830
0.770
ECG5000
0.938
0.935
0.935
0.934
0.933
0.937
0.941
0.928
0.924
ECGFiveDays
0.973
1.000
1.000
1.000
1.000
0.999
0.878
0.763
0.768
EOGHorisontalSigual
0.708
0.541
0.539
0.522
0.303
EOG VoGretticalSignall
0.591
0.467
0.503
0.472
0.434
U.392
0.376
0.298
0.448
Earthquakes
ElectricDevices
0.793
0.712
0.721
0.719
0.707
0.700
0.686
0.676
0.602
EthanolLevel
0.698
0.480
0.468
0.484
0.382
0.424
0.486
0.260
0.276
FaceAll
0.829
0.759
0.771
0.805
0.786
0.766
0.813
0.504
0.808
FaceFour
0.970
0.861
0. 0.932
0.932
0.920
0.659
0.773
0.511
0.830
FCCesUCR
0.924
Firty
0.834
0.771
0.771
0.774
0.732
0.653
0.653
0.525
0.690
0.926
0.937
0.891
0.817
FordA
0.934
0.940
0.936
0.948
0.928
0.902
0.930
0.568
0.555
FordR
0.799
0.789
0.794
0.807
0.793
0.733
0.815
0.507
0.620
Freeser TraularTrain
0.999
0.985
0.986
0.983
0.956
0.991
0.989
0.922
0.899
FreezerSmallTTain
0.985
0.891
0.870
0.872
0.933
0.982
0.979
0.920
0.753
Fungi
GestureMidAieDD
0.774
0.631
0.608
0.615
0.608
0.431
0.369
0.208
0.569
GestureMfiAAir22
0.469
0.515
0.546
0.362
0.234
0.138
GestureMidAAt:33
0.532
0.346
0.292
0.300
0.285
0.292
0.177
0.154
0.323
GesturePebble7i
0.951
0.878
0.930
0.884
0.919
0.378
0.395
0.500
0.791
20

---

Published in Transactions on Machine Learning Research (07/2024)
GunPoint
GesturePPebble22
0.997
0.980
0.980
0.987
0.848
0.899
0.980
0.967
0,316
0,430
0,993
0.:00
0,981 19600
0,380
0.827
0.671
0.907
CounPinnMMale MaleVereusFemall
0.998
1.000
1.000
0.994
1,000
0.987
1.000
0.968
0.997
0.994
0.991
0,984
0,984
0.997
1,000
0.997
0.918
GunPoint Versus
1.000
1.000
1.000
1.000
1.000
1.000
1,000
1,000
0.838
Ham
0.762
0.714
0.714
0.724
0.724
0.752
0.743
0.524
0.467
HandOutlines
0.963
0.919
0.922
0.930
0.922
0.930
0.724
0.735
0.881
Haptics
0.551
0.536
0,490
0.396
0.357
Herring
0.694
0.625
0.641
0.609
0.594
0.594
0.594
0.594
0.531
HouseTweety
0.958
0.941
0.916
0.941
0.933
0.782
0.790
0.815
0.924
InlineSkate
0.423
0.389
0.115
0.407
0.371
0.378
0.347
0.287
0.381
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
0.872
1.000
1.000
1,000
1.000
1.000
1,000
1,000
1.000
0.735
Insect SingbeeSnott
0.594
0.629
0.630
0.624
0.597
0.549
0.415
0.266
0.355
HalyPowerDDemane
0.925
0.960
0.954
0.928
0.955
0.845
Lirgeeit Appliacces
0.882
0.845
0.845
0.875
0.789
0.776
0.848
0.595
0.795
Lightning2
0.830
0,836
0.869
0.820
0.869
0.869
0.836
0.705
0.869
Lightning?
0.833
0.836
0.863
0.822
0.795
U.767
0.685
0.411
0.726
Mallat
0.965
0.915
0.914
0.873
0.951
0.871
0.922
0.713
0.934
Meat
0.987
0.950
0.950
0.967
0.950
0.917
0.883
0.900
0.933
MedicalImages
0,801
0. 0.792
0. 0.789
0.793
0.750
0.754
0.747
0.632
0.737
MelbournePedestten
0,966
0.959
0.956
MaddilePhalanOOuttin
0.630
0.636
0.036
0.630
0.656
0.643
0.630
0.617
0.500
0.859
0.811
D. 838
0.825
0.825
0.818
0.818
0.753
0.698
MiddlePhalanxTTV
0.587
0.591
0.584
0.578
0.591
0.571
0.610
0.506
0.506
MixesShapesRogguurTTaaa
0.975
0.915
0.917
0.922
0.905
0.911
0.855
0.879
0.842
MixedShapesSmmllTTrin
0.942
0.881
0.8G1
0.856
0.860
0.813
0.735
0.828
0.780
MoteStrain
0.964
0.857
0.861
0.863
0.851
0.825
0.8-13
0.768
0.835
0.930
NoilivalieFeeaaBBCCChhoee
0.928
0.940
0.938
0.935
0.919
0.912
0.913
0.832
0.865
OSULeaf
0,945
0,876
0.851
0.843
0.760
U.723
0.723
0.545
0,391
OliveOil
0.813
0.900
0.900
0.900
0.867
0.833
0.800
0.800
0.833
PLAID
0.945
0.551
0.561
0.549
0.555
0.495
0.445
0.419
0.840
DlalaoreeOuthheeeCorreett
0.850
0.795
0.809
0.823
0.784
0.787
0.8041
0.773
0.728
Phoneme
0.349
0.296
0.312
0.309
0.276
0.180
0.242
0.139
0.228
PickupGesture WitmotzZ
0,848
0.760
0.740
0.240
PigAirway PrePrrse
0.862
0.521
0,630
0.683
0.510
0.413
0.380
0.120
0.106
PigArtPressurre
0,994
0,962
0.966
0.966
0.928
0.808
0.524
0.774
0,245
PigCVP
0.947
0.803
0.812
0.870
0.788
0.649
0.615
0.596
0.154
Plane
1,000
1.000
1.000
0.990
0.990
1.000
1.000
0.933
1.000
PowerCons
0,992
0,967
0.
0.972
0.900
0.961
0.878
ProximalPhalann//e linAAeerrnnnn
0,885
0.844
0.834
0.829
0.844
U.854
0.839
0.854
0.805
0.931
0.887
0.900
0.859
0.866
0.873
0.770
ProximalPhslaux[II
0,831
0.785
0.824
0.805
0,771
0.810
0.800
0.780
0.761
RetrigerationDeviiess
0,643
0.587
0.589
0.589
0.515
0.565
0.563
0.483
0.464
Rock
0.896
0.660
0.700
0.700
0.580
0.580
0.600
0.680
0.600
Sype
0.601
0.405
0.411
0.397
0.416
0.509
0.419
0.419
0.397
SemnHHandGenddeCChe
0.963
0.962
0.882
0.725
SonnZHandMMovementCh
0.863
0.893
0.860
0.891
0.789
0.593
0.613
0.420
0.584
SomgHHandSubjoetthh
0.944
0.951
0.942
0.853
0.771
0.753
0.484
BaakeGeestueeiinooee
0.948
0.940
0.940
0.920
0.920
0.820
0.860
0.760
0,860
ShopeletSim
1.000
0.989
1.000
0.994
0.672
0.589
0.683
0.489
0.650
Shapes
0.927
0.897
0.902
0.905
0.848
0.788
0.773
0.733
0.768
EmallKitchenAApplaucee
0.851
0.723
0.731
0.733
0.677
0.725
0.691
0.592
0.6-13
SmootthSusppaee
0,980
0.960
0.827
SonyAAIBORaooSSurfice
0.948
0.874
0.903
0.900
0.902
0.804
0.899
0.724
0.725
Sony AinyATTRRbboSSurtrre
0.916
0.890
0.871
0.889
0.889
0.834
0.907
0.745
0.831
StarLightCurves
0.980
0.970
0.969
0.971
0.964
0.968
0.967
0.949
0.907
Strawberry
0.972
0.962
0.962
0.965
0.954
0.951
0.965
0.916
0.941
SwedishLeaf
0.965
0,939
0.941
0.942
0.914
0.880
0.923
0.738
0.792
Symbols
0.982
0.973
0.976
0.972
0.963
0.885
0.916
0.786
0.950
SyntheticConttol
0.997
0.987
1.000
0.990
0.490
T
0.961
0.930
0.917
0.947
0.939
0.864
0.930
0.807
0.772
ToeSegmentation2
0.946
0.915
0.802
0.900
0.900
0.831
0.877
0.615
0.838
Trace
L.000
1.000
1.000
1.000
0.990
1.000
1.000
1.000
1.000
TwGLeadECG
0.999
0.982
0.986
0.987
0.999
0.993
0.976
0.871
0.905
TwoPatterus
L.000
1.000
1.000
1.000
0.999
1.000
0.999
0.466
1.000
UMD
1,000
1.000
1.000
0.993
U.993
U.993
0.986
0.910
0.993
TWaveGestureLibraas
0.930
0.934
0.896
0.903
0.692
0.475
C'aaveGsttrrel MeaveGGsttureibraae Lihrary
0.859
0.810
0.795
0.801
0.785
0.781
0.733
0.569
0.728
CWaveGistureiibral
0,800
0.729
0.719
0.720
0.710
0.697
0.641
0.348
0.634
U WWavGGesturLLiraary2
0.821
0.761
0.770
0.768
0.757
U.721
0.690
0.635
0.658
Wafer
0.999
0.995
0.998
0.997
0.992
0.994
0.994
0.991
0.980
Wine
0.785
0.778
0.870
0.374
WordSynouyms
0.712
0.699
0.676
0.704
0.691
0.630
0.531
0.422
0.649
Worms
0.701
0.701
0.701
0.727
0.623
0.753
0.455
WormsTwor TwoClass
0.839
0.805
0.805
0.753
0.792
0.727
0.753
0.584
0.623
Yoga
0.904
0.880
0.887
0.877
0.837
0.812
0.791
0.830
0.837
Average
0.830
0.527
0.737
0.727
21

---

Published in Transactions on Machine Learning Research (07/2024)
Table 11: Full results for 26 UEA datasets of NuTTie and supervised counterparts.
Dataset
NuTTmm
HC2
ROCKET
Arsenal
DrCIF
CIF
HC1
TDE
STC
DTW-C
0.993
0.993
0.997
0.993
0,980
0.983
0.990
0.993
0.990
0.987
AtrialFibeillation
0.267
0,200
0.133
0.333
0.333
0.133
0.267
0.267
0.200
BasicMotions
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
0.975
0.975
Cricket
1.000
1.000
1.000
1.000
0.986
0
0.986
0.986
0.986
1.000
Duck DuckGesse
0.540
0.560
0.500
0.460
0.540
0.440
0.480
0.340
0.360
0.580
ERing
0.993
0.989
0.985
0.981
0.993
0.981
0.970
0.963
0.889
0.915
EigenWorms
0.916
0.947
0.908
0.901
0.924
0.916
0.634
0.939
0.779
0.618
Epilepsy
0.894
1.000
0.978
0.986
0.978
0.986
1.000
0.993
0.993
0.964
EchanolConcenttatton
0.494
0.772
0.445
0,460
0.692
0.734
0.791
0.555
0.821
0.323
FaoeDetection
0.684
0.660
0.648
0.653
0.620
0.627
0.656
0.564
0.646
0.529
FingerMovements
0.630
0.530
0.540
0.530
0.600
0.520
0.550
0.560
0.510
0.530
HandMooveeennDirettion
0.527
0.473
0.459
0,473
0.527
0.595
0.446
0.378
0.392
0.189
Handwriting
0.229
0.548
0.567
0.541
0.346
0.356
0.482
0.561
0.288
0.607
Heartbeat
0.785
0.732
0.741
0.741
0.790
0.780
0.722
0.746
0.722
0.717
LSST
0.721
0.643
0.622
0.642
0.356
0.573
0.575
0.570
0.587
0.551
Libras
0.967
0.933
0.900
0.906
0.894
0.911
0.900
0.850
0.861
0.872
MotorImagery
0.630
0.540
0.510
0.530
0.440
0.500
0.610
0.590
0.500
0.500
NATOPS
0.944
0.894
0.883
0.883
0.844
0.856
0.889
0.839
0.872
0.883
PEMS-SF
0.931
1.000
0.821
0.827
1.000
1.000
0.983
1.000
0.971
0.711
PenDigits
0.993
0.979
0.985
0.983
0.977
0.967
0.934
0.935
0.941
0.977
PhonemeSpectra
0.3449
0.290
0.259
0.277
0.308
0.265
0.321
0.246
0.295
0.151
RacketSports
0.941
0.908
0.914
0.901
0.901
0.882
0.888
0.836
0.888
0.803
SelfRegulationSCCPI
0.887
0.891
0.843
0,846
0.877
0.860
0.853
0.812
0.840
0.775
SelfRegnlationSCP2
0.611
0.500
0.494
0.494
0.494
0.500
0.461
0.500
0.533
0.539
StandWalkJump
0.733
0.467
0.600
0.533
0.533
0.400
0.333
0.333
0.467
0.200
UaavGeesturelarreyy
0.963
0.928
0.931
0.928
0.909
0.925
0.891
0.925
0.850
0.903
Average
0.752
0.748
0.721
0.716
0.732
0.726
0.711
0.703
0.701
0.654
D.2
Multivariate Time Series Classification
The full results, in terms of top-1 accuracy, for both supervised and self-supervised pretrained methods on each sub-dataset the UEA benchmark are presented in Table D and Table respectively.
D.3
Few-shot Learning
Figure shows the plot accuracy comparison between the proposed NuTime and a sophisticated meta- learning approach Narwaarya et al. 2020). Additionally, Table 13 provides the full classification results on datasets from the UCR archive for lew-shot learning.
E Attention Mechanism Visualization in NuTime
We provide the visualizations for allention scores between the [CLS] token and other patch tokens of the input series the last layer of NuTime in Figure 10 and to help interpret the learned representations.
22

---

Published in Transactions on Machine Learning Research (07 (07/2024) 2224)
Table 12: Full results for 29 UEA datasets of NuTime and self-supervised counterparts.
Dataset
NuTime
TS2Vec
T-Loss
TNC
TS-TCC
TST
DTW
ArtiealaryVordReeennitoon
0.994
0.987
0.943
0.973
0.953
0.977
0.987
AtrialFibrillatoon
0.347
0.200
0.133
0.133
0.267
0.067
0.200
BasicMotions
1.000
0.975
1.000
0.975
1.000
0.975
0.975
Character
0.994
0.995
0.993
0.967
0.985
0.975
0.989
Cricket
1.000
0.972
0.972
0.958
0.917
1.000
1.000
DeesDuckGeese
0.552
0.680
0.650
0.460
0.380
0.620
0.600
ERing
0.986
0.874
0.133
0.852
0.904
0.874
0.133
EigenWorms
0.910
0.847
0.840
0.840
0.779
0.748
0.618
Epilepsy
0.993
0.964
0.971
0.957
0.957
0.949
0.964
EthanolConcentration
0.466
0.308
0.205
0.297
0.285
0.262
0.323
FaceDetection
0.663
0.501
0.513
0.536
0.544
0.534
0.529
FingerMovements
0.612
0.480
0.580
0.470
0.460
0.560
0.530
HandMoovement
0.532
0.338
0.351
0.324
0.243
0.243
0.231
Handwriting
0.228
0.515
0.451
0.249
0.498
0.225
0.286
Heartbeat
0.784
0.683
0.741
0.746
0.751
0.746
0.717
JapaneseVowell
0.983
0.984
0.989
0.978
0.930
0.978
0.949
LSST
0.693
0.537
0.509
0.595
0.474
0.408
0.551
Libras
0.976
0.867
0.883
0.817
0.822
0.656
0.870
Motorlmagery
0.622
0.510
0.580
0.500
0.610
0.500
0.500
NATOPS
0.940
0.928
0.917
0.911
0.822
0.850
0.883
PEMS-SF
0.925
0.682
0.676
0.699
0.734
0.740
0.711
PenDigits
0.988
0.989
0.981
0.979
0.974
0.560
0.977
PhonemeSpectra
0.320
0.233
0.222
0.207
0.252
0.085
0.151
RacketSports
0.934
0.855
0.855
0.776
0.816
0.809
0.803
SellRegulatiionCCPI
0.899
0.812
0.843
0.799
0.823
0.754
0.775
SellRegulationSCCP2
0.603
0.578
0.539
0.550
0.533
0.550
0.539
SpokentrraicDDggits
0.993
0.988
0.905
0.934
0.970
0.923
0.963
JtandWalkJJum
0.667
0.467
0.333
0.400
0.333
0.267
0.200
UWWveGesturLLibbaayy
0.955
0.906
0.875
0.759
0.753
0.575
0.903
Average
0.778
0.712
0.675
0.677
0.682
0.635
0.650
23

---

Published in Transactions on Machine Learning Research (07/2024)
Table 13: Full results for few-shot learning on 41 UCR datasets.
Dataset
NuTime
FS-1
FS-2
ResNet
BOSS
DTW
ED
Adiac
0.781
0.671
0.674
0.539
0.709
0.540
0.538
Beef'
0.716
0.653
0.595
0.519
0.701
0.626
0.618
BeetleFly
0.809
0.900
0.958
0.702
0.789
0.614
0.667
BirdChicken
1.000
0.929
1.000
0.692
0.921
0.496
0.468
ChlorineConccnntration
0.502
0.329
0.331
0.342
0.356
0.338
0.339
Coffee
0.909
0.978
0.970
0.934
0.977
0.914
0.920
Xricket
0.550
0.594
0.544
0.555
0.491
0.567
0.348
Yricket
0,525
0.562
0.516
0,505
0.461
0.556
0.375
CricketZ
0.530
0.598
0.541
0.523
0.481
0.560
0.357
DistalPPalaarOOttiiegggCoooo
0.738
0.664
0.705
0.709
0.658
0.698
0.710
DisallPaalauxOOttiinCCCrrr
0.643
0.588
0.569
0.609
0.575
0.583
0.571
DistalPhalanxTW
0.508
0.463
0.481
0.476
0.437
0.448
0.444
ECG200
0.699
0.758
0.738
0.712
0.728
0.755
0.771
RCG5000
0.537
0.533
0.548
0.533
0.533
0.494
0.524
RCGFiveDays
0.781
0.939
0.928
0.916
0.909
0.666
0.685
ElevtricDevices
0.569
0.375
0.380
0.381
0.351
0.423
0.239
FaceAll
0.789
0.785
0.712
0.742
0.795
0.764
0.545
FaceFour
0.932
0.934
0.958
0.792
1.000
0.869
0.812
FordA
0.477
0.797
0.777
0.769
0.693
0.541
0.561
FordB
0.733
0.787
0.726
0.692
0.585
0.535
0.515
Insect Wingbeetsat
0.396
0.487
0.452
0.485
0.398
0.473
0.489
Meat
0.885
0.890
0.880
0.559
0.876
0.919
0.919
Medicallmages
0.625
0.592
0.585
0.620
0.488
0.675
0.579
MiddbePhalanxxOnnnnneegggououpp
0.371
0.547
0.515
0.527
0.478
0.558
0.529
MiddeePhallnnxCnntinCCCorrrt
0.631
0.529
0.531
0.540
0.526
0.550
0.563
MiddleePhalanxTW
0.380
0.353
0.351
0.341
0.348
0.339
0.338
PhalangesOuddinesSorreee
0.526
0.539
0.536
0.544
0.512
0.535
0.532
0.813
0.682
0.697
0.729
0.731
0.719
0.692
ProdinalPPhaanndOuthCCCCreet
0.710
0.634
0.638
0.650
0.645
0.626
0.633
ProximalPhalanxTTV
0.471
0.432
0.411
0.517
0.419
0.445
0.427
Strawberry
0.777
0.741
0.755
0.722
0.714
0.671
0.682
SwedishLeaf
0.845
0.776
0.778
0.765
0.776
0.690
0.599
SyntheticControl
0.937
0.971
0.948
0.960
0.867
0.958
0.736
TwoPatterns
0.903
0.831
0.811
0.874
0.692
0.970
0.361
UWaveGeestureLLibaaayxX
0.753
0.606
0.546
0.598
0.479
0.615
0.591
UWaveGGestureLLibaaayy
0.587
0.478
0.430
0.478
0.363
0.518
0.504
UWaveGestuurLLbbraay
0.570
0.599
0.541
0.570
0.489
0.551
0.536
Wafer
0.900
0.892
0.894
0.911
0.936
0.922
0.922
Wine
0.556
0.578
0.631
0.562
0.571
0.493
0.496
Yoga.
0.649
0.528
0.546
0.501
0.548
0.525
0.505
Average
0.675
0,663
0.653
0.627
0.625
0.618
0.566

---

Published in Transactions on Machine Learning Research (07 (07/2024) 2024)
(a) NuTime FS-1 23/0/17
(b) NuTime F5-2 26/1/13
10
1.0
NuTime
NuTime
0.8
is better here
0.8-
is better here
NuTime Accuracy
NuTime Accuracy
0.6
0.4
02
FS-1 Is better here
0.2
FS-2 is better here
00
0.0
FS-1 Accuracy
FS-2 Accuracy
Figure 9: Plot accuracy comparison of NuTime and (a) FS-1, (b) FS-2 on 41 datasets from the UCR archive. Each subfigure's title displays the win/tie//on loss comparison between NuTime and other methods. The two dotted lines indicate the 5% interval.
Beef
EOGHorizontaaSiggall
ElectricDevices
1.0
200
0.8
0-
100
0
1-
0.4
-2
-100
0.2
-3
100
200
300
400
500
100
200
300
400
500
100
200
300
400
500
SmallkitchenAppliances
ProximaPPhanmnOutieeegerrrop
HandOutlines
4-
2-
0.0
Figure 10: Attention scores between the [CLS] token and other patch tokens of the input series in the NuTime model last layer. A higher score indicates that the learned representation more closcly associated with those patches. Six datasets from the UCR archive are randomly selected for visualization.
25

---

Published in Transactions on Machine Learning Research (07/2022 /204)
CG(Positive Initial Return Stroke)
CG(Positive Initial Return Stroke)
CG(Positive Initial Return Stroke)
2-
2-
0-
1000
100
200
INNeatieee Initial Return Stroke)
|R (Regaatvve Initial Return Stroke)
IR(Negative Initial Return Strake)
4-
5.0
0
2.5
02
0.0
0-
0.0
Figure 11: Attention scores between the [CLS] token and other patch tokens of the input series in the NuTime model last layer. These series randomly selected from the LIGHTNING7 dataset. The time series are categorized into two classes: CG (Positive Initial Return Stroke) and IR (Negative Initial Return Stroke). CG-type time series in the first row are characterized by a sudden surge of radiation, followed by brief period of noise. IR-type series in the second row slowly build and then sharply decline in an exponential shape. The visualization implies that the NuTime learned representations which are helpful for classification.
26


Reviews:
[{"review_id": "ICuy3j4wqD", "reviewer": "Reviewer_3K4m", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The proposed module is model agnostic and generic.\n2. The motivation for the research is clear and the paper is well written.\n3. Evaluation experiments have been conducted thoroughly.\n\nWeaknesses\n1. The technical contribution of the proposed method is small.\n2. Contains hyperparameters that are difficult to determine.\n\nDetailed comments:\nThe proposed method is practical but technically incremental. The proposed method is based on the simple idea of computing the representation at each time-window for multiple scales and applying it to the Transformer. The following are questions.\n- How do you optimize the time window and scale candidates? These may have different optimal values for each dataset.\n- Can the proposed method be applied to data with different time lengths and time scales?\n- The response values may take various forms, not only scale. For example, discrete values or categorical features. Is it possible to extend to such cases?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an embedding module for a large number of time series of different scales. The proposed method is based on Transformer and extract features (mean and std, etc.) for each time-window from data. The effectiveness of the proposed method is verified using various datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The novelty of the proposed method needs to be emphasized more or new technical innovations need to be added.\n2. More discussion should be added on the details and limitations of the proposed method (see the above questions).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cOsSBSxPVf", "reviewer": "Reviewer_P6SZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The considered problem is an important step in generalized pre-training of any data that has unbounded range. The authors provide a number of good insights and the exposition is clear. \n\nThe paper suffers from a number of weaknesses that need to be addressed before publication. See requested changes section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission proposes a method for better encoding of time-series data for use in large scale pretraining. The method involves segment-wise normalization (mean subtraction and scaling by the std). To include the information lost via normalization, the authors propose to use a multiscale embedding of these scalar quantities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**On NME:** The author's proposal for the embedding layer is interesting. But the author's ablations do not answer the question of whether or not this is over-engineered. In Eq. 2 the authors show that the effect is very similar to just $\\frac{x}{\\sqrt{c_1 * x^2 + c_2^2}}$. Instead of relying on layer norm and w and b, why not just use a number of these (either randomly samples c_1 and c_2, appropriately in log space or handpicked) as your embedding? \n- Perform ablation tests that show that the complexity of NME is necessary. Or show that NME can be simplified. The ablations that I can think of are 1. y's as described as above but still using the weighted average. 2. Replace weighted averaging with concatenation (just pick a smaller y_i subspace).\n- In Fig. 3, it seems like the layer norm is shared between all z_i, but Eq. 2 implies each is normalized separately which makes more sense. I would suggest breaking up the layer norm block so that it is clear that each z is normalized individually.\n- I like Fig. 3a. It would be instructive to see the same figure but showing a few channels of the scalar value embedding e(x), perhaps both before and after training. For the various models trained with different number of scales. The x axis should be a log axis incorporating the entire dynamic range of data in the training set (E-4 to E4 if I understood correctly)\n\n**On instance-normalization:** A simple way to avoid losing information in instance-normalization is to include the information about the sample (mean and std) in an extra token (e.g. [CLS] [SCALE] etc). In the reviewer's experience, this method is the go-to for dealing with samples that have widely varying ranges. I would suggest that the authors perform this comparison. This would make their paper more useful and convincing for a wider audience.\n\n**On forecasting:** The elephant in the room. As forecasting is an important task and desired feature of timeseries foundation models, I believe a clear discussion as to the challenges of decoding in a high dynamic range dataset would make the paper more useful. It is the experience of the reviewer that increasing dynamic range in the output of a neural network is much more difficult than increasing the dynamic range of the input. I would even suggest including any thoughts/experiments in this direction that did not work, as they would be useful for your audience. Since the predominant mode of pre-training on sequences today is autoregressive or masked modeling, the question of decoding will be present in the minds of a large fraction of your readers.\n\n**Fine Tuning for 100 epochs**: I am a little surprised that the finetuning was done for 100 epochs, same as training. Was this necessary? Can you provide some loss curves or some other indication as to why it was done for so long?\n\nOthers:\n- The authors mention a number of times that they enumerate all possible scales and then mention that this allows them to better generalize to other domains where the scales might be different from the training set. Since all possible scales is an infinite set, I believe the authors mean all scales in the training dataset. As such, the proposed method is still subject to distribution shifts. A test dataset might take values in scales not seen in the training set. I strongly suggest that the authors fix this statement and tone down their claim.\n\n- In the abstract, I suggest changing to something like \"To embed scalar values - that my possess arbitrary numerical amplitudes - in a high-dimensional space\" for better readability.\n\n- Epochs: For training on large datasets, I think it is  preferrable to mention the total number of tokens (or timesteps) along with the number of epochs. This would make it easier to compare how much a network has been trained in papers that use datasets of different size. It will also make it clear whether the 100 epochs of the training are really the same amount of training as the 100 epochs of fine-tuning. \n\n- In implementation details: Specify how you are resizing the sub-sequence to 512. Are you using some kind of spline? For the scheduler, please specify the final value of the learning rate.  For multivariate data, please give more detail as to where exactly the linear layer is added.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g9reBgv1iY", "reviewer": "Reviewer_T9zk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Subjective evaluation:\n\n - I like the simple normalization idea + transformers.  Surprising it beats more engineered approaches.  \n - I like the focus on normalization and pre-processing, as something that can be isolated, and still take advantage of big hammers like transformers.  \n\n## TMLR criteria evaluation:\nOverall it approaches a problem of clear relevance to TMLR, and general importance.  The solution is clear, simple, and effective.  The methods are described clear enough, and the evaluation on standard benchmarks holds up, and shows state of the art performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new self-supervised method for time series analysis, called NuTime.  It leverages a standard and generic transformer network, with the key addition being how the data is pre-processed before feeding it into the network.  It chunks the time series into non-overlapping chunks, and each chunks sends in a normalized shape (the chunk of values after standard (x-mean)/std normalization), a mean value, and a standard deviation.  This removes some scale notions from the \"shape\", but also passes it to the transformer network separately.  \n\nWithin this, there is a parameter k, which is used to regularize the normalization (its square is added to the variance in the denominator).  This parameter seems to be needed, but can be set in a wide range.  \n\nThe results beats state of the art on several benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "###  Smaller writing suggestions and questions:\n\n1. While the evaluation measures seem to be standard data sets and tasks, they are really treated as a black box in this paper.  It would have been useful to explain a bit about what sorts of things are being classified, and why classification is a useful task in these settings.  \n\n 1.  BYOL seems to be an integral enough part of your approach.  I suggest to elaborate just a bit on how it works, and perhaps what the acronym means (although I guess just knowing that expansion does not help the reader too much).  \n\n 2.  Why did you choose to use **non**-overlapping windows, instead of shingled windows?  This is worth addressing in the paper.   \n\n 3. How do you know you have right range and resolution on k  (in [10^-4, 10^-3, .., 1, .., 10^4])?  I see the ablation study, but based on Fig 3(a), it seems the effect can be pretty large between values of k that you tried.  \nIs k=0 represented here (is that what Num Scales = 0 means?).  Please expand on this a bit more.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper was reviewed by three reviewers. After rebuttal, two reviewers were positive (accept and leaning accept), and one reviewer was negative (leaning reject). The negative reviewer cited lack of technical novelty, but acknowledged that the paper "may be practically useful, so it might be accepted." Based on the guidelines of TMLR, I think the paper can be accepted despite its apparent lack of novelty. I recommend "Accept with minor revision" because I hope the authors can release their code as suggested by Reviewer P6SZ. 

The reviewers initially had concerns about hyperparameter tuning, ablation studies, forecasting tasks, finetuning, etc. These concerns were addressed by the rebuttal.
Decision: Accept (poster)