Segment, Shuffle, and Stitch: A Simple Layer for Improving Time-Series Representations
Authors: Shivam Grover
Amin Jalali
Ali Etemad
Venue: NeurIPS 2024 poster (2024)
Keywords: Time Series
Deep Learning
Representation Learning
Temporal Mechanism
Primary area: deep_learning_architectures
TL;DR: This paper presents S3, a modular neural network layer that is designed to enhance time-series representation learning by rearranging its segments, yielding improved results across various tasks with minimal computational overhead.

Abstract: Existing approaches for learning representations of time-series keep the temporal arrangement of the time-steps intact with the presumption that the original order is the most optimal for learning. However, non-adjacent sections of real-world time-series may have strong dependencies. Accordingly, we raise the question: Is there an alternative arrangement for time-series which could enable more effective representation learning? To address this, we propose a simple plug-and-play neural network layer called Segment, Shuffle, and Stitch (S3) designed to improve representation learning in time-series models. S3 works by creating non-overlapping segments from the original sequence and shuffling them in a learned manner that is optimal for the task at hand. It then re-attaches the shuffled segments back together and performs a learned weighted sum with the original input to capture both the newly shuffled sequence along with the original sequence. S3 is modular and can be stacked to achieve different levels of granularity, and can be added to many forms of neural architectures including CNNs or Transformers with negligible computation overhead. Through extensive experiments on several datasets and state-of-the-art baselines, we show that incorporating S3 results in significant improvements for the tasks of time-series classification, forecasting, and anomaly detection, improving performance on certain datasets by up to 68\%. We also show that S3 makes the learning more stable with a smoother training loss curve and loss landscape compared to the original baseline. The code is available at https://github.com/shivam-grover/S3-TimeSeries.

Segment, Shuffle, and Stitch: A Simple Layer for Improving Time-Series Representations
Amin Jalali Queen's University, Canada {shivam -Shivam.grover, grover, amin. jalali, ali eteeemtdeeueeeesuuc..
Shivam Grover
Ali Etemad
Abstract
Existing approaches for learning representations of time-series keep the temporal arrangement of the time-steps intact with the presumption that the original order is the most optimal for learning. However, non-adjacent sections of real-world time-series may have strong dependencies. Accordingly, we raise the question: Is there an alternative arrangement for time-series which could enable more effective represenlation leurning? To address this, we propose a simple plug-and- play neural network layer called Segment, Shufile, and Stitch (S3) designed improve representation learning in time-series models. S3 works by creating non- overlapping segments from the original sequence and shuffling them in learned manner that is optimal for the task at hand. It then re-attaches the shuffled segments back together and performs a Icarned weighted sum with the original input to capture both the newly shuffled sequence along with the original sequence. S3 is modular and can be stacked to achieve different levels of granularity, and can be added to many forms of neural architectures including CNNs Transformers with negligible computation overhead. Through extensive experiments on several datasets and -ftth- baselines, significant improvements for the tasks of time-series classification, forecasting, and anomaly detection, improving performance
show that incorporating S3 results in
certain datasets by up to 68%. We also show that S3 makes the learning more stable with a smoother training loss curve and loss landscape compared the original baseline. The code is available hwt.ggihhhh....miieeeemmmeee
at https://github.. com /hiin/shimmm--aa www.ssagrrooor//SS----
Introduction
Time-series data serve an important role across diverse domains, including but not limited to health
human-compute computer interaction [4, 5], human activity recognition [6, 7]. climate analysis [8, 9], energy consumption [10, 11, 12]. traffic management [13, 14]. financial markets [15], and others. The pervasive nature of time-series data has resulted in considerable interest among researchers, leading to the development of a variety of deep learning solutions for classification [16, 17, 18] and forecasting tasks [10, 19, 20J. Neural architectures such as convolutional networks [16, 21], recurrent networks [22, 23], and Transformers [10, 20] can capture the essential spatial and temporal information from time-series. Notably, these approaches have frequently outperformed traditional approaches, including Dynamic Time Warping [24]. Bag of Stochastic Frontier Analysis Ensembles [26] in various scenarios.
analytics [1, 2, 3]. human-
Symbols [25], and the Collective of Transformation-Baan
While both traditional machine learning and deep learning solutions aim to extract effective goal- related representations prior to classification or forecasting, the general approach original temporal arrangement of the time-steps in the time-series intact, with the presumption that the original order is the most optimal. Moreover, most existing models do not have explicit mechanisms to explore the inter-relations between distant segments within each time-series, which may in fact have strong dependencies despite their lack of proximity. For example, CNN-based models for
keep the
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

time-series learning generally utilize fixed convolutional filters and receptive fields, causing them to only capture patterss within limited temporal window [27, 28]. As a result, when faced with time-series where important patterns or correlations span across longer time windows, these models often struggle to capture this information effectively [28]. Dilated convolutional neural networks partially solve this by increasing the receptive field through the dilation rate. However, they are still practically limited by their inherent architectures, as their receptive fields rely on the number of layers, which may not be large enough to fully capture the long-range dependencies and can lead to vanishing gradients as more layers are added [29]. Similarly, the out-of-the-box effectiveness of Transformers [30] in capturing long-ten dependencies highly depends on variety of factors such as sequence length. positional encoding, and tokenization strategies. Accordingly, we ask a simple question: Is there a better arrangement for the time-series that would enable more effective representation learning considering the classifccationforeeaaiii task at hand?
In this paper, we introduce simple and plug-and- network layer called Segment, Shuffle, and Stitch, or S3 for short, designed to enhance time-series representation learning. As the name suggests, S3 operates by segmenting the time-series into several segments, shufilingg these segments in the most optimal order controlled by learned shuffling parameters, and then stitching the shuffled segments. In addition to this, our module integrates the original time-series through a learned weighted sum operation with the shuffled version to also preserve the key information in the original order. S3 acts as modular mechanism intended to seamlessly integrate with any time-series model and as will demonstrate experimentally, results in a smoother training procedure and loss landscape. Since S3 is trained along with the backbone network, the shuffling parameter is updated in goal-centric manner, adapting to the characteristics of the data and the backbone model to better capture the temporal dynamics. Finally, S3 can be stacked create a more fine ene-grained shuffling with higher levels of granularity, has very few
to tune, and has negligible computation overhead.
For evaluation, we integrate S3 in variety of neural architectures including CNN-based and Transformer-basee models, and evaluate performance across various classification, univariate fore- casting, and multivariate forecasting datasets, observing that S3 results in substantial improvements when integrated into state-of-the. art models. Specifically, the results demonstrate that integrating S3 into state-of-thed thettartt methods can improve performance by up to 39.59% for classification, and by up to 68.71% % and 51.22% for univariate and multivariate forecasting, respectively. We perform detailed ablation and sensitivity studies to analyze different components of our proposed S3 layer.
Our contributions are summarized as follows:
We propose S3, a simple and modular network layer that can be plugged into existing neural architectures to improve time-series representation learning. By dynamically segmenting and shuffling the input time-series across the temporal dimension, S3 helps the model perform more effective task-centric representation learning.
By stacking multiple instances of S3, the model can perform shuffling at different granularity levels. Our proposed layer has very few hyperparameters and negligible added computational cost.
Rigorous experiments on various benchmark time-series datasets across both classification and forecasting tasks demonstrate that by incorporating S3 into existing state-of- wate-of-the--a models, we improve performance significantly. Experiments also show that by adding our proposed network layer, more stable training is achieved.
We make our code public to contribute to the field of time-series representation learning and enable fast and accurate reproducibility.
2 Related work
Deep learning architectures have recently made significant progress in the area of time-series rep- resentation learning. In the category of convolution- methods, DSN [16] introduces dynamic sparse connections to cover different receptive fields in convolution layers for time-series classifica- tion. In another convolution-bast based approach, SCINet [21] captures temporal features by partitioning each time sequence into two subsequences at each level to effectively model the complex temporal dynamics within hierarchical time-series data.
Transformer-basee based approaches are another category of solutions that have drawn considerable atten- tion. Informer [10] improves the capabilities of the vanilla Transformer on long input sequences by
2

---

introducing self-attention distillation using ProbSparse which is based on Kullback- diver- gence, and a generative style decoder. Autoformer [20] introduces decomposition blocks with an auto-correlation mechanism that allows progressive decomposition of the data. ContiFormer [31] integrates the continuous dynamics of Neural Ordinary Differential Equations with the attention mechanism of Transformers. PatchTST [32] enhances time-series forecasting by leveraging patch- ing techniques to split long input sequences into smaller patches, which are then processed by Transformer to capture long-term dependencies efficiently. Finally, [33] investigated the effective- ness of Transformers in dealing with long sequences for forecasting and highlighted the challenges Transformers encounter in this regard.
Recently, foundation models for time-series data, which are large-scale, pre- pre- traited models designed to capture complex temporal patterns, have gained popularity. A prominent example is Moment [34], which pre-trains Transformer encoder using a univariate setting on large and diverse collection of data called 'Pile". It leverages a masked reconstruction method and is capable of performing various downstream forecasting tasks after fine-tuning, demonstrating strong adaptability and forecasting accuracy.
Contrastive methods have recently demonstrated state-of-the-arr performances in time-series learning. TS2Vec [17] employs unsupervised hierarchical contrastive learning across augmented contextual views, capturing robust representations at various semantic levels. InfoTS [35] uses information- aware augmentations for contrastive learning which dynamically chooses the optimal augmentations. To address long-tere forecasting, [28] introduces contrastive approach that incorporates global autocorrelation alongside decomposition network. Finally, SoftCLT [36] is recent contrastive approach that uses the distances between time-series samples (instance-wer contrastive) along with the difference in timestamps (temporal contrastive), to capture the correlations between adjacent samples and improve representations.
Given the existence of seasonal and trend information in time-series, disentanglement is an approach that has been widely used across both classical machine learning [37, 38, 39] and deep learning solutions [40, 19]. For instance, CoST [40] attempts to capture periodic patterns using frequency domain contrastive loss to disentangle seasonal-trend representations in both time and frequency domains. Similarly, LaST [19] recently employed variational inference to learn and disentangle latent seasonal and trend components within time-series data.
3 Method
X
Problem definition. Given a set of N time- series instances as {x; } where x; E T has length of T and C channels, the goal is to optimally rearrange the segments of x; and form new sequence x, to better capture the underlying temporal relationships and depen- dencies within the time-series, which would consequently lead improved representations given the target task.
layer
Segment
Shuffle
weighted sum
Sticch
Segment
Learned weighted
Shuffle
Stich
Proposed mechanism. We propose S3, a simple neural network component for address- ing the aforementioned problem in three steps, as the name suggests, Segment, Shuffle, and Stitch, described below (see Figure 1).
Segment
Learned weighted
Shuffle
Stitch
The Segment module splits the input sequence x; into n non-overlapping segments, each containing 7 time-steps, where 7 T T/n. The set of segments can be represented by S, = St,,,, S E R(rxC)xn where S. = x 1)r jrl and S. E T
Figure 1: Stacking S3 layers. In this depiction, use n = 2, Φ 3, and 2 as the hyperparame- ters.
The segments are then fed into the Shuffle module, which uses a shuffle vector P {p, . optimal order for the task at hand. Each shuffling parameter P, in corresponds to segment s.
...)) E R rearrange the segments in the

---

in S,. P is essentially set of learnable weights optimized through the network's 's learning process, which controls the position and priority of the segment in the reordered sequence. The shuffling process is quite simple and intuitive: the higher the value of Py. the higher the priority of segment S.j in the shuffled sequence. The shuffled sequence Sphutlled can be represented as
Sphuflldd Sort(S, key P),
(I)
where the segments in S, are sorted according to the values in P. Permuting S, based on the sorted
order of P not differentiable by default, because it involves discrete operations and introduces discontinuities [41]. Soft sorting methods such as [42, 41, 43J approximate the sorted order by assigning probabilities that reflect how much larger each element is compared to others. While this approximation is differentiable in nature, it may introduce noise and inaccuracies, while making the sorting non-intuitive. To achieve differentiable sorting and permutation that are as accurate and intuitive as traditional methods, we introduce a few intermediate steps. These steps create a path for gradients flow through the shuffling parameters P while performing discrete permutations on S; based on the sorted order of P. We first obtain the indices that sort the elements of P using (rgsort(P). (P). We have a list of tensors = [$1 $2, S3, Sn. (for simplicity, we skip the index i) that we aim to reorder based on the list of indices o = 10 .2..... in differentiable way. We then create (T x C) X n. X n matrix U, which we populate by repeating each 8;, a times. Next, we form an n X n matrix Ω where each row j has a single non-zero element at position k o, which is Pk- We convert Ω binary matrix f by scaling each non-zero element using scaling factor Fy.a This process creates a path for the gradients to flow through P during backpropagation.
By performing the Hadamard product between and 5 we obtain a matrix V where each row j has one non-zero element i equals sg. Finally, by summing along the final dimension and transposing the outcome, we obtain the final shuffled matrix of size Syhufited E Rxxxxx To better illustrate, show simple example with S calculate V as
[si, 82, 83, s4] and given permutation
[3, 4, 1, We
81
82
83
84
0
0
S3
V U
S1
S2
82
$3
83
84
51
0
1
0
0
0
1
=
S1
0
0
0
0
84
(2)
52
$3
$4_
0
1
0
0_
0
B2
0
0_
and Sphuffled is obtained by
Sshafffed
CV (V*,)))T=
[ss
S1
S1
S2
We previously assumed that the set of shuffling parameters P is one-dimensional, containing scalar elements, each corresponding to one of the segments. By employing a higher-dimensional P, introduce additional parameters that enable the model capture complex representations that a single-dimensional P could struggle with. Therefore, we introduce a hyperparameter À to m, the size of P becomes n x n
determine the dimensionality of the vector P. When (repeated m times). We then perform a summation of P over the first m 1 dimensions to obtain a one-dimensional vector. Mathematically, this represented
P
Σ di=
Σ P Pd..de...dmm- dm-1=1
d2
This results a one-dimensional matrix P, which we then use to compute the permutation indices a = Argsort(P)) (P). This approach allows us to increase the number of shuffling parameters, thereby capturing more complex dependencies within the time-series data, without affecting the sorting operations.
In the final step, the Stitch module concatenates the shuffled segments Sphuilled to create a single shuffled sequence x; E TTC as X, Concat(Srhuniag To retain the information present in the original order along with the newly generated shuffled sequence, we perform a weighted sum between and w2 optimized through the training of the main network.
x; and x, with learnable weights For practical convenience, we can use a simple learnable Conv ID or MI.P layer taking x; and x; as inputs and generating the final time-series output x E T

---

Stacking S3 layers. Considering S3 as a modular layer, we can stack them sequentially within neural architecture. Let's define @ as a hyperparameetrr that determines the number of S3 layers. For simplicity and to avoid defining separate segment hyperparameter for each S3 layer, we define 0 which acts as a multiplier for the number of segments subsequent layers as
Ybg = 12 x 0--1 for 8 = 1.2,....
(5)
where n is the number of segments in the first S3 layer. When multiple S3 layers are stacked, each layer & from to é will segment and shuffle its input based on the output from the previous layer. We can formally represent the output of each layer l as x'e by
x
Pe); (WE,1, we,2)), for 1omf=11. 11,......
(6)
where X/ is the original time-series x, Py is the set of shuffling parameters for layer f, and (We,1, We,2) are the learnable weights for the sum operation between the concatenation of the shuffled segments and the original input, at layer f. Figure presents an example of three S3 layers (ø- = 3) applied to a time-series with n = 2 and 0 = 2.
All the Py values are updated along with the model parameters, and there is no intermediate loss for any of the S3 layers. This ensures that the S3 layers are trained according to the specific task and the baseline. In cases where the length of input sequence is not divisible by the number of segments n, we resort to truncating the first I mod n time-steps from the input sequence. In order to ensure that no data is lost and the input and output shapes are the same, we later add the truncated samples back at the beginning of the output of the final S3 layer.
4 Experiment setup
Evaluation protocol. For our experiments, we integrate S3 into existing state-of-ther offth--r-the-rt models for both time-series classification and forecasting. We first train and evaluate each model adhering strictly to their original setups and experimental protocols. We then integrate S3 at the beginning of the model, and train and evaluate it with the same setups and protocols as the original models. This meticulous approach ensures that any observed deviation in performance can be fairly attributed to the integration of S3. For classification, we measure the improvement as the percentage difference (Diff.) in accuracy resulting from S3, calculated as as (Accaasline+S3 AcCBaseline ) /CCcBaslinee For forecasting, since lower MSE is better, we use (MS BBsseline - MS / / S.M.Eeeaiiin Baseelnne A similar equation is used for measuring the percentage difference in MAE.
Classification datasets. For classification we use the following datasets: (1) The UCR archive [44] which consists of 128 univariate datasets, (2) the UEA archive [45] which consists of 30 multivariate datasets, and (3) three multivariate datasets namely EEG, EEG2, and HAR from the UCI archive [46]. For our experiments with pre-trained foundation model in Section 5, we also use the PTB-XL [47] dataset. The train/test splits for all classification datasets are as provided in the original papers.
Forecasting datasets. For forecasting, we use the following datasets: (1) both the univariate and multivariate versions of three ETT datasets [10], namely ETTh1 and ETTh2 recorded hourly, and ETTml recorded at every 15 minutes, (2) both the univariate and multivariate versions of the Electricity dataset [46], and (3) the multivariate version of the Weather dataset [48].
Anomaly detection datasets. We employ the widely used Yahoo [49] and KPI [50] datasets for anomaly detection tasks. The Yahoo dataset consists of 367 time-series sampled hourly, each with labeled anomaly points, while the KPI dataset contains 58 minutely sampled KPI curves from various internet companies. Our experiments are performed under normal settings, as outlined in (17,36)
Baselines. We select basclines from a variety of different time-series learning approaches. Specifically, for classification, we use four -tate-of-the-arr baseline methods, SoftCLT [36], TS2Vec [17], DSN [16], and InfoTS [35]. For forecasting, we use five state-of-the-an baseline methods, TS2 [17], LaST [19], Informer [10], PatchTST [32] and CoST [40]. For anomaly detection, we use two state- of-the-art baseline methods, SoftCLT [36] and TS2 [17]. Additionally, we use MOMENT [34] for experiments with foundation model. For each baseline method, integrate one three layers of S3 at the input level of the model and compare its performance with that of the original model.
Implementation details. All implementation details match those of the baselines. We used the exact hyperparameters of the baselines according to the original papers when they were specified in the papers when the code was available; alternatively, when the hyperparameters were not exactly
5

---

specified in the paper or in the code, we tried to maximize performance with our own search for the optimum hyperparameters. Additionally, for experiments involving PatchTST and LaST with the Electricity dataset, we use batch size of 8 due to memory constraints. Accordingly, some baseline results may slightly differ from those available in the original papers. Note that deviations in baseline results affect both the baseline model and baseline+S3. For the weighted sum operation in S3, we use Conv ID. Our code is implemented with PyTorch, and our experiments are conducted on a single NVIDIA Quadro RTX 6000 GPU. We release the code at: wttp:/iiiww..oo///aiiameeaeeeeee TimeSeries.
Results
Classification. The results of our experiments on time-series classification are presented in Table 1. In this table we observe that the addition of S3 results in substantial improvements across all baselines for all univariate and multivariate datasets. For UCR, UEA, EEG, EEG2, and HAR, we achieve average improve- ments of 3.89%, 5.25%, 32.03%, 9.675%. and 5.18% respectively over all models. The full classification results on UCR and UEA datasets for all baselines with and without S3 are mentioned in Appendix (Ta- ble Al and A2). These results highlight the efficacy of S3 in improving variety of different classification methods over diverse set of datasets despite negli- gible added computational complexity (we provide a detailed discussion on computation overhead later in this section). In addition to quantitative metrics, we also visualize the t-SNE plots for several datasets in Figure 2 and Appendix A1, which show that adding S3 results in representations with better class separability in the latent space.
Table 1: Comparison with baselines on time-
series classification
Method
UCR
UEA
EEG
EEG2
HAR
TS2Vec
0,819
0.695
0.593
0,845
0.930
S32
0.851
0.720
0.672
0,9733
0.935
3.90%
3.63%
19.47%
15.18%
11.31%
DSN
0.794
0.687
0.516
0.961
0.957
DSN+S3
0,833
0.736
0.717
0,980
0.971
Diff.
4.85%
7.24%
38, 38.90%
1.93%
1.41%
InfoTS
0.733
0.691
0.515
0.822
0.876
InfoTS+33
0.760
0.727
0.719
0.947
0.929
Dif..
3.71%
5.34%,
39.59%
15.21%
6.05%
SofCLT
0.825
0.699
0,516
0.893
0.918
Sof
0,845
0.734
0.672
0.950
0.936
DHT.
2.42%
4.76%
30.16%
6.38%
1.96%
Forecasting, Table 2 presents a comprehensive overview of the results for univariate forecasting on different datasets and horizons (H), with and without the incorporation of S3. We observe that S3 consistently leads to improvements for all baseline methods across all datasets, with an average improvement in MSE and MAE of 4.58% and 4.20% for Tee,ec, 4.02% and 3.01 % for LaST, 16.69% and 8. 8.9% for Informer, 1.64% and 1. 1.20% for PatchTST, and 3.01 % and 2.62% for CoST. Figure shows two forecasting outputs for Informer with and without S3, on two different horizon lengths for a sample from the ETTh dataset. In both cases, S3 improves the ability of the baseline Informer to generate time-series samples that better align with the ground truth. Similarly, Table 3 presents the results of multivariate forecasting. Consistent with univariate forecasting, S3 significantly enhances the performance of the bascline on multivariate forecasting and achieves average improvements in MSE and MAE of 13.71% % and 7.78% for TS2Vec, 9.88% and 5,45% for LaST, 25.13% and 14.82% for Informer, 1.62% and 1.17% for PatchTST, and 4.41% % and 1.97% for CoST.
Anomaly detection. Table 4 presents the results for anomaly detection on Yahoo and KPI datasets. The anomaly score is computed as the LI distance between two encoded representations derived from masked and unmasked inputs, following the methodology described in previous studies [17, 36]. We observe that the proposed S3 layer enhances the performance in terms of Fl for both datasets under the normal settings, with scores 0.7498 and 0.6892, respectively.
Loss behaviour. We make interesting observations while training the baseline models as well as the baseline+S3 variants. First, we observe that the training loss vs. epochs for baseline+S3 variants generally converge faster than the original basclines. See Figure 4 where we demonstrate examples of this behavior. Second, we observe that the training loss curves for baselinc+S3 are generally much smoother than the original baselines. This can again be observed in Figure 4. Additionally, we measure the standard deviations of the loss curves for SoftCLT with and without S3 on all the UCR datasets and measure an average reduction in standard deviation of 36.66%. The detailed values for the standard deviations of the loss curves are presented in Appendix A4. Lastly, according to [5 we investigate the loss landscape of the baselines and observe that the addition of S3 generally results in smoother loss landscape with fewer local minima (Figure 5). Additional examples are provided in Appendix A2.

---

Table 2: Univariate forecasting results for different baselines with and without S3.
TS2 +S3
LaST
Informer +S3
PStebTTT
CoST
Dataset
H
Metric
Base
Dif..
Base
+53
Dift.
Base
Dift.
Base
+53
Dift.
Base
+S3
Dift.
24
MSE
0.0449
0.0370
9.40%
0.0295
0,0293
0.80%
0,1169
0.0851
27.18%
0.0265
0.0258
2.57%
0.0386
0.0377
2.22%
MAF
0.1614
0.1459
9.58%
0.1297
0.1291
0.45%
0.2753
0.2263
17.76%
0.1237
0.1232
0.38%
0.1501
0.1486
0.99%
48
MSE
0.0714
0.0605
15.24%
0.0479
0.0464
3.07%
0.1358
0.1132
16.57%
0.0397
0.0389
1.99%
0.0592
0.0565
4.47%
MAE
0.2035
0.1884
7.45%
0.1630
0.1628
0.11%
0.2959
0,2727
7.81%
0.1503
0.1478
1.67%
0.1855
0,1818
2.02%
FT
168
MAE
MSE
0.1362
0.3235
0.1202
0.2663
17.65%
11.71%
0,0765
0.2071
0.0731
0.2069
4.40%
0.10%
0.3207
0,2434
24.10%
0,0681
0.1999
0.1982
0,0672
0.36%
1.28%
0.2322
0.2221
4.47%
4.38%
336
MAE
MSE
0.1699
0.3333
0.1426
0.2949
16.09%
0.0915
0.2409
0.0920
0.2366
2.89%
1.80%
0.1411
0.2990
0,2532
0.1021
15.30%
57.59%
0.2034
0.0820
0.0806
0.1974
2.21%
1.71%
0.2557
0.1083
0.1010
0.2440
3.92%
4.56%
MSE
0.1739
0.1734
0.29%
0.1417
0.1370
3.33%
0,269
0.0841
68.71%
0.0866
0.0855
1.24%
0.1297
0.1211 3,841
720
MAE
0.3307
0.3389
-1.58%%
0.3098
0.2989
3.50%
0.2159
0.1980
8.27%
0.2132
0.2098
1.59%
0.2840
0.2782
2.02%
24
MAE
MSE
0.0973
0.2363
0.0938
0.2344
3.49%
0.82%
0.0708
0,1956
0.0675
0.1901
4.58%
3,20%
0.0851
0.2230
0.2210
0.0844
0.75%
1.02%
0.0704
0,2036
0,0702
0,2012
0.34%
1.19%
0.0784
0.2059
0.2033
0.0766
2.37%
1.27%
MSE
0.1269
0.1242
1.77%
0.0995
0.0939
5.72%
0.1461
2.77%
0.
0.0963
0,0943
2.08%
0.1
0.1072
7.84%
48
MAF
0.2774
0.2761
0.72%
0,2394
0.2279
4.80%
0,30%
0.2990
1-249
0,24100
0,3410
0.445%
0.2589
0.2520
2.65%
ETTh2
336
168
MSE
MAE
MAR
MSE
0,2031
0.1974
0.36:38
0,3562
0.3551
0.1953
0.34477
0.1853
3.86%
6.12%
2.41%
3.50%
0,2213
0.3819
0.1796
0,3194
0.2049
0.3098
0.3579
0.1680
3 3.025%
7.39%
6.28%
6.40%
0.2637
0,4545
0,4133
0.3177
0:2295
0,2470
0,4433
0:3756
092999
5.79%
6.34%
2.44%
031184
0,1004
8.3370
0.58%
0.59%
1.12%
1.50%
0.1963
0,1805
0,3548
0.3389
0.1949
01316)
0.1677 7.10%
0.3542
0,74%
5.335
0.185
720
MSE
0.2081
0.1983
4.82%
0,2754
0.2712
1.53%
0.2678
0.2548
1.58%
0,2236
0,2302
1.52%
0.2018
0.1962 1.19%
MAE
0.37119
0.3620
2.36%
0.4299
0.4215
1.91%
0:4199
0.3975
5.34%
0,3549
0,3483
1.86%%
0,3664
0.3550
3.10%
24
MSE
000055
0.0152
0.0231
0.0229
16.22%
0.0246
0,0228
7.13%
0.0099
0.0097
1.55%
0.0146
0,0132
9.44%
MAE
0.0910
0.0928
-1.96%
0.1070
0.0982
8.25%
0,1111
0.1079
3.35%
0,1123
0.11005
1.60%
0.0890
0,08499
3.59%
48
MSE
0.0283
0.0290
-2.44%
0.0479
0.0465
3,08%
0.0561
0.0501
10.76%
0.0169
0.0167
1.18%
0.0252
0.0233
7.35%
MAE
0.1304
0.1350
-3.52%
0,1580
0,1517
3,99%
0.1719
0,1632
6.70%
0,1189
081180
0
0.1117
0.1139
1.01%
ETTml
96
MAE
MSE
0,1713
0.0480
0.0458
0.1643
4.07%
4.41%
0,2071
0.0765
0.1967
0.0715
5.01%
6.53%
0.2122
0.1124
0,2121
0,2420
0,1100
2.15%
0,1389
0.0362
0.0359
0.36%
0.90%
0.1162
0.0378
0.1391
0.0346
1.88%
8.64%
228
MSE
0.0992
0.0953
3.92%
0.1058
0.0853
9.949
0.1833
0.1762
3.87%
0.0627
0.5518
1.75%
0.0728
0,0669
8.10%
MAP
0.2495
0.2389
5.46%
0,2450
0.2231
10:40%
0.3968
0.3209
435%
0,1578
0.1556
0.2051
0,1946
5.11%
672
MAE
MSE
0,1568
0,3100
0.1529
0.3028
2.50%
2,33%
0,1699
0,3165
0,1589
0.3089
6.47%
2.75%
0.2686
0,4212
0,2582
0/4166
3.88%
1.10%
0,2072
0.0744
0.0733
0.2035
1.79%
1.67%
0,2486
0.1053
0.0969
0.2345
7.97%
5.66%
24
MSF
00.2546
0.2460
3.38%
0.1525
0.1486
2.58%
0,1937
0.1778
8. 8.18%
0.1434
0.1408
1.76%
0.2420
0.2369
2. 2.12%
MAE
0.2846
0.2710
4.75%
0,2797
0,2750
1.6%
0.3155
0.3053
3.15%
0.2713
0,2676
1.37%
0.2628
0.2604
0.91%
48
MSF
0,3001
0.2923
2.60%
0,1888
0,1856
1, 1,099
0:27600
0.2129
866
0.1765
0.1749
0.9115
0:2912
0.2888
0.81%
MAE
0.3151
0.3011
4.43%
0,3086
0.5023
2.03%
0.3794
0.3308
12.82%
0.2965
0.2962
0.79%
0.2977
0.2957
0.65%
Electricity
168
MSH
0143033
0.4055
0,2473
0.2415
2.35%
0.34012
0.3202
0,7484
01,2466
1.44%
1.58%
MAE
0.3872
0,3728
3.71%
0.3506
0.3423
2,36%
0,1292
0,4013
5.81%
0,3470
0,3156
0.38%
0.3781
0.3750
0.83%
336
MSE
0,5636
0.5383
4.49%
0,2907
0.2801
3.63%
0,4062
0,3520
13 34%
0, 0,2963
0,2925
1.26%
0.56M0
0,5563
2.22%
MAE
0.1712
0.1563
3.17%
0
0.3793
1.00%
0.163
10.02%
0,3868
0,3808
1.55%
0.1693
0,1620
L.57%
720
MSE
0.8641
0.8444
2,27%
0.3239
0.3180
1.82%
0,4799
0,3484
27.411
0.3393
0.3296
2.86%
0.8965
0.8783
2.03%
MAE
0.6538
0.6345
2.94%
0.4229
0.4197
0.76%
0.5126
0.4257
16.95%
0.4311
0.4237
1.70%
0.6471
0,6345
L.95%
Average
MAE
MSE
0,2093
0
0.2884
0.1997
4.58%
4.20%
0,1433
0.2700
0.1376
0.2619
3.01%
1.02%
0.3303
0,2124
0.3032
0.1769
16. 69%
8.1
0.2408
0.1216
0.1196
0.2380
1.64%
1.20%
0.2744
0.1933
0.1873
0.2670
2.62%
3.01%
10.0
TS2Vcc
10
T52Vec+53
TS2Vec
T52Vec+53
20
5.0
10
2.5
-2.
-10
-10
-5.0
(a) BodgerLLoopWcokeeee
(b) Chinatown
Figure 2: t-SNE visualizations of the learned representations of TS2Vec and TS2Vec+S3 for 4 ran- domly chosen test sets. Different colors represent different classes. It can be seen that representations belonging to different classes are more separable after adding $3.
Ablation. We perform ablation studies to investigate the impact of cach of the key modules of S3 and answer three questions: (1) Do we need the Segment module to divide the input into multiple segments, or will shuffling all the time-steps individually work just as well? (2) Do we need learned Shuffling parameters (P) to determine the optimal permutation of the time-series, or will random permutations work just as well? (3) Do we need the Stitch module to apply a learned weighted sum of the original sequence with the shuffled sequence, or will the shuffled sequences alone work just as well? Table 7 shows the results of three ablation studies where we observe that the removal of each component of S3 results in decline in performance.
Foundation models. In order to evaluate the impact of S3 on pre-trained foundation model, we integrate S3 at the beginning of the pre-trained MOMENT encoder [34]. We use linear probing for
7

---

Table 3: Multivariate forecasting results for different baselines with and without S3
VSZVNe +S3
LaST +S3
Informer +S3
PatchTST +S3
CoST +S3
Dalasel
H
Metric
Base
DifL.
Base
Dift.
Base
DiIr.
Base
Diff.
Base
Dift.
MSE
0.5360
0,5018
5.82%
0.3230
0.3349
-0.55%
0.5334
0.4928
7.79%
0.3297
0.3253
1.34%
0.3750
0,3719
0.84%
MAE
0.5026
0.4866
3.19%
0.36.2
0.3654
-0.61%
0.5253
0.49339
5,97%
0.3722
0.3672
1.36%
0.423%
0.4200
0.90%
48
MSE
0.5199
0.5907
0.5160
0.5596
6.17%
5.27%
0,4061
0.3804
0.3513
0,3792
6.69%
7.65%
0.6621
0.7471
0.6228
0.6738
5.92%
9.80%
0.3865
0.3532
0.3851
0.3520
0.29%
0.33%
0.1609
0.4250
0.1549
0.4252
1.30%
0.65%
ETTT
168
MSE
0.7498
0.7016
6.43%
0.5149
0.4633
10.02%
1.0853
0.9689
10.72%
0.4078
0.3989
2.18%
0,6303 0.6201
1 1.62%
MAR
0.6335
0.6011
6.43%
0.4906
0.4497
8.32%
0.8907
0.7828
5.72%
0:4212
0.409%
2.71%
0.5830
0,5755
1 1.29%
336
MAE
0.8841
0.7049
0.8631
0.6856
2.38%
2.74%
0.7102
0.6182
0,5828
0,5308
17.93%
14.10%
0.9550
1.3561
0.8339
1,0734
20,815%
6.61%
0,4319
0.4368
0,4254
0.4337
1.51%
0.70%
0.7898
0.6655
0.6586
0.7842
0.71%
1 1.06%
720
MSE
1.0471
1.0441
0.30%
0.9354
0.7765
16.98%
1,3860
1.2599
9.11%
0.4498
0.4367
2.90%
0.879
0,8740
0.63%
0.7884
0.7798
1.09%
0,7470
0 0.6652
11.00%
0.9575
0.9031
5.87%
0,4661
0,4542
2.55%
0.7392
0.7356
0.48%
MSE
0.4400
0.3516
14.37%
0,1760
1745
0.835%
00.3865
0.3696
4.38%
0.1741
0.169N
2.48%
0.3668
0.3537
4.10%
MAE
0.1786
0.1352
9.07%
0.2679
0.2667
0,46%
0.1753
0.1578
3.68%
0.2674
0,2513
4
0.1552
0.4423
2.83%
MSE
0.5969
0.5355
10.28%
0,26663
0.2242
15.80%
2,4910
2.1537
13.544%
0.2214
0.2220
-0.28%
0.6054
0,5709
5.70%
48
MAP
0.5952
0,5533
7.04%
1.04%
0,3084
0.3018
2.11%
1.2874
1,1683
9.25%
0.2994
0.3021
-0.91%
0.5916
0.5729
3.17%
ETT22
108
MAE
MSE
1.8673
1.0730
0.9924
1,6609
11 110058
7.52%
0,7635
0.6082
0, 0.6992
0,5757
5.35%
8.425
4,9837
1.9106
2, 2,8678
1,4202
25.67%
42,46/%
0.3242
0.3740
0.3219
0.3652
0.711%
2.34%
0.9816
1,5877
0.9516
1,5206
3.05%
4.23%
336
MSE
2.3067
1,7424
24.47%
1,2567
1.0107
19.58%
3.0175
3.1162
-3.25%
0.3302
0.3289
0.39%
1.8481
1/6036
13.23%
MAE
1.2217
1,3067
15,14
0.7978
0,7005
11.0668
1,4438
1 1,4891
-3.84%
0.3800
0.3786
0.39%
1,0756
1,0524
4. 4.02%
MSE
2.7273
2.2901
16.03%
1.1952
1.7197
11.92%
4.3677
3,3177
24.04%
0.3791
0.3745
1.21%
2.0718
1.9062
7.99%
720
MAE
1.4042
1,2268
12.64%%
1.1171
1,0020
4.93%
1.8087
1,5454
14.56%
0.4216
0.4198
0,43%
1.1100
1.0850
2.25%
24
MSE
0.538X
0.3959
76.525
0,1036
0.1002
3.19%
(199664
0,4995
46 088%
0,1960
D. 0.1962
-0.12%
0.2464
0.2401
2.57%
0.1958
0.1297
13.33/%
0,2098
0,2039
2.78%
0.7183
0,5-01
21,81%%
0.2770
0,2782
-0.13%
0.3290
0,3231
1.81%
18
MAR
MSE
0,5580
0.6498
0.4748
0.46998
1.5 14.8
26.994%
0.1386
0.2442
0.1331
0.2358
2. 2.866%
3 3.92%
0.3752
0.2599
0,3550
0.2311
11.022%
4 4,86
0.2591
0.3344
0.1229
0,2601
-0.38%
0.56%
0:3855
0.3294
0.3774
0.3190
2.005
3.16%
ETTml
0.6139
0,5194
19,33%
0,1823
0,1822
0.03%
1,7790
1,4871
16.11%
0.3258
0.3250
1.78%
0.3758
0,3605
4.08%
MAE
0.5677
0.5044
11.17%
0.2726
0.2714
0.415
1.9655
0.9602
8.16S
0.3479
0,3419
1.725
0,4196 0,4074
2.91%
228
MSF
0.6231
0.7224
0,5645
0.6149
148798
9.40%
0,3665
0,3062
0,3477
0.2823
7.825
5.13%
6.2431
2,1601 1,4082
3,0455
51.22%
34,80%
0.3948
0.3723
00900
0.3614
2.02%
1.19%
0.4798
0.4618
0,4735
0.4539
1.31%
1.72%
672
MSR
0.7530
0.6810
9. 0.566
0.9525
0.8008
6.47%
7.
7.3624
6.3159
14.21%
0:4173
0.40044
2.128
0.6110
0.6155
-0.67%
MAE
0.6418
0,6056
5.65%
0.7233
0.6546
9.17%
2:4195
2.1142
12.62%
0,4233
0,4183
1.19%
0.5686
0.5693
-0.11%
24
MSE 0.3408
MAR
0.3611
0.3236
0.2547
25 264
10.391
0,141M
0,1063
0.1374
0.10300
2.42%
3.15%
00.1620
00.2500
0,1918
0.1216
24, 9..%
18.40%
0.14
0.1954
0.1945
0.1432
0.46%
2.96%
0.3011
0.3623
0.2984
0.36015
0.51%
1.10%
MSE
0.8239
0,5103
38.05% 05.0
0,1319
0.1308
0.85%
0.3480
0.1562
55,10%
0,1229
0.1143
6.99%
0,364] 0,3532
2.98%
MAE
0.5876
0.4864
17.23%
0.1792
0.1777
0.811
0,4000
0,2467
38.33%%
0.1631
0.1602
1.77%
0.4139 0.4048
2.21%
Weather
168
MSE
1.1858
1,0399
12.31%
0.1990
0. 1981
0.47%
0:2240
0.2861
55.57%
0.1850
0.1831
0.4662 0.4589
1.57%
MAE
0.7884
0,7335
6,9895
-0.2431
0,2415
0,6638
0.4630
0.5232
30,1
0,2049
0,3019
1 1,4605
0. 0,4946
0,4844
2. 2.07%
MSE
1.7610
1.5445
12.29%
0.2576
0,2537
1.51%
0.5780
0./383
24.17%
0,2479
0,2466
0.56%
0.5035 0.1910
2.48%
336
MAE
0.9969
0.9278
6.93%
0.2892
0 0,2855
1.29%
0.5230
0,4833
13.32%
0.2826
0.2801
0.87%
0.5215 0,5100
2.21%
720
MSE
2.5061
2,1811
12.97%
0,3185
0.3130
2,049
1,0596
0,4460
57,855
0.3193
0.1123
2.20%
0.5385
0.5297
1 1.62%
MAE
1,2121
1,1522
7,24%
0,3310
0,3363
L.11%
0,7410
0.1624
37.60%
0,3310
0.3278
1.86%
0,5415
0,5392
0.99%
24
MSE
0,2960
0,2915
1.51%
0.1221
0.1215
0.49%
0.2553
0,2480
1.98%
0,0990
0.0990
0.00%
0.2420
0:2377
1.76%
MAE
0.3821
0.3793
0.735%
0,2158
0.2145
0.60%
0.3594
0,3521
2,04%
0,17755
0.1770
(1288%
0.2628
0.2585
1 1.66%
48
0.3186
0.3116
2.21%
0.1398
0.1394
0.29%
0.2968
0,2962
0.22%
0.1153
0.1166
-1.16%
0,2912 0.2897
0.51%
MAE
0,3982
0.3942
0. 0.99MS
0.2322
0.2312
0.43%
0.3856
0.3831
0.64%
0,219%
0.22008
-0.45%
0.2977
0.2939
1 1.26%
Electricity
168
MSE
0.4166
0.3415
0.4126
0,3373
0.94%
2.09%
0.1650
0.2536
0.1617
0.2522
0.55%
0.18%
0.4072
0.3258
0,3012
0.3931
3.46%
7.55%
0.2412
0.14170
0.2401
0,1419
0.47%
3.47%
0.3781
0.4084
0.1040
0.3731
1.06%
1.32%
336
MSE
0.360X
0.3551
1.59%
0.1857
0.1848
0.48%
0.3872
0,3404
12.10/%
D. 166666
0.1631
2. 1.10%
0.566K
0.5546
2.52%
MAE
0.1288
0.1258
0.71%
0.2739
0.2724
0.55%
0:4512
0.4186
7.23%
0.2602
0,2388
0.51%
0.1693
0.1623
1.51%
720
MAR
0.3840
0.4444
0.3795
0.4428
0. 0.47%
1.10%
0.2255
0.3103
0. 30422222
0.2215
1 1.97%
L.77%
0.4083
0.4548
0.3309
0.409/
18.95%
10.00%
0,2033
0. 0.2933
0.1955
0.2892
3.84%
1 1.40%
0.8965
D.6471
0.8781
0.6:44
2.06%
1 1.93%
Avenge
MAE
MSE
0,9339
0.6755
0.6229
0,8058
13.71%
7.78%
0.1326
0.4004
0,3785
0.3898
5.45%
9.88%
0.8793
1.6175
0.7491
1.2335
25.13/%
14.82%
0.3
0,2692
0,2648
0,3149
1.17%
1.62%%
0,5870
0.5206
0,5103
0.5611
4.41%
1.97%
Table 4: Anomaly Detection results.
Table 5: Results for time-series classifi- cation with pre-trained MOMENT [34].
Yahoo
KPI
Model
F1
Prec.
Roc.
Fi
Prec.
Rec.
MOMENT MOMENT+S3
Dift.
TS2Vee
0.7188
0.6879
0.7538
0.6643
0.9181
0.5205
PTB XL
Loss 0.8308
0.7202
13.31%
SS2
0.7196
0.6891
0.7542
0.6852
0.9185
0.5464
Ace 0.7176
0.7552
5.24%
SoACLT
0.7389
0.722
0.7589
0.6890
0.9060
0.5561
Loss 1.543
1.4320
7.75%
S0
0.7498
0.7428
0.7569
0.6892
0.9194
0,5512
Crop
Acc 0.734
0.7591
3.41%
fine-tuning, and to ensure a fair comparison, we strictly follow the setup and experimental protocols outlined in the original paper. In this process, the encoder is kept frozen, and only S3 and the final linear head are fine-tuned together for the specific task and dataset. We fine-tune and evaluate the models for classification on PTB-XL XL [47] and the Crop dataset from the UCR archive [44], and the E and ETTh2 datasets [10] for forecasting. The results for classification are presented in Table 5, where we observe that upon adding S3, both the test loss and accuracy see significant improvements of up to 13.31% and 5.24%, respectively. Table A3 (Appendix) shows the forecasting results, where incorporating S3 consistently improves performance. On the ETThl dataset, it yields
8

---

Gound Turn
droend
1.0
Homer133
informeeS3
inferme
co lozs
10
Vetue
20
20
-14
-2.0
(a) H- 24
(b) H= 168
(a) ProximalPhalanxTW
(b)
Symbols
Figure 3: Forecasting output by Informer and
Figure 4: Training loss against iterations two
Informer+S3 for a sample from ETTh1.
sample datasets from the UCR archive.
Accuracy
Arcuracy
0:
0.
0.02
Segments
Segments
(a) PowerCons, = 144
(b) OliveOil, t 570
(a) TS2Vec
(h) TS
Figure 6: Classification accuracy vs. number of segments for two sample datasets from UCR with different input sequence lengths
Figure 5: Visualisation of the loss landscape for the Beef dataset from the UCR archive.
average improvements of 2. 2.12% in MSE and 2.59% in MAE, and for ETTh2, the improvements are 4.96% MSE and 3.56% in MAE.
Hyperparameters. Next, we investigate the impact of the number of segments on performance. Considering all the experiments, we observe that no general rule of thumb can be advised for n. as its optimum value is naturally highly dependent on factors such as dataset complexity, length, basclines, and others. We present two examples in Figure 6 and more in Appendix A3 where we plot the accuracy vs. number of segments for datasets of different lengths from UCR. Here we only take into account the layer with the largest number of segments. As for the other hyperparameters, the number of S3 layers & has been set between 2, or across all experiments in this paper. 0, the multiplier for the number of segments in stacked S3 layers, has been set to 0.5, 1, 2 across all our experiments. And finally, has only been set to 2, or 3. The ranges of values for all the hyperparameters used in this work presented in Table 6. While optimizing hyperparameters is important, similar to any other form representation learning, we find that even sub-optimal tuning of S3 hyperparameters still yields meaningful improvements. We perform simple experiment where we apply uniform set of hyperparameters [n = 2. é 2, A = 1, 1] to all the baselines and baselines with added $3 presented in Table 8 where observe that despite not
Table 6: Range of hyper- parameters.
Range
12,4, 24
[1,2,3]
[0.5, 1,25
[1,2,3]
layers. The results for this experiment using the optimum hyperparameters, the addition of S3 still yields considerable performance boosts. The optimum hyperparameters used for each experiment are made available in the code.
Sensitivity to random seed. To evaluate the impact of the random seed on the performance of S3, we perform 10 separate trials with different seed values and present the standard deviations in Table 9, The first 4 rows show the classification basclines trained on the UCI datasets with and without S3. Similarly, the next 3 rows show forecasting baselines trained on the multivariate ETThl dataset (H=24) with and without S3, From this experiment, we observe that the addition of S3 has no considerable impact the sensitivity of the original basclines to the random seed.
Shuffling parameters. In Figure
we present a visual overview how the parameters P update along with the parent model during training on ETThl (H = 48) and ETTml (H = 48). Each parameter P corresponds to a segment of S. In both cases, we set n to and á to The baseline used LaST [19]. It can be seen that the individual weights in P rearrange the corresponding segments until they reach stable point (the optimal order) after which the order of the segments maintained. Similarly, Figure A4 (Appendix) shows how W1 and wa update during training. Finally, in Figure A5

---

Table 7: Ablation results for classification.
Table 8: Classification results with Table Variation seed.
common hyperparameters.
Method
w/o S3
w/S3
Method
Method
EEG
EEG2
HAR
TSZVec
SonCLT
0.0044
0.077
0.2675
0.0058
1S2VeesS3
TS2Nec+S3
TS2Nec+S3 Segment
8506
0365
TS2Vec4S3
TS2Vec
0.625
0.593
0.100
0.845
0.922
0.926
InfoTS
DSN
00057
00041
0.0662
0.0055
1S2Vec153
0.544
0.71
0.501
Difl.
11.11%
7.69%
0.36%
TS2
Informer
00144
0,019
0.0122
0.0028
D3 Sepment
DSN-S3
0910
DSN
0.516
0,961
0.957
LAST
00000
0.0122
0.28
DSN+S3
0.567
0.978
0.974
Table 10: Param. count.
0.556
Dift.
9.83%
1.78%
1.72%
InfoTS-S3 weo
InfoTS-33 wio Sepment
0.929
0.988
InfoTS+S3
InfoTS
0.593
0.515
0.842
0.822
0.898
0.876
TS2Ves
SoftCT
Method
Raveline
641k
613k
579
53
24
InfoTS-S3 Stiihh
0,587
Dift.
15.15%
2.43%
2.45%
InfTTS
606k
SeiCLT+83
SeiCI Segment
0.883
SoRCIT
SofCLT
0.640
0,500
0.890
0.893
0.923
0,918
Vee
Informer
DSN
638k
106k.
So9CLT+S3 Stuch
0,913
0
Dift.
15.15%
-0,33%
0.54%
LaST
Value for eacch parameter In
0.60006
Value each para parameter
0.60002
0.60004
0.60000
0.60002
0.59998
71
0.59998
71.
0.59994
0.59992
0.59996
Pp
0
250
500
750
1000
1250
1500
250
500
750
1000
1250
1500
Iteration
Iteration
(a) ETTh1 (H=
(b) ETTml (H= 48)
Figure 7: The progression of the shuffling parameters during training for LaST+S3.
(Appendix), we show sample visualizations of how S3 segments, shuffles, and stitches a time-series once the optimal shuffling parameters are obtained.
Computation overhead. Our proposed network layer adds very few learnable parameters to the baseline models, which stem from a learnable shuffling vector P along with wa and W2, per each $3 layer. To emphasize the negligible computation overhead of $3 in comparison to the baseline models, we show the total number of parameters in the baseline and the total learnable parameters that S3 adds, in Table 10 for classification on the EEG2 dataset and forecasting on the multivariate ETThl dataset (H 24).
6 Conclusion
Summary. We propose S3, a simple plug-and-play neural network component that rearranges the time-series in three steps: Segment the time-series. Shuffle the segments, and Stitch the shuffled time-series by concatenating the segments and performing a learned sum with the original time- series. Through extensive experiments on time-series classification and forecasting with state-of- the-art methods, we show that S3 improves the learning capabilities of the baseline with negligible computation overhead. We also show empirically that S3 helps in faster and smoother training leading to better performance.
Limitations. While present the efffcctiveness of S3 diverse set of baselines classification, forecasting, and anomaly detection tasks, we acknowledge that the evaluation of other time-series tasks such imputation remain for future work. Additionally applying S3 to learning representations from other forms of time-series such as videos is an interesting direction for future research.
Broader impact. Since S3 is a plug-and-pla network layer with negligible added parameters, it allows researchers from various domains related to time-series such as health signal processing, biometrics, climate analysis, financial markets, and others to use this module in their existing models without the need for redesign. The low computation overhead also makes S3 a suitable choice for edge-devices.
10

---

Acknowledgements
This research was partially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) through the Discovery Grant program.
References
[1] Dani Kiyasseh, Tingting Zhu, and David A Clifton. Clocs: Contrastive learning of cardiac signals across space, time, and patients. In International Conference on Machine Learning, pages 5606-5615. PMLR, 2021.
[2] Anubhav Bhatti, Behnam Behinaein, Paul Hungler, and Ali Etemad. Aux: Attentive cross- connections for fusion of wearable signals in emotion recognition. ACM Transactions on Computing for Healthcare, 5(3):1-24, 2024.
[3] Debaditya Shome, Pritam Sarkar, and Ali Etemad. Region-disentangg diffusion model for high-fidelity pPg-to-ecg translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15009-15019, 2024.
[4] Debaditya Shome and Ali Ftemad. Speech emotion recognition with distilled prosodic and linguistic affect representations. IEEE International Conference on Acoustics, Speech and Signal Processing, pages 1197--1988,, 2024.
[5] Pritam Sarkar and Ali Ftemad. Self-super eeg representation learning for emotion recogni- tion. IEEE Transactions on Affective Computing, 13(3): 2022.
[6] Pritam Sarkar, Ahmad Beirami, and Ali Ftemad. Uncovering the hidden dynamics of video self-supervised learning under distribution shifts. Advances in Neural Information Processing Systems, 36, 2024.
[7] Hycokhyen Kwon, Catherine Tong, Harish Haresamudram, Yan Gao, Gregory D. Abowd, Nicholas D. Lane, and Thomas Plötz. Imutube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition. Proceedings of the ACM Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(3), September 2020.
[8] Sijie He, Xinyan Li, Timothy DelSole, Pradeep Ravikumar, and Arindam Banerjee. Sub- seasonal climate forecasting via machine learning: Challenges, analysis, and advances. In AAAI Conference Artificial Intelligence, volume 35, pages 169-177, 2021.
[9] Haitao Lin, Zhangyang Gao, Yongjie Xu, Lirong Wu, Ling Li, and Stan Li. Conditional local convolution for spatio-temporal meteorological forecasting. In AAAI Conference on Artificial Intelligence, volume 36, pages 7470-7478, 2022.
[10] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In AAAI Conference on Artificial Intelligence, volume 35, pages 11 1110-11115,, 2021. 1, 5, 21
[11] Seok-Jun Bu and Sung- Bae Cho. Time series forecasting with multi-headed attention-based deep learning for residential energy consumption. Energies, 13(18):4722, 2020.
[12] Zhiyuan Wang, Xovee Xu, Goce Trajcevski, Kunpeng Zhang, Ting Zhong, and Fan Zhou. Pref: Probabilistic electricity forecasting via copula-augmented state space model. In AAAI Conference on Artificial Intelligence, volume 36, pages 12200-1220,, 2022.
[13] Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. Advances in Neural Information Processing Systems, 17844484-78 2020.
[14] Razvan-Gabriel Cirstea, Bin Yang, Chenjuan Guo, Tung Kieu, and Shirui Pan. Towards spatio- temporal aware traffic time series forecasting. In IEFE International Conference on Data Engineering, pages 2900-2913, 2022.

---

[15] Dawei Cheng, Fangzhou Yang, Sheng Xiang, and Jin Liu. Financial time series forecasting with multi mudti-mo graph neural network. Pattern Recognition, 121:108218, 2022. 1
[16] Qiao Xiao, Boqian Wu, Yu Zhang, Shiwei Liu, Mykola Pechenizkiy, Elena Mocanu, and Decebal Constantin Mocanu. Dynamic sparse network for time series classification: Learning what to "see". Advances in Neural Information Processing Systems, 35: 15:16649-16862, 2022. 1, 2,5
[17] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. AAAI Conference on Artificial Intelligence, volume 36, pages 8980-8987, 208.. 1, 3, 6,
[18) Xuchao Zhang, Yifeng Gao, Jessica Lin, and Chang-Ties Tien Lu. TTppne: Multivariate time series classification with attentional prototypical network. In AAAI Conference on Artificial Intelligence, volume 34, pages
2020.
[19] Zhiyuan Wang, Xovee Xu, Weifeng Zhang, Goce Trajcevski, Ting Zhong, and Fan Zhou. Learning latent seasonal-trend representations for time series forecasting. Advances Neural Information Processing Systems, 35:38775-38788 2022, 1, 3, 9,
[20] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-2243 2021. 1,3
[21] Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction, Advances in Neural Information Processing Systems, 35:5816 15:5816-5828, 2022. 1,
[22] Qingxiong Tan, Mang Ye, Baoyao Yang. Siqi Liu, Andy Jinhua Ma, Terry Cheuk-Fung Fung Yip, Grace Lai-Hung Wong, and PongChi Yuen. Data-gru: Dual-attention time-aware gated recurrent unit for irregular multivariate time series. In AAAI Conference on Artificial Intelligence, volume 34, pages 930-937, 2020. 1
[23] Donghui Chen, Ling Chen, Youdong Zhang, Bo Wen, and Chenghu Yang. multiscale interactive recurrent network for time-series forecasting. IEEE Transactions on Cybernetics, 52(9))8793-8803 2021.
[24] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre- Alain Muller. Deep learning for time series classification: a review. Data Mining and Knowledge Discovery, 33(4):9177963. 2019. 1
[25] Patrick Schäfer. The boss concerned with time series classification in the presence of noise. Data Mining and Knowledge Discovery, 29: 1505-1530, 2015.
[26] Anthony Bagnall, Jason Lines, Jon Hills, and Aaron Bostrom. Time-series classification with cote: the collective of transformation--aaee ensembles. IEEE Transactions on Knowledge and Data Engineering, 27(9):2522-2533 2015.
[27] Aosong Feng and Leandros Tassiulas. Adaptive graph spatial-temporal transformer network for traffic forecasting. In ACM International Conference on Information and Knowledge Managemení, pages 3933-3937, 2022. 2
[28] Junwoo Park, Daehoon Gwak, Jaegul Choo, and Edward Choi. Self-supervised contrastive learning for long-term forecasting. In International Conference on Learning Representutions, 2024. 2.3
[29] Ali Salehi and Madhusudhanan Balasubramanian. Ddcnet: Deep dilated convolutional neural network for dense prediction. Nearocomputing, 523:1 16-129, 2023.
[30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017. 2
12

---

[31] Yuqi Chen, Kan Ren, Yansen Wang, Yuchen Fang, Weiwei Sun, and Dongsheng Li. Contiformer: Continuous- transformer for irregular time series modeling. Advances in Neural Information Processing Systems, 36, 2024. 3
[32] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Reprexentations. 3, 5, 21
[33] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI Conference on Artificial Intelligence, volume 37, pages 11121-11128, 2023.
[34] Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: family of open time-series foundation models. arXiv preprint arXiv:2402.03885, 2024. 3, 5, 7,
[35] Dongsheng Luo, Wei Cheng. Yingheng Wang, Dongkuan Xu, Jingehao Ni, Wenchao Yu, Xuchao Zhang, Yanchi Liu, Yuncong Chen, Haifeng Chen, et al. Time series contrastive learning with information-aware augmentations. In AAAI Conference on Artificial Intelligence, volume 37, pages 3534-4542, 2023. 3,5
[36] Seunghan Lee, Taeyoung Park, and Kibok Lee. Soft contrastive learning for time series. In The International Conference on Learning Representations, 2024. 3, 5, 6, 21
[37] Renfei He, Limao Zhang, and Alvin Wei Ze Chew. Modeling and predicting rainfall time series using seasonal -orennd--ddddddd-- decomposition and machine learning. Knowledge-Band Based Systems, 251::09125,, 2022. 3
[38] Robert B Cleveland, William S Cleveland, Jean E McRae, Irma Terpenning, al. Stl: A seasonal-trens decomposition. Journal of Official Statistics, ((1):3773, 1990. 3
[39] Shuai Liu, Xiucheng Li, Gao Cong, Yile Chen, and Yue Jiang. Multivariate time-series imputation with disentangled temporal representations. In International Conference on Learning Representations, 2022. 3
[40] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. CoST: Con- trastive learning of disentangled seasonal-trend representations for time series forecasting, In International Conference on Learning Representations, 2022. 3,5
[41] Marco Cuturi, Olivier Teboul, and Jean- Phan-Philippe Vert. Differentiable ranking and sorting using optimal transport. Advances in Neural Information Processing Systems, 32, 2019.
[42] Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. Fast differentiable sorting and ranking. In International Conference on Machine Learning, pages 950- 950-959, 2020. 4
[43] Michael Taylor, John Guiver, Stephen Robertson, and Tom Minka. Softrank: optimizing non- smooth rank metrics. International Conference on Web Search and Data Mining, pages 77-86, 2008. 4
[44] Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series archive. IEEE Journal of Automatica Sinica, 6(6): 1293:12931335 2019. 5, 8
[45] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. ArXiv Preprint 18 2018.
[46] Dheeru Dua and Casey Graff. Uc irvine machine learning repository, 2017. 5
[47] Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojciech Samek, and Tobias Schaeffter. Ptb-xl, large publicly available electrocardiography dataset. Scientific Data, 7(1):1115, 2020. 5,
13

---

[48]
Max Planck Institute for Biogeochemistry. Jena climate and weather data. https://www bgo- jena. jena.mpg........................ Sttp..e/wetter/..
[49] Nikolay Laptev, Saeed Amizadeh, and lan Flint. Generic and scalable framework for automated time-series anomaly detection. In ACM International Conference Knowledge Discovery and Data Mining, pages
2015. 5
[50] Hansheng Ren, Bixiong Xu, Yujing Wang, Chao Yi, Congrui Huang, Xiaoyu Kou, Tony Xing, Mao Yang, Jie Tong, and Qi Zhang. Time-series anomaly detection service microsoft. In ACM International Conference on Knowledge Discovery & Data Mining, pages 3009-3017, 2019.
[51] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in Neural Information Processing Systems, 31, 2018. 6, 19
[52) Xiyuan Zhang, Ranak Roy Chowdhury, Jingho Shang, Rajesh Gupta, and Dezhi Hong. Towards diverse and coherent augmentation for time-series forecasting. In International Conference on Acoustics, Speech, and Signal Processing, pages 1-5, 2023. 21
[53] Brian Kenji Iwana and Seiichi Uchida. An empirical survey of data augmentation for time series classification with neural networks. Plos One, 16(7)xx0254841, 2021.
[54] Anh Tong, Thanh Nguyen-Tanne Toan Tran, and Jaesik Choi. Learning fractional white noises in neural stochastic differential equations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. 21
14

---

A Appendix
A.1 Additional results
We show the full results for all baselines with and without S3 on 128 UCR datasets in Table Al and on 30 UEA datasets in Table A2. In Figure Al we show 2 more t-SNE plots for Te2Vec trained on randomly chosen UCR datasets. A better clustering of different classes can be observed in the learned representations after S3 is incorporated. In Table A4 we compare the standard deviation values for the training loss of SoftCLT and SoftCLT+S3. From this table we observe that after incorporating S3, the training becomes much more stable with significantly less variations. Figure A2 shows 2 more comparisons of the loss landscape of VSc trained with and without S3. In Figure A3 we present 6 more samples for the impact of n on the performance of the model.
Table Al: Per-dataset breakdown of results for the UCR archive.
Dataset
TS2Vec
1S2Vee
DSN
DSN+53
InfoIS
InfoISo83
SoftCLT
SoRCLT+83
ACSFI
0.778
0.830
0.790
0.810
0,800
0.800
0.800
0.560
Adiac
0.737
0.785
0.767
0.859
0.703
0.631
0.772
0.795
AlGiestreWilmnee
0.773
0.700
0.710
0.755
0.117
0.211
0.743
0.794
0.779
0.797
0.704
0.797
0.146
0.406
0.777
0.786
DiCismmeeinnee
0.723
0.773
0.744
0.753
0.160
0.203
0.740
0.746
Hoad
0.34
0.886
0.500
0.926
0.846
0.823
0.663
RME
0.987
1,000
0.827
0.950
0.993
1.000
1.000
1,000
Reef
0.700
0.967
0.33
0.500
0.700
0.833
0.731
0.967
HecrleFly
0.000
0.950
0.500
0.950
0.800
0.950
0,850
0,500
RindChicken
0.500
1,000
0.900
1,000
0.750
0,900
0.500
1,000
CRF
1,000
1,000
0.988
1.000
1,000
1.000
0.968
0.099
Car
0.733
0,500
0.500
0.933
0.717
0.750
0.767
0,500
Chinstoonn
0.965
0.959
0.980
CiloennCCenenntration
0.815
0.836
0.833
0.894
0.754
0.750
0.754
0,820
CaCCBCGTano
0.817
0.535
0.796
0
0.801
0.857
0.824
0.954
GvfTee
1,000
1,000
1,000
1,000
0.964
1,000
1,000
1,000
Computers
0.628
0.680
0.808
0.688
CricketX
0.795
0.808
0.500
0.828
0,700
0.718
0.785
0,810
CrickeY
0.764
0.785
0.908
0.836
0.733
0.721
0.769
0.756
Cricketz
0.500
0.805
0.831
0.828
0.751
0.715
0.790
0.808
Crup
0.755
0.740
0.730
0.759
DiscomSixeRaluctionn
0.984
0.997
0.915
0.907
0.961
0,900
0.974
0.990
0.727
0.770
0.788
0.845
0.741
0.741
0.748
0.755
0.732
0.793
0.772
0,833
0.757
0.772
0.728
0. 0.797
DisuiPhullan TW
0.676
0.719
0.790
0.676
DodgeriLoupaay
0.563
0,650
0413
0:435
0.625
0.513
0.500
0.575
Dadger aupGame
0.548
0.920
0.768
0,779
0.783
0.797
0.862
0.577
Dadborl nopWeekknd
0.964
0
0.986
0.986
0.978
0.971
0.949
0.986
Earthquokes
0.748
0.748
0.710
0.829
0.748
0.748
ECG200
0.10
0
0.8
0.930
0.880
0,800
0.890
0.950
RCG5000
0.933
0:940
0.945
0.944
0.941
0.942
0.939
0.941
ECyFiveDays
1.000
1,000
0.986
1.000
0.866
0.964
1,000
1.000
FlveiricDeviios
0.519
0.724
0.733
0.503
0.514
0.503
0,583
0,500
0.544
0.561
0.561
BOGVerticaSSigaa
0.500
0.508
0.412
0.497
0.525
0.539
0.525
0.525
Fithnoll.co evel
0.420
0.626
0.644
0.676
0.306
0.630
0,658
0.659
FaceAll
0.768
0.818
0.792
0.789
0,820
0,823
FaceFuur
0.852
0.841
0.966
0.966
0.920
0.943
0.322
0.875
FaCesTCR
0.926
0.947
0.965
0.979
0.893
0.873
0.940
0.935
Wands
0.771
0.815
0.826
0.822
0,767
0.763
0.793
0.509
0.931
0.989
0.967
0.546
FoedA
0.935
0.933
0.947
0.954
0.815
0.501
0.977
0.966
FordB
0.796
0.806
0:933
0,940
0.606
0.573
0.788
0.795
FrovarReguhrTaain
0.985
0.950
0.99
0.991
0.982
0.961
0.991
0.996
PreczerSmallTain
0
0.941
0.796
0.928
0,740
0.972
0.981
Fungi
0.957
0.968
1,000
1.000
1,000
1,000
0.973
0.977
GesureMidAAnD
0.615
0.715
0.638
0,746
0.077
0.092
0.723
0.700
GestureMidAin22
0.469
0.508
0.577
0.610
0.085
0.092
0.562
0.615
GesuraMidAinD3
0.308
0.300
0.423
0.077
0.392
0,477
Gestua/PebllZ1
0.507
0,892
0.337
0.924
0,314
0.218
0913
01872
GestrePPbola22
0.348
0.892
0.778
0.804
0.310
0.437
0.561
0.892
GunPoini
0.980
0.993
0.993
0.993
0.960
0.987
0.993
0.993
0.965
0.594
0.981
1,000
0.956
0.956
1,000
1,000
1.000
0.997
1.000
1,000
1,000
1,000
1,000
CumPoiniOlddeeea Young
1,000
1,000
1.000
1.000
1,000
1,000
1,000
1,000
Ham
0.333
0.510
0.752
0.790
0.519
Handins
0.924
0.919
0.852
0.848
0.908
0.850
0.549
Haptics
0.316
0.545
0.565
0.594
0
0.438
0.503
0.512
Herring
0.594
0.888
0.672
0.734
0.578
0.594
0.641
0L688
House Twenty
0.908
0.966
0.924
0.983
0.891
0.874
0.933
0.941
InlineSitate
0.407
0.420
0.311
0.480
0.333
0.429
0.402
InseetEPGeegularraai Train
1.000
1.000
1.000
1.000
1,000
1,000
1.000
1,000
1,000
1,000
0.434
1.000
1,000
1,000
1,000
1,000
In Wingbeertood
0.634
0.641
0.581
0.636
0.647
01638
Inlly PowerDemand
0.524
0.971
0.959
0.965
0.945
0.964
0.972
Large LargeKitchenta KitchnnApplianees
0.567
0.861
0.580
0.928
0.821
0.827
0.827
0.567
Lighning2
0.352
0
0.770
03.015
0.721
0.787
0.336
03352
Lighning?
0.349
0.836
0.308
0.890
0.822
0.849
0.822
0.308
15

---

Tablle Al comtinued from previous page
Dataset
TS2Vee
1S2Vee483
DSN
DSN+83
InfiT8
InfoTS483
SohCLT
SoRCLTS53
Maila
0.222
0.940
0.961
0.972
0.122
0.944
0.15
0.946
Meat
0.950
1,000
0.567
0.950
0.933
0.917
0.950
0.983
Medicallmages
0.788
0.801
0.753
0.770
0.770
0.745
0.793
0.792
MelbournePesssttee Pedestrttin
0.957
0.48
0.9/9
0.636
0.666
0.720
0,790
0.649
0.649
0.630
0.675
0.818
0.821
0.793
0.828
0.735
0.759
0.821
0.838
MiddleehalammTW TW
0.571
0.623
0.581
0,639
0.10
0.610
0.578
0.10
0.520
0.926
0.97
0.983
0.96
0.925
0.29
MixedSiapssSmmllInn Truin
0.846
0.852
0.929
0.950
0.856
0.577
0.891
0.881
Motesmmin
0.858
0.884
0.907
0.851
0.885
0.552
0.874
0.876
Nondmwasiveeeeullee Ghhorrx
0.928
0.944
0.926
0.926
0.04
0.333
01937
0.932
0,948
0916
0.892
0.926
0.946
OlivOOd
0.500
0.333
0.467
0.967
0.667
0.933
0.833
0.856
OSULea
0.811
0,839
0.909
0.992
0.603
0.522
0.818
0,331
CullngeeQlllnssSureet
0.793
0.817
0.832
0.837
0.801
0.800
0.501
0.825
Phoneme
0.304
0.258
0.325
0.341
0.197
0.299
0.304
PickupGietture Wiimotel
0.820
0.540
0.820
0.843
0.320
0,400
0.880
0,900
0.611
0.663
0.471
0.889
0.139
0.163
0.332
0.361
PrrArrPeesuure
0.966
0.962
0.520
0.284
0.966
01962
PGCCVP
0.793
0.817
0.269
0.254
0.139
0.188
0.832
0.798
PLAID
0.553
0.547
0:214
0.205
0.077
0.179
0.512
0.538
Plane
0.590
1,000
1.400
1,000
0.81
0.981
0.981
1,000
PowerCons
0.961
1.000
0.928
0.967
0.899
0.994
0.994
0.991
Proxime paa.aamiPPPaaannttiieeeeeieeeo
0.829
0.868
0.868
0.878
0350
0.854
Proxima Phanaam.iPaalnnnCae CutiannnOttiieaCoooo.
0,883
0.500
0.11
0.924
0,845
0.873
0.869
0911
ProsimalPhalimeIWW TW
0.500
0.829
0.828
0.825
0.790
0.524
0.824
0.820
RehiigenatiooDeviiess
0.589
0
0.528
0.584
0.584
Rock
0.740
0.740
0.500
0.600
0.500
0.867
0.740
ScrentType
0.416
0.448
0.997
0.645
0.440
0.400
0.451
0.445
Songgannkienderh22
0.947
0.948
0.618
0.574
0.937
0.963
0.937
0.958
Hamg
0.362
0.876
0.431
0.436
SemnHaandSihjeeshh2
0.951
0.920
0.666
0.456
0.69
0.929
0.947
0.966
Stake Gesueeeeeeii
0.940
0.960
0.940
0.914
0.400
0.500
0.920
0.960
ShapeletSion
0.989
1,000
0.767
0.872
0.528
0.500
1,000
0.994
StupesAll
0.985
0.903
0.905
Small Applinnppllianess
0.733
0.733
0.821
0.835
0.739
0.731
0.112
0.752
Sm ShShhhuueeeeeee
0,980
0.987
0.947
0.912
0.987
0.940
0
0,980
Sony ATBORohotSurfeee
0.890
0.930
0.965
0.972
0.839
0.895
0.884
0.864
ScuyAlBBRRoooSiu AobooSutSurt
0:923
0.929
Starl CariCCrvess
0.968
0.939
0.081
0.983
0.959
0.968
0.065
Stawberr
0.962
0.968
0.972
0.984
0,957
0.962
0,054
0,962
SwedishLeaf
0.929
0.538
0.999
0.981
0.936
0.934
0.947
0.938
Symbols
0.973
0.927
SyutherticCouurll
0.993
0.993
0.993
0.907
0.980
0.997
0.933
ioeSegmentationl
0.944
0.912
0.974
0.978
0,741
0.825
0.934
01952
The TreSegmentation22
0.900
0.900
0.946
0.962
0.846
0.900
0,877
0.938
1.000
1,000
1,000
1.000
1,000
1,000
1,000
Twol LMNECG
0.987
0.900
0.996
0.999
0.753
0.910
0.977
0.994
TwoPPteres
1,000
0.994
0.949
1.000
0.099
1.000
1,000
1,000
UMD
0.593
1,000
0.979
0.934
1,000
1,000
1,000
1,000
U'WaveGesttrel ibmar
0.956
UWaveGesmeel Jhmry'X X
0.790
0.817
0J833
0,833
0.806
0,811
WWweeGesuueL..bmn Y
0.719
0.739
0.771
0,776
0,723
0.730
0.736
0.346
0.764
0.750
0.784
0.78%
0.732
0.754
0.772
0.769
Wafarr
0.096
0.992
Wine
0,870
0.944
0.556
0.722
0.759
0.815
0.519
0.981
WonlSynonyms
0.680
0.708
0.774
0.732
0.687
0.687
0.701
0.718
Worms
0.649
0
0.619
0.663
0.649
0.610
0.714
0.779
WormsTwuClass
0.751
0.779
Yoga
0.872
0.583
0.919
0.879
0.841
0,881
0.800
Average
0,819
0,851
0.794
0.833
0.733
0.760
0,825
0.545
16

---

Table A2: Per-dataset breakdown of classification results for the UEA dataset.
Dataset
T82Vec
TS2Vec+83
DSN
DSN+S3
InfoTS
InfaTS+83
SoftCIT
SofCIT
Articulary WeroulayyWodRecoonniio
0.987
0.983
0.980
0.993
0.983
0.993
0.993
0.997
AtriaFFibellation
0.200
0.400
0.267
0,266
0,400
0.200
0:400
BasicMotions
0.975
0.983
1,000
0,975
1,000
1,000
0.975
1,000
Character Trajectorees
0.995
0.996
0.994
0.994
0.683
0.701
0.984
0.997
Cricket
0.986
1.000
0.986
0.944
0.958
0.986
0.972
0.986
DuckDuckGeese
0.500
0.500
0.620
0.660
0,560
0.580
0.500
0.420
WigenWorms
0.720
0.864
Epilopsy
0.964
0.964
1.000
0.884
0.957
1,000
0.935
0.971
ERing
0.874
0.848
0.907
0.956
0,926
0.944
0.911
0.937
OlhamaiCunceentaation
0.300
0.312
0.221
0.680
0.259
0.304
0
0.304
Face Detection
0.498
0.516
0.638
0.652
0.525
0.564
0.534
0.534
FingerMowements
0.480
0.590
0.500
0.500
0.570
0.600
0.600
HandMowementtiieee Directiion
0.270
0.338
0.297
0.649
0,408
0.541
0.419
0.486
Handwriting
Heartbeat
0.663
0.756
0.766
0,741
0.756
0,780
0,683
0.761
Wingeea
0.466
0.437
0,322
0.483
0,466
0.469
0.485
0.478
Japanese Vowels
0.984
0.986
0.992
0,970
0.962
0.992
0.984
0.992
Libras
0.867
0.889
0.961
0.894
0.833
0.878
0.878
0.911
LSST
0.543
0.563
0.550
0.543
0.559
0.516
0.563
MotorImagery
0.510
0.540
0.490
0.640
0,580
0.610
0.530
0.560
NATOPS
0,922
PEMS-SF
0.694
0.723
0.786
0.861
0.740
0.844
0.682
0.746
PenDigits
0.990
0.989
0.987
0.982
0,983
0.987
0.987
0. 0.988
PhecemeSSpetrr
0.240
0.240
0.315
0.114
0.233
0.282
0.223
0.231
PacketSports
0.855
0.868
0.863
0.865
0.895
0.914
0.862
0.855
ScllRegullttioSSCP1
0.785
0.850
0.700
0.867
Sell SellRegulationSCPP
0.578
0,578
0,506
0.583
0,567
0.606
0.494
0.589
SrakennArbbic
StandWalkJump
0.467
0.533
0.400
0,533
0,400
0.467
0.333
0.600
UWaveGesturel.ibrrr
0.909
0.919
0.925
0.903
0.906
0.928
0.925
0.934
Average
0.695
0.720
0.687
0.737
0.691
0.727
0.699
0.734
Table A3: Forecasting results for MOMENT with and without S3,
Datasel II
Metric PatchTST DLineur TimesNel FEDFormer Stationary
LighTTS MOMENT MOMENT+83 DifI.
MSE
0.375
0.384
0.513
2.90%
96
MAE
0.399
0.399
0.402
0.419
0.491
0.432
0.437
0.422
3.28%
192
MAE
0.421
0.416
0.429
0.448
0.504
0.462
0.487
0.464
4.60%
MSE
0.422
0.439
0.491
0.459
0.588
0.518
0.426
0.425
0.06%
ETTh
336
MAE
0.436
0.443
0.469
0.465
0.535
0.488
0.432
0.432
0.00%
MSE
0.447
0.472
0.521
0.506
0.643
0.547
0.423
0.411
1 1.90%
720
96
MSE
0.274
0.289
0.340
0.358
0.476
0.397
0.378
0.348
7.80%
MAE
0.336
0.353
0.374
0.397
0.458
0.437
0.411
0.390
4.98%
MSE
0.339
0.383
0.402
0.429
0,512
0.520
0.382
0.372
2.42%
192
MSE 0.329
0.448
0.452
0.496
0.552
0.626
0.379
0,360
4.95%
ETTM2
336
3.65%
MSE 0.379
0.605
0.462
0.463
0.562
0.863
0.373
0.357
4.25%
MAF
0.422
0.551
0.468
0.474
0.560
0.672
0.404
0.396
1.94%
17

---

Table A4: Comparison of the measured standard deviation of the training loss for TS2Vec and TS2 Vec+S3, over the first 75 UCR datasets.
Dutaset
TS2Vec
TS2Vec+S3
Dif..
2.39
34.65%
Wiimotex
L.19
0.88
26.38%
AllGesture WiimoreZ
1.43
1.35
5.71%
BME
23.59
21.24
10.00%
BirdChicken
1.66
2.45
44.43%
4.26
ChlorineComenntration
11.05
9.16
17.03%
Computers
10,93
9.85
9.92%
CricketX
2.82
1.51
46.49%
Cricket
19.42
13.91
28.35%
Cricket7.
4,66
3.34
28.44%
DistPPhalanOOntinneAeeoooo
13.17
0.92
93 93.00%
4.32
0.95
77.98%
DodgerLoopp oopeekeend
557.35
798.64
-43,29%
ECG200
1,85
1,67
9.64%
ECG5000
1.83
1,30
29.05%
FOGHorizonnalipnal
3.58
3.14
12. 16%
EOGVerticaSignall
2.83
1.66
41.09%
Farthquakes
EthanolLevel
1,60
1,48
7.27%
FaceAll
12.12
10.66
12.03%
FaceFoour
4.64
1.62
65.14%
Fifty Words
5.10
4.47
12.31%
FordA
2,94
2.58
13%
FordB
1.41
1.31
7.24%
GestureMidAinD3
HAR
2,40
1.32
45.06%
Haptics
1.42
1,06
25.90%
Herring
2.60
2.64
1.77%
Twentyy
9.22
8.35
9. 38%
InlineSkate
6.10
5,71
6.50%
InscctaP@ReeottrrTaain
11.61
4.14
64.37%
InselEEPOmmlllTrinn
PenyoondDD
13.87
0.86
93.83%
LargeKichhnAApplaneee
13.10
7.66
41.51%
Lightning2
94.45
6.31
93.32%
Lightning?
2.13
1.37
35,4 46.4
Mear
4.75
4.31
9.19%
Medicallmages
1.23
1.43
16..77%
MelloureePeesttii
MuudiePPalanxOutiiee
4.11
0.92
77.71%
Middle PualaanOOulineemrmmm
4.21
0.93
77, 77.95%
MiddleePhllanx TW
4.09
0.93
77.39%
MixedShapeRRggglaff/ Train
2.48
2.38
4.1 17%
MixedShapesSmall Traii
MoteStrain
8.07
0.98
87. .7.9
Nout
3.20
2.80
12.73%
OSUL
1.72
1.52
11 11.90%
OliveOil
19.02
11.44
39, 84%
PLAID
4.67
2.98
36.0 36.09%
Ptulange
Phoneme
2.30
1.87
18.91%
MekkkpGestneWimmeee
PigAirway Pressure
6.32
5.59
11.57%
PigAAPrressure
1.57
1.50
4.1 19
PigCVP
1.89
35.55%
Plane
15.28
10:49
31.31%
PowerCons
13.26
0.95
92.85%
12.95
0.94
92.724
ProsimalPhalansTTW TWW
1.40
0.92
34.04%:
RefrigerationDeviictt
6.66
4.45
33.25%
Rock
8.27
9.23
-11.55%
ScreenType
9.70
5.49
43.44%
SengllandGGnneehh2
SemgHandMovemennCh
19.03
17.21
9. 9.59%
SemeHandSubjeethh2
8.78
7,93
68%
Geate WiimoteZ
2.41
0.83
65.65%
ShapeletSim
3.32
1.12
66.18%
ShapesAll
1.32
Small muullKiccheAgppianeee
84.86
47.19
44.39%
SmoothSubspace
Sony I
12.81
1,47
88.50%
18

---

1S2Vec +53
TS2Vec
T52V
-38
-50
-10
100 225.2222255522252222555
(a) TwustT
(h) BasicMotions
Figure A1: t-SNE visualizations the learned representations for Te2 and TS2Vec+S3 for randomly chosen test sets. Different colors represent different classes, where we observe better grouping of representations belonging each class after the addition of S3.
TS2Vec
TS2Vec+S3
TS2Vec
TS2Vec+53
(a) WiimteZZ
(b) Symbols
Figure A2: Visualisation of the loss landscape following [51], for TS2Vec and TS2Vec+S3 on two UCR datasets. It can be observed that the loss landscape with S3 is considerably smoother than the baseline without S3.
0.90
0.75
0.80
Accuracy
Accuracy
Accuracy
0.70
0.80
0.75
0.60
0.70
0.70
50
100
150
200
10
15
20
25
50
100
150
200
Number of Segments
Number of Segments
Number of Segments
BectleFly (t 512)
RacketSport (t 30)
(c) Rock (t 2844)
2.65
2.60
2.55
MSE
MSE
2.50
2.4
1.4
2.40
2.2
1.2
100
200
50
100
150
200
50
100
150
200
Number Segments
Number Segments
Number Segments
(d) ETTh2 (H= 720)
(c) ETTh2 (H= 336)
(f) ETTh2 (H= 48)
Figure A3: Performance vs. number of segment. In (a), (b), and (c), UCR datasets for classification used, and in (d), (e). and (f) the ETTh2 dataset for forecasting used with different horizon lengths. higher value for accuracy is better, and a lower value MSE is better.
19

---

A.2 Weighted average parameters during training
In Figure A4, we present visual overview of how the weighted average parameters wi and W2 update along with the rest of the model during training on ETThl (H 24) and ETTh2 (H 720) in multivariate settings. We use LaST [19] as baseline. observed that the weights converge to their final values at the end of training.
1.4
2.0
1.0
Weight
W
Weight
W
1.0
0.8
W2
0.5
0.0
0.4
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
0.0
0.2
0.4
0.6
0.8
1.0
12
1.4
1.6
lteration
-10*
Iteration
x10
(a) ETTh2(M) H 720
(b) ETThI(M) H =
Figure A4: The progression of the weighted average parameters W/ and W2-
A.3
Visualisation of the segments
Figure AS visualizes three sample time- series and how their respective segments rearranged using S3. The figure depicts the original sequence, the shuffled sequence, and the final output.
Input
Input
Input
10
32
16
Shuffled
Shuffled
Shuffled
Output
Output
Output
Figure A5: Sample visualizations of S3 on UCI datasets (left and middle) and the ETThl dataset (right) with VS2 as the baseline. We have 8 segments for the UCI datasets, and 16 for the ETTh1.
A.4 Comparison with data augmentation methods
The proposed S3 layer is fundamentally different from data augmentation techniques, in that unlike traditional augmentation methods, S3 introduces learnable parameters that are jointly optimized with the rest of the model, allowing S3 to adapt dynamically throughout the training process. To further explore the differences berween and data augmentation, we conduct an experiment where, before training on the E multivariate dataset, we apply the following augmentations: (1) shuffling augmentation with varying numbers of segments, (2) shuffling augmentation combined with mixup, (3) noise augmentation with zero mean, variance of 1, and magnifying coefficient of 0.01, and (4)
20

---

7.5
Table AS: Comparison of S3 with different data augmentation techniques on the ETTml multivariate dataset. All values are MSE.
Difference
2.5
Model
TS2Vee
LaST
0.0
Baseline
0.6616
0.3366
Baseline shuffle (seg = 8)
0.6170
0,3670
500
1000
1500
2000
2500
Baseline shuffic (seg 8) mixup
0.6105
0.3556
No. training samples
Baseline shuffle (seg = 16)
0.6245
0,3652
Baseline shuffle (sog = 16) mixup
0.6203
0,3523
Figure A6: Dataset size improvement due to S3. Here, different subsets of the LSST dataset from the UEA archive are randomly selected to create different sized variants. The results are averaged over three runs.
Baseline noise
0.6302
0.3294
Baseline moise mixup
0.6064
0,3352
Baseline [52]
0.6600
0.3349
Baselinc $3
0.3959
0.1002
noise augmentation combined with mixup. The results are presented in Table A5 where we observe that S3 outperforms such data augmentation strategies. Additionally, data augmentation techniques typically have stronger impact on smaller datasets, often due to the lack of variation and diversity, but their effect is less pronounced on larger datasets [53]. To evaluate this, we use the LSST dataset from UEA and create 10 different subsets by randomly dropping varying amounts of data, ranging from 20% to 99%. We then retrain SoftCLT [36], both with and without S3, on these subsets as well the original dataset. The results, averaged over three runs, presented in Figure A6. These results show no clear trend in performance gain relative to dataset size, indicating that S3 benefits datasets regardless of their size.
A.5 Case-study on factors influencing the improvement by S3
To investigate the scenarios where S3 is most effective, we conduct additional experiments considering three key factors: (1) the length of time-series sequences, (2) the degree of non-linearity, and (3) the presence of long-term temporal dependencies. For (I), we examine the percentage difference relative sequence length. For (2), consider the percentage difference relative to the mean of squared residuals from linear model. Lastly, for (3), we analyze the percentage difference in relation to the Hurst exponent [54]. We use PatchTST [32] for experiments (2) and (3), while Informer [10] is used for experiment (1) since PatchTTST cannot be used due to its reliance on equal-length the ETT, Weather, and Electricity datasets. The results of this analysis are shown in Table A6, where m represents the slope of the linear relationship. and R is Pearson's correlation coefficient. We observe direct positive relationships between the percentage difference upon adding S3 and both sequence length and long-term temporal dependency, while non-linearity in the time-series shows no considerable relationship. For sequence length, the moderate correlation suggests that while the effect is not very strong, it is consistent, indicating that sequence length likely a relevant factor for percentage difference. Long-term temporal dependency has strong impact percentage difference, as indicated by the steep slope. However, the lower correlation suggests that other factors the data may also influence this relationship. We hypothesize that when long- long-term temporal dependencies are simple to learn or highly repetitive, the impact of S3 will be less pronounced, as re-ordering may not be necessary learn such simple dynamies. On the other hand, for more complex long-term dependencies, S3 has much stronger impact, as reflected by the high slope.
inputs. We conduct all three experiments
Table A6: Linear analysis between three factors (sequence length, non-linearity, and long-term temporal dependency) and percentage difference.
Factor
R
Sequence length
+0.04
+0,55
Non-linearity
0.00
+0.05
Long-term temporal dependency
+2.89
+0.26
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, see Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
This paper has no theoretical assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The implementation details are detailed in Section 4, and the full documented code is provided.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer:
[Yes]
reproducible, they Justification: The code is provided with sufficient documentation to be able to run it in few simple steps.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
httpssccccbppccccceeeeeeeeecseee htt///giiesseeeee http:/uudee//oeduuiiieeeeee /Cc////oodmsssseeeeeeeeeee wwwssssiicccyyycccc for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to unders tand the results?
Answer: [Yes]
Justification: We use the same experimental settings as the baselines that have incorpo- rated S3 into, except for experiments on PatchTST and LaST with the electricity dataset where use a batch size of 8. We have clarified this in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In Table we show the standard deviation performance metric for all base- lines single dataset for both classification and forecasting by repeating the experiments for 10 different seed values, observing very small sensitivity to seed values. We do not perform this for each experiment in main paper, it would be computationally too expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The compute resources is presented Section 4. Additionally we have separate study in Table 10 in which we analyze the exact computational overhead w.r.t. the baselines.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https ttttps://neuue htt////eerrpp..eee
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See section. 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Apart from our code, that is well documented, we do not release any assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper doess not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "Lhb1G9uQox", "reviewer": "Reviewer_FZgx", "summary": "The paper introduces a simple but effective differentiable module that performs pre-processing to input multivariate time-series before being fed into any differentiable model for arbitrary task. The pre-processing involves segmenting, shuffling the segments and stiching them together. The novelty include making this seemingly discrete operations into a differentiable module. This simple idea yields significant improvement in performance of different kinds of models over variety of datasets.", "questions": "See weaknesses", "limitations": "1. Lack of understanding on the segment permutations generated and why they are better for the model performance atleast qualitatively", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The method is simple and easy to add to most deep learning models\n2. The technical details are well-motivated and explained\n3. The method also improves training efficiency and convergence time along with performance with very little increate in model complexity\n4. Experimental results across different tasks are strong", "weaknesses": "1. Visualization and any qualitative study on the shuffling and segments generalted by S3 would greatly benefit the readers.\n2. How well does it optimize transformer based models, especially those that already do segmentation like PatchTST since the attention module captures the relations all pairs of segments already?\n3. Does the representations due to S3 generalize to multiple tasks at a time or do we need to retrain for each task?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. Below we provide a careful point-by-point response to each question. We would be happy to provide additional discussions/information in the author-reviewer discussion period should you have any follow-up questions.\n\n> Visualizations and qualitative analysis.\n\nWe have now included several visualizations in **Figure R1** of the enclosed document. The figure shows how the segments are rearranged according to what the model thinks is optimal for the task.\n\n> Performance on transformer-based models.\n\nWe have now added S3 to PatchTST on the ETT (univariate and multivariate), weather, and electricity datasets and obtained the results presented in **Table R1** of the accompanying document. We observe that S3 is able to further improve the performance of the additional baselines. Also, please note that in the initial set of experiments originally presented in our paper, Informer [1] is a transformer-based model, where we observed significant improvements when S3 was added (please see Tables 2 and 3 in our paper).\n\n> Does S3 need to be trained for each task?\n\nWe naturally retrain S3 for each task since the backbone model itself requires to be retrained as well. The S3 layer is simply added to the backbone and retrained just like all the other layers of the model. Having said this, we performed an experiment where we selected the exact same hyperparameters for all the tasks to observe how well a common hyperparameter could perform. We present this result in Table 6 of the original paper, where we observe that although the results are expectedly lower than the optimum hyperparameters, we still see meaningful gains.\n\n**References**\n\n[1] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai hang. Informer: Beyond efficient transformer for long sequence time-series forecasting. AAAI, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RhsdsSBMVs", "reviewer": "Reviewer_xbmW", "summary": "The paper paper introduces a new approach called Segment, Shuffle, and Stitch (S3) to enhance time-series representation learning. The method involves segmenting the time-series into non-overlapping parts, shuffling them optimally, and stitching them back together along with the original sequence.\n\nKey contributions include:\n\n- Proposing the S3 mechanism to improve time-series representation learning by dynamically reordering segments.\n- Demonstrating that S3 can be integrated with existing neural architectures like CNNs and Transformers, resulting in significant performance improvements.\n- Showing through extensive experiments that S3 enhances performance in time-series classification and forecasting tasks, with improvements up to 68%.", "questions": "- The 3 points in weaknesses are also questions in the sense that they ask for some new experiments to be performed. Addressing those points would be my first recommendation.\n- Intuitively, it feels like this work is to some extent a form of bootstrap (as data augmentation) combined with a mixup-like sample interpolation. I may be wrong on this and am happy to discuss. If so, could the authors do more of an ablation study connected to this. I.e. how does the approach outperform other (non-permutation)-based data augmentation strategies combined with the same summation operation?\n\nEdit: I have read the author's rebuttal. They have addressed questions I had and I am as a result raising my score to a 6.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Code is available, making reproducing this paper easier.\n- Paper is clear.\n- Results appear good, when considered on the set of baselines and dataset picked by the authors.", "weaknesses": "- Tables 1 and 2 focus on the ETT datasets, which are only a (highly intra-correlated) subset of the common forecasting datasets: Electricity, Traffic, Weather, Illness...\n- I see no mention of CoST in the results tables, despite being cited in the paper. This is usually a very strong baseline for contrastive approaches. Including it would certainly paint a more complete picture of the results landscape. On a related note this also applies to e.g. more recent transformer baselines. Informer is relevant, but also very far from state of the art.\n- Error bars would help one better contextualize the results.\n- The lack of an ablation study makes understanding the reason this works more complicated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. Below we provide a careful point-by-point response to each question. We would be happy to provide additional discussions/information in the author-reviewer discussion period should you have any follow-up questions.\n\n> Other forecasting datasets.\n\nAs per your comment, we have now performed additional experiments on the popular Electricity and Weather datasets, and present the results in **Table R2** in the accompanying doc. We observe that adding S3 results in overall improvements over all the baselines for both datasets.\n\n> CoST and other transformer baselines.\n\nAs per your comment, we have now added S3 to CoST and PatchTST [1] on the ETT (univariate and multivariate), Weather, and Electricity datasets, and present the results in **Table R3** of the accompanying doc. We observe improvements for all the datasets.\n\n> Error bars.\n\nThe sources of variation in the performance of S3 are $\\mathbf{P}$, $\\mathbf{w}_1$, and $\\mathbf{w}_2$ as well as the random seed for the baseline model to which S3 is added. We have performed an experiment where we present the standard deviations of several classification and forecasting models in Table 7 of the original paper over 5 random initializations. We observe that (a) the standard deviations are generally very small, and (b) any possible variance stems from the baseline model as opposed to the S3 module.\n\n> Ablation and comparison against augmentation.\n\n**(1)** We performed a detailed ablation of each component of S3 in Table 5 of the original paper where we observed a drop in performance when each one is ablated. \n\n**(2)** In regards to the question on augmentation, we acknowledge that at first glance S3 seems to share similarities to data augmentation. However, S3 has meaningful **learnable** parameters that **train along with the rest of the model** to enable a segmentation and mixup that is (a) **variable** throughout the training process, and (b) **customized** to that model and task, setting it highly apart. To further test this, we have now performed a new experiment where we apply shuffling augmentation (with different number of segments) before training, shuffling augmentation plus mixup, noise augmentation (with a mean of zero and a variance of 1, with a magnifying coefficient of 0.01), and noise augmentation plus mixup, on the ETTm1 multivariate dataset. Please see Table R2 in the accompanying doc where we observe that S3 outperforms such data augmentation strategies. \n\n**(3)** Lastly, a common characteristic of data augmentation techniques is that they tend to improve performance well for smaller datasets (often due to the lack of variations and diversity) but not so much for larger datasets [2]. To evaluate this, we took the LSST dataset from UEA and created 10 different subsets by dropping different amounts of data ranging from 20% to 99%. We then retrained SoftCLT [3] with and without S3 on these subsets as well as the original dataset. The results (averaged over 3 runs) are presented in **Figure R2** of the accompanying doc. The results demonstrate that there is no evident trend in the performance gain with respect to the dataset size, indicating that S3 does not only benefit smaller datasets.\n\n**References**\n\n[1] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, 2023.\n\n[2] Iwana BK, Uchida S. An empirical survey of data augmentation for time series classification with neural networks. Plos one. 2021\n\n[3] Seunghan Lee, Taeyoung Park, and Kibok Lee. Soft contrastive learning for time series. ICLR, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PQ6MFEkGOn", "reviewer": "Reviewer_5XMz", "summary": "This paper proposes a new neural network design element which segments, shuffles, and stitches time series for improved representation learning. They evaluate their methods on forecasting and classification tasks, and show that S3 benefits some widely used baselines.", "questions": "I do not have questions per se, but I am listing some things that I am curious about below:\n\nI would also encourage the authors to evaluate the benefits of S3 on some recent time series foundation models such as MOMENT [2], Chronos [3], Moirai [4], TimesFM [5], and/or LagLLama [6]. The MOMENT model does both classification and forecasting, so it might be interesting to see how S3 benefits pre-trained models, say by just training the S3 layer and freezing the pre-trained backbone (or some variation of this experiment).\n\nOn a similar note, I wonder if S3 improves generalization and hurts memorization, or vice versa. It would be interesting to do some transfer learning experiments where you train on some time series data and evaluate the model on other time series data (see MOMENT or PatchTST for inspiration). \n\n### References\n[2] Goswami, Mononito, et al. \"Moment: A family of open time-series foundation models.\" arXiv preprint arXiv:2402.03885 (2024).\n[3] Ansari, Abdul Fatir, et al. \"Chronos: Learning the language of time series.\" arXiv preprint arXiv:2403.07815 (2024).\n[4] Woo, Gerald, et al. \"Unified training of universal time series forecasting transformers.\" arXiv preprint arXiv:2402.02592 (2024).\n[5] Das, Abhimanyu, et al. \"A decoder-only foundation model for time-series forecasting.\" arXiv preprint arXiv:2310.10688 (2023).\n[6] Rasul, Kashif, et al. \"Lag-llama: Towards foundation models for time series forecasting.\" arXiv preprint arXiv:2310.08278 (2023).", "limitations": "The authors have a very brief description of limitations of their study.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. To the best of my knowledge, the idea is novel, and fundamentally challenges and changes how to learn representations for time series data\n2. The paper is well written and easy to follow\n3. Experiments are well-designed, and results are promising", "weaknesses": "I have not found any major weaknesses in the methodology or experimental design. However,  I think that the paper might benefit from showing what the S3 module is actually learning. For example, the authors can include the segmented, shuffled, and stitched time series on a particular dataset as an example, along with the weighted time series (used as input to the model), and the original time series. This might provide some intuition as to how this design element improves predictive performance. \n\nI think there's always scope to improve experimental design. TS2Vec is a excellent choice for classification, but not for forecasting. I would recommend that the authors use methods such as PatchTST (transformer-based) or iTransformer, TimesNet (CNN-based), N-BEATs or N-HITS (MLP-based) etc. for time series forecasting. For classification, it would also be good to compare with fully supervised methods such as ResNet1D (see [1]). \n\n### References\n[1] Ismail Fawaz, Hassan, et al. \"Deep learning for time series classification: a review.\" Data mining and knowledge discovery 33.4 (2019): 917-963.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. Below we provide a careful point-by-point response to each question. We would be happy to provide additional discussions/information in the author-reviewer discussion period should you have any follow-up questions.\n\n> Visualisation of S3\n\nWe have now included several visualizations in **Figure R1** in the enclosed document that demonstrate how the segments are rearranged according to what the model thinks is optimal for the task. The figures allow for a visual comparison between the original sequence, the shuffled sequence, and the final output.\n\n> Additional baselines for forecasting, and fully supervised baseline for classification\n\nWe have implemented S3 with PatchTST, CoST [1] on the ETT (univariate and multivariate), Weather, and Electricity datasets for forecasting, and the results are shown in Table R1 of the enclosed document. We observe that S3 is able to further improve the performance of the additional baselines. For a fully supervised method, our original submission (Table 1) included the baseline DSN which was fully supervised.\n\n> S3 with pre-trained foundation models\n\nThank you for the very interesting suggestion! Given the very limited time during the rebuttal period and the resources needed to explore large foundation models, we were unable to perform these experiments at this time. We do agree that this area would be very interesting to explore for S3 and is indeed something we had discussed as an exciting future direction to take this work. We hope to follow up our current work with a follow up study on time-series foundation models.\n\n> Experiments on transfer learning\n\nWe have now performed an additional experiment in this regard: following the protocol used in TS2Vec [2], we trained the TS2Vec+S3 encoder on FordA for classification. We then froze the encoder and used the model along with fine-tuning of the classification head and S3 to classify the sequences for the other 127 UCR datasets. The average accuracy score over all 127 datasets for TS2Vec without S3 is 0.8037, and with S3 the accuracy score is 0.8160. Based on these results, it appears that S3 is indeed effective toward both in-dataset (in-distribution) and cross-dataset (out-of-distribution) settings. Please note that since the goal of S3 is to customize the reordering of the time-series specific to each dataset and task, the S3 layers would need to be fine-tuned as well (when the classification head is being re-trained), while the rest of the model stays frozen. We will add the full table of individual results on all the datasets in the final paper.\n\n**References**\n\n[1] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. ICLR, 2022.\n\n[2] Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. AAAI, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kb7xCYTdD8", "reviewer": "Reviewer_2e8Y", "summary": "This paper introduces a plug-and-play mechanism called Segment, Shuffle, and Stitch (S3) designed to enhance time-series representation learning in existing models. S3 operates by dividing the original sequence into non-overlapping segments and shuffling them in a learned manner that is optimal for the given task. It then reattaches the shuffled segments and performs a learned weighted sum with the original input to capture both the newly shuffled sequence and the original sequence. This proposed model can enhance the performance of specific models in classification and prediction tasks.", "questions": "What are the essential differences between the proposed method and other data augmentation methods?", "limitations": "See Weakness.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is easily comprehensible and straightforward.\n\nSufficient experiments are conducted to confirm the effectiveness of the method.", "weaknesses": "Lack of comparative methods:\nIn fact, the proposed method seems to share the same spirit as data augmentation methods in the time series field[1-4]. Why hasn't any data augmentation method been compared?\n\n\nSelection of baseline models:\nThe selected baseline model, Informer, seems somewhat outdated. Why not choose a more recent model, e.g., iTransformer[5] or PatchTST[6]?\n\n\nDataset for prediction task:\nThe author conducted experiments on three ETT datasets, but for prediction tasks, more datasets should be considered, e.g., traffic, electricity, and weather.\n\n\nTime-Series Representation Claim:\n As the author pointed out, more tasks should be considered for time series representation learning.\n\n\n[1]FRAUG: FREQUENCY DOMAIN AUGMENTATION FOR TIME SERIES FORECASTING  [2]Time Series Data Augmentation for Deep Learning: A Survey  [3]SimPSI: A Simple Strategy to Preserve Spectral Information in Time Series Data Augmentation [4]TOWARDS DIVERSE AND COHERENT AUGMENTATION FOR TIME-SERIES FORECASTING [5]ITRANSFORMER: INVERTED TRANSFORMERS ARE EFFECTIVE FOR TIME SERIES FORECASTING [6]A TIME SERIES IS WORTH 64 WORDS: LONG-TERM FORECASTING WITH TRANSFORMERS", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. Below we provide a careful point-by-point response to each question. We would be happy to provide additional discussions/information in the author-reviewer discussion period should you have any follow-up questions.\n\n> Comparison with data augmentation methods.\n\nWe acknowledge that at first glance S3 seems to share similarities to data augmentation. However, S3 has meaningful **learnable** parameters that **train along with the rest of the model** to enable a segmentation and mixup that is (a) **variable** throughout the training process, and (b) **customized** to that model and task, setting it highly apart. To further test this, we have now performed new experiments where we apply shuffling augmentation before training, shuffling augmentation plus mixup, noise augmentation, and noise augmentation plus mixup. In this experiment, we have also compared S3 to the augmentation method presented in the paper cited in your comment [1]. Please see **Table R2** in the accompanying doc where we observe that S3 outperforms such data augmentation strategies. Moreover, a common characteristic of data augmentation techniques is that they tend to improve performance well for smaller datasets (often due to the lack of variations and diversity) but not so much for larger datasets [2]. To evaluate the impact of S3 based on the size of the training set, we took the LSST dataset from UEA and created 10 different subsets by dropping different amounts of data ranging from 20% to 99%. We then retrained SoftCLT [3] with and without S3 on these subsets as well as the original dataset. The results (averaged over 3 runs) are presented in **Figure R2** of the accompanying doc. The results demonstrate that there is no evident trend in the performance gain with respect to the dataset size, indicating that S3 does not only benefit smaller datasets.\n\n> Experiments on a more recent transformer model\n\nWe have now added S3 to PatchTST and CoST [4] for forecasting on the ETT (univariate and multivariate), Weather, and Electricity datasets. The results are shown in **Table R1** of the enclosed document. In these additional experiments, we observe a similar trend where the addition of S3 considerably improves performance.\n\n> Experiments on additional dataset for forecasting \n\nWe have now performed additional experiments for forecasting electricity and weather datasets for several baselines. The results are shown in **Table R3** of the enclosed document.\n\n> Experiments on anomaly detection\n\nTo further expand the tasks beyond univariate and multivariate classification and forecasting, we have now performed additional experiments on two popular baselines with and without S3 for anomaly detection on the KPI and Yahoo datasets to further evaluate the improvements in representation learning that S3 brings. The results for this are shown in **Table R2** of the enclosed document.\n\n**References**\n\n[1] Xiyuan Zhang, Ranak Roy Chowdhury, Jingbo Shang, Rajesh Gupta, and Dezhi Hong.\nTowards diverse and coherent augmentation for time-series forecasting. ICASSP, 2023.\n\n[2] Iwana BK, Uchida S. An empirical survey of data augmentation for time series classification with neural networks. Plos one. 2021\n\n[3] Seunghan Lee, Taeyoung Park, and Kibok Lee. Soft contrastive learning for time series. ICLR, 2024.\n\n[4] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. ICLR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WUjKloq9SX", "reviewer": "Reviewer_dMD7", "summary": "This paper introduces a new method for time-series representation learning that enhances the modeling of non-adjacent segment dependencies. Specifically, the proposed method segments, shuffles in a learned manner and stitches the shuffled segments to combine with original time series. The proposed method is model-agnostic without adding significant parameter overhead and shows performance improvement across multiple classification and forecasting base models.", "questions": "1. The results in Tables 1, 2, and 3 seem to indicate more significant improvements in multivariate than in univariate time series tasks. Any reason behind this?\n\n2. What does the \"number of segments\" represent in Figure 6 and Figure A3? Is it the number of segments for the first layer or the final layer? If it refers to \"n\", then in Figure A3, this number seems to perform the best when it is larger than 100 for some datasets?\n\n3. Could you describe the inference process for the S3 method? Additionally, what are the computational overheads for training and inference times for S3?", "limitations": "The paper mentions potential expansions into tasks like imputation and anomaly detection. Further details on limitations from the reviewer are discussed in Weaknesses and Questions.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method permutes the original segments to better capture inter-relations between distant segments. It is model-agnostic and introduces minimal parameter overhead to the original model.\n\n2. Extensive experiments on various base models for both classification and forecasting tasks demonstrate the effectiveness of the proposed method.", "weaknesses": "1. It it not clear how the sorting process, specifically the calculation of permutation $\\sigma$ from $P$, is made differentiable.\n\n2. The compared forecasting baselines such as Informer are no longer state-of-the-art methods. Adding more recent baselines such as Time-LLM, GPT4TS, DLinear, PatchTST would provide a clearer understanding of the proposed method's comparative benefits.\n\n3. The basic assumption for S3 is that modeling non-adjacent dependencies is important. However, the paper lacks detailed case studies that demonstrate the specific types of non-adjacent dependencies effectively captured by S3, which are not addressed by existing models. Additionally, there is no case study to validate that the learned shuffling weights accurately represent these segment dependencies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. Below we provide a careful point-by-point response to each question. We would be happy to provide additional discussions/information in the author-reviewer discussion period should you have any follow-up questions.\n\n> Clarification on the differentiability of the sorting process (calculation of σ from P)\n\nWhile the calculation of **σ** from **P** is not differentiable, it is not used directly for the permutation of the segments. Instead, we use **σ** to populate the intermediate zero matrix **Ω** with elements from **P**. Matrix **Ω** is then turned into a binary permutation matrix which, when multiplied with the list of segments, permutes them in the correct order. Crucially, this matrix multiplication operation is differentiable, and **Ω**, which was built with elements from **P**, builds a bridge for the gradient to flow from the final output back to the elements of **P**. So even though the creation of **σ** itself is not differentiable, gradients can still flow back from the final output, through the permutation matrix, to **P** through the elements that were selected while creating **σ**. \n\n> Adding more recent forecasting baselines \n\nWe have now added S3 to PatchTST and CoST [1] for forecasting on the ETT (univariate and multivariate), Weather, and Electricity datasets. The results are shown in **Table R1** of the enclosed document. In these additional experiments, we observe a similar trend where the addition of S3 improves performance.\n\n> Case study.\n\nWe now present 3 sample time-series before and after the S3 layer in **Figure R1** of the accompanying doc (we will add more to the final paper). Here, we observe that S3 is optimized differently to learn a unique shuffling pattern that suits the baseline model and task. Given that the learnable parameters of S3 are optimized through training of the network and with the goal of maximizing the performance on that specific task, the final learned shuffling pattern serves the purpose of shuffling the time-series such that adjacent learning of originally non-adjacent segments results in more effective representations, as evidenced by the consistent improvement in results. The fact that the shuffling pattern in S3 is optimized with the learnable network and for each specific task is in fact an advantage as different models and different tasks may benefit from different re-structuring of the data.\n\n> Univariate vs. multivariate results.\n\nThank you for this interesting observation. One possible explanation for more improvements on multivariate time-series can be that multivariate datasets inherently contain more complex dynamics and interdependencies (both temporally and between variables), which can be more challenging for models to understand. Accordingly, S3 can further help with learning of the more complex temporal dependencies, further improving performance. Empirically, we can observe that models tend to struggle more with multivariate datasets from Tables 2 and 3 in the original paper. For instance, TS2Vec has an average MSE value of 0.1184 on ETTh1 univariate forecasting, and 0.7612 on ETTh1 multivariate forecasting, which is significantly larger than the former. While we acknowledge that comparing metric values for a model across different datasets or tasks can be misleading due to potential inconsistencies in data scales or nature of the task, in this case, both the dataset and the task are more or less the same. Accordingly, the key difference between univariate and multivariate settings here would be the complexity, i.e., the dataset and tasks are more complex in multivariate due to the higher number of dimensions. \n\n> Clarification on number of segments in Figure 6 and A3  \n\nThe “number of segments” in Figure 6 and A3 from the original paper is the maximum number of segments over all S3 layers in the model. For instance, let us assume 3 layers of S3 with $n_0$=4, $n_1$=8, $n_2$=16 (where $n_i$ is the number of segments for layer $i$). Here, the maximum number of segments is 16. Similarly, if $n_0$=24, $n_1$=12, $n_2$=6, then the maximum number of segments is 24. We will make sure to clarify this in the final paper.\n\n> Inference process\n\nDuring inference, we integrate S3 as the first layer of the model. The input sequences go through S3 first where they are segmented into several segments (the number of segments was optimized during training). These segments are then shuffled using the set of shuffling parameters that were optimized and fixed at the end of the training phase, and the learned weighted average yields the final sequence. This sequence is then fed into the subsequent stages of the baseline model for further processing and output generation. So essentially, S3 acts like any other learnable layer in a network where it is optimized during training and used with the fixed parameters during inference.\n\n> Computational overhead\n\nS3 adds only a few hundred parameters, the exact count of which depends on the specific hyperparameters selected. We show the number of added parameters in Table 8 of the paper, where we observe that in comparison to hundreds of thousands to millions of parameters in the baselines, S3 adds very few parameters, which can be considered negligible. In terms of inference time, given the ratio of added parameters vs. the original number of parameters in the baseline models, the added time is also negligible.\n\n> Experiments on Anomaly detection\n\nTo further expand the tasks beyond univariate and multivariate classification and forecasting, we have now performed additional experiments for the task of anomaly detection on the KPI and Yahoo datasets. The results for this are shown in **Table R4** of the enclosed document, where we observe that the addition of S3 results in considerable performance gains.\n\n**References**\n\n[1] Woo, et al., CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. ICLR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their time and for providing us with constructive feedback. We are happy to see the engaging comments given by all the reviewers. We have carefully addressed all the concerns raised under the individual response section. Following, we provide a summary of our responses.

* **Adding S3 to PatchTST and CoST for forecasting**:  As per the suggestion of **Reviewers FZgx, xbmW, 5XMz, and 2e8Y**, we added S3 to PatchTST and CoST for the task of forecasting. The results are presented in **Table R1** of the enclosed document.
* **Comparison with data augmentation**: As per the questions by **Reviewers xbmW and 2e8Y**, we provide a discussion on the key differences between S3 and data augmentation, and also performed several experiments to compare S3 against data augmentation techniques. The results for these are outlined in **Table R2** and **Figure R2** of the enclosed document.
* **Expanding forecasting baselines on Electricity and Weather datasets**: As per the feedback of **Reviewers xbmW and 2e8Y**, we have now performed additional experiments for several forecasting baselines on the Electricity and Weather datasets. The results for these are presented in **Table R3** of the enclosed document.
* **Visualization and qualitative analysis**: As per the suggestion of **Reviewers FZgx and 5XMz**, we have included several visualizations (**Figure R1** in the enclosed document) which show how the segments are rearranged with S3 and allow for a visual comparison between the original sequence, the shuffled sequence, and the final output.
* **Experiments on anomaly detection**: As per the suggestion of **Reviewers 2e8Y and dMD7**, we have now performed experiments on anomaly detection on the KPI and Yahoo datasets. The results for this are shown in **Table R4** of the enclosed document.
* **Additional clarifications**: we provide additional clarifications regarding differentiability, inference overhead, and others.

Meta-review: The paper presents a simple but effective differentiable module called Segment, Shuffle, and Stitch (S3), to combine these discrete operations in analysing time-series data sets. This addresses a key problem in multi-variate time-series data analysis for several architectures, including PatchTST and CoST. The proposed method is validated on several multi-variate and univariate time-series data sets. 

However, the model requires retraining with each task (Authors response to Reviewer FZgx) and S3 has not been studied in conjunction with pre-trained foundation models (Authors response to Reviewer 5XMz). The method proposed in this paper has the potential to impact  pre-processing of time-series data sets, while analysing time-series data sets that are ubiquitous in several applications. This can be achieved through lesser computational resources.
Decision: Accept (poster)