Title: Segment, Shuffle, and Stitch: A Simple Layer for Improving Time-Series Representations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Time Series

Deep Learning

Representation Learning

Temporal Mechanism
Abstract: Existing approaches for learning representations of time-series keep the temporal arrangement of the time-steps intact with the presumption that the original order is the most optimal for learning. However, non-adjacent sections of real-world time-series may have strong dependencies. Accordingly, we raise the question: Is there an alternative arrangement for time-series which could enable more effective representation learning? To address this, we propose a simple plug-and-play neural network layer called Segment, Shuffle, and Stitch (S3) designed to improve representation learning in time-series models. S3 works by creating non-overlapping segments from the original sequence and shuffling them in a learned manner that is optimal for the task at hand. It then re-attaches the shuffled segments back together and performs a learned weighted sum with the original input to capture both the newly shuffled sequence along with the original sequence. S3 is modular and can be stacked to achieve different levels of granularity, and can be added to many forms of neural architectures including CNNs or Transformers with negligible computation overhead. Through extensive experiments on several datasets and state-of-the-art baselines, we show that incorporating S3 results in significant improvements for the tasks of time-series classification, forecasting, and anomaly detection, improving performance on certain datasets by up to 68\%. We also show that S3 makes the learning more stable with a smoother training loss curve and loss landscape compared to the original baseline. The code is available at https://github.com/shivam-grover/S3-TimeSeries.

Segment, Shuffle, and Stitch: A Simple Layer for Improving Time-Series Representations

Abstract

Existing approaches for learning representations of time-series keep the temporal arrangement of the time-steps intact with the presumption that the original order is the most optimal for learning. However, non-adjacent sections of real-world time-series may have strong dependencies. Accordingly, we raise the question: Is there an alternative arrangement for time-series which could enable more effective representation learning? To address this, we propose a simple plug-andplay neural network layer called Segment, Shuffle, and Stitch (S3) designed to improve representation learning in time-series models. S3 works by creating nonoverlapping segments from the original sequence and shuffling them in a learned manner that is optimal for the task at hand. It then re-attaches the shuffled segments back together and performs a learned weighted sum with the original input to capture both the newly shuffled sequence along with the original sequence. S3 is modular and can be stacked to achieve different levels of granularity, and can be added to many forms of neural architectures including CNNs or Transformers with negligible computation overhead. Through extensive experiments on several datasets and state-of-the-art baselines, we show that incorporating S3 results in significant improvements for the tasks of time-series classification, forecasting, and anomaly detection, improving performance on certain datasets by up to 68%. We also show that S3 makes the learning more stable with a smoother training loss curve and loss landscape compared to the original baseline. The code is available at https://github.com/shivam-grover/S3-TimeSeries.

1 Introduction

Time-series data serve an important role across diverse domains, including but not limited to health analytics [1, 2, 3], human-computer interaction [4, 5], human activity recognition [6, 7], climate analysis [8, 9], energy consumption [10, 11, 12], traffic management [13, 14], financial markets [15], and others. The pervasive nature of time-series data has resulted in considerable interest among researchers, leading to the development of a variety of deep learning solutions for classification [16, 17, 18] and forecasting tasks [10, 19, 20]. Neural architectures such as convolutional networks [16, 21], recurrent networks [22, 23], and Transformers [10, 20] can capture the essential spatial and temporal information from time-series. Notably, these approaches have frequently outperformed traditional approaches, including Dynamic Time Warping [24], Bag of Stochastic Frontier Analysis Symbols [25], and the Collective of Transformation-Based Ensembles [26] in various scenarios.

While both traditional machine learning and deep learning solutions aim to extract effective goalrelated representations prior to classification or forecasting, the general approach is to keep the original temporal arrangement of the time-steps in the time-series intact, with the presumption that the original order is the most optimal. Moreover, most existing models do not have explicit mechanisms to explore the inter-relations between distant segments within each time-series, which may in fact have strong dependencies despite their lack of proximity. For example, CNN-based models for

time-series learning generally utilize fixed convolutional filters and receptive fields, causing them to only capture patterns within a limited temporal window [27, 28]. As a result, when faced with time-series where important patterns or correlations span across longer time windows, these models often struggle to capture this information effectively [28]. Dilated convolutional neural networks partially solve this by increasing the receptive field through the dilation rate. However, they are still practically limited by their inherent architectures, as their receptive fields rely on the number of layers, which may not be large enough to fully capture the long-range dependencies and can lead to vanishing gradients as more layers are added [29]. Similarly, the out-of-the-box effectiveness of Transformers [30] in capturing long-term dependencies highly depends on a variety of factors such as sequence length, positional encoding, and tokenization strategies. Accordingly, we ask a simple question: Is there a better arrangement for the time-series that would enable more effective representation learning considering the classification/forecasting task at hand?

In this paper, we introduce a simple and plug-and-play network layer called S egment, S huffle, and S titch, or S3 for short, designed to enhance time-series representation learning. As the name suggests, S3 operates by segmenting the time-series into several segments, shuffling these segments in the most optimal order controlled by learned shuffling parameters, and then stitching the shuffled segments. In addition to this, our module integrates the original time-series through a learned weighted sum operation with the shuffled version to also preserve the key information in the original order. S3 acts as a modular mechanism intended to seamlessly integrate with any time-series model and as we will demonstrate experimentally, results in a smoother training procedure and loss landscape. Since S3 is trained along with the backbone network, the shuffling parameter is updated in a goal-centric manner, adapting to the characteristics of the data and the backbone model to better capture the temporal dynamics. Finally, S3 can be stacked to create a more fine-grained shuffling with higher levels of granularity, has very few hyper-parameters to tune, and has negligible computation overhead.

For evaluation, we integrate S3 in a variety of neural architectures including CNN-based and Transformer-based models, and evaluate performance across various classification, univariate forecasting, and multivariate forecasting datasets, observing that S3 results in substantial improvements when integrated into state-of-the-art models. Specifically, the results demonstrate that integrating S3 into state-of-the-art methods can improve performance by up to 39.59% for classification, and by up to 68.71% and 51.22% for univariate and multivariate forecasting, respectively. We perform detailed ablation and sensitivity studies to analyze different components of our proposed S3 layer.

Our contributions are summarized as follows:

We propose S3, a simple and modular network layer that can be plugged into existing neural architectures to improve time-series representation learning. By dynamically segmenting and shuffling the input time-series across the temporal dimension, S3 helps the model perform more effective task-centric representation learning.

By stacking multiple instances of S3, the model can perform shuffling at different granularity levels. Our proposed layer has very few hyperparameters and negligible added computational cost.

Rigorous experiments on various benchmark time-series datasets across both classification and forecasting tasks demonstrate that by incorporating S3 into existing state-of-the-art models, we improve performance significantly. Experiments also show that by adding our proposed network layer, more stable training is achieved.

We make our code public to contribute to the field of time-series representation learning and enable fast and accurate reproducibility.

2 Related work

Deep learning architectures have recently made significant progress in the area of time-series representation learning. In the category of convolution-based methods, DSN [16] introduces dynamic sparse connections to cover different receptive fields in convolution layers for time-series classification. In another convolution-based approach, SCINet [21] captures temporal features by partitioning each time sequence into two subsequences at each level to effectively model the complex temporal dynamics within hierarchical time-series data.

Transformer-based approaches are another category of solutions that have drawn considerable attention. Informer [10] improves the capabilities of the vanilla Transformer on long input sequences by

introducing self-attention distillation using ProbSparse which is based on Kullback-Leibler divergence, and a generative style decoder. Autoformer [20] introduces decomposition blocks with an auto-correlation mechanism that allows progressive decomposition of the data. ContiFormer [31] integrates the continuous dynamics of Neural Ordinary Differential Equations with the attention mechanism of Transformers. PatchTST [32] enhances time-series forecasting by leveraging patching techniques to split long input sequences into smaller patches, which are then processed by a Transformer to capture long-term dependencies efficiently. Finally, [33] investigated the effectiveness of Transformers in dealing with long sequences for forecasting and highlighted the challenges Transformers encounter in this regard.

Recently, foundation models for time-series data, which are large-scale, pre-trained models designed to capture complex temporal patterns, have gained popularity. A prominent example is Moment [34], which pre-trains a Transformer encoder using a univariate setting on a large and diverse collection of data called 'Pile'. It leverages a masked reconstruction method and is capable of performing various downstream forecasting tasks after fine-tuning, demonstrating strong adaptability and forecasting accuracy.

Contrastive methods have recently demonstrated state-of-the-art performances in time-series learning. TS2Vec [17] employs unsupervised hierarchical contrastive learning across augmented contextual views, capturing robust representations at various semantic levels. InfoTS [35] uses informationaware augmentations for contrastive learning which dynamically chooses the optimal augmentations. To address long-term forecasting, [28] introduces a contrastive approach that incorporates global autocorrelation alongside a decomposition network. Finally, SoftCLT [36] is a recent contrastive approach that uses the distances between time-series samples (instance-wise contrastive) along with the difference in timestamps (temporal contrastive), to capture the correlations between adjacent samples and improve representations.

Given the existence of seasonal and trend information in time-series, disentanglement is an approach that has been widely used across both classical machine learning [37, 38, 39] and deep learning solutions [40, 19]. For instance, CoST [40] attempts to capture periodic patterns using frequency domain contrastive loss to disentangle seasonal-trend representations in both time and frequency domains. Similarly, LaST [19] recently employed variational inference to learn and disentangle latent seasonal and trend components within time-series data.

3 Method

Problem definition. Given a set of N timeseries instances as { x i } N i =1 , where x i ∈ R T × C has a length of T and C channels, the goal is to optimally rearrange the segments of x i and form a new sequence x ′ i to better capture the underlying temporal relationships and dependencies within the time-series, which would consequently lead to improved representations given the target task.

Proposed mechanism. We propose S3, a simple neural network component for addressing the aforementioned problem in three steps, as the name suggests, Segment, Shuffle, and Stitch, described below (see Figure 1).

The Segment module splits the input sequence x i into n non-overlapping segments, each containing τ time-steps, where τ = T/n . The set of segments can be represented by S i = { s i, 1 , s i, 2 , . . . , s i,n } ∈ R ( τ × C ) × n where s i,j = x i [( j -1) τ : jτ ] and s i,j ∈ R τ × C .

Segment

ffl

e

Stitch

Segment

ffl

e

Stitch

Segment

ffl

e

Stitch

S3 layer

Learned

weighted

sum

Learned

weighted

sum

Learned

weighted

sum

x

x

′

The segments are then fed into the Shuffle module, which uses a shuffle vector P = { p 1 , p 2 , . . . , p n } ∈ R n to rearrange the segments in the optimal order for the task at hand. Each shuffling parameter p j in P corresponds to a segment s i,j

in S i . P is essentially a set of learnable weights optimized through the network's learning process, which controls the position and priority of the segment in the reordered sequence. The shuffling process is quite simple and intuitive: the higher the value of p j , the higher the priority of segment s i,j is in the shuffled sequence. The shuffled sequence S shuffled i can be represented as

where the segments in S i are sorted according to the values in P . Permuting S i based on the sorted order of P is not differentiable by default, because it involves discrete operations and introduces discontinuities [41]. Soft sorting methods such as [42, 41, 43] approximate the sorted order by assigning probabilities that reflect how much larger each element is compared to others. While this approximation is differentiable in nature, it may introduce noise and inaccuracies, while making the sorting non-intuitive. To achieve differentiable sorting and permutation that are as accurate and intuitive as traditional methods, we introduce a few intermediate steps. These steps create a path for gradients to flow through the shuffling parameters P while performing discrete permutations on S i based on the sorted order of P . We first obtain the indices that sort the elements of P using σ = Argsort ( P ) . We have a list of tensors S = [ s 1 , s 2 , s 3 , ... s n ] (for simplicity, we skip the index i ) that we aim to reorder based on the list of indices σ = [ σ 1 , σ 2 , ....σ n ] in a differentiable way. We then create a ( τ × C ) × n × n matrix U , which we populate by repeating each s i , n times. Next, we form an n × n matrix Ω where each row j has a single non-zero element at position k = σ j which is p k . We convert Ω to a binary matrix ˜ Ω by scaling each non-zero element to 1 using a scaling factor 1 Ω j,k . This process creates a path for the gradients to flow through P during backpropagation.

By performing the Hadamard product between U and ˜ Ω , we obtain a matrix V where each row j has one non-zero element k equals to s k . Finally, by summing along the final dimension and transposing the outcome, we obtain the final shuffled matrix of size S shuffled i ∈ R τ × C × n . To better illustrate, we show a simple example with S = [ s 1 , s 2 , s 3 , s 4 ] and a given permutation σ = [3 , 4 , 1 , 2] . We calculate V as

and S shuffled i is obtained by

We previously assumed that the set of shuffling parameters P is one-dimensional, containing n scalar elements, each corresponding to one of the segments. By employing a higher-dimensional P , we can introduce additional parameters that enable the model to capture complex representations that a single-dimensional P could struggle with. Therefore, we introduce a hyperparameter λ to determine the dimensionality of the vector P . When λ = m , the size of P becomes n × n ×··· × n (repeated m times). We then perform a summation of P over the first m -1 dimensions to obtain a one-dimensional vector. Mathematically, this is represented as

This results in a one-dimensional matrix ˜ P , which we then use to compute the permutation indices σ = Argsort ( ˜ P ) . This approach allows us to increase the number of shuffling parameters, thereby capturing more complex dependencies within the time-series data, without affecting the sorting operations.

In the final step, the Stitch module concatenates the shuffled segments S shuffled i to create a single shuffled sequence ˜ x i ∈ R T × C as ˜ x i = Concat ( S shuffled i ) . To retain the information present in the original order along with the newly generated shuffled sequence, we perform a weighted sum between x i and ˜ x i with learnable weights w 1 and w 2 optimized through the training of the main network. For practical convenience, we can use a simple learnable Conv1D or MLP layer taking x i and ˜ x i as inputs and generating the final time-series output x ′ i ∈ R T × C .

Stacking S3 layers. Considering S3 as a modular layer, we can stack them sequentially within a neural architecture. Let's define ϕ as a hyperparameter that determines the number of S3 layers. For simplicity and to avoid defining a separate segment hyperparameter for each S3 layer, we define θ which acts as a multiplier for the number of segments in subsequent layers as

where n is the number of segments in the first S3 layer. When multiple S3 layers are stacked, each layer ℓ from 1 to ϕ will segment and shuffle its input based on the output from the previous layer. We can formally represent the output of each layer ℓ as x ′ ℓ by

where x ′ 0 is the original time-series x , P ℓ is the set of shuffling parameters for layer ℓ , and ( w ℓ, 1 , w ℓ, 2 ) are the learnable weights for the sum operation between the concatenation of the shuffled segments and the original input, at layer ℓ . Figure 1 presents an example of three S3 layers ( ϕ = 3 ) applied to a time-series with n = 2 and θ = 2 .

All the P ℓ values are updated along with the model parameters, and there is no intermediate loss for any of the S3 layers. This ensures that the S3 layers are trained according to the specific task and the baseline. In cases where the length of input sequence x is not divisible by the number of segments n , we resort to truncating the first T mod n time-steps from the input sequence. In order to ensure that no data is lost and the input and output shapes are the same, we later add the truncated samples back at the beginning of the output of the final S3 layer.

4 Experiment setup

Evaluation protocol. For our experiments, we integrate S3 into existing state-of-the-art models for both time-series classification and forecasting. We first train and evaluate each model adhering strictly to their original setups and experimental protocols. We then integrate S3 at the beginning of the model, and train and evaluate it with the same setups and protocols as the original models. This meticulous approach ensures that any observed deviation in performance can be fairly attributed to the integration of S3. For classification, we measure the improvement as the percentage difference (Diff.) in accuracy resulting from S3, calculated as ( Acc Baseline+S3 -Acc Baseline ) /Acc Baseline. For forecasting, since lower MSE is better, we use ( MSE Baseline -MSE Baseline+S3 ) /MSE Baseline. A similar equation is used for measuring the percentage difference in MAE.

Classification datasets. For classification we use the following datasets: (1) The UCR archive [44] which consists of 128 univariate datasets, (2) the UEA archive [45] which consists of 30 multivariate datasets, and (3) three multivariate datasets namely EEG , EEG2 , and HAR from the UCI archive [46]. For our experiments with pre-trained foundation model in Section 5, we also use the PTB-XL [47] dataset. The train/test splits for all classification datasets are as provided in the original papers.

Forecasting datasets. For forecasting, we use the following datasets: (1) both the univariate and multivariate versions of three ETT datasets [10], namely ETTh1 and ETTh2 recorded hourly, and ETTm1 recorded at every 15 minutes, (2) both the univariate and multivariate versions of the Electricity dataset [46], and (3) the multivariate version of the Weather dataset [48].

Anomaly detection datasets. We employ the widely used Yahoo [49] and KPI [50] datasets for anomaly detection tasks. The Yahoo dataset consists of 367 time-series sampled hourly, each with labeled anomaly points, while the KPI dataset contains 58 minutely sampled KPI curves from various internet companies. Our experiments are performed under normal settings, as outlined in [17, 36].

Baselines. Weselect baselines from a variety of different time-series learning approaches. Specifically, for classification, we use four state-of-the-art baseline methods, SoftCLT [36], TS2Vec [17], DSN [16], and InfoTS [35]. For forecasting, we use five state-of-the-art baseline methods, TS2Vec [17], LaST [19], Informer [10], PatchTST [32] and CoST [40]. For anomaly detection, we use two stateof-the-art baseline methods, SoftCLT [36] and TS2Vec [17]. Additionally, we use MOMENT [34] for experiments with foundation model. For each baseline method, we integrate one to three layers of S3 at the input level of the model and compare its performance with that of the original model.

Implementation details. All implementation details match those of the baselines. We used the exact hyperparameters of the baselines according to the original papers when they were specified in the papers or when the code was available; alternatively, when the hyperparameters were not exactly

specified in the paper or in the code, we tried to maximize performance with our own search for the optimum hyperparameters. Additionally, for experiments involving PatchTST and LaST with the Electricity dataset, we use a batch size of 8 due to memory constraints. Accordingly, some baseline results may slightly differ from those available in the original papers. Note that deviations in baseline results affect both the baseline model and baseline+S3. For the weighted sum operation in S3, we use Conv1D. Our code is implemented with PyTorch, and our experiments are conducted on a single NVIDIA Quadro RTX 6000 GPU. We release the code at: https://github.com/shivam-grover/S3TimeSeries.

5 Results

Classification. The results of our experiments on time-series classification are presented in Table 1. In this table we observe that the addition of S3 results in substantial improvements across all baselines for all univariate and multivariate datasets. For UCR, UEA, EEG, EEG2, and HAR, we achieve average improvements of 3.89%, 5.25%, 32.03%, 9.675%, and 5.18% respectively over all models. The full classification results on UCR and UEA datasets for all baselines with and without S3 are mentioned in Appendix (Table A1 and A2). These results highlight the efficacy of S3 in improving a variety of different classification methods over a diverse set of datasets despite negligible added computational complexity (we provide a detailed discussion on computation overhead later in this section). In addition to quantitative metrics, we also visualize the t-SNE plots for several datasets in

Figure 2 and Appendix A1, which show that adding S3 results in representations with better class separability in the latent space.

Forecasting. Table 2 presents a comprehensive overview of the results for univariate forecasting on different datasets and horizons (H), with and without the incorporation of S3. We observe that S3 consistently leads to improvements for all baseline methods across all datasets, with an average improvement in MSE and MAE of 4.58% and 4.20% for TS2Vec, 4.02% and 3.01% for LaST, 16.69% and 8.19% for Informer, 1.64% and 1.20% for PatchTST, and 3.01% and 2.62% for CoST. Figure 3 shows two forecasting outputs for Informer with and without S3, on two different horizon lengths for a sample from the ETTh1 dataset. In both cases, S3 improves the ability of the baseline Informer to generate time-series samples that better align with the ground truth. Similarly, Table 3 presents the results of multivariate forecasting. Consistent with univariate forecasting, S3 significantly enhances the performance of the baseline on multivariate forecasting and achieves average improvements in MSE and MAE of 13.71% and 7.78% for TS2Vec, 9.88% and 5.45% for LaST, 25.13% and 14.82% for Informer, 1.62% and 1.17% for PatchTST, and 4.41% and 1.97% for CoST.

Anomaly detection. Table 4 presents the results for anomaly detection on Yahoo and KPI datasets. The anomaly score is computed as the L1 distance between two encoded representations derived from masked and unmasked inputs, following the methodology described in previous studies [17, 36]. We observe that the proposed S3 layer enhances the performance in terms of F1 for both datasets under the normal settings, with scores of 0.7498 and 0.6892, respectively.

Loss behaviour. We make interesting observations while training the baseline models as well as the baseline+S3 variants. First, we observe that the training loss vs. epochs for baseline+S3 variants generally converge faster than the original baselines. See Figure 4 where we demonstrate examples of this behavior. Second, we observe that the training loss curves for baseline+S3 are generally much smoother than the original baselines. This can again be observed in Figure 4. Additionally, we measure the standard deviations of the loss curves for SoftCLT with and without S3 on all the UCR datasets and measure an average reduction in standard deviation of 36.66%. The detailed values for the standard deviations of the loss curves are presented in Appendix A4. Lastly, according to [51], we investigate the loss landscape of the baselines and observe that the addition of S3 generally results in a smoother loss landscape with fewer local minima (Figure 5). Additional examples are provided in Appendix A2.

Dataset

H

Metric

TS2Vec

LaST

Informer

PatchTST

CoST

Base

+S3

Diff.

Base

+S3

Diff.

Base

+S3

Diff.

Base

+S3

Diff.

Base

+S3

Diff.

ETTh1

24

MSE

0.0409

0.0370

9.40%

0.0295

0.0293

0.80%

0.1169

0.0851

27.18%

0.0265

0.0258

2.57%

0.0386

0.0377

2.22%

MAE

0.1614

0.1459

9.58%

0.1297

0.1291

0.45%

0.2753

0.2263

17.76%

0.1237

0.1232

0.38%

0.1501

0.1486

0.99%

48

MSE

0.0714

0.0605

15.24%

0.0479

0.0464

3.07%

0.1358

0.1132

16.57%

0.0397

0.0389

1.99%

0.0592

0.0565

4.47%

MAE

0.2035

0.1884

7.45%

0.1630

0.1628

0.11%

0.2959

0.2727

7.81%

0.1503

0.1478

1.67%

0.1856

0.1818

2.02%

168

MSE

0.1362

0.1202

11.71%

0.0765

0.0731

4.40%

0.1604

0.0944

41.12%

0.0681

0.0672

1.28%

0.0920

0.0879

4.47%

MAE

0.3235

0.2663

17.65%

0.2071

0.2069

0.10%

0.3207

0.2434

24.10%

0.1999

0.1982

0.86%

0.2322

0.2221

4.38%

336

MSE

0.1699

0.1426

16.09%

0.0948

0.0920

2.89%

0.1411

0.1021

27.59%

0.0820

0.0806

1.71%

0.1083

0.1040

3.92%

MAE

0.3333

0.2949

11.54%

0.2409

0.2366

1.80%

0.2990

0.2532

15.30%

0.2034

0.1974

2.21%

0.2557

0.2440

4.56%

720

MSE

0.1739

0.1734

0.29%

0.1417

0.1370

3.33%

0.269

0.0841

68.71%

0.0866

0.0855

1.24%

0.1297

0.1247

3.84%

MAE

0.3307

0.3359

-1.58%

0.3098

0.2989

3.50%

0.2159

0.1980

8.27%

0.2132

0.2098

1.59%

0.2840

0.2782

2.02%

ETTh2

24

MSE

0.0973

0.0938

3.49%

0.0708

0.0675

4.58%

0.0851

0.0844

0.75%

0.0704

0.0702

0.34%

0.0784

0.0766

2.37%

MAE

0.2363

0.2344

0.82%

0.1956

0.1902

3.20%

0.2230

0.2210

1.02%

0.2036

0.2012

1.19%

0.2059

0.2033

1.27%

48

MSE

0.1269

0.1242

1.77%

0.0995

0.0939

5.72%

0.1503

0.1461

2.77%

0.0963

0.0943

2.08%

0.1163

0.1072

7.84%

MAE

0.2781

0.2761

0.72%

0.2394

0.2279

4.80%

0.3036

0.2990

1.54%

0.2410

0.2399

0.44%

0.2589

0.2520

2.65%

168

MSE

0.1974

0.1853

6.12%

0.1796

0.1680

6.40%

0.2637

0.2470

6.34%

0.1604

0.1595

0.58%

0.1805

0.1677

7.10%

MAE

0.3562

0.3437

3.50%

0.3194

0.3098

3.02%

0.4120

0.3756

8.83%

0.3184

0.3148

1.12%

0.3339

0.3161

5.33%

336

MSE

0.2031

0.1953

3.86%

0.2213

0.2049

7.39%

0.3177

0.2993

5.79%

0.1847

0.1812

1.89%

0.1963

0.1949

0.74%

MAE

0.3638

0.3551

2.41%

0.3819

0.3579

6.28%

0.4543

0.4432

2.44%

0.3398

0.3378

0.59%

0.3548

0.3542

0.18%

720

MSE

0.2084

0.1983

4.82%

0.2754

0.2712

1.53%

0.2678

0.2548

4.88%

0.2236

0.2202

1.52%

0.2048

0.1962

4.19%

MAE

0.3719

0.3620

2.36%

0.4297

0.4215

1.91%

0.4199

0.3975

5.34%

0.3549

0.3483

1.86%

0.3664

0.3550

3.10%

ETTm1

24

MSE

0.0150

0.0152

-1.74%

0.0231

0.0229

16.22%

0.0246

0.0228

7.13%

0.0099

0.0097

1.55%

0.0146

0.0132

9.44%

MAE

0.0910

0.0928

-1.96%

0.1070

0.0982

8.25%

0.1117

0.1079

3.33%

0.1123

0.1105

1.60%

0.0880

0.0849

3.59%

48

MSE

0.0283

0.0290

-2.44%

0.0479

0.0465

3.08%

0.0561

0.0501

10.76%

0.0169

0.0167

1.18%

0.0252

0.0233

7.35%

MAE

0.1304

0.1350

-3.52%

0.1580

0.1517

3.99%

0.1749

0.1632

6.70%

0.1189

0.1184

0.42%

0.1177

0.1130

4.01%

96

MSE

0.0480

0.0458

4.41%

0.0765

0.0715

6.53%

0.1124

0.1100

2.15%

0.0362

0.0359

0.90%

0.0378

0.0346

8.64%

MAE

0.1713

0.1643

4.07%

0.2071

0.1967

5.04%

0.2421

0.2433

-0.51%

0.1389

0.1384

0.36%

0.1462

0.1391

4.88%

228

MSE

0.0992

0.0953

3.92%

0.1058

0.0953

9.94%

0.1833

0.1762

3.87%

0.0527

0.0518

1.75%

0.0728

0.0669

8.10%

MAE

0.2495

0.2359

5.46%

0.2490

0.2231

10.40%

0.3368

0.3209

4.73%

0.1578

0.1556

1.39%

0.2051

0.1946

5.11%

672

MSE

0.1568

0.1529

2.50%

0.1699

0.1589

6.47%

0.2686

0.2582

3.88%

0.0744

0.0732

1.67%

0.1053

0.0969

7.97%

MAE

0.3100

0.3028

2.33%

0.3165

0.3089

2.75%

0.4212

0.4166

1.10%

0.2072

0.2035

1.79%

0.2486

0.2345

5.66%

Electricity

24

MSE

0.2546

0.2460

3.38%

0.1525

0.1486

2.58%

0.1937

0.1778

8.18%

0.1434

0.1408

1.76%

0.2420

0.2369

2.12%

MAE

0.2846

0.2710

4.75%

0.2797

0.2750

1.69%

0.3153

0.3053

3.15%

0.2713

0.2676

1.37%

0.2628

0.2604

0.91%

48

MSE

0.3001

0.2923

2.60%

0.1888

0.1856

1.69%

0.2760

0.2129

22.86%

0.1765

0.1749

0.91%

0.2912

0.2888

0.81%

MAE

0.3151

0.3011

4.43%

0.3086

0.3023

2.03%

0.3794

0.3308

12.82%

0.2985

0.2962

0.79%

0.2977

0.2957

0.65%

168

MSE

0.4303

0.4054

5.79%

0.2473

0.2415

2.35%

0.3402

0.3202

5.87%

0.2484

0.2436

1.94%

0.4084

0.4019

1.58%

MAE

0.3872

0.3728

3.71%

0.3506

0.3423

2.36%

0.4292

0.4043

5.81%

0.3470

0.3456

0.38%

0.3781

0.3750

0.83%

336

MSE

0.5636

0.5383

4.49%

0.2907

0.2802

3.63%

0.4062

0.3520

13.34%

0.2963

0.2925

1.26%

0.5690

0.5563

2.22%

MAE

0.4712

0.4563

3.17%

0.3831

0.3793

1.00%

0.4634

0.4169

10.02%

0.3868

0.3808

1.55%

0.4693

0.4620

1.57%

720

MSE

0.8641

0.8444

2.27%

0.3239

0.3180

1.82%

0.4799

0.3484

27.41%

0.3393

0.3296

2.86%

0.8965

0.8783

2.03%

MAE

0.6538

0.6345

2.94%

0.4229

0.4197

0.76%

0.5126

0.4257

16.95%

0.4311

0.4237

1.70%

0.6471

0.6345

1.95%

Average

MSE

0.2093

0.1997

4.58%

0.1434

0.1376

4.02%

0.2124

0.1769

16.69%

0.1216

0.1196

1.64%

0.1933

0.1873

3.01%

MAE

0.3011

0.2884

4.20%

0.2700

0.2619

3.01%

0.3303

0.3032

8.19%

0.2408

0.2380

1.20%

0.2744

0.2670

2.62%

8

6

4

2

0

2

4

6

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

TS2Vec

8

6

4

2

0

2

4

6

6

4

2

0

2

4

6

8

10

TS2Vec+S3

(a) DodgerLoopWeekend

20

15

10

5

0

5

10

15

20

20

10

0

10

20

TS2Vec

20

10

0

10

20

15

10

5

0

5

10

TS2Vec+S3

(b) Chinatown

Figure 2: t-SNE visualizations of the learned representations of TS2Vec and TS2Vec+S3 for 4 randomly chosen test sets. Different colors represent different classes. It can be seen that representations belonging to different classes are more separable after adding S3.

Ablation. We perform ablation studies to investigate the impact of each of the key modules of S3 and answer three questions: ( 1 ) Do we need the Segment module to divide the input into multiple segments, or will shuffling all the time-steps individually work just as well? ( 2 ) Do we need learned Shuffling parameters ( P ) to determine the optimal permutation of the time-series, or will random permutations work just as well? ( 3 ) Do we need the Stitch module to apply a learned weighted sum of the original sequence with the shuffled sequence, or will the shuffled sequences alone work just as well? Table 7 shows the results of three ablation studies where we observe that the removal of each component of S3 results in a decline in performance.

Foundation models. In order to evaluate the impact of S3 on pre-trained foundation model, we integrate S3 at the beginning of the pre-trained MOMENT encoder [34]. We use linear probing for

MOMENT MOMENT+S3

PTB-XL

Crop

Loss

Acc

Loss

Acc

0.8308

0.7202

0.7176

1.543

0.734

0.7552

1.4320

0.7591

Diff.

13.31%

5.24%

7.75%

3.41%

fine-tuning, and to ensure a fair comparison, we strictly follow the setup and experimental protocols outlined in the original paper. In this process, the encoder is kept frozen, and only S3 and the final linear head are fine-tuned together for the specific task and dataset. We fine-tune and evaluate the models for classification on PTB-XL [47] and the Crop dataset from the UCR archive [44], and the ETTh1 and ETTh2 datasets [10] for forecasting. The results for classification are presented in Table 5, where we observe that upon adding S3, both the test loss and accuracy see significant improvements of up to 13.31% and 5.24%, respectively. Table A3 (Appendix) shows the forecasting results, where incorporating S3 consistently improves performance. On the ETTh1 dataset, it yields

0

10

20

Time Step

1.4

1.2

1.0

0.8

Value

Ground Truth

Informer+S3

Informer

(a) H

= 24

0

50

100

150

Time Step

2.0

1.5

1.0

Value

Ground Truth

Informer+S3

Informer

(b) H

= 168

(a) TS2Vec

(b) TS2Vec+S3

0

50

100

150

200

Iteration

5

10

15

Training loss

SoftCLT+S3

SoftCLT

(a) ProximalPhalanxTW

0

50

100

150

200

Iteration

10

20

30

Training loss

SoftCLT+S3

SoftCLT

(b) Symbols

0

50

100

Number of Segments

0.985

0.990

0.995

1.000

(a) PowerCons,

t

= 144

0

50

100

150

200

Number of Segments

0.84

0.86

0.88

0.90

0.92

(b) OliveOil,

t

= 570

average improvements of 2.12% in MSE and 2.59% in MAE, and for ETTh2, the improvements are 4.96% in MSE and 3.56% in MAE.

Hyperparameters. Next, we investigate the impact of the number of segments n on performance. Considering all the experiments, we observe that no general rule of thumb can be advised for n , as its optimum value is naturally highly dependent on factors such as dataset complexity, length, baselines, and others. We present two examples in Figure 6 and more in Appendix A3 where we plot the accuracy vs. number of segments for datasets of different lengths from UCR. Here we only take into account the layer with the largest number of segments. As for the other hyperparameters, the number of S3 layers ϕ has been set between 1, 2, or 3 across all experiments in this paper. θ , the multiplier for the

number of segments in stacked S3 layers, has been set to 0.5, 1, or 2 across all our experiments. And finally, λ has only been set to 1, 2, or 3. The ranges of values for all the hyperparameters used in this work are presented in Table 6. While optimizing hyperparameters is important, similar to any other form of representation learning, we find that even sub-optimal tuning of S3 hyperparameters still yields meaningful improvements. We perform a simple experiment where we apply a uniform set of hyperparameters [ n = 2 , ϕ = 2 , θ = 1 , λ = 1] to all the baselines and baselines with added S3 layers. The results for this experiment are presented in Table 8 where we observe that despite not using the optimum hyperparameters, the addition of S3 still yields considerable performance boosts. The optimum hyperparameters used for each experiment are made available in the code.

Sensitivity to random seed. To evaluate the impact of the random seed on the performance of S3, we perform 10 separate trials with different seed values and present the standard deviations in Table 9. The first 4 rows show the classification baselines trained on the UCI datasets with and without S3. Similarly, the next 3 rows show forecasting baselines trained on the multivariate ETTh1 dataset (H=24) with and without S3. From this experiment, we observe that the addition of S3 has no considerable impact on the sensitivity of the original baselines to the random seed.

Shuffling parameters. In Figure 7, we present a visual overview of how the parameters of P update along with the parent model during training on ETTh1 (H = 48) and ETTm1 (H = 48). Each parameter in P corresponds to a segment of S . In both cases, we set n to 16 and ϕ to 1. The baseline used is LaST [19]. It can be seen that the individual weights in P rearrange the corresponding segments until they reach a stable point (the optimal order) after which the order of the segments is maintained. Similarly, Figure A4 (Appendix) shows how w 1 and w 2 update during training. Finally, in Figure A5

.

Method

EEG

HAR

TS2Vec+S3

0.672

0.973

0.935

TS2Vec+S3 w/o Segment

0.552

0.872

0.865

TS2Vec+S3 w/o

P

0.562

0.864

0.906

TS2Vec+S3 w/o Stitch

0.584

0.871

0.901

DSN+S3

0.717

0.980

0.971

DSN+S3 w/o Segment

0.516

0.929

0.910

DSN+S3 w/o

P

0.490

0.953

0.828

0.550

0.934

0.856

InfoTS+S3

0.719

0.947

0.929

InfoTS+S3 w/o Segment

0.535

0.865

0.898

InfoTS+S3 w/o

P

0.499

0.845

0.869

InfoTS+S3 w/o Stitch

0.515

0.821

0.887

SoftCLT+S3

0.672

0.950

0.936

SoftCLT+S3 w/o Segment

0.548

0.919

0.879

SoftCLT+S3 w/o

P

0.523

0.905

0.883

SoftCLT+S3 w/o Stitch

0.600

0.913

0.921

Method

EEG

HAR

TS2Vec

0.593

0.845

0.926

TS2Vec+S3

0.625

0.910

0.922

Diff.

11.11%

7.69%

-0.36%

DSN

0.516

0.961

0.957

DSN+S3

0.567

0.978

0.974

Diff.

9.83%

1.78%

1.72%

InfoTS

0.515

0.822

0.876

InfoTS+S3

0.593

0.842

0.898

Diff.

15.15%

2.43%

2.45%

SoftCLT

0.500

0.893

0.918

SoftCLT+S3

0.640

0.890

0.923

Diff.

15.15%

-0.33%

0.54%

Method

w/o S3

w/ S3

SoftCLT

0.0044

0.0058

TS2Vec

0.0077

0.0075

InfoTS

0.0067

0.0082

DSN

0.0041

0.0035

TS2Vec

0.0019

0.0028

Informer

0.0144

0.0142

LaST

0.0009

0.0012

Table 10:

Param. count

Method

Baseline

S3

TS2Vec

24

SoftCLT

641k

579

InfoTS

7

DSN

106k

13

TS2Vec

638k

12

Informer

11.3M

6

LaST

130k

16

0

250

500

750

1000

1250

1500

Iteration

0.59992

0.59994

0.59996

0.59998

0.60000

0.60002

0.60004

Value for each parameter in

P

P

0

P

1

P

2

P

3

P

4

P

5

P

6

P

7

P

8

P

9

P

10

P

11

P

12

P

13

P

14

P

15

(a) ETTh1 (H = 48)

0

250

500

750

1000

1250

1500

Iteration

0.59996

0.59998

0.60000

0.60002

0.60004

0.60006

Value for each parameter in

P

P

0

P

1

P

2

P

3

P

4

P

5

P

6

P

7

P

8

P

9

P

10

P

11

P

12

P

13

P

14

P

15

(b) ETTm1 (H = 48)

(Appendix), we show sample visualizations of how S3 segments, shuffles, and stitches a time-series once the optimal shuffling parameters are obtained.

Computation overhead. Our proposed network layer adds very few learnable parameters to the baseline models, which stem from a learnable shuffling vector P along with w 1 and w 2 , per each S3 layer. To emphasize the negligible computation overhead of S3 in comparison to the baseline models, we show the total number of parameters in the baseline and the total learnable parameters that S3 adds, in Table 10 for classification on the EEG2 dataset and forecasting on the multivariate ETTh1 dataset (H = 24).

6 Conclusion

Summary. We propose S3, a simple plug-and-play neural network component that rearranges the time-series in three steps: S egment the time-series, S huffle the segments, and S titch the shuffled time-series by concatenating the segments and performing a learned sum with the original timeseries. Through extensive experiments on time-series classification and forecasting with state-ofthe-art methods, we show that S3 improves the learning capabilities of the baseline with negligible computation overhead. We also show empirically that S3 helps in faster and smoother training leading to better performance.

Limitations. While we present the effectiveness of S3 on a diverse set of baselines on classification, forecasting, and anomaly detection tasks, we acknowledge that the evaluation of other time-series tasks such as imputation remain for future work. Additionally applying S3 to learning representations from other forms of time-series such as videos is an interesting direction for future research.

Broader impact. Since S3 is a plug-and-play network layer with negligible added parameters, it allows researchers from various domains related to time-series such as health signal processing, biometrics, climate analysis, financial markets, and others to use this module in their existing models without the need for redesign. The low computation overhead also makes S3 a suitable choice for edge-devices.

References

Dani Kiyasseh, Tingting Zhu, and David A Clifton. Clocs: Contrastive learning of cardiac signals across space, time, and patients. In International Conference on Machine Learning , pages 5606-5615. PMLR, 2021. 1

Anubhav Bhatti, Behnam Behinaein, Paul Hungler, and Ali Etemad. Attx: Attentive crossconnections for fusion of wearable signals in emotion recognition. ACM Transactions on Computing for Healthcare , 5(3):1-24, 2024. 1

Debaditya Shome, Pritam Sarkar, and Ali Etemad. Region-disentangled diffusion model for high-fidelity ppg-to-ecg translation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 15009-15019, 2024. 1

Debaditya Shome and Ali Etemad. Speech emotion recognition with distilled prosodic and linguistic affect representations. In IEEE International Conference on Acoustics, Speech and Signal Processing , pages 11976-11980, 2024. 1

Pritam Sarkar and Ali Etemad. Self-supervised ecg representation learning for emotion recognition. IEEE Transactions on Affective Computing , 13(3):1541-1554, 2022. 1

Pritam Sarkar, Ahmad Beirami, and Ali Etemad. Uncovering the hidden dynamics of video self-supervised learning under distribution shifts. Advances in Neural Information Processing Systems , 36, 2024. 1

Hyeokhyen Kwon, Catherine Tong, Harish Haresamudram, Yan Gao, Gregory D. Abowd, Nicholas D. Lane, and Thomas Plötz. Imutube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies , 4(3), September 2020. 1

Sijie He, Xinyan Li, Timothy DelSole, Pradeep Ravikumar, and Arindam Banerjee. Subseasonal climate forecasting via machine learning: Challenges, analysis, and advances. In AAAI Conference on Artificial Intelligence , volume 35, pages 169-177, 2021. 1

Haitao Lin, Zhangyang Gao, Yongjie Xu, Lirong Wu, Ling Li, and Stan Z Li. Conditional local convolution for spatio-temporal meteorological forecasting. In AAAI Conference on Artificial Intelligence , volume 36, pages 7470-7478, 2022. 1

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI Conference on Artificial Intelligence , volume 35, pages 11106-11115, 2021. 1, 2, 5, 8, 21

Seok-Jun Bu and Sung-Bae Cho. Time series forecasting with multi-headed attention-based deep learning for residential energy consumption. Energies , 13(18):4722, 2020. 1

Zhiyuan Wang, Xovee Xu, Goce Trajcevski, Kunpeng Zhang, Ting Zhong, and Fan Zhou. Pref: Probabilistic electricity forecasting via copula-augmented state space model. In AAAI Conference on Artificial Intelligence , volume 36, pages 12200-12207, 2022. 1

Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. Advances in Neural Information Processing Systems , 33:17804-17815, 2020. 1

Razvan-Gabriel Cirstea, Bin Yang, Chenjuan Guo, Tung Kieu, and Shirui Pan. Towards spatiotemporal aware traffic time series forecasting. In IEEE International Conference on Data Engineering , pages 2900-2913, 2022. 1

Dawei Cheng, Fangzhou Yang, Sheng Xiang, and Jin Liu. Financial time series forecasting with multi-modality graph neural network. Pattern Recognition , 121:108218, 2022. 1

Qiao Xiao, Boqian Wu, Yu Zhang, Shiwei Liu, Mykola Pechenizkiy, Elena Mocanu, and Decebal Constantin Mocanu. Dynamic sparse network for time series classification: Learning what to 'see'. Advances in Neural Information Processing Systems , 35:16849-16862, 2022. 1, 2, 5

Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In AAAI Conference on Artificial Intelligence , volume 36, pages 8980-8987, 2022. 1, 3, 5, 6

Xuchao Zhang, Yifeng Gao, Jessica Lin, and Chang-Tien Lu. Tapnet: Multivariate time series classification with attentional prototypical network. In AAAI Conference on Artificial Intelligence , volume 34, pages 6845-6852, 2020. 1

Zhiyuan Wang, Xovee Xu, Weifeng Zhang, Goce Trajcevski, Ting Zhong, and Fan Zhou. Learning latent seasonal-trend representations for time series forecasting. Advances in Neural Information Processing Systems , 35:38775-38787, 2022. 1, 3, 5, 9, 20

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems , 34:22419-22430, 2021. 1, 3

Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems , 35:5816-5828, 2022. 1, 2

Qingxiong Tan, Mang Ye, Baoyao Yang, Siqi Liu, Andy Jinhua Ma, Terry Cheuk-Fung Yip, Grace Lai-Hung Wong, and PongChi Yuen. Data-gru: Dual-attention time-aware gated recurrent unit for irregular multivariate time series. In AAAI Conference on Artificial Intelligence , volume 34, pages 930-937, 2020. 1

Donghui Chen, Ling Chen, Youdong Zhang, Bo Wen, and Chenghu Yang. A multiscale interactive recurrent network for time-series forecasting. IEEE Transactions on Cybernetics , 52(9):8793-8803, 2021. 1

Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and PierreAlain Muller. Deep learning for time series classification: a review. Data Mining and Knowledge Discovery , 33(4):917-963, 2019. 1

Patrick Schäfer. The boss is concerned with time series classification in the presence of noise. Data Mining and Knowledge Discovery , 29:1505-1530, 2015. 1

Anthony Bagnall, Jason Lines, Jon Hills, and Aaron Bostrom. Time-series classification with cote: the collective of transformation-based ensembles. IEEE Transactions on Knowledge and Data Engineering , 27(9):2522-2535, 2015. 1

Aosong Feng and Leandros Tassiulas. Adaptive graph spatial-temporal transformer network for traffic forecasting. In ACM International Conference on Information and Knowledge Management , pages 3933-3937, 2022. 2

Junwoo Park, Daehoon Gwak, Jaegul Choo, and Edward Choi. Self-supervised contrastive learning for long-term forecasting. In International Conference on Learning Representations , 2024. 2, 3

Ali Salehi and Madhusudhanan Balasubramanian. Ddcnet: Deep dilated convolutional neural network for dense prediction. Neurocomputing , 523:116-129, 2023. 2

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems , 30, 2017. 2

Yuqi Chen, Kan Ren, Yansen Wang, Yuchen Fang, Weiwei Sun, and Dongsheng Li. Contiformer: Continuous-time transformer for irregular time series modeling. Advances in Neural Information Processing Systems , 36, 2024. 3

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations . 3, 5, 21

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI Conference on Artificial Intelligence , volume 37, pages 11121-11128, 2023. 3

Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. arXiv preprint arXiv:2402.03885 , 2024. 3, 5, 7, 8

Dongsheng Luo, Wei Cheng, Yingheng Wang, Dongkuan Xu, Jingchao Ni, Wenchao Yu, Xuchao Zhang, Yanchi Liu, Yuncong Chen, Haifeng Chen, et al. Time series contrastive learning with information-aware augmentations. In AAAI Conference on Artificial Intelligence , volume 37, pages 4534-4542, 2023. 3, 5

Seunghan Lee, Taeyoung Park, and Kibok Lee. Soft contrastive learning for time series. In The International Conference on Learning Representations , 2024. 3, 5, 6, 21

Renfei He, Limao Zhang, and Alvin Wei Ze Chew. Modeling and predicting rainfall time series using seasonal-trend decomposition and machine learning. Knowledge-Based Systems , 251:109125, 2022. 3

Robert B Cleveland, William S Cleveland, Jean E McRae, Irma Terpenning, et al. Stl: A seasonal-trend decomposition. Journal of Official Statistics , 6(1):3-73, 1990. 3

Shuai Liu, Xiucheng Li, Gao Cong, Yile Chen, and Yue Jiang. Multivariate time-series imputation with disentangled temporal representations. In International Conference on Learning Representations , 2022. 3

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. CoST: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. In International Conference on Learning Representations , 2022. 3, 5

Marco Cuturi, Olivier Teboul, and Jean-Philippe Vert. Differentiable ranking and sorting using optimal transport. Advances in Neural Information Processing Systems , 32, 2019. 4

Mathieu Blondel, Olivier Teboul, Quentin Berthet, and Josip Djolonga. Fast differentiable sorting and ranking. In International Conference on Machine Learning , pages 950-959, 2020. 4

Michael Taylor, John Guiver, Stephen Robertson, and Tom Minka. Softrank: optimizing nonsmooth rank metrics. In International Conference on Web Search and Data Mining , pages 77-86, 2008. 4

Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. The ucr time series archive. IEEE Journal of Automatica Sinica , 6(6):1293-1305, 2019. 5, 8

Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. ArXiv Preprint arXiv:1811.00075 , 2018. 5

Dheeru Dua and Casey Graff. Uc irvine machine learning repository, 2017. 5

Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojciech Samek, and Tobias Schaeffter. Ptb-xl, a large publicly available electrocardiography dataset. Scientific Data , 7(1):1-15, 2020. 5, 8

Max Planck Institute for Biogeochemistry. Jena climate and weather data. https://www. bgc-jena.mpg.de/wetter/ . 5

Nikolay Laptev, Saeed Amizadeh, and Ian Flint. Generic and scalable framework for automated time-series anomaly detection. In ACM International Conference on Knowledge Discovery and Data Mining , pages 1939-1947, 2015. 5

Hansheng Ren, Bixiong Xu, Yujing Wang, Chao Yi, Congrui Huang, Xiaoyu Kou, Tony Xing, Mao Yang, Jie Tong, and Qi Zhang. Time-series anomaly detection service at microsoft. In ACM International Conference on Knowledge Discovery & Data Mining , pages 3009-3017, 2019. 5

Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in Neural Information Processing Systems , 31, 2018. 6, 19

Xiyuan Zhang, Ranak Roy Chowdhury, Jingbo Shang, Rajesh Gupta, and Dezhi Hong. Towards diverse and coherent augmentation for time-series forecasting. In International Conference on Acoustics, Speech, and Signal Processing , pages 1-5, 2023. 21

Brian Kenji Iwana and Seiichi Uchida. An empirical survey of data augmentation for time series classification with neural networks. Plos One , 16(7):e0254841, 2021. 21

Anh Tong, Thanh Nguyen-Tang, Toan Tran, and Jaesik Choi. Learning fractional white noises in neural stochastic differential equations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. 21

A Appendix

A.1 Additional results

We show the full results for all baselines with and without S3 on 128 UCR datasets in Table A1 and on 30 UEA datasets in Table A2. In Figure A1 we show 2 more t-SNE plots for TS2Vec trained on 2 randomly chosen UCR datasets. A better clustering of different classes can be observed in the learned representations after S3 is incorporated. In Table A4 we compare the standard deviation values for the training loss of SoftCLT and SoftCLT+S3. From this table we observe that after incorporating S3, the training becomes much more stable with significantly less variations. Figure A2 shows 2 more comparisons of the loss landscape of TS2Vec trained with and without S3. In Figure A3 we present 6 more samples for the impact of n on the performance of the model.

8

6

4

2

0

2

4

6

10

8

6

4

2

0

2

4

TS2Vec

8

6

4

2

0

2

4

6

8

6

4

2

0

2

4

6

8

TS2Vec+S3

(a) HouseTwenty

150

100

50

0

50

100

150

150

100

50

0

50

100

TS2Vec

26.5 26.0 25.5 25.0 24.5 24.0 23.5

4.8

4.6

4.4

4.2

4.0

3.8

TS2Vec+S3

(b) BasicMotions

TS2VecGLYPH<10>

TS2Vec+S3GLYPH<10>

(a) ShakeWiimoteZ

TS2VecGLYPH<10>

TS2Vec+S3GLYPH<10>

(b) Symbols

0

50

100

150

200

Number of Segments

0.65

0.70

0.75

0.80

0.85

0.90

(a) BeetleFly (

t

= 512

)

5

10

15

20

25

Number of Segments

0.70

0.75

0.80

(b) RacketSport (

t

= 30

)

0

50

100

150

200

Number of Segments

0.55

0.60

0.65

0.70

0.75

(c) Rock (

t

= 2844

)

0

50

100

150

200

Number of Segments

2.40

2.45

2.50

2.55

2.60

2.65

MSE

(d) ETTh2 (H

= 720

)

0

50

100

150

200

Number of Segments

2.2

2.4

2.6

MSE

(e) ETTh2 (H

= 336

)

0

50

100

150

200

Number of Segments

1.2

1.4

1.6

1.8

MSE

(f) ETTh2 (H

= 48

)

A.2 Weighted average parameters during training

In Figure A4, we present a visual overview of how the weighted average parameters w 1 and w 2 update along with the rest of the model during training on ETTh1 (H = 24) and ETTh2 (H = 720) in multivariate settings. We use LaST [19] as the baseline. It is observed that the weights converge to their final values at the end of training.

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Iteration

×

10 4

0.4

0.6

0.8

1.0

1.2

1.4

Weight

w

1

w

2

(a) ETTh2(M) H = 720

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Iteration

×

10 4

0.0

0.5

1.0

1.5

2.0

Weight

w

1

w

2

(b) ETTh1(M) H = 24

A.3 Visualisation of the segments

Figure A5 visualizes three sample time-series and how their respective segments are rearranged using S3. The figure depicts the original sequence, the shuffled sequence, and the final output.

1

2

3

4

5

6

7

8

Input

5

2

3

1

8

7

6

4

Shuffled

Output

1

2

3

4

5

6

7

8

Input

4

5

6

7

1

3

2

8

Shuffled

Output

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Input

13

16

2

10

14

3

12

6

4

7

11

8

1

9

5

15

Shuffled

Output

A.4 Comparison with data augmentation methods

The proposed S3 layer is fundamentally different from data augmentation techniques, in that unlike traditional augmentation methods, S3 introduces learnable parameters that are jointly optimized with the rest of the model, allowing S3 to adapt dynamically throughout the training process. To further explore the differences between S3 and data augmentation, we conduct an experiment where, before training on the ETTm1 multivariate dataset, we apply the following augmentations: (1) shuffling augmentation with varying numbers of segments, (2) shuffling augmentation combined with mixup, (3) noise augmentation with zero mean, variance of 1, and a magnifying coefficient of 0.01, and (4)

0

500

1000

1500

2000

2500

No. of training samples

0.0

2.5

5.0

7.5

Difference (%)

noise augmentation combined with mixup. The results are presented in Table A5 where we observe that S3 outperforms such data augmentation strategies. Additionally, data augmentation techniques typically have a stronger impact on smaller datasets, often due to the lack of variation and diversity, but their effect is less pronounced on larger datasets [53]. To evaluate this, we use the LSST dataset from UEA and create 10 different subsets by randomly dropping varying amounts of data, ranging from 20% to 99%. We then retrain SoftCLT [36], both with and without S3, on these subsets as well as on the original dataset. The results, averaged over three runs, are presented in Figure A6. These results show no clear trend in performance gain relative to dataset size, indicating that S3 benefits datasets regardless of their size.

A.5 Case-study on factors influencing the improvement by S3

To investigate the scenarios where S3 is most effective, we conduct additional experiments considering three key factors: (1) the length of time-series sequences, (2) the degree of non-linearity, and (3) the presence of long-term temporal dependencies. For (1), we examine the percentage difference relative to sequence length. For (2), we consider the percentage difference relative to the mean of squared residuals from a linear model. Lastly, for (3), we analyze the percentage difference in relation to the Hurst exponent [54]. We use PatchTST [32] for experiments (2) and (3), while Informer [10] is used for experiment (1) since PatchTST cannot be used due to its reliance on equal-length inputs. We conduct all three experiments on the ETT, Weather, and Electricity datasets. The results of this analysis are shown in Table A6, where m represents the slope of the linear relationship, and R is Pearson's correlation coefficient. We observe direct positive relationships between the percentage difference upon adding S3 and both sequence length and long-term temporal dependency, while non-linearity in the time-series shows no considerable relationship. For sequence length, the moderate correlation suggests that while the effect is not very strong, it is consistent, indicating that sequence length is likely a relevant factor for percentage difference. Long-term temporal dependency has a strong impact on percentage difference, as indicated by the steep slope. However, the lower correlation suggests that other factors in the data may also influence this relationship. We hypothesize that when long-term temporal dependencies are simple to learn or highly repetitive, the impact of S3 will be less pronounced, as re-ordering may not be necessary to learn such simple dynamics. On the other hand, for more complex long-term dependencies, S3 has a much stronger impact, as reflected by the high slope.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, see Section 6

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper has no theoretical assumptions.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The implementation details are detailed in Section 4, and the full documented code is provided.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is provided with sufficient documentation to be able to run it in few simple steps.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to unders tand the results?

Answer: [Yes]

Justification: We use the same experimental settings as the baselines that we have incorporated S3 into, except for experiments on PatchTST and LaST with the electricity dataset where we use a batch size of 8. We have clarified this in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In Table 9 we show the standard deviation in performance metric for all baselines on a single dataset for both classification and forecasting by repeating the experiments for 10 different seed values, observing very small sensitivity to seed values. We do not perform this for each experiment in our main paper, as it would be computationally too expensive.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The compute resources is presented Section 4. Additionally we have a separate study in Table 10 in which we analyze the exact computational overhead w.r.t. the baselines.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: See section. 6

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Apart from our code, that is well documented, we do not release any assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new method for time-series representation learning that enhances the modeling of non-adjacent segment dependencies. Specifically, the proposed method segments, shuffles in a learned manner and stitches the shuffled segments to combine with original time series. The proposed method is model-agnostic without adding significant parameter overhead and shows performance improvement across multiple classification and forecasting base models.\n\nSTRENGTHS:\n1. The proposed method permutes the original segments to better capture inter-relations between distant segments. It is model-agnostic and introduces minimal parameter overhead to the original model.\n\n2. Extensive experiments on various base models for both classification and forecasting tasks demonstrate the effectiveness of the proposed method.\n\nWEAKNESSES:\n1. It it not clear how the sorting process, specifically the calculation of permutation $\\sigma$ from $P$, is made differentiable.\n\n2. The compared forecasting baselines such as Informer are no longer state-of-the-art methods. Adding more recent baselines such as Time-LLM, GPT4TS, DLinear, PatchTST would provide a clearer understanding of the proposed method's comparative benefits.\n\n3. The basic assumption for S3 is that modeling non-adjacent dependencies is important. However, the paper lacks detailed case studies that demonstrate the specific types of non-adjacent dependencies effectively captured by S3, which are not addressed by existing models. Additionally, there is no case study to validate that the learned shuffling weights accurately represent these segment dependencies.\n\nQUESTIONS:\n1. The results in Tables 1, 2, and 3 seem to indicate more significant improvements in multivariate than in univariate time series tasks. Any reason behind this?\n\n2. What does the \"number of segments\" represent in Figure 6 and Figure A3? Is it the number of segments for the first layer or the final layer? If it refers to \"n\", then in Figure A3, this number seems to perform the best when it is larger than 100 for some datasets?\n\n3. Could you describe the inference process for the S3 method? Additionally, what are the computational overheads for training and inference times for S3?\n\nLIMITATIONS:\nThe paper mentions potential expansions into tasks like imputation and anomaly detection. Further details on limitations from the reviewer are discussed in Weaknesses and Questions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a plug-and-play mechanism called Segment, Shuffle, and Stitch (S3) designed to enhance time-series representation learning in existing models. S3 operates by dividing the original sequence into non-overlapping segments and shuffling them in a learned manner that is optimal for the given task. It then reattaches the shuffled segments and performs a learned weighted sum with the original input to capture both the newly shuffled sequence and the original sequence. This proposed model can enhance the performance of specific models in classification and prediction tasks.\n\nSTRENGTHS:\nThe paper is easily comprehensible and straightforward.\n\nSufficient experiments are conducted to confirm the effectiveness of the method.\n\nWEAKNESSES:\nLack of comparative methods:\nIn fact, the proposed method seems to share the same spirit as data augmentation methods in the time series field[1-4]. Why hasn't any data augmentation method been compared?\n\n\nSelection of baseline models:\nThe selected baseline model, Informer, seems somewhat outdated. Why not choose a more recent model, e.g., iTransformer[5] or PatchTST[6]?\n\n\nDataset for prediction task:\nThe author conducted experiments on three ETT datasets, but for prediction tasks, more datasets should be considered, e.g., traffic, electricity, and weather.\n\n\nTime-Series Representation Claim:\n As the author pointed out, more tasks should be considered for time series representation learning.\n\n\n[1]FRAUG: FREQUENCY DOMAIN AUGMENTATION FOR TIME SERIES FORECASTING  [2]Time Series Data Augmentation for Deep Learning: A Survey  [3]SimPSI: A Simple Strategy to Preserve Spectral Information in Time Series Data Augmentation [4]TOWARDS DIVERSE AND COHERENT AUGMENTATION FOR TIME-SERIES FORECASTING [5]ITRANSFORMER: INVERTED TRANSFORMERS ARE EFFECTIVE FOR TIME SERIES FORECASTING [6]A TIME SERIES IS WORTH 64 WORDS: LONG-TERM FORECASTING WITH TRANSFORMERS\n\nQUESTIONS:\nWhat are the essential differences between the proposed method and other data augmentation methods?\n\nLIMITATIONS:\nSee Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new neural network design element which segments, shuffles, and stitches time series for improved representation learning. They evaluate their methods on forecasting and classification tasks, and show that S3 benefits some widely used baselines.\n\nSTRENGTHS:\n1. To the best of my knowledge, the idea is novel, and fundamentally challenges and changes how to learn representations for time series data\n2. The paper is well written and easy to follow\n3. Experiments are well-designed, and results are promising\n\nWEAKNESSES:\nI have not found any major weaknesses in the methodology or experimental design. However,  I think that the paper might benefit from showing what the S3 module is actually learning. For example, the authors can include the segmented, shuffled, and stitched time series on a particular dataset as an example, along with the weighted time series (used as input to the model), and the original time series. This might provide some intuition as to how this design element improves predictive performance. \n\nI think there's always scope to improve experimental design. TS2Vec is a excellent choice for classification, but not for forecasting. I would recommend that the authors use methods such as PatchTST (transformer-based) or iTransformer, TimesNet (CNN-based), N-BEATs or N-HITS (MLP-based) etc. for time series forecasting. For classification, it would also be good to compare with fully supervised methods such as ResNet1D (see [1]). \n\n### References\n[1] Ismail Fawaz, Hassan, et al. \"Deep learning for time series classification: a review.\" Data mining and knowledge discovery 33.4 (2019): 917-963.\n\nQUESTIONS:\nI do not have questions per se, but I am listing some things that I am curious about below:\n\nI would also encourage the authors to evaluate the benefits of S3 on some recent time series foundation models such as MOMENT [2], Chronos [3], Moirai [4], TimesFM [5], and/or LagLLama [6]. The MOMENT model does both classification and forecasting, so it might be interesting to see how S3 benefits pre-trained models, say by just training the S3 layer and freezing the pre-trained backbone (or some variation of this experiment).\n\nOn a similar note, I wonder if S3 improves generalization and hurts memorization, or vice versa. It would be interesting to do some transfer learning experiments where you train on some time series data and evaluate the model on other time series data (see MOMENT or PatchTST for inspiration). \n\n### References\n[2] Goswami, Mononito, et al. \"Moment: A family of open time-series foundation models.\" arXiv preprint arXiv:2402.03885 (2024).\n[3] Ansari, Abdul Fatir, et al. \"Chronos: Learning the language of time series.\" arXiv preprint arXiv:2403.07815 (2024).\n[4] Woo, Gerald, et al. \"Unified training of universal time series forecasting transformers.\" arXiv preprint arXiv:2402.02592 (2024).\n[5] Das, Abhimanyu, et al. \"A decoder-only foundation model for time-series forecasting.\" arXiv preprint arXiv:2310.10688 (2023).\n[6] Rasul, Kashif, et al. \"Lag-llama: Towards foundation models for time series forecasting.\" arXiv preprint arXiv:2310.08278 (2023).\n\nLIMITATIONS:\nThe authors have a very brief description of limitations of their study.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper paper introduces a new approach called Segment, Shuffle, and Stitch (S3) to enhance time-series representation learning. The method involves segmenting the time-series into non-overlapping parts, shuffling them optimally, and stitching them back together along with the original sequence.\n\nKey contributions include:\n\n- Proposing the S3 mechanism to improve time-series representation learning by dynamically reordering segments.\n- Demonstrating that S3 can be integrated with existing neural architectures like CNNs and Transformers, resulting in significant performance improvements.\n- Showing through extensive experiments that S3 enhances performance in time-series classification and forecasting tasks, with improvements up to 68%.\n\nSTRENGTHS:\n- Code is available, making reproducing this paper easier.\n- Paper is clear.\n- Results appear good, when considered on the set of baselines and dataset picked by the authors.\n\nWEAKNESSES:\n- Tables 1 and 2 focus on the ETT datasets, which are only a (highly intra-correlated) subset of the common forecasting datasets: Electricity, Traffic, Weather, Illness...\n- I see no mention of CoST in the results tables, despite being cited in the paper. This is usually a very strong baseline for contrastive approaches. Including it would certainly paint a more complete picture of the results landscape. On a related note this also applies to e.g. more recent transformer baselines. Informer is relevant, but also very far from state of the art.\n- Error bars would help one better contextualize the results.\n- The lack of an ablation study makes understanding the reason this works more complicated.\n\nQUESTIONS:\n- The 3 points in weaknesses are also questions in the sense that they ask for some new experiments to be performed. Addressing those points would be my first recommendation.\n- Intuitively, it feels like this work is to some extent a form of bootstrap (as data augmentation) combined with a mixup-like sample interpolation. I may be wrong on this and am happy to discuss. If so, could the authors do more of an ablation study connected to this. I.e. how does the approach outperform other (non-permutation)-based data augmentation strategies combined with the same summation operation?\n\nEdit: I have read the author's rebuttal. They have addressed questions I had and I am as a result raising my score to a 6.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper introduces a simple but effective differentiable module that performs pre-processing to input multivariate time-series before being fed into any differentiable model for arbitrary task. The pre-processing involves segmenting, shuffling the segments and stiching them together. The novelty include making this seemingly discrete operations into a differentiable module. This simple idea yields significant improvement in performance of different kinds of models over variety of datasets.

STRENGTHS:
1. The method is simple and easy to add to most deep learning models
2. The technical details are well-motivated and explained
3. The method also improves training efficiency and convergence time along with performance with very little increate in model complexity
4. Experimental results across different tasks are strong

WEAKNESSES:
1. Visualization and any qualitative study on the shuffling and segments generalted by S3 would greatly benefit the readers.
2. How well does it optimize transformer based models, especially those that already do segmentation like PatchTST since the attention module captures the relations all pairs of segments already?
3. Does the representations due to S3 generalize to multiple tasks at a time or do we need to retrain for each task?

QUESTIONS:
See weaknesses

LIMITATIONS:
1. Lack of understanding on the segment permutations generated and why they are better for the model performance atleast qualitatively
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12