Segment, Shuffle, and Stitch: A Simple Layer for Improving Time-Series Representations

Authors: shivam  grover
amin  jalali
ali  etemad
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 18:07:59

Abstract: Existing approaches for learning representations of time-series keep the temporal arrangement of the time-steps intact with the presumption that the original order is the most optimal for learning. However, non-adjacent sections of real-world time-series may have strong dependencies. Accordingly, we raise the question: Is there an alternative arrangement for time-series which could enable more effective representation learning? To address this, we propose a simple plug-and-play neural network layer called Segment, Shuffle, and Stitch (S3) designed to improve representation learning in time-series models. S3 works by creating non-overlapping segments from the original sequence and shuffling them in a learned manner that is optimal for the task at hand. It then re-attaches the shuffled segments back together and performs a learned weighted sum with the original input to capture both the newly shuffled sequence along with the original sequence. S3 is modular and can be stacked to achieve different levels of granularity, and can be added to many forms of neural architectures including CNNs or Transformers with negligible computation overhead. Through extensive experiments on several datasets and state-of-the-art baselines, we show that incorporating S3 results in significant improvements for the tasks of time-series classification, forecasting, and anomaly detection, improving performance on certain datasets by up to 68\%. We also show that S3 makes the learning more stable with a smoother training loss curve and loss landscape compared to the original baseline. The code is available at https://github.com/shivam-grover/S3-TimeSeries.

Main paper: 
 introduction: 
 Time-series data serve an important role across diverse domains, including but not limited to health analytics [1, 2, 3] , human-computer interaction [4, 5] , human activity recognition [6, 7] , climate analysis [8, 9] , energy consumption [10, 11, 12] , traffic management [13, 14] , financial markets [15] , and others. The pervasive nature of time-series data has resulted in considerable interest among researchers, leading to the development of a variety of deep learning solutions for classification [16, 17, 18] and forecasting tasks [10, 19, 20] . Neural architectures such as convolutional networks [16, 21] , recurrent networks [22, 23] , and Transformers [10, 20] can capture the essential spatial and temporal information from time-series. Notably, these approaches have frequently outperformed traditional approaches, including Dynamic Time Warping [24] , Bag of Stochastic Frontier Analysis Symbols [25] , and the Collective of Transformation-Based Ensembles [26] in various scenarios.While both traditional machine learning and deep learning solutions aim to extract effective goalrelated representations prior to classification or forecasting, the general approach is to keep the original temporal arrangement of the time-steps in the time-series intact, with the presumption that the original order is the most optimal. Moreover, most existing models do not have explicit mechanisms to explore the inter-relations between distant segments within each time-series, which may in fact have strong dependencies despite their lack of proximity. For example, CNN-based models for time-series learning generally utilize fixed convolutional filters and receptive fields, causing them to only capture patterns within a limited temporal window [27, 28] . As a result, when faced with time-series where important patterns or correlations span across longer time windows, these models often struggle to capture this information effectively [28] . Dilated convolutional neural networks partially solve this by increasing the receptive field through the dilation rate. However, they are still practically limited by their inherent architectures, as their receptive fields rely on the number of layers, which may not be large enough to fully capture the long-range dependencies and can lead to vanishing gradients as more layers are added [29] . Similarly, the out-of-the-box effectiveness of Transformers [30] in capturing long-term dependencies highly depends on a variety of factors such as sequence length, positional encoding, and tokenization strategies. Accordingly, we ask a simple question: Is there a better arrangement for the time-series that would enable more effective representation learning considering the classification/forecasting task at hand?In this paper, we introduce a simple and plug-and-play network layer called Segment, Shuffle, and Stitch, or S3 for short, designed to enhance time-series representation learning. As the name suggests, S3 operates by segmenting the time-series into several segments, shuffling these segments in the most optimal order controlled by learned shuffling parameters, and then stitching the shuffled segments. In addition to this, our module integrates the original time-series through a learned weighted sum operation with the shuffled version to also preserve the key information in the original order. S3 acts as a modular mechanism intended to seamlessly integrate with any time-series model and as we will demonstrate experimentally, results in a smoother training procedure and loss landscape. Since S3 is trained along with the backbone network, the shuffling parameter is updated in a goal-centric manner, adapting to the characteristics of the data and the backbone model to better capture the temporal dynamics. Finally, S3 can be stacked to create a more fine-grained shuffling with higher levels of granularity, has very few hyper-parameters to tune, and has negligible computation overhead.For evaluation, we integrate S3 in a variety of neural architectures including CNN-based and Transformer-based models, and evaluate performance across various classification, univariate forecasting, and multivariate forecasting datasets, observing that S3 results in substantial improvements when integrated into state-of-the-art models. Specifically, the results demonstrate that integrating S3 into state-of-the-art methods can improve performance by up to 39.59% for classification, and by up to 68.71% and 51.22% for univariate and multivariate forecasting, respectively. We perform detailed ablation and sensitivity studies to analyze different components of our proposed S3 layer.Our contributions are summarized as follows:• We propose S3, a simple and modular network layer that can be plugged into existing neural architectures to improve time-series representation learning. By dynamically segmenting and shuffling the input time-series across the temporal dimension, S3 helps the model perform more effective task-centric representation learning. • By stacking multiple instances of S3, the model can perform shuffling at different granularity levels. Our proposed layer has very few hyperparameters and negligible added computational cost.• Rigorous experiments on various benchmark time-series datasets across both classification and forecasting tasks demonstrate that by incorporating S3 into existing state-of-the-art models, we improve performance significantly. Experiments also show that by adding our proposed network layer, more stable training is achieved. • We make our code public to contribute to the field of time-series representation learning and enable fast and accurate reproducibility. 
related work: 
 Deep learning architectures have recently made significant progress in the area of time-series representation learning. In the category of convolution-based methods, DSN [16] introduces dynamic sparse connections to cover different receptive fields in convolution layers for time-series classification. In another convolution-based approach, SCINet [21] captures temporal features by partitioning each time sequence into two subsequences at each level to effectively model the complex temporal dynamics within hierarchical time-series data.Transformer-based approaches are another category of solutions that have drawn considerable attention. Informer [10] improves the capabilities of the vanilla Transformer on long input sequences by introducing self-attention distillation using ProbSparse which is based on Kullback-Leibler divergence, and a generative style decoder. Autoformer [20] introduces decomposition blocks with an auto-correlation mechanism that allows progressive decomposition of the data. ContiFormer [31] integrates the continuous dynamics of Neural Ordinary Differential Equations with the attention mechanism of Transformers. PatchTST [32] enhances time-series forecasting by leveraging patching techniques to split long input sequences into smaller patches, which are then processed by a Transformer to capture long-term dependencies efficiently. Finally, [33] investigated the effectiveness of Transformers in dealing with long sequences for forecasting and highlighted the challenges Transformers encounter in this regard.Recently, foundation models for time-series data, which are large-scale, pre-trained models designed to capture complex temporal patterns, have gained popularity. A prominent example is Moment [34] , which pre-trains a Transformer encoder using a univariate setting on a large and diverse collection of data called 'Pile'. It leverages a masked reconstruction method and is capable of performing various downstream forecasting tasks after fine-tuning, demonstrating strong adaptability and forecasting accuracy.Contrastive methods have recently demonstrated state-of-the-art performances in time-series learning. TS2Vec [17] employs unsupervised hierarchical contrastive learning across augmented contextual views, capturing robust representations at various semantic levels. InfoTS [35] uses informationaware augmentations for contrastive learning which dynamically chooses the optimal augmentations. To address long-term forecasting, [28] introduces a contrastive approach that incorporates global autocorrelation alongside a decomposition network. Finally, SoftCLT [36] is a recent contrastive approach that uses the distances between time-series samples (instance-wise contrastive) along with the difference in timestamps (temporal contrastive), to capture the correlations between adjacent samples and improve representations.Given the existence of seasonal and trend information in time-series, disentanglement is an approach that has been widely used across both classical machine learning [37, 38, 39] and deep learning solutions [40, 19] . For instance, CoST [40] attempts to capture periodic patterns using frequency domain contrastive loss to disentangle seasonal-trend representations in both time and frequency domains. Similarly, LaST [19] recently employed variational inference to learn and disentangle latent seasonal and trend components within time-series data. Problem definition. Given a set of N timeseries instances as {x i } N i=1 , where x i ∈ R T ×C has a length of T and C channels, the goal is to optimally rearrange the segments of x i and form a new sequence x ′ i to better capture the underlying temporal relationships and dependencies within the time-series, which would consequently lead to improved representations given the target task.Proposed mechanism. We propose S3, a simple neural network component for addressing the aforementioned problem in three steps, as the name suggests, Segment, Shuffle, and Stitch, described below (see Figure 1 ).The Segment module splits the input sequence x i into n non-overlapping segments, each containing τ time-steps, where τ = T /n. The set of segments can be represented by S i = {s i,1 , s i,2 , . . . , s i,n } ∈ R (τ ×C)×n where s i,j = x i [(j -1)τ : jτ ] and s i,j ∈ R τ ×C .The segments are then fed into the Shuffle module, which uses a shuffle vector P = {p 1 , p 2 , . . . , p n } ∈ R n to rearrange the segments in the optimal order for the task at hand. Each shuffling parameter p j in P corresponds to a segment s i,j in S i . P is essentially a set of learnable weights optimized through the network's learning process, which controls the position and priority of the segment in the reordered sequence. The shuffling process is quite simple and intuitive: the higher the value of p j , the higher the priority of segment s i,j is in the shuffled sequence. The shuffled sequence S shuffled i can be represented asEQUATIONwhere the segments in S i are sorted according to the values in P. Permuting S i based on the sorted order of P is not differentiable by default, because it involves discrete operations and introduces discontinuities [41] . Soft sorting methods such as [42, 41, 43] approximate the sorted order by assigning probabilities that reflect how much larger each element is compared to others. While this approximation is differentiable in nature, it may introduce noise and inaccuracies, while making the sorting non-intuitive. To achieve differentiable sorting and permutation that are as accurate and intuitive as traditional methods, we introduce a few intermediate steps. These steps create a path for gradients to flow through the shuffling parameters P while performing discrete permutations on S i based on the sorted order of P. We first obtain the indices that sort the elements of P using σ = Argsort(P). We have a list of tensors S = [s 1 , s 2 , s 3 , ...s n ] (for simplicity, we skip the index i) that we aim to reorder based on the list of indices σ = [σ 1 , σ 2 , ....σ n ] in a differentiable way. We then create a (τ × C) × n × n matrix U, which we populate by repeating each s i , n times. Next, we form an n × n matrix Ω where each row j has a single non-zero element at position k = σ j which is p k . We convert Ω to a binary matrix Ω by scaling each non-zero element to 1 using a scaling factor 1 Ω j,k . This process creates a path for the gradients to flow through P during backpropagation.By performing the Hadamard product between U and Ω, we obtain a matrix V where each row j has one non-zero element k equals to s k . Finally, by summing along the final dimension and transposing the outcome, we obtain the final shuffled matrix of size S shuffled i ∈ R τ ×C×n . To better illustrate, we show a simple example with S = [s 1 , s 2 , s 3 , s 4 ] and a given permutation σ = [3, 4, 1, 2] . We calculate V asEQUATIONand S shuffled i is obtained byEQUATIONWe previously assumed that the set of shuffling parameters P is one-dimensional, containing n scalar elements, each corresponding to one of the segments. By employing a higher-dimensional P, we can introduce additional parameters that enable the model to capture complex representations that a single-dimensional P could struggle with. Therefore, we introduce a hyperparameter λ to determine the dimensionality of the vector P. When λ = m, the size ofP becomes n × n × • • • × n (repeated m times).We then perform a summation of P over the first m -1 dimensions to obtain a one-dimensional vector. Mathematically, this is represented asEQUATIONThis results in a one-dimensional matrix P, which we then use to compute the permutation indices σ = Argsort( P). This approach allows us to increase the number of shuffling parameters, thereby capturing more complex dependencies within the time-series data, without affecting the sorting operations.In the final step, the Stitch module concatenates the shuffled segments S shuffled i to create a single shuffled sequence xi ∈ R T ×C as xi = Concat(S shuffled i ). To retain the information present in the original order along with the newly generated shuffled sequence, we perform a weighted sum between x i and xi with learnable weights w 1 and w 2 optimized through the training of the main network. For practical convenience, we can use a simple learnable Conv1D or MLP layer taking x i and xi as inputs and generating the final time-series outputx ′ i ∈ R T ×C .Stacking S3 layers. Considering S3 as a modular layer, we can stack them sequentially within a neural architecture. Let's define ϕ as a hyperparameter that determines the number of S3 layers. For simplicity and to avoid defining a separate segment hyperparameter for each S3 layer, we define θ which acts as a multiplier for the number of segments in subsequent layers asEQUATIONwhere n is the number of segments in the first S3 layer. When multiple S3 layers are stacked, each layer ℓ from 1 to ϕ will segment and shuffle its input based on the output from the previous layer. We can formally represent the output of each layer ℓ as x ′ ℓ byEQUATIONwhere x ′ 0 is the original time-series x, P ℓ is the set of shuffling parameters for layer ℓ, and (w ℓ,1 , w ℓ,2 ) are the learnable weights for the sum operation between the concatenation of the shuffled segments and the original input, at layer ℓ. Figure 1 presents an example of three S3 layers (ϕ = 3) applied to a time-series with n = 2 and θ = 2.All the P ℓ values are updated along with the model parameters, and there is no intermediate loss for any of the S3 layers. This ensures that the S3 layers are trained according to the specific task and the baseline. In cases where the length of input sequence x is not divisible by the number of segments n, we resort to truncating the first T mod n time-steps from the input sequence. In order to ensure that no data is lost and the input and output shapes are the same, we later add the truncated samples back at the beginning of the output of the final S3 layer. 
experiment setup: 
 Evaluation protocol. For our experiments, we integrate S3 into existing state-of-the-art models for both time-series classification and forecasting. We first train and evaluate each model adhering strictly to their original setups and experimental protocols. We then integrate S3 at the beginning of the model, and train and evaluate it with the same setups and protocols as the original models. This meticulous approach ensures that any observed deviation in performance can be fairly attributed to the integration of S3. For classification, we measure the improvement as the percentage difference (Diff.) in accuracy resulting from S3, calculated as (Acc Baseline+S3 -Acc Baseline )/Acc Baseline . For forecasting, since lower MSE is better, we use (M SE Baseline -M SE Baseline+S3 )/M SE Baseline . A similar equation is used for measuring the percentage difference in MAE.Classification datasets. For classification we use the following datasets: (1) The UCR archive [44] which consists of 128 univariate datasets, (2) the UEA archive [45] which consists of 30 multivariate datasets, and (3) three multivariate datasets namely EEG, EEG2, and HAR from the UCI archive [46] . For our experiments with pre-trained foundation model in Section 5, we also use the PTB-XL [47] dataset. The train/test splits for all classification datasets are as provided in the original papers.Forecasting datasets. For forecasting, we use the following datasets: (1) both the univariate and multivariate versions of three ETT datasets [10] , namely ETTh1 and ETTh2 recorded hourly, and ETTm1 recorded at every 15 minutes, (2) both the univariate and multivariate versions of the Electricity dataset [46] , and (3) the multivariate version of the Weather dataset [48] .Anomaly detection datasets. We employ the widely used Yahoo [49] and KPI [50] datasets for anomaly detection tasks. The Yahoo dataset consists of 367 time-series sampled hourly, each with labeled anomaly points, while the KPI dataset contains 58 minutely sampled KPI curves from various internet companies. Our experiments are performed under normal settings, as outlined in [17, 36] .Baselines. We select baselines from a variety of different time-series learning approaches. Specifically, for classification, we use four state-of-the-art baseline methods, SoftCLT [36] , TS2Vec [17] , DSN [16] , and InfoTS [35] . For forecasting, we use five state-of-the-art baseline methods, TS2Vec [17] , LaST [19] , Informer [10] , PatchTST [32] and CoST [40] . For anomaly detection, we use two stateof-the-art baseline methods, SoftCLT [36] and TS2Vec [17] . Additionally, we use MOMENT [34] for experiments with foundation model. For each baseline method, we integrate one to three layers of S3 at the input level of the model and compare its performance with that of the original model. Implementation details. All implementation details match those of the baselines. We used the exact hyperparameters of the baselines according to the original papers when they were specified in the papers or when the code was available; alternatively, when the hyperparameters were not exactly specified in the paper or in the code, we tried to maximize performance with our own search for the optimum hyperparameters. Additionally, for experiments involving PatchTST and LaST with the Electricity dataset, we use a batch size of 8 due to memory constraints. Accordingly, some baseline results may slightly differ from those available in the original papers. Note that deviations in baseline results affect both the baseline model and baseline+S3. For the weighted sum operation in S3, we use Conv1D. Our code is implemented with PyTorch, and our experiments are conducted on a single NVIDIA Quadro RTX 6000 GPU. We release the code at: https://github.com/shivam-grover/S3-TimeSeries. Classification. The results of our experiments on time-series classification are presented in Table 1 . In this table we observe that the addition of S3 results in substantial improvements across all baselines for all univariate and multivariate datasets. For UCR, UEA, EEG, EEG2, and HAR, we achieve average improvements of 3.89%, 5.25%, 32.03%, 9.675%, and 5.18% respectively over all models. The full classification results on UCR and UEA datasets for all baselines with and without S3 are mentioned in Appendix (Table A1 and A2). These results highlight the efficacy of S3 in improving a variety of different classification methods over a diverse set of datasets despite negligible added computational complexity (we provide a detailed discussion on computation overhead later in this section). In addition to quantitative metrics, we also visualize the t-SNE plots for several datasets in Figure 2 and Appendix A1, which show that adding S3 results in representations with better class separability in the latent space. 
results: 
 Forecasting. Table 2 presents a comprehensive overview of the results for univariate forecasting on different datasets and horizons (H), with and without the incorporation of S3. We observe that S3 consistently leads to improvements for all baseline methods across all datasets, with an average improvement in MSE and MAE of 4.58% and 4.20% for TS2Vec, 4.02% and 3.01% for LaST, 16.69% and 8.19% for Informer, 1.64% and 1.20% for PatchTST, and 3.01% and 2.62% for CoST. Figure 3 shows two forecasting outputs for Informer with and without S3, on two different horizon lengths for a sample from the ETTh1 dataset. In both cases, S3 improves the ability of the baseline Informer to generate time-series samples that better align with the ground truth. Similarly, Table 3 presents the results of multivariate forecasting. Consistent with univariate forecasting, S3 significantly enhances the performance of the baseline on multivariate forecasting and achieves average improvements in MSE and MAE of 13.71% and 7.78% for TS2Vec, 9.88% and 5.45% for LaST, 25.13% and 14.82% for Informer, 1.62% and 1.17% for PatchTST, and 4.41% and 1.97% for CoST.Anomaly detection. Table 4 presents the results for anomaly detection on Yahoo and KPI datasets. The anomaly score is computed as the L1 distance between two encoded representations derived from masked and unmasked inputs, following the methodology described in previous studies [17, 36] . We observe that the proposed S3 layer enhances the performance in terms of F1 for both datasets under the normal settings, with scores of 0.7498 and 0.6892, respectively.Loss behaviour. We make interesting observations while training the baseline models as well as the baseline+S3 variants. First, we observe that the training loss vs. epochs for baseline+S3 variants generally converge faster than the original baselines. See Figure 4 where we demonstrate examples of this behavior. Second, we observe that the training loss curves for baseline+S3 are generally much smoother than the original baselines. This can again be observed in Figure 4 . Additionally, we measure the standard deviations of the loss curves for SoftCLT with and without S3 on all the UCR datasets and measure an average reduction in standard deviation of 36.66%. The detailed values for the standard deviations of the loss curves are presented in Appendix A4. Lastly, according to [51] , we investigate the loss landscape of the baselines and observe that the addition of S3 generally results in a smoother loss landscape with fewer local minima (Figure 5 ). Additional examples are provided in Appendix A2. Ablation. We perform ablation studies to investigate the impact of each of the key modules of S3 and answer three questions: (1) Do we need the Segment module to divide the input into multiple segments, or will shuffling all the time-steps individually work just as well? (2) Do we need learned Shuffling parameters (P) to determine the optimal permutation of the time-series, or will random permutations work just as well? (3) Do we need the Stitch module to apply a learned weighted sum of the original sequence with the shuffled sequence, or will the shuffled sequences alone work just as well? Table 7 shows the results of three ablation studies where we observe that the removal of each component of S3 results in a decline in performance.Foundation models. In order to evaluate the impact of S3 on pre-trained foundation model, we integrate S3 at the beginning of the pre-trained MOMENT encoder [34] . We use linear probing for fine-tuning, and to ensure a fair comparison, we strictly follow the setup and experimental protocols outlined in the original paper. In this process, the encoder is kept frozen, and only S3 and the final linear head are fine-tuned together for the specific task and dataset. We fine-tune and evaluate the models for classification on PTB-XL [47] and the Crop dataset from the UCR archive [44] , and the ETTh1 and ETTh2 datasets [10] for forecasting. The results for classification are presented in Table 5 , where we observe that upon adding S3, both the test loss and accuracy see significant improvements of up to 13.31% and 5.24%, respectively. Table A3 (Appendix) shows the forecasting results, where incorporating S3 consistently improves performance. On the ETTh1 dataset, it yields Hyperparameters. Next, we investigate the impact of the number of segments n on performance. Considering all the experiments, we observe that no general rule of thumb can be advised for n, as its optimum value is naturally highly dependent on factors such as dataset complexity, length, baselines, and others. We present two examples in Figure 6 and more in Appendix A3 where we plot the accuracy vs. number of segments for datasets of different lengths from UCR. Here we only take into account the layer with the largest number of segments. As for the other hyperparameters, the number of S3 layers ϕ has been set between 1, 2, or 3 across all experiments in this paper. θ, the multiplier for the number of segments in stacked S3 layers, has been set to 0.5, 1, or 2 across all our experiments. And finally, λ has only been set to 1, 2, or 3. The ranges of values for all the hyperparameters used in this work are presented in Table 6 . While optimizing hyperparameters is important, similar to any other form of representation learning, we find that even sub-optimal tuning of S3 hyperparameters still yields meaningful improvements. We perform a simple experiment where we apply a uniform set of hyperparameters [n = 2, ϕ = 2, θ = 1, λ = 1] to all the baselines and baselines with added S3 layers. The results for this experiment are presented in Table 8 where we observe that despite not using the optimum hyperparameters, the addition of S3 still yields considerable performance boosts. The optimum hyperparameters used for each experiment are made available in the code.Sensitivity to random seed. To evaluate the impact of the random seed on the performance of S3, we perform 10 separate trials with different seed values and present the standard deviations in Table 9 . The first 4 rows show the classification baselines trained on the UCI datasets with and without S3. Similarly, the next 3 rows show forecasting baselines trained on the multivariate ETTh1 dataset (H=24) with and without S3. From this experiment, we observe that the addition of S3 has no considerable impact on the sensitivity of the original baselines to the random seed.Shuffling parameters. In Figure 7 , we present a visual overview of how the parameters of P update along with the parent model during training on ETTh1 (H = 48) and ETTm1 (H = 48). Each parameter in P corresponds to a segment of S. In both cases, we set n to 16 and ϕ to 1. The baseline used is LaST [19] . It can be seen that the individual weights in P rearrange the corresponding segments until they reach a stable point (the optimal order) after which the order of the segments is maintained. Similarly, Figure A4 (Appendix) shows how w 1 and w 2 update during training. Finally, in Figure A5 (Appendix), we show sample visualizations of how S3 segments, shuffles, and stitches a time-series once the optimal shuffling parameters are obtained.Computation overhead. Our proposed network layer adds very few learnable parameters to the baseline models, which stem from a learnable shuffling vector P along with w 1 and w 2 , per each S3 layer. To emphasize the negligible computation overhead of S3 in comparison to the baseline models, we show the total number of parameters in the baseline and the total learnable parameters that S3 adds, in Table 10 for classification on the EEG2 dataset and forecasting on the multivariate ETTh1 dataset (H = 24). 
conclusion: 
 Summary. We propose S3, a simple plug-and-play neural network component that rearranges the time-series in three steps: Segment the time-series, Shuffle the segments, and Stitch the shuffled time-series by concatenating the segments and performing a learned sum with the original timeseries. Through extensive experiments on time-series classification and forecasting with state-ofthe-art methods, we show that S3 improves the learning capabilities of the baseline with negligible computation overhead. We also show empirically that S3 helps in faster and smoother training leading to better performance.Limitations. While we present the effectiveness of S3 on a diverse set of baselines on classification, forecasting, and anomaly detection tasks, we acknowledge that the evaluation of other time-series tasks such as imputation remain for future work. Additionally applying S3 to learning representations from other forms of time-series such as videos is an interesting direction for future research.Broader impact. Since S3 is a plug-and-play network layer with negligible added parameters, it allows researchers from various domains related to time-series such as health signal processing, biometrics, climate analysis, financial markets, and others to use this module in their existing models without the need for redesign. The low computation overhead also makes S3 a suitable choice for edge-devices.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to unders tand the results?Answer: [Yes] Justification: We use the same experimental settings as the baselines that we have incorporated S3 into, except for experiments on PatchTST and LaST with the electricity dataset where we use a batch size of 8. We have clarified this in Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: In Table 9 we show the standard deviation in performance metric for all baselines on a single dataset for both classification and forecasting by repeating the experiments for 10 different seed values, observing very small sensitivity to seed values. We do not perform this for each experiment in our main paper, as it would be computationally too expensive.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See section. 6Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: Apart from our code, that is well documented, we do not release any assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification:Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We show the full results for all baselines with and without S3 on 128 UCR datasets in Table A1 and on 30 UEA datasets in Table A2 . In Figure A1 we show 2 more t-SNE plots for TS2Vec trained on 2 randomly chosen UCR datasets. A better clustering of different classes can be observed in the learned representations after S3 is incorporated. In Table A4 we compare the standard deviation values for the training loss of SoftCLT and SoftCLT+S3. From this table we observe that after incorporating S3, the training becomes much more stable with significantly less variations. Figure A2 shows 2 more comparisons of the loss landscape of TS2Vec trained with and without S3. In Figure A3 we present 6 more samples for the impact of n on the performance of the model. Figure A3 : Performance vs. number of segment. In (a), (b), and (c), 3 UCR datasets for classification were used, and in (d), (e), and (f) the ETTh2 dataset for forecasting was used with 3 different horizon lengths. A higher value for accuracy is better, and a lower value MSE is better.In Figure A4 , we present a visual overview of how the weighted average parameters w 1 and w 2 update along with the rest of the model during training on ETTh1 (H = 24) and ETTh2 (H = 720) in multivariate settings. We use LaST [19] as the baseline. It is observed that the weights converge to their final values at the end of training.0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 Iteration The proposed S3 layer is fundamentally different from data augmentation techniques, in that unlike traditional augmentation methods, S3 introduces learnable parameters that are jointly optimized with the rest of the model, allowing S3 to adapt dynamically throughout the training process. To further explore the differences between S3 and data augmentation, we conduct an experiment where, before training on the ETTm1 multivariate dataset, we apply the following augmentations: (1) shuffling augmentation with varying numbers of segments, (2) shuffling augmentation combined with mixup, (3) noise augmentation with zero mean, variance of 1, and a magnifying coefficient of 0.01, and (4) A5 where we observe that S3 outperforms such data augmentation strategies. Additionally, data augmentation techniques typically have a stronger impact on smaller datasets, often due to the lack of variation and diversity, but their effect is less pronounced on larger datasets [53] . To evaluate this, we use the LSST dataset from UEA and create 10 different subsets by randomly dropping varying amounts of data, ranging from 20% to 99%. We then retrain SoftCLT [36] , both with and without S3, on these subsets as well as on the original dataset. The results, averaged over three runs, are presented in Figure A6 . These results show no clear trend in performance gain relative to dataset size, indicating that S3 benefits datasets regardless of their size.To investigate the scenarios where S3 is most effective, we conduct additional experiments considering three key factors: (1) the length of time-series sequences, (2) the degree of non-linearity, and (3) the presence of long-term temporal dependencies. For (1), we examine the percentage difference relative to sequence length. For (2), we consider the percentage difference relative to the mean of squared residuals from a linear model. Lastly, for (3), we analyze the percentage difference in relation to the Hurst exponent [54] . We use PatchTST [32] for experiments (2) and (3), while Informer [10] is used for experiment (1) since PatchTST cannot be used due to its reliance on equal-length inputs. We conduct all three experiments on the ETT, Weather, and Electricity datasets. The results of this analysis are shown in Table A6 , where m represents the slope of the linear relationship, and R is Pearson's correlation coefficient. We observe direct positive relationships between the percentage difference upon adding S3 and both sequence length and long-term temporal dependency, while non-linearity in the time-series shows no considerable relationship. For sequence length, the moderate correlation suggests that while the effect is not very strong, it is consistent, indicating that sequence length is likely a relevant factor for percentage difference. Long-term temporal dependency has a strong impact on percentage difference, as indicated by the steep slope. However, the lower correlation suggests that other factors in the data may also influence this relationship. We hypothesize that when long-term temporal dependencies are simple to learn or highly repetitive, the impact of S3 will be less pronounced, as re-ordering may not be necessary to learn such simple dynamics. On the other hand, for more complex long-term dependencies, S3 has a much stronger impact, as reflected by the high slope. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: The paper presents a simple but effective differentiable module called Segment, Shuffle, and Stitch (S3), to combine these discrete operations in analysing time-series data sets. This addresses a key problem in multi-variate time-series data analysis for several architectures, including PatchTST and CoST. The proposed method is validated on several multi-variate and univariate time-series data sets. 

However, the model requires retraining with each task (Authors response to Reviewer FZgx) and S3 has not been studied in conjunction with pre-trained foundation models (Authors response to Reviewer 5XMz). The method proposed in this paper has the potential to impact  pre-processing of time-series data sets, while analysing time-series data sets that are ubiquitous in several applications. This can be achieved through lesser computational resources.

Reviews: review: paper_summary: paper_summary: The paper introduces a simple but effective differentiable module that performs pre-processing to input multivariate time-series before being fed into any differentiable model for arbitrary task. The pre-processing involves segmenting, shuffling the segments and stiching them together. The novelty include making this seemingly discrete operations into a differentiable module. This simple idea yields significant improvement in performance of different kinds of models over variety of datasets.
strength_weakness: strength_weakness: 1. Visualization and any qualitative study on the shuffling and segments generalted by S3 would greatly benefit the readers.
2. How well does it optimize transformer based models, especially those that already do segmentation like PatchTST since the attention module captures the relations all pairs of segments already?
3. Does the representations due to S3 generalize to multiple tasks at a time or do we need to retrain for each task?
questions: questions: See weaknesses
limitations: limitations: 1. Lack of understanding on the segment permutations generated and why they are better for the model performance atleast qualitatively
paper_summary: paper_summary: The paper paper introduces a new approach called Segment, Shuffle, and Stitch (S3) to enhance time-series representation learning. The method involves segmenting the time-series into non-overlapping parts, shuffling them optimally, and stitching them back together along with the original sequence.

Key contributions include:

- Proposing the S3 mechanism to improve time-series representation learning by dynamically reordering segments.
- Demonstrating that S3 can be integrated with existing neural architectures like CNNs and Transformers, resulting in significant performance improvements.
- Showing through extensive experiments that S3 enhances performance in time-series classification and forecasting tasks, with improvements up to 68%.
strength_weakness: strength_weakness: - Tables 1 and 2 focus on the ETT datasets, which are only a (highly intra-correlated) subset of the common forecasting datasets: Electricity, Traffic, Weather, Illness...
- I see no mention of CoST in the results tables, despite being cited in the paper. This is usually a very strong baseline for contrastive approaches. Including it would certainly paint a more complete picture of the results landscape. On a related note this also applies to e.g. more recent transformer baselines. Informer is relevant, but also very far from state of the art.
- Error bars would help one better contextualize the results.
- The lack of an ablation study makes understanding the reason this works more complicated.
questions: questions: - The 3 points in weaknesses are also questions in the sense that they ask for some new experiments to be performed. Addressing those points would be my first recommendation.
- Intuitively, it feels like this work is to some extent a form of bootstrap (as data augmentation) combined with a mixup-like sample interpolation. I may be wrong on this and am happy to discuss. If so, could the authors do more of an ablation study connected to this. I.e. how does the approach outperform other (non-permutation)-based data augmentation strategies combined with the same summation operation?

Edit: I have read the author's rebuttal. They have addressed questions I had and I am as a result raising my score to a 6.
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposes a new neural network design element which segments, shuffles, and stitches time series for improved representation learning. They evaluate their methods on forecasting and classification tasks, and show that S3 benefits some widely used baselines.
strength_weakness: strength_weakness: I have not found any major weaknesses in the methodology or experimental design. However,  I think that the paper might benefit from showing what the S3 module is actually learning. For example, the authors can include the segmented, shuffled, and stitched time series on a particular dataset as an example, along with the weighted time series (used as input to the model), and the original time series. This might provide some intuition as to how this design element improves predictive performance. 

I think there's always scope to improve experimental design. TS2Vec is a excellent choice for classification, but not for forecasting. I would recommend that the authors use methods such as PatchTST (transformer-based) or iTransformer, TimesNet (CNN-based), N-BEATs or N-HITS (MLP-based) etc. for time series forecasting. For classification, it would also be good to compare with fully supervised methods such as ResNet1D (see [1]). 

### References
[1] Ismail Fawaz, Hassan, et al. "Deep learning for time series classification: a review." Data mining and knowledge discovery 33.4 (2019): 917-963.
questions: questions: I do not have questions per se, but I am listing some things that I am curious about below:

I would also encourage the authors to evaluate the benefits of S3 on some recent time series foundation models such as MOMENT [2], Chronos [3], Moirai [4], TimesFM [5], and/or LagLLama [6]. The MOMENT model does both classification and forecasting, so it might be interesting to see how S3 benefits pre-trained models, say by just training the S3 layer and freezing the pre-trained backbone (or some variation of this experiment).

On a similar note, I wonder if S3 improves generalization and hurts memorization, or vice versa. It would be interesting to do some transfer learning experiments where you train on some time series data and evaluate the model on other time series data (see MOMENT or PatchTST for inspiration). 

### References
[2] Goswami, Mononito, et al. "Moment: A family of open time-series foundation models." arXiv preprint arXiv:2402.03885 (2024).
[3] Ansari, Abdul Fatir, et al. "Chronos: Learning the language of time series." arXiv preprint arXiv:2403.07815 (2024).
[4] Woo, Gerald, et al. "Unified training of universal time series forecasting transformers." arXiv preprint arXiv:2402.02592 (2024).
[5] Das, Abhimanyu, et al. "A decoder-only foundation model for time-series forecasting." arXiv preprint arXiv:2310.10688 (2023).
[6] Rasul, Kashif, et al. "Lag-llama: Towards foundation models for time series forecasting." arXiv preprint arXiv:2310.08278 (2023).
limitations: limitations: The authors have a very brief description of limitations of their study.
paper_summary: paper_summary: This paper introduces a plug-and-play mechanism called Segment, Shuffle, and Stitch (S3) designed to enhance time-series representation learning in existing models. S3 operates by dividing the original sequence into non-overlapping segments and shuffling them in a learned manner that is optimal for the given task. It then reattaches the shuffled segments and performs a learned weighted sum with the original input to capture both the newly shuffled sequence and the original sequence. This proposed model can enhance the performance of specific models in classification and prediction tasks.
strength_weakness: strength_weakness: Lack of comparative methods:
In fact, the proposed method seems to share the same spirit as data augmentation methods in the time series field[1-4]. Why hasn't any data augmentation method been compared?


Selection of baseline models:
The selected baseline model, Informer, seems somewhat outdated. Why not choose a more recent model, e.g., iTransformer[5] or PatchTST[6]?


Dataset for prediction task:
The author conducted experiments on three ETT datasets, but for prediction tasks, more datasets should be considered, e.g., traffic, electricity, and weather.


Time-Series Representation Claim:
 As the author pointed out, more tasks should be considered for time series representation learning.


[1]FRAUG: FREQUENCY DOMAIN AUGMENTATION FOR TIME SERIES FORECASTING  [2]Time Series Data Augmentation for Deep Learning: A Survey  [3]SimPSI: A Simple Strategy to Preserve Spectral Information in Time Series Data Augmentation [4]TOWARDS DIVERSE AND COHERENT AUGMENTATION FOR TIME-SERIES FORECASTING [5]ITRANSFORMER: INVERTED TRANSFORMERS ARE EFFECTIVE FOR TIME SERIES FORECASTING [6]A TIME SERIES IS WORTH 64 WORDS: LONG-TERM FORECASTING WITH TRANSFORMERS
questions: questions: What are the essential differences between the proposed method and other data augmentation methods?
limitations: limitations: See Weakness.
paper_summary: paper_summary: This paper introduces a new method for time-series representation learning that enhances the modeling of non-adjacent segment dependencies. Specifically, the proposed method segments, shuffles in a learned manner and stitches the shuffled segments to combine with original time series. The proposed method is model-agnostic without adding significant parameter overhead and shows performance improvement across multiple classification and forecasting base models.
strength_weakness: strength_weakness: 1. It it not clear how the sorting process, specifically the calculation of permutation $\sigma$ from $P$, is made differentiable.

2. The compared forecasting baselines such as Informer are no longer state-of-the-art methods. Adding more recent baselines such as Time-LLM, GPT4TS, DLinear, PatchTST would provide a clearer understanding of the proposed method's comparative benefits.

3. The basic assumption for S3 is that modeling non-adjacent dependencies is important. However, the paper lacks detailed case studies that demonstrate the specific types of non-adjacent dependencies effectively captured by S3, which are not addressed by existing models. Additionally, there is no case study to validate that the learned shuffling weights accurately represent these segment dependencies.
questions: questions: 1. The results in Tables 1, 2, and 3 seem to indicate more significant improvements in multivariate than in univariate time series tasks. Any reason behind this?

2. What does the "number of segments" represent in Figure 6 and Figure A3? Is it the number of segments for the first layer or the final layer? If it refers to "n", then in Figure A3, this number seems to perform the best when it is larger than 100 for some datasets?

3. Could you describe the inference process for the S3 method? Additionally, what are the computational overheads for training and inference times for S3?
limitations: limitations: The paper mentions potential expansions into tasks like imputation and anomaly detection. Further details on limitations from the reviewer are discussed in Weaknesses and Questions.

Comments: title: Thank you for the excellent work and the rebuttal!
Thank you
comment: We are very happy to hear that our responses have addressed the reviewer's concerns, and appreciate the increase in score.
We would like to sincerely thank the reviewer for their support and encouraging comments. We agree regarding the new experiment - we will certainly add these experiments to the paper (either main section or appendix).
Thank the authors for the new experimental results and explanations. These solve most of my concerns. I have raised my score.
Dear Authors, 

I really appreciate the time and effort that you have put into the study. I really like it, and I would like to maintain my current score to reflect my very positive assessment of this paper. I really appreciate figure R1, and I think it should find its way into the paper. 
Also the transfer learning experiments while preliminary, are very promising, and I would encourage the authors to add an element of this to their revised manuscript. 

Best, 

Reviewer 5XMz
We would like to thank the reviewer for their comments and further engaging with us on this matter. We think we now have a better understanding about the scope of the question. 

To analyze the scenarios in which S3 is most effective, we have now performed additional experiments, and considered 3 factors: (1) length of time-series sequences, (2) non-linearity, and (3) long-term temporal dependency. For (1), we considered the %improvement vs. sequence length. For (2) we considered the %improvement vs. the mean of squared residuals w.r.t. a linear model. And finally for (3), we considered the %improvement vs. Hurst exponent [1]. We used PatchTST (transformer-based) as the baseline model given that it is the most recent work in the area, and used the ETT, Weather, and Electricity datasets. For sequence length, however, PatchTST uses equal input lengths for everything, so we used Informer (another transformer-based model) instead for which the input length is varied depending on the dataset and forecasting horizon. The outcome of this analysis is shown in the table below, where m is the slope of a linear relationship and R is Pearson's correlation coefficient. We observe that there are direct positive relationships between improvement by S3 versus sequence length, and also long-term temporal dependency, while the non-linearity in the time-series shows no relationship. For sequence length, the moderate correlation suggests that while the impact is not very strong, it is reliable, implying that sequence length is likely a relevant factor for %improvement. Long-term temporal dependency has a substantial impact on %improvement, as indicated by the large slope. Although, the lower correlation suggests that this relationship could be influenced by other factors in the data. We hypothesize that should these long-term temporal dependencies be simple to learn or highly repetitive, the impact of S3 will be less significant as re-ordering will not be required to learn such simple dynamics while for more complex long-term dependencies, S3 has a much stronger impact (as per the high slope).

| x                             |\| y            |\| m     |\| R     |
| ----------------------------- | ------------ | ----- | ----- |
| Sequence length               |\| %improvement |\| +0.04 |\| +0.55 |
| Non-linearity                 |\| %improvement |\| 0.00  |\| +0.05 |
| Long-term temporal dependency |\| %improvement |\| +2.89 |\| +0.26 |


To answer your second question, we have analyzed how w1 and w2 change over training of ETTh1 multivariate dataset with PatchTST. While unfortunately we cannot provide you with a new figure at this stage (due to NeurIPS rules), we have tried to provide a discretized version in the table below. We observe that the values steadily converge to the final results without too much fluctuation (some small fluctuations in the beginning are generally observed). 

| Iteration    | w1     | w2     |
| ------------ | ------ | ------ |
| 0            | 0.1510 | 0.1490 |
| 1000         | 0.2846 | 0.0683 |
| 2000         | 0.5074 | 0.1505 |
| 3000         | 0.7129 | 0.3286 |
| 4000         | 0.8140 | 0.4210 |
| 5000         | 0.4210 | 0.5552 |
| 6000         | 0.9498 | 0.9498 |
| 7000         | 0.9647 | 0.5858 |
| 8000         | 0.9809 | 0.6215 |
| 9000         | 0.9906 | 0.6368 |
| 10000        | 0.9838 | 0.6632 |
| 11000        | 0.9827 | 0.6762 |
| 12000        | 0.9884 | 0.6969 |
| 13000        | 0.9960 | 0.6994 |
| 14000        | 0.9953 | 0.7068 |
| 15000        | 0.9992 | 0.7188 |
| 16000        | 0.9970 | 0.7286 |
| 17000        | 1.0004 | 0.7348 |
| Final values | 1.0004 | 0.7348 |

**References**

[1] Tong, et al., “Learning fractional white noises in neural stochastic differential equations", NeurIPS, 2022.
Thank the authors for the clarifications and new experimental results! These solve most of my concerns. 

However, the current case study still does not show the benefits of shuffling time series. Typically, forecasting task requires the model to obey the temporal dynamics, so it is counterintuitive to shuffle the order and perturb the temporal dynamics. Moreover, models such as Transformer are able to capture long-range dependencies without the need to shuffle the segment order. Therefore, I was hoping to see in which scenarios shuffling the order better models non-adjacent dependencies compared to, say, modeling long-range dependencies using a Transformer. 

Additionally, not sure if I miss it, is there any visualization to show how w1 and w2 change over the training process?
Since we posted the rebuttal for your attention, we were able to add S3 to a foundation model, Moment [1], and use linear probing (fine-tuning the final linear layer of Moment) on their pre-trained *MOMENT-1-large* model along with S3 on the PTB-XL [2] dataset. The table below presents the results where we observe a considerable gain by adding S3, indicating potential for future research in this area.

|               |\| Moment |\| Moment + S3 |\| Improvement |
| ------------- | ------ | ----------- | ----------- |
| Test loss (lower is better)     |\| 0.8308 |\| 0.7329      |\| 11.79%      |
| Test accuracy (higher is better) |\| 0.7176 |\| 0.7497      |\| 4.48%       |


**References**

[1] Goswami, et al., “Moment: A family of open time-series foundation models”, ICML, 2024.

[2] Wagner, et al., “PTB-XL, a large publicly available electrocardiography dataset”, Scientific Data, 2020.
We sincerely thank the reviewer for the great comments which have resulted in improving the paper. We appreciate that our rebuttal has answered your questions and are grateful for increasing your score.
We sincerely thank the reviewer for the great comments which have resulted in improving the paper. We appreciate that our rebuttal has answered your questions and are grateful for increasing your score.
I thank the authors for the detailed response to my and other reviewers' questions. This has helped strengthen my position for the paper's merit. I increased my score to 6.
I have read the author's response to my points, and as mentioned in the main review I am raising my score.