TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Series Forecasting
Authors: Qinglong Liu
Cong Xu
Wenhao Jiang
Kaixuan Wang
Lin Ma
Haifeng Li
Venue: ICML 2025 poster (2025)
Keywords: deep learning
time series forecasting
non-stationarity
attention
Primary area: deep_learning->sequential_models_time_series

Abstract: Real-world time series inherently exhibit significant non-stationarity, posing substantial challenges for forecasting. To address this issue, this paper proposes a novel prediction framework, TimeStacker, designed to overcome the limitations of existing models in capturing the characteristics of non-stationary signals. By employing a unique stacking mechanism, TimeStacker effectively captures global signal features while thoroughly exploring local details. Furthermore, the framework integrates a frequency-based self-attention module, significantly enhancing its feature modeling capabilities. Experimental results demonstrate that TimeStacker achieves outstanding performance across multiple real-world datasets, including those from the energy, finance, and weather domains. It not only delivers superior predictive accuracy but also exhibits remarkable advantages with fewer parameters and higher computational efficiency.

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Series Forecasting
Qinglong Liu'
Cong Xu'
Wenhao Jiang
Kaixuan Wang' Lin Ma
Haifeng Li
Abstract
ral network models have been developed, exhibiting remark-
Real-world time series inberently exhibit signif- icant non-stationarity, posing substantial chal- lenges for forecasting. To address this issue, this paper proposes a novel prediction frame- work, TimeStacker, designed to overcome the limitations of existing models in capturing the characteristies of non-stationary signals. By em- ploying a unique stacking mechanism, TimeS- tacker effectively captures global signal features while thoroughly exploring local details. Further- more, the framework integrates frequency-ban based self-attention module, significantly enhancing its feature modeling capabilities. Experimental re- sults demonstrate that TimeStacker achieves out- standing performance across multiple real-world datasets, including those from the energy, finance, and weather domains. It not only delivers supe- rior predictive accuracy but also exhibits remark- able advantages with fewer parameters and higher computational efficiency.
able performance in time series forecasting, For example, MLP-based approaches such as Deinear(Zeng et al., 2023), SOFTS(Han et al., 2024a), SparseTSF(Lin et al., 2024) and WWneMixer(Wang et al., 2024a), and Transformer-bant Yan, 2023),
architectures include Crossformer(Zhang PatchTST(Nie et al., 2022), SAMformer(Ibbert et al.), and iTransformer(LLu et al., 2023). These models have achieved state-of-the performance in time series forecasting due to their advanced architectural designs and innovative method- ologies.
DOMATN
Windee s20-L
Angihado
Windoww sise. L/2
Original Signal
better representttion
Frequency
Frequerry
signa?
Angiinge
If ve shack all the fully represent The
Frinuctr
1. Introduction
Time series forecasting, which involves inferring future trends and patterns from historical observations, is widely applied in diverse domains, including weather forecast- ing(Wu et al., 2023), energy scheduling(Chou & Tran, 2018), traffic management(Zhou et al., 2021), medical anal- ysis(Cepulionis & Lukoševičiūté, 2016). and financial eco- (onins et al., 2022). However, the complexity of real-world systems often results in non-stationary time se- ries( (Wang et al., 2024b), which complicates accurate pre- diction using traditional methods and presents substantial challenges for time series forecasting.
Figure I. Under observations with different window sizes, the fre- quency of the same non-stationary signal exhibits varying pat- terns. The signal represents the original non-stationary signal, while the red components illustrate the frequeney patterns obtained through Short-Time Fourier Transform (STFT) with window sizes of L. 2/L. and 4/ L, respectively.
Despite the substantial advancements made by these meth- ods in time series forecasting, the majority of studies pri- marily explore temporal correlations, often overlooking the frequency-domain characteristies of non-stationary sig- nals. Based on stochastic process theory(Cox, 2017), the frequency of non-stationary signals fluctuates over time, and frequency-domate representations are more effective than time-domain signals in capturing signal characteristics within specific time intervals. Thus, analyzing the frequency variation patterns of non-stationary signals is essential for time series forecasting.
With the rapid advancement of deep learning, numerous neu-
Facully of Computing, Harbin Institute of Technology. Harbin, China, Harbin, China. Correspondence to: Haifeng Li <li- haifeng hit.edu.en>>
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Series Forecasting
However, the uncertainty principle of time-frequency anal- ysis(Cohen, 1995) precludes the precise observation of signal's frequency at a specific moment. To overcome this limitation, the short-time Fourier transform (STFT) is com- monly employed to segment the original signal (i.e., divide it into patches) for frequency analysis within specific time intervals. The selection of patch size significantly influences the ability to capture frequency variation patterss: larger patches are more effective in capturing global signal fea- tures, whereas smaller patches better reveal local details, As depicted in Figure 1, signal frequencies corresponding to different patch sizes are visualized. Hence, identifying an optimal patch size for effectively extracting and represent- ing signal patterns remains a key challenge revealing the intrinsic regularities of sequences.
tional efficiency than other benchmark models. This frame- work offers viable solution for time series forecasting of non-stationary signals.
2. Related Work
Time series forecasting models have undergone substantial evolution over time. Early approaches time series fore- casting were primarily based on statistical theories, which typically assumed that time series exhibit stationarity or linear relationships(Boxx et al., 2015). These methods pre- dicted trends, seasonality, and stochastic variations by mod- eling these components. Representative models include AutoRegressive Integrated Moving Average (ARIMA)(Lees & Tong, 2011), 1), Exponential Smoothing (De Livera et al., 2011), and Seasonal AutoRegressive Integrated Mov- ing Average (SARIMA)Dubey al., 2021).
To overcome these challenges, this paper introduces novel framework, TimeStacker. Rather than selecting single optimal patch size, patches of varying sizes are sequentially stacked and aggregated based on frequency. Through itera- tive stacking, the most expressive patterns within the signal are progressively emphasized, enhancing the representation of the overall time series. Specifically, patterns within the signal are aggregated layer by layer in descending order of patch size, allowing the model to capture global features while retaining local details.
The emergence of deep learning facilitated significant ad- vancements in time series forecasting, particularly with the introduction of Transformer models(Li et al., 2019). In contrast to traditional approaches, deep learning models are capable of automatically extracting features and ef- fectively capturing complex nonlinear relationships. For example, Recurrent Neural Networks (RNNs)(Sagheer & Kotb, 2019) capture temporal dependencies in time series through their recursive structure, making them particularly effective for modeling short-term dependencies. Convolu- tional Neural Networks (CNNs)(Sezer et al., 2020) utilize one- -inedmmenio convolutional operations to extract local features, effectively capturing short-term patterns and regu- larities in sequences.
To further optimize the stacking process, a frequency-based enhanced self-attention module is designed to aggregate same size. Within this module, signal similar- ity is computed in the frequency domain, whereas aggrega- tion operations are conducted in the time domain. This ap- proach effectively mitigates the detrimental effects of inher- ent Fourier transform eтrоrs and spectral leakage in signal modeling. Experimental results indicate that TimeStacker attains state-of-the thr performance across most forecasting tasks while maintaining fewer parameters and significantly greater efficiency than other models.
patches of the
The introduction of the Transformer architecture represented fundamental paradigm shift in time series forecasting, For instance, PStchTST partitioned time series into independent patches embedded in high-dimensional spaces while pre- serving channel independence, enabling all series to share weights and establishing foundation for subsequent re- search. Crossformer improved multivariate forecasting ca- pabilities by capturing dimensiamaii patch dependen- cies in multivariate time series, iTransformer revisited the hierarchical design of traditional Transformer architectures, utilizing self-attention mechanisms to model vater-variable relationships and employing feedforward networks to cap- ture nonlinear variable transformations.
The main contributions of this paper are summarized as follows:
i) A novel framework, TimeStacker, is proposed to compre- hensively capture the variation patterns of frequency scales. By stacking patches sequentially from large to small, it facilitates a simple yet effective approach to time series forecasting.
fi) A novel rrqquency-based self-attention module is de- signed to more effectively compute the similarity between patches based on their frequencies. Additionally, this mod- ule mitigates the detrimental effects of inherent Fourier transform errors and spectral leakage in signal modeling.
Beyond Transformer- basedd architectures, recent years have seen substantial advancements in MLP-based models. DLin- ear highlighted the effectiveness of linear layers in time series forecasting, particularly excelling in long-sequence modeling. TSF simplifies the forecasting task by disentangling the periodic and trend components of time se- ries data through cross-period sparse prediction techniques. TimeMixer introduced fully MLP-based model designed
iii) Experimentally, TimeStacker demonstrates state-of-the- art predictive accuracy across most real- world datasets while utilizing fewer parameters and exhibiting higher computa-
2

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Series Forecasting
Patch size gradually decreases
XL
Input
Stocker Block
Stocker Block
W HMM Manh
Patch size,
Patch
Vartetea
Inter-Patch Frequency bazed Attertion Hodale
Smoohh Loyer
Inter-Petch Frequency based Attention Module
Predictor
Imsrmalizstionn
Time Points
Charnel Independent
Charnel Independent
Figure 2. Overall Architecture of Staker. The overall architecture comprises multiple Stacker Blocks. Each Stacker Block consists Smooth Layer and an Inter-Pater Frequency-basee Attention Module, responsible for smoothing the time series and aggregating patches, respectively. Within each block, patches of the same size are aggregated based on their frequency characteristics. Subsequent blocks sequentially process patches decreasing sizes.
to explore multiscale temporal information in time series across various temporal domains.
the data have a mean of O and a variance of 1, thereby mitigating the impact of scale differences on model training.
Channel Independence. Channel independence is a fun- damental strategy in time series forecasting. The core prin- ciple involves treating each variable in multivariate time series as independent channel and modeling each channel separately rather than as whole. This approach was first introduced by PatchTST((Han et al., 2024b), which demon- strated its effectiveness in time series forecasting and has since been widely adopted subsequent neural network models for time series prediction. This method mitigates noise interference between channels and reduces modeling complexity, thereby facilitating more effective representa- tion of individual variable characteristics.
3. Method
This section begins with the task definition, followed by an overview of the preliminaries, the overall structure of TimeStacker, implementation details, theoretical analysis, and complexity analysis. The overall architecture of the proposed model is depicted in Figure 2.
3.1. Problem Definition
The time series forecasting problem formulated as fol- lows: Given a time series Xt-T+1# = ( xr) E R DxT, where represents specific timestamp, Dis the number of variables, and Hı E R denotes the observed value time f, the objective is predict the future values X {fr+1..... } Dxr, where repre- sents the prediction horizon.
3.3. Overall Architecture
Patches of different sizes capture the frequency variation patterns of time series to different extents. A decrease in patch size enhances the temporal resolution of the sequence while reducing its frequency resolution. Stacking patches enables a more comprehensive analysis of complex varia- tion patterns in time series, thereby enhancing forecasting accuracy.
3.2. Preliminaries
Normalization. Time series datasets often exhibit vary- ing numerical ranges, which can result in unequal model attention to different data and lead to biased parameter up- dates. Revin(Kim et al., 2021) demonstrated that appropriate normalization significantly enhances time series forecast- ing performance and plays crucial role in model training. Therefore, this model employs the same normalization and denormalization approach Revin, utilizing mean and vari- ance for standardization. The formulas are as follows:
As illustrated in Figure 2, the overall framework comprises L StackerBlocks, a normalization normalization-denommizzazzz mod- ulc, and predictor module. The StackerBlock is designed to capture variation patterns within patches of the same size and is elaborated in Section 3.4. The normalization- denormalization module, as discussed in Section 3.1, per- forms data preprocessing, while the predictor module is implemented as linear layer.
x' = (X) X --H
(1)
Consider univariate time series of length T, represented as X= = {z). ......, ITT E gIxT .Given L non-overlapping patch sizes defined by the vector P = {pL. {PL-P2,- P2...... PL), where P1 > P2 > ... > Pr and each element P is re- quired be divisor of T, the goal is to extract features and progressively stack patches to capture variation pat-
X denormal (x') = X' +
(2)
Here, X' denotes the normalized time series, μ represents the mean of the input time series, and a represents the standard deviation. This normalization process ensures that
3

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Scries Forecasting
Hadamard Product
comprises two core components: the Smooth Layer and the Inter-Patch Frequency- Attention Module.
W,
Inter-Patch Correlation Matrix
Patch Size
Query V
In real-world time series, outliers can significantly impact the model's ability to capture sequence variation patters,. To mitigate this issue, the Smooth Layer is introduced, uti- lizing time points within a patch to reduce the influence of outliers. Specifically, the Smooth Layer is implemented using convolution operation with a kernel size of pt The formal representation is given by:
Softmax
DFT
MatMul
Key
Mathuul
W,
Linear Projection
SmoothLayer, (x()) = W * 2 +
(7)
Figure 3. Internal Structure of Freenttention. This module initially computes the similarity between patches in the frequency domain. It then aggregates these patches in the time domain based on the computed similarity.
To exploit frequency variation information within the se- quence, the Inter-Patch Frequency-Baser Attention Mod- ule (FreqAttention) is introduced. The structure of Fre- qAttention is depicted in Figure 3. Unlike traditional self- attention mechanisms, similarity in this approach is com- puted in the frequency domain. The formal representation is given by:
terns effectively. The time series X is first standardized as follows:
x (1) normal(X)
g000 = =F F(x11)
Let Pi be the first element of P, representing the initial patch size. The standardized time series (') is partitioned into subsequences of size P1 yielding:
(8)
Q = Wg 3 200 K = KW ② Z00
(9)
xx(2) = 51 (1) 82 (1) ,...,,......... Sa, ( RIX
W ① 20) (W& @ Z00T
(4)
Cor = Softmar
dd
Here k,- I. Next, (1) is fed into the StackerBlock pi associated with p1 which extracts variation patterns within patches of the same size, yielding, Similarly, let be the next patch size, and re-segment X² into patches, yielding:
(10)
Here, F()) denotes the Fourier transform, represents the Hadamard product, and W;, Wa E R 1 are learnable parameters used to compute the query and key vectors, and de is scalar matching the dimension of di From signal processing perspective, the Hadamard product enables W,, and Wa to function as learnable filters, extract- ing frequency components relevant to subsequent sequences. Consequently, CorMar E Ig x represents the frequency- based similarity between patches and reflects the sequence's 's variation patterns in the frequency domain. Aggregating the time series in the time domain using CorMat at yields the following formal expression:
T
(2) E glaxps K2
(5)
P2
After applying the same operations, x is obtained. This process iterated L times, with each step further parti- tioning patches based on the preceding iteration. Through iterative patch stacking, the variation patterns of the time series are progressively captured by integrating patches of different sizes. The formal representation is as follows:
V = xOW
(1I)
Stocker 1 (x)(11))
Stacker B (11-1))
Freg
x( Concat
Stocker Blocki (1(1-11)
Softmar W. W, M. (WLo WwW
(12)
3
7
(6)
Stacker Stacke (2)-
Here, W. E RP is learnable parameter for computing the value vector. By combining Equations (7) to (12), the formal expression of the StackerBlock is given by:
- 112,....
3.4. Stacker Block
Stacker Block, (X())) = (x(1) + +x((10 + 20) (13)
To effectively capture variation patterns across patches (inter-patch), the StackerBlock is introduced. This module
4

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Scries Forecasting
Table 1. Main Results. All results are based on input sequences of length 96 and are calculated as the average across four different prediction lengths (96, 192, 336, 7203. The prediction performance is evaluated using MSE or MAE as metrics, where lower values indicate closer alignment between the predicted and actual sequences. Complete experimental results are provided in Appendix D.I.
TimeStucker
SOFTS
SparseTFF
Transformer
TimeMixer
SAMformer
FatchTST
Crossformer
DLincur
RLinear
Models
(ONPs))
(2024)
(20124)
(2024)
(2024)
(2124)
(2023)
(2023)
(2123)
(2023)
Metric
MSE
MAR
MSH
MAE
MSE
MAR
MSE
MAE
MSH
MAE
MSE
MAR
MSE
MAR
MSR
MAE
MSR
MAR
MSE
MAR
ETTa1
0.433
0.449
0.44
0.111
0.425
0.447
0.447
0.440
0.414
0.432
0.469
0.151
0.529
0.522
0.456
0.452
0.446
0.431
ETT22
0.368
0.390
0.375
0:400
0.421
0.438
0.383
0.407
0.364
0.395
0.383
0.4D1
0.587
0.407
0.942
0.684
0.559
0,515
0.374
0.398
ETTml
0.381
0.381
0.393
0.403
0.125
0.101
0.407
0.410
0.381
0.395
0.115
0.107
0.387
0.400
0.513
0.196
0.403
0.407
0.414
0.407
ETTm2
0.274
0.316
0.287
0.330
0.297
0.331
0.288
0.332
0.205
0.323
0.285
0.327
0.281
0.326
0.757
0.610
0.350
0.401
0.286
0.327
Traflic
0.508
0.335
0.409
0.267
0.578
0,350
0.428
0.282
0.484
0.297
0.595
0.382
0.481
0.304
0.550
0,304
0.625
0.383
0.626
0.378
Electricity
0.191
0.275
0.174
0.264
0.222
0.289
0.178
0.270
0.182
0.272
0.217
0.295
0.205
0.290
0.244
0.334
0.212
0,300
0.219
0.298
Weather
0.243
0.264
0.255
0.278
0.290
0.302
0.258
0.278
0.240
0.271
0.264
0.285
0.289
0.348
0.259
0.315
0.265
0.317
0.272
0.291
Exchange
0.336
0.389
0.348
0.395
0.365
0.401
0.360
0.403
0.355
0.399
0.346
0.399
0.367
0.404
0,940
0.707
0.354
0.414
0.378
0.417
3.5. Theoretical Analysis
lows:
1 4x
Definition 3.1. The statistical properties of non-stationary signals (c.g., mean, variance, autocorrelation function) change over time, and their frequency characteristics also dynamically evolve with time. These can be expressed using the Fourier series as follows:
At Ar
(15)
Here At denotes the standard deviation in the time domain, while, and Aʃ denotes the standard deviation in the fre- quency domain. This reveals an inherent limitation in pre- cisely measuring both the temporal location and frequency components of a signal: achieving arbitrarily high time and frequency resolution simultaneously is fundamentally impossible.
xxt)- = at)++) R=1
an (t)cooss (2xnfot) + bt)(s) (2mrffot)
(14)
To overcome this limitation, TimeStacker sequentially stacks patches of varying sizes, from large to small, ef- fectively reducing Af (frequency resolution) while progres- sively enhancing time resolution. This approach enables the model to capture the dynamic evolution of the spectrum over time at multiple temporal resolutions, allowing it to focus on both fine-grained local details and overarching global trends.
Let att denote a non-stationary signal, where an,(t) and (t) are time-varying nonlinear functions that capture the dynamic variations in the signal components. This implies that the frequency content of a non-stationary signal varies over time.
From a rrqquencydomaale perspective, time series forecast- ing fundamentally involves analyzing the latent frequency characteristics within historical sequences to identify the temporal variation patterns of their Fourier coefficients a, (t) and bn (t). These coefficients encode the amplitude and phase information of the signal at specific frequencies, serv- ing as key indicators of its time-varying properties in the frequency domain. Therefore, capturing and modeling the variation patterns of these coefficients can effectively re- veal the periodicity and trends in time series, establishing a robust foundation for accurate future forecasting.
Specifically, larger patches yield lower time resolution and higher frequency resolution, enabling the model to capture global periodicity and long-term trends within the signal. Conversely, smaller patches yield higher time resolution, allowing the model to accurately capture dynamic variations in local signal components.
3.6. Complexity Analysis
The Inter-Patch Frequency-based Attention Module distin- guishes itself from traditional self-attention mechanisms primarily in the computation of Q and K. To facilitate ex- planation, we use the calculation of Q as an example for analyzing computational complexity.
The time-frequency uncertainly principle states that a signal cannot achieve arbitrarily high reso- lution simultaneously in both the time and frequency do- mains, reflecting the resolution limit of a signal in the time- frequency domain. The mathematical expression is as fol-
Theorem 3.2.
Let the input for similarity computation be X E Rkxx.
5

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Scries Forecasting
Table 2. Ablation Experiment Results. Ablation experiments were conducted the FreqArtention module of TimeStacker, involving replacemen( (EPLAAC)) and removal (W/O) operations based on traditional self-attention MAE and MSE were used as evaluation metrics, with all input sequence lengths set to 96.
W/O
REPLACE
W/O
REPLACE
Method
TimeStacker
FreqAttentin
FreqAttention MSE MAE
Hadamard
Hadamard
MSE MAE
MSE MAE
MSE MAE
MSE MAE
ETTh1
96
0.388 0.395
0.386 0.395
0.384 0.393
0.381 0.393
0.379 0.385
192
0.436 0.420
0.431 0,424
0,435 0.425
0.431 0.424
0.429 0.416
336
0.476 0.444
0.472 0.445
0.472 0.444
0,474 0.447
0.459 0.436
720
0.480 0.471
0.474 0.469
0.466 0.464
0.476 0.468
0.464 0.455
ETTh2
96
0.288 0.338
0.296 0.343
0.292 0.338
0.293 0.342
0,280 0.327
192
0.398 0.397
0.389 0.396
0.390 0.399
0.393 0.394
0.373 0.385
336
0.418 0.426
0.416 0.424
0.419 0.426
0.414 0.423
0.407 0.416
720
0.418 0.437
0.419 0.437
0.420 0.439
0.418 0.436
0.412 0.431
Exchange
96
0.087 0.206
0.085 0.203
0.086 0.204
0.085 0.203
0.084 0.200
192
0.179 0.303
0.173 0.293
0.171 0.294
0.171 0.293
0.171 0.293
336
0.323 0.415
0.317 0.411
0.320 0.412
0.316 0.410
0.314 0.408
720
0.807 0.675
0.791 0.669
0.780 0.666
0.796 0.672
0.776 0.656
The computation of Q is detailed in Equations (8) and (9). The computational complexity of the Fast Fourier Transform (FFT) is O (kplog2p). while the complexity of the Hadamard product is O (I (1 + 1)) approximately O (k&). Substituting
iTransformer(Liu et al., 2023). These datasets span vari- ous domains, including energy, transportation, and weather. Specifically, the datasets used in the experiments include ETT (comprising four subsets: ETTh1, ETTh2, ETTm1, and ETTm2), Weather, Traffic, Electricity, and Exchange. Each dataset was divided into training, validation, and test sets in a 6:2:2 ratio. For more details on the datasets, refer to Appendix A.
kp into the formulas, the com-
plexity of computing Q is derived as O (L (logyP + 1 )), which approximates to O (Llogzp). In comparison, the computational complexity of traditional self-attention is O (kp"), which simplifies to O (Lp). This demonstrates that the proposed method significantly reduces complexity, particularly for nigh-diimnssonal inputs, where its advan- tages become especially evident.
Implementation Details. TimeStacker employs Mean Ab- solute Error (MAE) the loss function and utilizes Adam as the optimizer. The learning rate is set to x 10 the weight decay to 1 x 10-3, and to 1 x 10-8, The first and second moment decay rates are set to 0.9 and 0.999, respectively. All experiments were implemented using Py- Torch 2. 2.0(Paszke et al., 2019) and conducted on an NVIDIA RTX 4080 GPU with 16GB of memory. Detailed parameter settings can be found in Appendix B.2.
This method leverages the frequency characteristics of sig- nals to alleviate the performance bottlenecks in traditional self-attention models, which are caused by high computa- tional complexity in the time domain. This optimization provides more efficient computational pathway for time series forecasting, maintaining the model's predictive accu- racy and robustness.
4.2. Experimental Results
Baseline. We compared TimeStacker with state-of-the-ar and representative models proposed in the past two years to evaluate its effectiveness. The main baselines include: (1) Transformer- et al., 2024), SAMfommer(Ibert et al.), Patc et al., 2022), and Crossformer(Zhang & Yan, 2023); (2) Linear- layer-based models such as SparseTSF(Lin et al., 2024), SOFTS(Han et al., 2024a), TimeMixer Wang et al., 2024a), DLinear(Zeng et al., 2023), and RLinear(Li et al., 2023).
4. Experiments
In this section, extensive experiments and analyses are con- ducted on real-world datasets from various domains, encom- passing both long-term and short-term forecasting tasks, to comprehensively evaluate the performance and computa- tional efficiency of TimeStacker.
models such as Pathformer(Cher
4.1. Experimental Setup
Main Results. Table presents the time series forecast- ing results. Red highlights the best performance, while blue with underlining indicates the second- performance. Lower Mean Squared Error (MSE) and MAE values cor-
Dataset, To evaluate the performance of TimeStacker, ex- periments were conducted on multiple widely used real- world datasets, following a processing approach similar to
6

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Series Forecasting
0.4
ETTh1
Imeitacerttun
ETTh2
0.42
SAbfomer
Time innetareiuuug
Exchange
irandfrmer
S0FTS
0.44
limellorr
0.43
0.25
0.42
335
SkVismer
0.4
Innretemer
0.34
3.4
28
102
136
720
96
152
550
720
Back Langth
Length
Lick BawkBackk.orrr Lenggh
Figure 4. Impact of book-Back Lengths. All models use look-back window lengths of 96, 192, 336, and 720, with MSE serving as the evaluation metric. The results are calculated as the average across four different prediction lengths {96, 192, 336, 720), to assess the performance of different models under varying look-back lengths.
volved replacement (REPLACE) and removal (W/O) opera- tions, as outlined below:
Table 3. The impact of different look-back window lengths on model prediction performance. The look- back window lengths are [96. 192, 336, 720}, and the prediction lengths are (96, 192, 336,
7203. The evaluation metric is MSE
W/O FreqArtention: Removed the FreqAttention mod- ule from TimeStacker.
Method
96
192
336
720
MSE
MSE
MSE
MSE
REPIACE FreqAttention: Replaced the Frtention module in TimeStacker with traditional Self-Attention.
ETThi
96
0.379
0.370
0.362
0.364
192
0.429
0.424
0.401
0.399
336
0.459
0.448
0.428
0.42
W/O Hadamard: Removed the Hadamard product op- eration from the FreqAttention module.
720
0.464
0.447
0.432
0.427
ETTh2
96
0.280
0.283
0.270
0.267
192
0.373
0.366
0.348
0.336
REPLACE Hadamard: Replaced the Hadamard prod- uct operation in the Freq module with a linear transformation.
336
0.407
0.397
0.362
0.354
720
0.412
0.409
0.399
0.389
Exchange
96
0.084
0.084
0.085
0.085
192
0.171
0.173
0.174
0.171
336
0.314
0.317
0.320
0.321
As shown in Table 2, incorporating frequency information into similarity computation significantly enhances the ac- curacy of time series forecasting compared to traditional self-attention methods. Introducing the Hadamard prod- uet in the computation of Q and K further reveals hidden pattern features within the sequence. The proposed Fre- qAttention exhibits a distinct advantage in capturing time series dynamics, confirming its effectiveness in modeling the complex characteristics of dynamic signals.
720
0.776
0.756
0.792
0.772
respond to higher predictive accuracy. Compared to other models, TimeStacker demonstrates superior performance across multiple datasets (ETThI-ETTH) Weather, Ex- change). Notably, in comparison with iTransformer, TimeS- tacker achieves significant improvements. For example, on the ETTml dataset, MAE reduced by 7.07%; on the Weather dataset, MAE is reduced by 5.04%; and on the Exchange dataset, MSE is reduced by 6.67%. These results clearly demonstrate that TimeStacker not only excels in time series forecasting accuracy but also exhibits strong gener- alization capabilities. In particular, in critical application domains such as energy, weather, and finance, TimeStacker showcases broad applicability and practical utility.
Impact of Look-Back Lengths. In general, longer look- back windows provide more information about the sequence, allowing the model to capture sequence features more com- prehensively. However, longer sequences may also intro- duce additional noise, potentially affecting prediction accu- racy. To examine the impact of look-back window length on model performance, experiments were conducted using varying window lengths of [96, 192, 336, 720).
The detailed results, presented in Table 3, indicate that pre- diction accuracy on the Exchange dataset remained largely unchanged regardless of window length. In contrast, for the ETThl and ETTh2 datasets, accuracy significantly im- proved as the took-back window length increased. These
4.3. Model Analysis
Ablation Study. A series of ablation experiments were conducted to evaluate the core components of TimeStacker, namely the Atteniitntion module. These experiments in-
7

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Scries Forecasting
findings suggest that TimeStacker effectively utilizes the additional sequence information from longer windows to enhance forecasting performance.
ing mechanism and balancing modeling between the time and frequency domains, TimeStacker effectively captures the complex characteristies of non-stationary time series, To further enhance the stacking mechanism and fully uti- lize frequency information, a frequency- module was designed. This module not only mitigates the impact of spectral leakage but also enhances feature repre- sentation by comprehensively modeling both frequency and time domain information.
Additionally, our model was compared with SOFT, TimeMixer, i'Transformer, and SAMformer. As illustrated in Figure 4, the results demonstrate the superior ability of Stacer in capturing sequence features. Further details can be found in Appendix D.2.
self-attention
Additionally, extensive experiments were conducted on vari- ous real-world datasets. The results demonstrate that T-meS- tacker effectively extracts the dynamic characteristics of non non-stationary time series and achieves a comprehensive feature representation. Performance analysis further reveals that this approach delivers high prediction accuracy while significantly reducing computational complexity, highlight- ing its potential for applications in resource-constrained scenarios.
Mamoory Foctorint
439G 12ra
4CMIB
200MB
108
PRechTST
Diliner FURR Vilen
BUinerr
thansfereer
TranGtacherourdd
SAMtormmr
Sparects
BOITS
Tiasekiioer
Limitation and Future Work. While TimeStacker effec- tively captures time series features, comparative experi- ments indicate a slight decline in predictive performance as the number of variables in the time series increases. To investigate this phenomenon, additional experiments were conducted (in Appendix D.5) by varying the number of vari- ables and testing the model on the Traffic and Electricity datasets. The results suggest that the model may encounter performance bottlenecks when handling multivariate time series.
Thanang Mere
Figure S. Performance Comparison of Models. All models were evaluated on the ETThl dataset using an experimental setup with look- back length 96, a prediction length of 720, and batch size of 128. The final evaluation results were obtained by averaging the outcomes five independent trials. The evaluation metrics included memory usage during runtime, execution time, and MSE.
Model Effectiveness. In Section 3.6, the computational complexity of the core module in TimeStacker was derived as O (Llog2p). To provide a more comprehensive evalu- ation of the model's efficiency, additional analyses were conducted on memory usage and training time. The ETTh1 dataset was used for this evaluation, with all models config- ured to a look-back length of 96, prediction length of 720, and a batch size of 128. The results were averaged over five independent trials.
Therefore, future work will focus on optimizing multi- channel prediction strategies by designing additional mod- ules to enhance TimeStacker's performance on multivariate time series tasks.
A Acknowledgements
This study is supported by the Young Scientists Fund of the National Natural Science Foundation of China(Grant No. No.62306094), Independent Research Exploration Projects of Songjiang Laboratory(Grant No.SI. 20230220,, Project supported by the Special Funds of the National Natural Science Foundation of China(Grant No. 32441112).
As illustrated in Figure 5, the models with the lowest mem- ory usage were SSF and DLinear, each consuming only 13.7 MB, while the fastest model was RLinear, with an iteration time of just 59 ms. In comparison, TimeStacker required 24.9 MB of memory and had an iteration time of 115 ms, while achieving the lowest MSE among all models. Clearly, TimeStacker demonstrates strong predictive perfor- mance and computational efficiency, achieving the lowest prediction error while maintaining relatively low memory consumption and runtime.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
5. Conclusion, Limitation, and Future Works
Conclusion. This paper introduces the TimeStacker frame- work, which leverages the frequency variations of non- stationary signals over time. By employing a unique stack-
8

---

TimeStacker:
A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Scries Forecasting
References
sharpness-aware minimization and channel-wise atten- tion. In Forty-first International Conference on Machine Learning.
Box, G. E., Jenkins, G. M., Reinsel, G, C., and Ljung, G. M. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
Kim, T., Kim, J., Tae, Y., Park, C., Choi, J.-H., and Choo, J. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
Čepulionis, P. and Lukoševičiūte, K. Flectrocardiogram time series forecasting and optimization using ant colony optimization algorithm. Mathematical Models in Engi- neering, (1):69-77, 2016.
Lee, Y.-S. and Tong, L.-I. Forecasting time series using a methodology based on autoregressive integrated moving average and genetic programming. Knowledge- -asse Systems, 24(1):66-72, 2011.
Challu, C., Olivares, K. G., Oreshkin, B. N., Ramirez, F. G., Canseco, M. M., and Dubrawski, A. Nhits: Neural hier- archical interpolation for time series forecasting. 37(6): 6989 6997, 2023.
Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.- X., and Yan, X. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecast- ing. Advances neural information processing systems, 32, 2019.
Chen, P., Zhang, Y., Cheng, Y., Shu, Y., Wang, Y., Wen, Q Yang, B., and Guo, C. Pathformer: Multi-scale transform- ers with adaptive pathways for time series forecasting. arXiv preprini arXiv:2402.05956 2024.
Cheng, D., Yang, F., Xiang, S., and Liu, J. Financial time series forecasting with multi-modality graph neural net- work. Pattern Recognition, 121:108218, 2022.
Li, Z., Qi. S., Li, Y., and Xu, Z. Revisiting long-term time series forecasting: An investigation linear mapping. arXiv preprint arXiv:2305. 10721, 2023.
Chou, J.-S. and Tran, D.-S. Forecasting energy consumption time series using machine learning techniques based on usage patterns of residential householders. Energy, 165: 709-726, 2018
Lin, S., Lin, W., Wu, W., Chen, H., and Yang. J. Sparsetsf: Modeling long-term time series forecasting with 1k pa- rameters. arXiv preprint arXiv:2405.08 04.0000690,, 2024.
Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., and Long, M. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625, 2023.
Cohen, L.. Tieeufrry analysis, volume 778. Prentice Hall PTR New Jersey, 1995.
Cox, D. R. The theory of stochastic processes. Routledge, 2017.
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211. 1..3300 2022.
De Livera, A. M., Hyndman, R. J., and Snyder, R. D. Fore- casting time series with complex seasonal patterns using exponential smoothing. Journal of the American statisti- cal association, 106(496):1533-1577, 2011.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Dubey, A. K., Kumar, A., García-Diaz, V., Sharma, A. K., and Kanhaiya, K. Study and analysis of sarima and Istm in forecasting time series data. Sustainable Energy Technologies and Assessments, 47:101474, 2021.
Sagheer, A. and Koth, M. Time series forecasting of petroleum production using deep Istm recurrent networks. Neurocomputing, 323:203-213. 2019. 2019.
Han, I.,, Chen, X.-Y,, Ye, H.-J., and Zhan, D.-C. Softs: Efficient multivariate time series forecasting with series- core fusion. arXiv preprint arXiv:2404. 14197, 2024a.
Sezer, O. B., Gudelek, M. U.. and Ozbayoglu. A. M. Fi- nancial time series forecasting with deep learning: A systematic literature review: 2005-2019. Applied soft computing, 90: 2020.
Han, L., Ye, H.-J., and Zhan, D.-C. The capacity and robust- trade-off: Revisiting the channel independent strat- egy for multivariate time series forecasting. IEEE Trans- actions Knowledge and Data Engineering, 2024b.
Wang, S., Wu, H., Shi, X., Hu, T., Luo, H., Ma, L., Zhang J. Y., and Zhou, J. Timemixer: Decomposable multi- scale mixing for time series forecasting, arXiv preprint arXiv:2405. /4616, 2024a.
Ilbert, R., Odonnat, A., Feofanov, V., Virmaux, A., Paolo, G., Palpanas, T., and Redko, Samformer: Unlocking the potential of transformers in time series forecasting with

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Scries Forecasting
Wang, Y., Wu, H., Dong, J., Liu, Y., Long, M., and Wang, Deep time series models: A comprehensive survey and benchmark. arXiv preprint arXiv:2407.13277 2024b.
Wu, H., Zhou, H., Long, M., and Wang. Interpretable weather forecasting for worldwide stations with unified deep model. Nature Machine Intelligence, 5(6):602-611, 2023.
Zeng, A., Chen, M., Zhang. I.., and Xu, Q. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 11121-11128 128, 2023.
Zhang. Y. and Yan, Crossformer: Transformer utilizing cross-dimension dependeney for multivariate time series forecasting. In The eleventh international conference on learning representations, 2023.
Zhou, K., Wang, W., Huang, L., and Liu, B. Comparative study on the time series forecasting of web traffic based on statistical model and generative adversarial model. Knowledge Based Systems, 213:106467, 2021.
Zhou, T., Ma. Z., Wen, Q. Sun, L., Yao, T. Yin, W. Jin, R., al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in neural information processing systems, 35:12677-12690, 2022a.
Zhou, T., Ma, Z., Wen, Q. Wang, X., Sun, L., and Jin, R. Fedformer: Frequency enhanced decomposed trans- former for long-term series forecasting. pp. 27268-27286. 2022b.
10

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Scries Forecasting
A. DATASET DESCRIPTION
Time series forecasting experiments were conducted on widely used real-world datasets (details provided in Table 4), The datasets are described follows:
ETT Dataset: Comprising two hourly datasets (ETTh) and two 1 15-minute datasets (ETTm), this dataset includes load features of seven oil and electricity transformers recorded from July 2016 to July 2018.
Traffic Dataset: This dataset contains hourly road occupancy rates recorded by sensors San Francisco freeways from 2015 to 2016.
Electricity Dataset: Recording hourly electricity consumption for 321 customers, this dataset spans the period from 2012 to 2014.
Weather Dataset: Comprising 21 weather indicators, including air temperature and humidity, this dataset was recorded every 10 minutes throughout 2020.
Exchange- rate Dataset: This dataset contains daily exchange rates for eight countries, collected from 1990 to 2016.
Table 4. Variates denotes the number of variables in each dataset, Dataset Size denotes the number of time points in the dataset, Frequency denotes the sampling frequency of the dataset, and Information denotes the category information the dataset.
Dataset
ETTh1
ETTh2
ETTml
ETTm2
Traffic
Electricity
Weather
Exchange-rate
Variates
7
7
7
862
321
8
Dataset Size
14,307
14,307
57,507
57,507
17,451
26,211 I
52,603
7,207
Frequency
Hourly
Hourly
15min
15min
Hourly
Hourly Electricity
10min
Daily
Information
Electricity
Electricity
Electricity
Electricity
Transportation
Weather
Economy
B. IMPLEMENT DETAILS
B.1. TimeStacker
The complete algorithmic process of TimeStacker is outlined as follows, It takes the time series æ as input and generates the corresponding prediction T.
Algorithm TimeStacker
Input: Historical look-back window 1:-T41: RT, Patch size list P (Pi, P2. ................................ Pr Output: Forecasting horizon Z++1*+H R.
N Normalize(z) |* Normalizer the input sequence with mean and variance */
for p; in P do
x Smyoth / Apply convld with kernel size of p3 */
I Reshape(x, (pt, T/p)))
T = Alletion))) 1* Aggregate different patches #/
end for
Reshape
Denormaliseeppredicoooff je Apply linear layer for prediction */
B.2. Model Configuration
TimeStacker primarily consists of multiple Stacker Blocks, with the number of blocks determined by the Patchlist parameter. The number of elements Patchlist defines the depth of TimeStacker, while the size of each element determines the observation window for the corresponding block. Given the varying sampling frequencies and physical characteristics of different datasets, customized configurations are adopted for each dataset. Detailed information is provided Table 5.

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Scries Forecasting
Table Model Configuration.
DataSet
ETThl
ETTh2
ETTml
ETTm2
Trafiic
Electricity
Weather
Exchanng
Patch Size List
(96.48.32.24.66.12)
(96.48.32.24.16.12)
(48.32.24.16)
(48.32.24.16)
(96.48.32.16.12)
(96,48,24.12)
(96,48.32.24.12)
(96,48,24)
C. EXPERIMENT DETAILS
C.1. Metric Details
To comprehensively evaluate model performance across different datasets, Mean Absolute Error (MAE) and Mean Squared Error (MSE) were selected evaluation metries. These metrics assess prediction accuracy from different perspectives, offering comprehensive and intuitive basis for model comparison.
MAE:
MAE
21
(16)
MSE:
MSE =
-
(17)
Here, Iú ,T, RCxL denote the ground truth and predicted values, respectively, where L represents the number of time points, and C is the number of channels. The MSE primarily emphasizes the squared differences between predicted and actual values, thereby amplifying the impact of larger errors. This makes MSE particularly sensitive to outliers and suitable for evaluating the model's ' ability to capture global trends. On the other hand, the MAE directly computes the average absolute differences between predicted and actual values. It focuses on assessing the model's overall control of error magnitude in practical applications and is less affected by outliers.
C.2. Baseline
Representative methods from the past two years in time series forecasting were selected as baseline approaches to com- prehensively evaluate the performance of the proposed model. detailed introduction to these methods provided
below:
DLinear: Utilizes simple yet effective single-layer linear model to capture temporal relationships between input and output sequences.
Crossformer: Segments multivariate time series data along each dimension, embeds them into feature vectors, and employs a two-stage attention mechanism to efficiently capture both intra- and inter-series dependencies.
Splits time series data into subsequence-leve patches to extract local semantics, adopting a channel-independent strategy where each channel shares the same embedding and Transformer weights across all sequences.
PatchTST:
RLinear: Uses linear mapping to model periodic features in multivariate time series, demonstrating robustness across different periods as input length increases.
SAMformer: Enhances the model's generalization ability by leveraging sharpness- optimization techniques.
TimeMixer: Addresses complex temporal variations time series forecasting through multi-scale mixing perspective, improving complementary predictions from multi-scale sequences by decoupling variations.
iTransformer: Reverses the Transformer structure by encoding each individual series variable tokens without modifying any existing modules.
SparseTTFE Simplifies the forecasting task by decoupling the periodicity and trend of time series data using cross-period sparse prediction techniques.
SOFTS: Introduces novel centralized structure transfer information across channels, addressing the limitations of
12

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Series Forecasting
channel-independent approaches in leveraging inter-channel correlations and mitigating robustness challenges in channel- dependent methods.
C.3. Experiment details
TimeStacker adopts MAE the loss function and uses Adam as the optimizer, with a learning rate set to 1 10-3, weight decay set to 10-3 3, and epsilon set to 1 x 10 8. The first-order and second-order moments are configured as 0.9 and 0.999, respectively, All experiments are implemented in PyTorch 2.0 and executed on an NVIDIA RTX 4080 GPU with 16GB of memory.
In the experiments, the same dataset processing approach as TimeMixer was followed, ensuring that datasets were divided into training, validation, and test sets in a strict temporal sequence with 6:2:2 ratio to prevent data leakage. The look-back window length was fixed 96, and the forecasting horizons were set to (96.192,336.720).
D. FULL RESULTS
D.1. Complete Experimental Result
The complete experimental results are presented in Table 12. Experiments were conducted on six widely used real-world datasets spanning domains such as energy, traffic, and weather to comprehensively validate the effectiveness of the proposed method. To further assess model performance, comparisons were made against several representative models in the field.
The results demonstrate that the proposed method achieves outstanding performance across multiple datasets, consistently surpassing most baseline models in both predictive accuracy and computational efficiency. Notably, the method exhibits significant advantages in handling complex non-stationary time series, effectively capturing both global trends and local details within the signals. These findings further underscore the generalizability and robustness of the approach, offering an efficient and accurate solution for time series forecasting tasks.
The results of the mean and standard deviation of MSE and MAE for multiple runs of all datasets are shown in Table 6.
Table 6. Mean/Standard deviation for MSE and MAE across multiple runs.
DataSet
ETThl
ETTh2
ETTml
ETTm2
Traffic
Electricity
Weather
Exchange
MSE
0.433/0.00145
0.368/0.00091
0.274/0.0006 0.381/0.00119
0.508/0.00052
0.194/0.0005
0.243/0.00092
0.336/0.00101
MAE
0.423/0.00167
0.390/0.00057
0.381/0.00052
0.316/0.00022
0.335/0.00087
0.275/0.00077
0.264/0.00044
0.389/0.00137
D.2. Impact of Look-Back Lengths
The impact of look-back length on model performance was examined, with detailed experimental results presented in Table 11. Prediction experiments were conducted using look-back lengths of (96, 192, 336, 720). and the results were compared against SOFT, TimeMixer, iTransformer, and SAMformer. The results demonstrate that as the look-back length increases, the predictive accuracy of TimeStacker improves significantly. Compared to other models, TimeStacker effectively utilizes the extended sequence information, resulting in a substantial enhancement in prediction performance. This outcome validates the advantages and robustness of the proposed method in time series modeling.
D.3. Impact of Patch Size List
To demonstrate how TimeStacker adapts to various non-stationary signals, we configured the parameter Patch Size List and conducted experiments on the ETTml dataset. The results are shown in Table 7. These results indicate that employing various window combinations can more effectively capture the underlying dynamic patterns of the sequence, thereby improving prediction performance.
D.4. Complexity Analysis
To enable a more in-depth analysis of the model's computational complexity, the input length was increased to evaluate its temporal and spatial complexity (GPU Memory (MB) /Taining Time (ms/iter)). The experimental results are shown in
13

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Scries Forecasting
Table Impact of different patch size list.
Patch Size List
[16, 16, 16, 16]
[16, 16, 16, 24)
[16, 16, 16, 32]
[16, 16, 16, 48]
[16, 16, 24, 32)
[16, 16, 24, 48]
[16, 24, 32, 48]
MSR MAE
0.465
0.468
0.468
0.465
0.463
0.463
0.460
0.433
0.439
0.436
0.431
0,431
0.430
0.428
Table 8. Complexity Analysis(GPU Memory(MBy/Tainin Time(ms/iter)).
Input Length
Timestacker
Sparse
TimeMixer
DLinear
PatchTST
Crossformer
192
28.8/134
15.6/108
518.3/180
13.6/45
145.8/87.8
5214/238
384
29.3/133
16.1/112
875.4/193
16.7/49
334.7/90.1
5734/273
768
34.3/137
20.6/115
1763/202
23.2/99
830.0/93.3
6814/342
1536
59.2/137
28.7/127
3744/282
34.8/108
2404/137
9016/1007
3072
110.7/134
44.1/131
7376/462
59.1/108
7832/1315
12470/3121
Table
D.5. Additional Experiments
To investigate the impact the number of variables on model performance, we selected the Traffic and Electricity datasets, where our model performed less effectively, and conducted comparative experiments with existing models including SOFTS, iTransformer, and TimeMixer. These experiments focused specifically on single-variable time series forecasting tasks.
The results not only revealed the differences in how various models handle single-channel time series but also provided clearer understanding of how the number of variables affects model performance. Under single-channel conditions, our model demonstrated stronger ability to capture the dynamic changes of individual sequences, whereas certain other models appeared to rely on correlations between multiple channels to enhance predictive accuracy. Furthermore, these findings offer valuable insights for optimizing model design, such as exploring strategies to balance feature modeling capabilities between single-channel and multi-channel sequences. This series of experiments helps clarify the applicability of different models across various datasets and variable scales, offering more targeted solutions for time series forecasting tasks.
To comprchensively evaluate the performance of the proposed model, it also compared with the representative multiresolution method, N-HiTS, and the representative frequency-based methods, FEDformer and FiLM, on the ETTm2, Electricity, Traffic, and Weather datasets. The experimental results are shown in Table 9.
Table Additional Comparative Experineets(MSEEMME))
Input Length
Timestacker
N-HiTS
FEDformer
FiLM
ETTm2
0.274/0.316
0.279/0.330
0.305/0.349
0.287/0.329
Electricity 0.194/0.275
0.186/0.287
0.214/0.327
0.223/0.302
Traffic
0.508/0.335
0.452/0.311
0.610/0.376
0.637/0.384
Weather
0.243/0.264
0.249/0.274
0.309/0.360
0.271/0.299
E. SHOW CASE
A visualization of TimeStacker's prediction results was conducted across all datasets. As illustrated in Figure 6, in the 96-to-96 forecasting task, TimeStacker exhibited consistent performance across different datasets, clearly demonstrating its superior predictive capability.
To further highlight the capability of TimeStacker in capturing non-stationary signals, synthetic signals exhibiting nonlinear frequency variations over time were generated and evaluated alongside an MLP as a baseline. As illustrated in Figure can be observed that TimeStacker successfully captures frequency components, though slight phase misalignment is observed, whereas MLP-based models struggle to adapt rapidly changing frequencies.
14

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Scries Forecasting
Table 10. Additional Experiments. All results are based on input sequences of length 96 and are calculated as the average across four different prediction lengths (96. 192, 336, 7203.
Models
TimeStacker
SOFTS
Sparse
TSF
iTransformer
TimeMixer
Metric
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
Traffic
96
0.167
0.235
0.165
0.227
0.281
0.326
0.170
0.232
0.166
0.223
192
0.157
0.223
0.158
0.223
0.236
0.282
0.158
0.22
0.157
0.230
336
0.151
0.223
0.158
0.226
0.223
0.272
0.155
0.216
0.154
0.230
720
0.169
0,242
0.174
0.244
0.242
0.292
0.182
0.254
0.171
0.242
AVG
0.161
0.231
0.164
0.230
0.246
0.293
0.166
0.231
0.162
0.231
Electricity
96
0.296
0,384
0.292
0.381
0.477
0.504
0.299
0.4
0.297
0.391
192
0.304
0.392
0.301
0.389
0.457
0.493
0.299
0.391
0.314
0.401
336
0.364
0.42
0.365
0.418
0.489
0.512
0.362
0.426
0.367
0.431
720
0.425
0.473
0.427
0.473
0.501
0.517
0,426
0.477
0.424
0,46
AVG
0.347
0.417
0.346
0.415
0.481
0.507
0.347
0.424
0.351
0.421
Table 11. Impact of -Bckk Lengths. The backtracking window lengths were set to [96, 192, 336, 720). and the prediction lengths were set to {96, 192, 336, 7203. The evaluation metrics used were MSE and MAE. The best performance of the model is highlighted red.
Dauset
Exclunge
ETTai
ETT12
Larghs
Memoo
MSB:
MAI!
MSH
MAR:
MSS
MAR
MSE:
MAR
MSB
MAU
MSII
MARI
MSS
MAE
MSE
MAI
MSE.
MAR
MSII
MARI
MSE
MAR
MSE:
MAI
96
0.082
0.199
0.090
0,210
0.089
0.211
0.120
0.251
0.375
0.400
0.378
0,359
0,570
0.395
0.366
0.391
0.259
0.341
0.286
0,338
0.276
0.334
0,339
0.587
192
0.137
0.297
0.181
0,173
0.299
0.429
0.414
DA15
0,411
0.424
0.367
0.368
0,347
0.382
TineMixer
336
0.324
0.408
0.329
0,488
0.355
0.420
0444
0.4S8
0.422
0.427
0.454
0.440
0.366
0.413
0.365
0.401
0.374
720
0.837
0.691
0.817
0,679
0.887
0 6,000
0.904
0.741
0-458
0.4R2
465
0.462
0,447
0.455
0.490
0.472
0.412
0.434
0 441
0.436
0.405
0.434
0,410
0.443
AVG
0.355
0.399
0.354
0.403
0,369
0.407
0,383
0.425
0.447
0.440
0.434
0.434
0.416
0:425
0,415
0.432
0.365
0.795
0.365
0.394
0,345
0.318
0,378
0.412
0.201
0.01C
0.233
0.361
0,5A0
11,06
0.3M3
0.2M3
U.354
0.312
192
0.172
0.294
0.181
0,305
0. TMO
0.315
0.1K3
0.309
0.455
0.431
0U431
0.432
0.425
11.33
0.429
01438
0.313
0.394
0 0
00407
0.343
0.007
0.366
01415
SOFTS
336
0.324
0.412
0.314
0,413
0.339
0,377
04454
0.400
0.452
01451
0.442
0.439
11:43
0.457
0.558
0.410
0.426
9.399
0.345
0.466
0,404
0.424
03811
0.622
OL703
0.463
0.435
U.544
0,429
AVG
0.348
0.305
0.329
0,377
0.424
0.427
11.37
0.436
0.374
0.006
0.363
01402
96
0.086
0.206
0.000
0.211
0.095
0 220
0,000
0.233
0.356
0.405
387
0.402
0,394
0.407
0.300
0.413
0.207
0.340
0 2.2
0.347
0.296
0.352
0.300
0.500
192
0.177
0.299
0.193
0.316
0.192
0.309
0.193
0.31O
0:441
0.436
4.445
0.429
0.444
D:435
0.418
0L431
0.350
0.400
0. 3.3
0.399
0.389
0.405
0.300
0.415
(Trandomeer
396
0.331
0.345
0.456
0.445
720
0.847
0.691
0.757
0.657
0.594
0.645
0.706
0.681
0.491
6.464
0.467
0.463
0.499
0.490
0-422
0.441
0.426
0447
0.490
0.465
AVG
0.360
0.403
0.546
0.404
8,329
D -00
0. 360
01404
0-454
0.488
0.439
0.435
0.458
0:437
0.485
0L446
0 0.353
0.417
0 305
0.401
0.377
11:005
0.768
(1420
96
0.088
0.205
0.093
0.122
0.254
0.385
11,05
0.389
0.255
0.283
0.345
0.279
0.346
192
0.176
0.299
0.186
0.307
0.229
0.339
0.48
0.419
11424
0.425
0.436
0.401
0.36
0.397
0.35
0.39
5AM0omms
336
0.322
0.44
0.333
0.42
0,347
0.43
0.385
0.452
0.475
0.445
0.461
0,442
0.451
11.46
0.456
0L4SS
0.419
0.428
0. 5.355
0.421
0.364
0.206
0.365
0.365
0.409
1,009
720
0.799
0.673
0.904
0.678
0.556
0.701
0.957
0.743
0.478
0.463
0.464
0.459
0.454
IL:68
0.467
0L482
0.421
0.44
0.415
0,441
0.405
0.436
0.4
0.439
AVG
0.346
0.399
0.354
0.423
0L447
0.435
0.434
0L446
0.389
0.353
0.397
0.398
96
0.84
0.200
0.084
0.201
0.085
0.304
0.065
0.305
0.189
0.
0.164
0.312
0:37
0.91%
0.367
0.931
197
0.131
0.701
0.175
0.205
0.174
112044
0.171
0.201
0-479
0:416
0.414
0.401
11400
0L418
0:335
0.185
01 165
0.
0.56K
0.344
0.3%
01 0.524
TimeStacker
196
0.314
0.408
0.117
0.411
0.433
0.321
C1408
0-499
0.445
0.479
0.424
0.43
CL47A
0-407
0.416
907
0-413
0.363
0.947
0.354
1.996
AvG
0.336
0.109
0,333
013887
0.543
1L292
0
0L391
BAN
0.421
EL422
01423
0.006
ILA17
0.403
0L419
0.308
0,198
0.366
13
0,345
IL3NS
0,387
IL382
15

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Scries Forecasting
Table 12. Complete Experimental Result. All results are based on input sequences of length 96 and are calculated the average across four different prediction lengths (96, 192, 336, 7203.
TimeStacker
SOFTS
SpaneTTFF
irunsfommer
TimeMiser
SAMformar
PatchTST
Crosslommer
DLinkar
RLincar
Models
(ours)
(2024)
(2024)
(2024)
(2024)
(2024)
(2023)
(2023)
(2023)
(2023)
Metric
MSE
MAE
MSE
MAB
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
MSE
MAE
96
0,379
0.385
0.381
0.399
0.388
0.387
0.386
0.405
0.375
0,400
0.383
0.392
0:414
0:419
0.423
0.448
0.386
0,400
0.386
0.395
192
0.429
0.416
0.431
0.438
0.417
0.441
0:436
0.429
0.421
0.438
0.423
0.160
0.445
0.471
0.474
0.437
0.432
336
0.459
0.436
0.400
0.452
0.460
0.438
0.487
0.458
0.484
0.458
0,475
0.445
0.501
0,466
0.570
0.546
0.481
0.459
0.799
0.446
720
0.464
0.455
0.499
0.488
0.468
0.457
0.503
0.491
0.498
0.482
0.478
0.468
0.500
0.488
0.653
0.621
0.519
0.516
0.481
0.470
AVG
01433
01,433
0.449
0.442
0:441
0.425
0.454
0.447
0.447
0,440
0.444
0.432
0.469
0.454
0.529
0.522
0:456
0,452
0.446
0.44
96
0.280
0.327
0.297
0.347
0.304
0.346
0.297
0.349
0,289
0.341
0.289
0.338
0.302
0.348
0.745
0.584
0.333
0.387
0.288
0.338
ETTh2
192
0.373
0.385
0.373
0.394
0.409
0.403
0.380
0.400
0.372
0.392
0.401
0.398
0.388
0.400
0.877
0.656
0.477
0.476
0.374
0.390
336
0.402
0.416
0.410
0.426
0.426
0.430
0.428
0.432
0.386
0.414
0.419
0.428
0.426
0,433
1 1.043
0.731
0.594
0.541
0.415
0.426
720
0:112
0.431
0.411
0.433
0:121
0.138
0.427
0.445
0.112
0.431
0.121
0.110
0.131
0.416
1.104
0.763
0.831
0.657
0.420
0.410
AVG
0,368
0.390
0.373
0. 0.400
03442
0,433
0,383
0.407
0.364
0.395
0.383
0.401
0.387
0:407
0.942
0.684
0.559
0.515
0.574
0.398
96
0.311
0.337
0.325
0. 0.361
0. 366
0,369
0,314
0.368
0.320
0.357
0,352
0.374
0.329
0.367
0,401
0.426
0.345
0.372
0.355
0.376
BTTm]
192
0.361
0.367
0.375
0.389
0/101
0,387
0.377
0.391
0,361
0.381
0.392
0.392
0.367
0.385
0,450
0.451
0.380
336
0.389
0.391
0.405
0.412
0.432
0.406
0.426
0.420
0.390
0.404
0.425
0.413
0.399
0,410
0.532
0.515
0,413
0,413
0.424
0.415
720
0.460
01.428
0.466
0.447
0344%
0.442
0.491
0.459
0.454
0.441
0.49
0.449
01.454
0.430
0 0.666
0.589
0:474
0.453
0.487
0.450
AVG
00381
0,381
0.393
0.403
0.425
0.401
0.407
0.410
0
0.395
0.415
0.407
0.387
0.400
0.513
0.406
0.403
0.407
0.414
0.407
96
0.171
0.250
0.180
0.261
0.198
0.272
0.180
0.264
0.175
0.258
0.181
0.264
0.175
0.259
0.287
0.366
0,193
0.292
0.182
0.265
ETTm2
192
0.236
0.292
0.346
0.
300
03255
0
0,250
0.309
0.232
0.299
0.245
0.305
0.241
0:302
0.414
0.402
0.284
0,362
0.246
0.304
336
0.293
0.329
0.319
0.352
0.315
0.343
0,311
0.348
0.208
0.340
0.305
0.341
0.305
0.343
0.597
0.542
0.369
0.427
0.307
0.342
720
0.395
0.391
0,405
0.401
0,416
0.399
0.412
0.407
0.391
0.396
0.409
0.308
0.402
04100
1.750
1.042
0.554
0.522
0.407
0.398
AVG
0.274
0,316
0.287
0.330
0.797
0.391
0.258
0.332
0:275
0.323
0.285
0.327
0.281
0.326
0.757
0.610
0.350
0.401
012866
0.327
96
0.496
0.351
0.376
0.559
0.335
0.395
0.268
0.462
0.285
0.552
0.367
0.462
0.295
0.522
0.290
0.650
0.396
0.649
0.389
Traflic
192
0.491
0,331
0.261
0.567
0.346
0.417
0.276
0.473
0.296
0.569
0.368
0.466
0.296
0.530
0.293
0.598
0.370
0.366
336
0.505
0.334
0.415
0.269
0.575
0.349
0.433
0.283
0.498
0.296
0.586
0.376
0.482
0.304
0.558
0.305
0.605
0.373
0.609
0.369
720
0.541
0.343
0.447
0.287
0.609
0,368
0.467
0.300
0,506
0,313
0.63
0.401
0.514
0.322
0.589
0.328
0.645
0.394
0.647
0.387
AVG
0,508
0.335
0.409
0.267
0.578
0,350
0.488
0,282
0.484
0.297
0.595
0.382
0.481
0.304
0.550
0.304
0.625
0.383
0.626
0.378
96
0,168
0.251
0.143
0,233
0,202
0.261
0.148
0,240
0.153
0.247
0.199
0.277
0.181
0,270
0.219
0.314
0.197
0.282
0.201
0.281
Electricity
192
0.176
0.262
0.158
0.248
0.207
0.277
0.162
0.243
0.166
0.256
0.199
0.279
0.188
0.284
0.231
0.322
0.196
0.285
0.201
0.283
336
0.195
0.278
0.178
0.269
0.219
0.292
0.178
0.269
0.185
0.277
0.214
0.294
0,204
0.293
0.246
0.337
0,209
0.301
0.215
0.298
730
0:235
0.310
0.218
0.305
0.761
0.324
0.225
0.317
0.225
0.310
0.257
0.328
0.746
0.374
0.2800
0.363
0.245
0.333
0.257
0.331
0.191
0.275
0.222
0.289
0.178
0.270
0.182
0.272
0.217
0.295
0.205
0.290
0.214
0.334
0.212
0.300
96
0.161
0.198
0.208
0,213
0,250
0.174
0.214
0.163
0.209
0.193
0.205
0.177
0.218
0.158
0.230
0.196
0.255
0.232
Weather
192
0.207
0.241
0.217
0.253
0.259
0.282
0.221
0.254
0.208
0.250
0.242
0.274
0.225
0.259
0.206
0.277
0.237
0.296
0.340
0.271
336
0.261
0,281
0.282
0,300
0.308
0.315
0.278
0.296
0.251
0.287
0.281
0.309
0.278
0.297
0.272
0.335
0,283
0.335
0.292
0.307
720
0.343
0.334
0.356
0.351
0,380
0.361
0.358
0.347
0.339
0.341
0.358
0.351
0.354
0.348
0.398
0.418
0.345
0.381
0.364
0.353
AVG
0.243
0.264
0.255
0.278
0.290
0.302
0.258
0.278
0.240
0.271
0.264
0.285
0.259
0,348
0.259
0.315
0,265
0,317
0.272
0.291
96
0.081
0.200
0.081
0.201
0.093
0.217
0.086
0.206
0.082
0.199
0.088
0.209
0.088
0,205
0.256
0.367
0.088
0,218
0.093
0.217
Exchange
192
0.171
0.293
0.102
0.294
0.179
0.304
0.177
0.299
0.177
0.297
0.176
0.299
0.176
0.299
0.470
0.509
0.176
0.315
0.184
0.307
336
0.314
0.406
0.324
0.412
0.319
0,410
0.351
0.417
0.324
0.408
0.322
0.414
0.301
0.397
1,268
0.
0.313
0.427
0.351
0.432
720
0.776
0.656
0.811
0.672
0.823
0.683
0.847
0.691
0.837
0.691
0.799
0.673
0.901
0.714
1.767
1.068
0.839
0.695
0.886
0.714
0.336
0.389
0.348
0.395
0,365
0.401
0.360
0.403
0.355
0.399
0.346
0.399
0.367
0,404
0.940
0.707
0.354
0.414
1st Count
15
0
2.
16

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Scries Forecasting
-22
-30
(a) ETThl
(b) ETTh2
(c) ETTml
Peukiden
23-
(d) ETTm2
(c) Exchange
(f) Electricity
0.8
0.0
Traffic
(h) Weather
Figure 6. Visualization of forecasting results on the real-world dataset with look- back window length L = 96 and prediction length
17

---

TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Scries Forecasting
-15
(a) TimkSteker
(b) MLP
Figure 7. Visualization of results synthetic data. Non-stationary signals with time-varying frequencies were used for training and testing. (a) presents visualization produced by TimeStacker, while (b) presents the visualization produced by the ML.P.
18


Reviews:
[{"review_id": "0PZOhxqECi", "reviewer": "Reviewer_Z5kW", "summary": "The paper introduces TimeStacker, a novel time series forecasting framework designed to handle nonstationary signals effectively. The proposed approach utilizes a multi-level stacking mechanism, aggregating patches of varying sizes to capture both local and global frequency-domain features. Additionally, a frequency-based self-attention module (FreqAttention) is introduced, which computes similarity in the frequency domain while aggregating in the time domain. The authors claim that TimeStacker achieves state-of-the-art performance across several real-world datasets, outperforming recent Transformer- and MLP-based forecasting models with fewer parameters and better computational efficiency.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your appreciation and constructive comments.\n\n**Q1:**\n\nThe synthetic data is constructed by randomly selecting 30 frequencies, with the amplitude corresponding to each frequency varying over time. The experimental results are visualized in this anonymous URL (https://i.postimg.cc/pT66jQHH/synthetic-exp.png). More detailed information will be provided in the appendix.\n\n\n**Q2:**\n To demonstrate how TimeStacker adapts to various non-stationary signals, we configured the parameter *Patch Size List* and conducted experiments on the ETTm1 dataset. The results are shown in the following:\n\n| Patch Size List | [16,16,16,16] | [16,16,16,24] | [16,16,16,32] | [16,16,16,48] | [16,16,24,32] | [16,16,24,48] | [16,24,32,48] |\n| --------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n| MSE             | 0.465         | 0.468         | 0.468         | 0.465         | 0.463         | 0.463         | 0.460         |\n| MAE             | 0.433         | 0.439         | 0.436         | 0.431         | 0.431         | 0.430         | 0.428         |\n\nThese results indicate that employing various window combinations can more effectively capture the underlying dynamic patterns of the sequence, thereby improving prediction performance.\n\n\n\nThe mean and standard deviation of the results from multiple runs(5 different seed) of experiments are provided as following (mean/std):\n\n| Dataset | ETTh1         | ETTh2         | ETTm1         | ETTm2         | Traffic       | Electricity   | Weather       | Exchange      |\n| ------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n| MSE     | 0.433/0.00145 | 0.368/0.00091 | 0.381/0.00119 | 0.274/0.00061 | 0.508/0.00052 | 0.194/0.00056 | 0.243/0.00092 | 0.336/0.00101 |\n| MAE     | 0.423/0.00167 | 0.390/0.00057 | 0.381/0.00052 | 0.316/0.00042 | 0.335/0.00087 | 0.275/0.00077 | 0.264/0.00042 | 0.389/0.00137 |\n\nOnce again, we appreciate your thorough review of our manuscript. The typo in “theSmooth Layer” has been corrected.", "claims_and_evidence": "The author claims that their method achieves state-of-the-art performance across multiple real-world datasets. This is mostly validated by the empirical experiments on 8 datasets, though the proposed method doesn't achieve the best on traffic and electricity datasets.\n\nThe author also claims that the proposed method has fewer parameters and higher computational efficiency, this is supported by comparison in training time and memory footprint. The comparison shows that the proposed method is somewhat efficient but is not the most efficient one.", "methods_and_evaluation_criteria": "The TimeStacker framework is designed to handle nonstationary patterns in time series data. The core components align well with this goal:\n- Multi-level patch stacking captures both local and global features, addressing the issue of fluctuating frequency characteristics in nonstationary time series.\n- FreqAttention (frequency-based self-attention) leverages frequency-domain similarity rather than time-domain patterns, which is a reasonable approach for handling signals with evolving spectral properties.\n\n\nThe evaluation is on 8 datasets and use MSE and MAE as the metrics which is common in time series analysis. However, the test data is not specifically for non-stationary data. Some tests with synthetic data are encouraged.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiment design and analysis are solid. The proposed method is compared with 9 state-of-the-art methods on 8 different datasets. The authors have conducted extensive ablation studies. Experiments of efficiency and look-back lengths let the readers understand the performance of TimeStacker better.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "In line 165, \"theSmooth Layer\" misses a space", "questions_for_authors": "1. Can you add synthetic experiments on non-stationary data?\n2. Can you provide an ablation study showing how TimeStacker adapts to different types of nonstationary signals compared to other models?\nCan you provide mean, standard deviation for MSE and MAE across multiple runs?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x0cbxIsLPc", "reviewer": "Reviewer_FZmL", "summary": "This paper is another incremental work in developing Transformer-based time-series architectures and follow some widely used yet problematic benchmarks, such as ETT, Exchange, Weather, etc.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to comment on our work. We would like to clarify several points and explain the motivations behind methodology and evaluation protocol.\n\n**Q1:** *“incremental” Transformer-based time-series model.*\n\n**R1:** We acknowledge that Transformer-based approaches have become pervasive in time-series research. However, the TimeStacker model goes beyond a straightforward incremental tweak by **introducing a stacking mechanism and a frequency-based self-attention module**. These contributions distinguish TimeStacker from earlier Transformer-based architectures, as they specifically address the challenges of non-stationarity modeling—issues that are well documented in both academic and applied settings.\n\n**Q2:** *Every meteorologist ... dice.*\n\n**R2:** Why can meteorologists infer that global warming is accelerating[1] based on historical weather data? The difficulty in precisely characterizing a phenomenon does not imply that it is uncharacterizable. The goal of machine learning is to learn how to make reasonable inferences from historical data, much like human experts, thereby alleviating tedious tasks (learning from human beings and learning for human beings). Consequently, our aim is not to predict the weather exactly two weeks in advance, but rather to enable the model to make sound inferences from historical data—assisting laypersons in data analysis and decision making while allowing experts to focus on deeper theoretical research.\n\n**Q3:** *“widely used yet problematic” datasets (ETT, Exchange, Weather, etc.) ... benchmarks.*\n\n**R3:** We respectfully disagree with the notion that employing these long-standing community benchmarks—which have been cited and scrutinized by thousands of published works—invalidates our research or the broader domain of Transformer-based time-series forecasting. Although no benchmark is perfect, these datasets encompass varied characteristics and provide common reference points that promote cumulative progress in the field. Reproducible research benefits from established baselines, and abandoning them would break continuity with a large body of existing work.\n\nWe acknowledge the insights presented in the “NIPS 2024 Time Series Workshop” talk and value any new perspectives it may offer. However, a single presentation—especially one from a specialized forum—cannot unilaterally dismiss the robust, peer-reviewed benchmarks that have been used by the global research community for many years. While we recognize that these datasets have limitations (as do all benchmarks), they continue to offer a practical and widely accepted foundation for performance comparison. Given that many top-tier conference papers utilize these benchmarks, it is essential for any new approach, including ours, to demonstrate its merits on them. Our objective is to **explore the historical evolution of sequences in order to capture their underlying dynamic patterns**, rather than merely performing forecasting.\n\n**Q4:** *performs well on Exchange and Weather but “falls short” on Traffic and Electricity ... predictable.*\n\n**R4:** We respectfully note that our experiments do not indicate that TimeStacker “falls short” on these datasets; our reported results are generally comparable to or better than many baselines. Moreover, TimeStacker is specifically designed for modeling non-stationary signals rather than for multivariate data. As demonstrated in our experiments in **Appendix D.3**, when we reduced the number of variables in the Traffic and Electricity datasets, our approach achieved superior performance under the same conditions. Our goal in including Traffic and Electricity is to offer comprehensive comparisons on standard community benchmarks, thereby ensuring that readers gain a complete understanding of TimeStacker’s performance across different data regimes. TimeStacker has shown its strengths in more volatile domains where underlying patterns are less predictable. This aligns with our primary focus: robustly handling non-stationarity rather than excelling on any single type of time-series problem.\n\n**Q5:** *Restricting input length ... ones.*\n\n**R5:** The choice of input window length generally follows standard practices in the literature. Moreover, the window length should be viewed as a reflection of the model’s ability rather than a mere hyperparameter. In our experiments, we have evaluated multiple window lengths (in **Section 4.3, Model Analysis**) and report results using window sizes commonly adopted in prior works to ensure comparability.\n\n**Q6:** *Regarding the use of Revin for regularization.*\n\n**R6:** As indicated by Equations (1) and (2), this operation simply standardizes the model input to a common scale without altering the stationarity of the signal. In non-stationary signal research, this is considered fundamental.\n\n[1] Xu, Yangyang, et al. “Global Warming Will Happen Faster than We Think.” *Nature*, Dec. 2018.", "claims_and_evidence": "This paper claims its proposed approach may better tackle non-stationary signals in time-series forecasting.", "methods_and_evaluation_criteria": "The benchmark datasets and the whole research stream of many compared baselines have some problems.\n\nI suggest the authors to watch the talk in NeurIPS 2024 Time Series Workshop, https://cbergmeir.com/talks/neurips2024/, and adjust your evaluation benchmarks.\n- Actually, the exchange dataset is not a proper testbed to compare deep forecasting models. A naive baseline will excel, a lot. Why your deep learning model can win in this case but fall short on Traffic and Electricity, where more predictable patterns exist. \n- Your model also excels on Weather dataset, however, every meteorologist will tell you that everything further than 2 weeks into the future is essentially rolling a dice. Forecasting 720 points means 720 / 24 = 30 days out.\n- The input window length is a hyperparameter. Restricting to small input lengths (such as 64) is favouring more complex models over simpler ones.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "My major concerns are about the invalidity of evaluation benchmarks and compared baselines.", "relation_to_broader_scientific_literature": "Limited relation.", "essential_references_not_discussed": "If this paper cares about non-stationarity, maybe some existing normalization methods should be mentioned, discussed, and compared if possible.\n\nFor example, can your proposed architectures handle non-stationarity without using RevIN [1]? If not, why the proposed modules tackle the non-stationarity challenge and how to demonstrate that?\n\n[1] Kim, T., Kim, J., Tae, Y., Park, C., Choi, J.-H., and Choo, J. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please share your thoughts or comments after watching the talk or slides in https://cbergmeir.com/talks/neurips2024/.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PVKxAShpwT", "reviewer": "Reviewer_fBGw", "summary": "The paper \"TimeStacker: A Novel Framework with Multilevel Observation for Capturing Nonstationary Patterns in Time Series Forecasting\" introduces TimeStacker, a forecasting framework that addresses the challenges of nonstationary time series by integrating multi-resolution stacking and frequency-based self-attention. By sequentially aggregating patches of varying sizes, TimeStacker captures both global trends and local variations, while its frequency-based attention module enhances feature extraction by computing similarity in the frequency domain. Grounded in time-frequency analysis and the uncertainty principle, the model outperforms state-of-the-art forecasting methods across diverse real-world datasets, achieving superior accuracy with lower computational complexity.\n\n## After rebuttal \n\n I will maintain my score", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your recognition of our work and your constructive feedback.\n\n\n\n**Q1&W1：**\n\nOur approach fundamentally differs from multi-resolution and Fourier‑based methods. While the latter emphasize extracting static features from various frequency bands—that is, observing the signal at different granularities—TimeStacker focuses on capturing the dynamic evolution within the input signal to reveal its underlying transformation patterns. For this reason, we refer to our method as **“multi‑level”** rather than **“multi‑resolution”**.\n\nIn TimeStacker, a sequence\n\n$$\nX = \\{x_1, x_2, x_3, ..., x_t\\}\n$$\n\n is first partitioned into K segments (ensuring that t is divisible by K), resulting in a new sequence\n\n$$\n\\hat{X}_k= \\{X_1, X_2, ..., X_{t/k}\\},X_i = \\{x_{(i-1)\\times l + 1}, x_{(i-1)\\times 1 + 2}, ..., x_{i\\times l}\\}, l=t/K\n$$\n\nNext, we compute the similarity between these subsequences in the **frequency domain**—essentially using a window of size K with stride K to observe the internal evolution of the sequence. Based on this similarity, we aggregate the subsequences in the **time domain** to produce an output sequence of length t. By continuously varying the window shape (i.e., reducing the observation window K) and repeating the process across multiple levels, we break the constraints imposed by Theorem 3.2 to capture the variation patterns of the Fourier coefficients *a* and *b* as defined in Definition 3.1.\n\n\n\nIn contrast, multi-resolution methods such as N‑HiTS primarily extract signal features via downsampling. For example, given a sequence\n\n$$\nX = \\{X_1, X_2, X_3, ..., X_t\\},\n$$\n\n downsampling with a stride of K produces a new sequence\n  \n\n$$\n\\hat{X}_k = \\{x_1, x_{2k}, x_{3k}, ..., x_t\\},\n$$\n\n\n followed by interpolation for prediction. The process is then repeated with smaller strides to capture information at different granularities along the entire sequence.\n\n\n\n**Q2&W2&W3：**\n\nAt a detailed level, TimeStacker leverages frequency‑domain information to observe the dynamic evolution of a sequence. Its process can be summarized as follows:\n\n> Sequence → Transformation (Frequency Domain) → Compute variation patterns (Fourier coefficients *a* and *b*) among subsequences → Aggregate in the Time Domain based on these patterns → Sequence → Prediction\n\nThis approach also helps mitigate errors that can arise from the inverse transformation using discrete orthogonal bases.\n\nIn contrast, other Fourier‑based architectures (e.g., FEDformer, FiLM) project the time‑domain sequence onto a Fourier (or other orthogonal) basis, enhance the features in the frequency domain, and then apply an inverse transformation to return to the time domain. Their process can be abstractly described as:\n\n> Sequence → Transformation (Frequency Domain) → Feature Enhancement → Inverse Transformation (Time Domain) → Sequence → Prediction\n\n\n\nBelow is a preliminary comparison between TimeStacker and baseline models (N‑HiTS, FEDformer, and FiLM). The first value in each cell represents MSE and the second represents MAE. A more comprehensive comparison will be added to the main text and appendix.\n\n| Dataset     | TimeStacker   | N‑HiTS        | FEDformer     | FiLM          |\n| ----------- | ------------- | ------------- | ------------- | ------------- |\n| ETTm2       | 0.274 / 0.316 | 0.279 / 0.330 | 0.305 / 0.349 | 0.287 / 0.329 |\n| Electricity | 0.194 / 0.275 | 0.186 / 0.287 | 0.214 / 0.327 | 0.223 / 0.302 |\n| Traffic     | 0.508 / 0.335 | 0.452 / 0.311 | 0.610 / 0.376 | 0.637 / 0.384 |\n| Weather     | 0.243 / 0.264 | 0.249 / 0.274 | 0.309 / 0.360 | 0.271 / 0.291 |\n\n\n\n**Q3：**\n\nWe define the output of our smoothing layer as ***SmoothLayer(x) + x***, as presented in Equation (13) of our paper, incorporating a residual connection. This mechanism ensures that even if *SmoothLayer(·)* excessively smooths the signal—thereby potentially losing local features—the residual branch can effectively compensate by reintroducing these features. Consequently, the approach minimizes noise interference while preserving key information, thus enhancing the model’s expressive power.\n\n\n\n**W3：**\n\nWe appreciate your suggestions. We will add the relevant essential references to the main text and plan to release the code publicly in the near future. Additionally, we will include a complete comparison with N‑HiTS, FEDformer, and FiLM, and, based on the feedback from all reviewers, we will augment our experimental data to more comprehensively demonstrate the advantages of our approach and further enrich the paper.", "claims_and_evidence": "The paper asserts the necessity of analyzing time series in the frequency domain and adopting a multi-resolution perspective, claims that are well-supported by **Definition 3.1 and Theorem 3.2**, which theoretically establish the importance of capturing time-frequency variations in nonstationary signals. Furthermore, the experimental results empirically validate these claims by demonstrating TimeStacker’s superior performance across multiple real-world datasets. However, while the proposed approach effectively addresses the stated challenges, it bears resemblance to existing methodologies that employ similar multi-resolution strategies, which somewhat limits its novelty.", "methods_and_evaluation_criteria": "The proposed method adopts a multi-resolution approach to analyzing time series in the frequency domain, which is a well-motivated strategy for handling nonstationary signals. However, to strengthen the contribution, a clearer distinction between TimeStacker and existing models such as N-HiTS and TimeMixer—which also leverage multi-resolution techniques—would be beneficial. Regarding the evaluation criteria, the paper employs widely accepted metrics (e.g., MSE, MAE) and benchmark datasets commonly used in time-series forecasting research, ensuring a fair and standardized comparison against existing methods.", "theoretical_claims": "I have examined **Definition 3.1 and Theorem 3.2**, both of which serve as the theoretical foundation for TimeStacker’s motivation. **Definition 3.1** effectively formulates the need for frequency-domain analysis by representing nonstationary time series as time-varying Fourier components, while **Theorem 3.2**, derived from the time-frequency uncertainty principle, justifies the necessity of a multi-resolution approach. These theoretical claims are mathematically sound and align with well-established principles in signal processing. Additionally, they provide a clear rationale for the model’s design choices. I did not identify any fundamental issues with these proofs, but a deeper comparison with alternative formulations could further reinforce their validity.", "experimental_designs_or_analyses": "The experimental design presented in the paper is generally well-structured and valid, employing widely recognized benchmarks and evaluation metrics commonly used in time-series forecasting. The results effectively demonstrate the advantages of the proposed method. However, to further strengthen the empirical analysis, it would be beneficial to include an ablation study on the multi-resolution stacking steps, providing insights into how each resolution level contributes to the final performance. Additionally, a direct performance comparison with N-HiTS, which also utilizes a hierarchical decomposition strategy, would help clarify TimeStacker’s relative advantages and better position it within the landscape of multi-resolution forecasting models.", "relation_to_broader_scientific_literature": "The paper introduces a novel perspective on time-series forecasting by emphasizing the importance of frequency-domain analysis and multi-resolution decomposition, challenging the traditional focus on purely temporal correlations. This approach aligns with prior research on multi-scale modeling (e.g., N-HiTS, TimeMixer) but distinguishes itself by explicitly leveraging frequency-based self-attention and progressive stacking to capture both global trends and local variations. By grounding its methodology in time-frequency analysis and the uncertainty principle, the paper contributes to a broader shift in the field, encouraging researchers to rethink conventional time-series modeling paradigms. While the proposed framework builds on existing ideas, it provides a cohesive and theoretically justified approach, which could inspire further advancements in handling nonstationary time-series data.", "essential_references_not_discussed": "Challu, Cristian et al. “N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting.” AAAI 2023", "other_strengths_and_weaknesses": "**Strengths:**  \n1. The paper is well-structured and intuitive, making it easy to follow. The proposed multi-resolution stacking and frequency-based self-attention mechanisms are clearly explained, allowing readers to grasp the motivation behind TimeStacker without excessive complexity.  \n2. The mathematical foundations provided through Definition 3.1 and Theorem 3.2 effectively support the proposed approach, strengthening its conceptual motivation and distinguishing it from purely empirical contributions.  \n\n **Weaknesses:**  \n1. While the paper presents a well-motivated framework, its core ideas (multi-resolution analysis and frequency-domain modeling) are not fundamentally new, as similar approaches have been explored in models like **N-HiTS** and Fourier-based architectures (e.g., FEDformer, FiLM). A clearer articulation of how TimeStacker differs from or improves upon these methods would enhance its originality.  \n2. The paper does not adequately differentiate itself from prior work that employs hierarchical decomposition and frequency modeling. Explicit comparisons—both in theoretical discussion and experimental evaluation—would strengthen the argument for its contribution.  \n3. Additional points regarding experimental design, missing citations, and potential improvements have been detailed in responses to individual review questions.  \n\nOverall, while the paper is well-written and theoretically grounded, addressing its novelty concerns and improving its positioning relative to prior multi-resolution models would make the contribution more compelling.", "other_comments_or_suggestions": "Some figures appear blurry and lack clarity, making it difficult to interpret fine details. Improving the resolution and contrast of the figures would enhance readability.", "questions_for_authors": "1. How does TimeStacker differentiate itself from other multi-resolution approaches in time-series forecasting? \n   - Several existing methods, such as N-HiTS and other hierarchical decomposition models, already leverage multi-resolution processing. Could you clearly articulate the key differences between TimeStacker and these models in terms of both methodology and performance?  \n   - A more explicit comparison could help clarify the novelty and contribution of the proposed approach.  \n\n2. In what scenarios does TimeStacker outperform existing frequency-domain-based models?\n   - The paper emphasizes the advantages of analyzing time-series data in the frequency domain, but how does TimeStacker compare against prior frequency-aware models such as FEDformer, FiLM, or other Fourier-based architectures?  \n   - Are there specific types of datasets, forecasting horizons, or signal characteristics where TimeStacker’s design proves particularly effective? Including such insights would strengthen the empirical justification for the proposed method.  \n\n3. Does applying smoothness in the inter-patch frequency-based attention module conflict with the goal of capturing local features?  \n   - The smooth layer is stated to reduce noise within patches, yet patches themselves are meant to capture localized variations in the time series.  \n   - Would excessive smoothing risk removing important short-term patterns or distort fine-grained structures?  \n   - Could you provide any empirical justification or ablation studies showing how this trade-off impacts performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4eiiLIoDWE", "reviewer": "Reviewer_PxiA", "summary": "The paper introduces TimeStacker, a new framework designed to enhance time series forecasting by effectively capturing nonstationary patterns. The core innovation lies in its stacking mechanism, which sequentially aggregates patches of varying sizes to balance global and local signal representations. Additionally, the framework employs a frequency-based self-attention module that improves feature modeling by computing similarity in the frequency domain while aggregating in the time domain. Experimental results across multiple real-world datasets (energy, finance, weather) demonstrate that TimeStacker achieves state-of-the-art performance, surpassing existing models in both predictive accuracy and computational efficiency while using fewer parameters.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your appreciation and  for the valuable suggestions.\n\n**Cross-Dataset Performance:**\n\nFor the ETT series data, we present the average performance across datasets (the first row indicates MSE and the second row indicates MAE) as shown below:\n\n| TimeStaker | SOFTS | SparseTSF | iTransformer | TimeMixer | SAMformer | PatchTST | Crossformer | DLinear | RLinear |\n| ---------- | ----- | --------- | ------------ | --------- | --------- | -------- | ----------- | ------- | ------- |\n| 0.364      | 0.376 | 0.396     | 0.383        | 0.367     | 0.382     | 0.381    | 0.685       | 0.442   | 0.380   |\n| 0.378      | 0.394 | 0.398     | 0.399        | 0.388     | 0.392     | 0.397    | 0.578       | 0.444   | 0.392   |\n\n\n\n**Complexity Analysis:**\n\nWe have conducted an in-depth analysis on how increasing the input sequence length affects memory consumption and training time, with the results averaged over three runs. Partial results are shown in the table below (GPU Memory(MB)/Training Time(ms/iter):\n\n| Input Length | TimeStaker | SparseTSF | PatchTST   | Crossformer |\n| ------------ | ---------- | --------- | ---------- | ----------- |\n| 192          | 28.8/134   | 15.6/108  | 145.8/87.8 | 5214/238    |\n| 384          | 29.3/133   | 16.1/112  | 334.7/90.1 | 5734/273    |\n| 768          | 34.3/137   | 20.6/115  | 830.0/93.3 | 6814/342    |\n| 1536         | 59.2/137   | 28.7/127  | 2403.6/137 | 9016/1007   |\n| 3072         | 110.7/134  | 44.1/131  | 7832/1315  | 12470/3121  |\n\nWe will provide clearer visualizations and more comprehensive data in the appendix.\n\n\n\n**Regarding Multivariate Data:**\n\nOur work primarily focuses on time series modeling and does not include specialized designs for multivariate data. Integrating information from multiple variables is a complex task that we plan to address in our future work.\n\nOnce again, thank you for your valuable feedback.", "claims_and_evidence": "The claims made in the submission are largely supported, through experiments on multiple datasets. The results consistently demonstrate that TimeStacker outperforms state-of-the-art models in predictive accuracy while maintaining computational efficiency. The inclusion of ablation studies further strengthens the validity of the proposed frequency-based self-attention mechanism. But, one potential limitation is the claim that TimeStacker effectively handles multivariate time series. While the model performs well on most datasets, the authors acknowledge a decline in performance as the number of variables increases, suggesting a possible bottleneck. Additionally, while the theoretical justification of the stacking mechanism is well-founded, more in-depth comparisons with alternative approaches for handling nonstationary signals could strengthen the argument. Overall, the empirical results are compelling, but further validation on larger and more complex datasets would reinforce the generalizability of the proposed method.", "methods_and_evaluation_criteria": "The methods and evaluation criteria align well with the problem of time series forecasting. TimeStacker introduces a novel stacking mechanism and frequency-based self-attention, both of which are well-motivated by the challenges of nonstationary signals. The choice of benchmark datasets ensures a comprehensive evaluation across diverse real-world applications. The use of Mean Squared Error (MSE) and Mean Absolute Error (MAE) as performance metrics is standard in time series forecasting and appropriately assesses both the accuracy and robustness of predictions. Ablation studies and comparisons with recent state-of-the-art models provide further validation. While the evaluation framework is well-structured, the model's performance on highly multivariate datasets could have been explored further to assess scalability.", "theoretical_claims": "The authors reference the time-frequency uncertainty principle to justify their multi-scale stacking approach, which is a well-established concept in signal processing. The mathematical formulation of the FreqAttention module, including the use of Fourier transforms and Hadamard products for computing similarity, appears logically sound and aligns with existing principles in time-series analysis.", "experimental_designs_or_analyses": "The evaluation framework is well-structured, with comparisons against multiple state-of-the-art models across diverse benchmark datasets, ensuring assessment of TimeStacker’s performance. The use of standard forecasting metrics, MSE and MAE, that supports the reliability of the results. Additionally, the ablation studies provide insights into the contribution of the frequency-based self-attention module. But, a deeper analysis of computational complexity trade-offs compared to baseline models would strengthen the argument. Overall, the experimental design is good, but more investigation into scalability and complexity would enhance the paper.", "relation_to_broader_scientific_literature": "The proposed TimeStacker framework is related to prior work in deep learning-based forecasting models, such as MLP-based approaches (DLinear, TimeMixer) and Transformer-based models (PatchTST, Crossformer). The contribution connects with broader trends in time-frequency analysis and multi-scale modeling, which have been explored in statistical and signal processing literature. By integrating these concepts into a computationally efficient deep learning framework, the paper advances the field by offering a scalable and interpretable solution for nonstationary time series forecasting.", "essential_references_not_discussed": "No major reference missing", "other_strengths_and_weaknesses": "No additional points", "other_comments_or_suggestions": "Table 1, could benefit from having a average across datasets to see overall improvement compared.", "questions_for_authors": "No additional questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces TimeStacker, a time series forecasting transformer designed to handle nonstationary signals. Empirically, TimeStacker achieves strong or state of the art performance on several real-world datasets.

The paper received 4 reviews with ratings: Accept, Weak Accept, Weak Reject, Weak Reject. The reviews are split.

One of the main concerns raised is how TimeStacker differs from previous works that have used multi-resolution and frequency-based approaches. The main contribution of the approach is a method to combine different window sizes using attention in the frequency domain. The discussion and rebuttal below clarified that this approach is distinct from typical multi-resolution patch approaches used elsewhere in NLP, time series forecasting, computer vision, etc. I am satisfied with regards to the novelty of the paper.

Further, concerns were raised about the validity of the tasks and comparison to naive or simple baselines. I am sympathetic to Reviewer FZmL's concerns about the exchange dataset in particular (forecasting exchange rates two years into the future is optimistic at best). More generally, as a community we must be massively overfitting to these few datasets. Nevertheless, with updates to include naive baselines and other reviewer feedback, the paper can be accepted.
Decision: Accept (poster)