Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting
Authors: Bong Gyun Kang
Dongjun Lee
HyunGi Kim
Dohyun Chung
Sungroh Yoon
Venue: NeurIPS 2024 poster (2024)
Keywords: Time series forecasting
Long-range dependency
Low-pass filter
Spectral attention
Long term trend
Primary area: deep_learning_architectures
TL;DR: Introduce low-pass filter based spectral attention to address long-range dependency in time series forecasting

Abstract: Sequence modeling faces challenges in capturing long-range dependencies across diverse tasks. Recent linear and transformer-based forecasters have shown superior performance in time series forecasting. However, they are constrained by their inherent inability to effectively address long-range dependencies in time series data, primarily due to using fixed-size inputs for prediction. Furthermore, they typically sacrifice essential temporal correlation among consecutive training samples by shuffling them into mini-batches. To overcome these limitations, we introduce a fast and effective Spectral Attention mechanism, which preserves temporal correlations among samples and facilitates the handling of long-range information while maintaining the base model structure. Spectral Attention preserves long-period trends through a low-pass filter and facilitates gradient to flow between samples. Spectral Attention can be seamlessly integrated into most sequence models, allowing models with fixed-sized look-back windows to capture long-range dependencies over thousands of steps. Through extensive experiments on 11 real-world time series datasets using 7 recent forecasting models, we consistently demonstrate the efficacy of our Spectral Attention mechanism, achieving state-of-the-art results.

Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting
Bong Gyun Kang
Dongjun Lee
HyunGi Kim?
DoHyun Chung?
Sungroh Yoon
Interdisciplinary Program in Artificial Intelligence, Scoul National University Department of Electrical and Computer Engineering, Scoul National University Department of Future Automotive Mobility, Seoul National University
Abstract
Sequence modeling faces challenges in capturing long-range dependencies across diverse tasks. Recent linear and transformer-based forecasters have shown superior performance in time series forecasting. However, they constrained by their inherent inability to effectively address long-range dependencies in time series data, primarily due using fixed-size inputs for prediction. Furthermore, they typically sacrifice essential temporal correlation among consecutive training samples by shuffling them into mini-batches. To overcome these limitations, introduce a fast and effective Spectral Attention mechanism, which preserves temporal correlations among samples and facilitates the handling of long-range information while maintaining the base model structure. Spectral Attention preserves long- period trends through low-pass filter and facilitates gradient flow between samples. Spectral Attention can be seamlessly integrated into most sequence models, allowing models with fixed-sized look-back windows capture long- range dependencies over thousands of steps. Through extensive experiments on 11 real-world time series datasets using recent forecasting models, we consistently demonstrate the efficacy of our Spectral Attention mechanism, achieving state-of- the-art results.
Introduction
Time series forecasting (TSF) stands core task in machine learning. ubiquitous in lives through applications such as weather forecasting, traffic flow estimation, and financial investment. Over
time, TSF techniques have evolved from statistical (5) 10 [8] and machine learning approaches [2]
2,20 to deep learning models like Recurrent Neural Networks [5]
25 41 and Convolution based
Networks 12 45
Following the success of Transformers 44 in various domains, Transformer-
based models have also become mainstream in the time series domain [23 27 36 49 54
55 56)
Recently, methodologies based on Multi-layer Perceptron have received renewed attention g
288 (52). However, despite the advancements, long-term dependency modeling in TSF remains challenging [53].
Unlike image models, where data are randomly sampled from the image distribution and are thus independent of each other [16), TSF models sample data from the continuous signal, dependent
DDenotes equal contribution. (luckypanda, elite1717) @nw.aac.k CCorresponding author. sryoon @ ac.k.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Time Series Data
(b) Conventional
(c) Ours
Look-Back Window Prediction
Decorrelated
Preserve Correlation
(Input)
(Label)
mini-batch
t+3
Whole Time Series Sequence
Look-Back Window (Input)
Look-Back Window (Input)
Figure I: (a) Training data sampled for each time step from continuous sequences, exhibiting high temporal correlations. (b) Conventional approaches simply ignore this temporal information with shuffled batch. (c) We address the temporal correlation between the samples for the first time, enabling the model to consider long-range dependencies that surpass the look-back window.
on the time variable & as shown in Figure Db. This leads to a high level of correlation between each training sample, which consists of a fixed-sized look-back window before t (as input) and the subsequent prediction sequence after & (as label). Therefore, the conventional approach of shuffling the consecutive samples into mini-batches deprives the model of utilizing the crucial inherent temporal correlation between the samples. This restricts the model's consideration to only fixed-size book-bck window for sequence modeling, limiting the ability to address long-range dependencies (Figure [T))
Recent studies pointed out that simply increasing the look-back window leads to substantially detri- mental effects such as increased model size and longer training and inference times [53]. This is particularly challenging for transformer forecasters, which exhibit quadratic time/memory complex-
ity [36 44 [4)] and even for efficient models using techniques like Sparse Attention, which have
Onlogg)) complexity [23 27 55 Furthermore, if the commonly used look-back window of 96 extended fivefold, the model can only utilize time steps of less than 500, making it difficult to consider long-range dependencies spanning thousands or the entire dataset. Also, increasing the look-back window may not be beneficial, often leading to decreased performance [53], highlighting the fact that current models are not sufficient in capturing long-range dependencies.
To address this limitation, we propose Spectral Attention, which can be applied to most TSF models and enables the model to utilize long-range temporal correlations in sequentially obtained training data. With the stream of consecutive training samples (Figure [Tp). Spectral Attention stores an exponential moving average (EMA) of the activations with various smoothing factors at each time step. This serves as a low-pass filter, inherently embedding long-range information over a thousand steps. Attention is then applied to the stored EMA activations of various smoothing factors (low-pass filters with different frequencies), enabling the model to learn which periodic trends to consider when predicting the future, thereby enhancing its performance. Spectral Attention is even applicable to models such as iTransformer [3]] which do not preserve the temporal order of time series data internally.
We further extend Spectral Attention, where computations depend on the activation of the previous timesteps, to Batched Spectral Attention, enabling parallel training across multiple timesteps. This extension makes Spectral Attention faster and more practical and allows for the direct utilization of temporal relationships among consecutive data samples within a mini-batch in the training base model. Batched Spectral Attention, the EMA is unfolded over time to perform Spectral Attention simultaneously across multiple time steps. This unfolding allows gradients at time to propagate through the Spectral Attention module to the previous time step within the mini-batch, achieving effects similar to Backpropagation Through Time (BPTT) [47 and extends the model's effective input window.
Our approach preserves the base TSF model architecture and learning objective while enabling the model to leverage long-term trends spanning thousands of steps. By effectively utilizing the temporal correlation of training samples, our method allows gradients to flow back in time beyond the look- back window, extending to the entire mini-batch. Also, our method requires little additional training time and memory. We conducted experiments on 7 recent TSF models and 11 real-world datasets and demonstrated consistent performance enhancement all architecture, achieving state-of-thed results. We summarize the main contributions follows:

---

We propose Spectral Attention, which addresses long-range dependencies spanning thou- sands of steps through frequency filtering and attention mechanisms, leveraging the temporal correlation among the consecutive samples.
We propose Batched Spectral Attention, which enables parallel training across multiple timesteps and expends the effective input window, allowing the gradient to flow through time within the mini-batch.
Batched Spectral Attention applicable to most existing TSF models and practical in real-world scenarios with minimal additional memory and comparable training time. Also, it allows finetuning with a trained TSF model.
Batched Spectral Attention demonstrates consistent model-agnostic performance improve- ments, particularly showcasing superior performance on datasets with significant long-term trend variations.
Related Works
Classic TSF models. Statistical TSF methods, such as ARIMA [35], Holt-Winters [18), and Gaussian Process [10], assume that temporal variations adhere to predefined patterns. However, their practical applicability is largely limited by the complex nature of real-world data. Machine learning approaches, such as Support Vector Machines [4] and Random Forests [[4] have proven be
effective even compared to early artificial neural networks [3] 17 40. Convolutional network-baa based methods leverage convolution kernels to capture temporal variations sliding along the temporal Recurrent neural network (RNN) grasp changes over time via state transitions across different time steps. [25] 41). However, RNN-based models exhibit limitations in modeling long-range dependencies due to challenges such as vanishing gradients and error accumulation [0 Recently, transformer and linear-based models have emerged alternatives, demonstrating
dimension 12 45
43
superior performance compared to recurrent models [46]] 13
Transformer and Linear based models. Transformer-based models 44 address temporal rela- tionships between time points using the attention. LogSSarreTransffrmeee [27]. Reformer [23), and Informer (55) have been proposed to make the Transformer architecture efficient, addressing the quadratic time complexity. The Autoformer (49 incorporates series decomposition as an inner block of Transformer and aggregates similar sub-series by utilizing the cnto-Coorrelaton mechanism. PatchTST (36 introduces patching, a hhannel-independent approach that processes each variable sepa- rately and focuses tross-time attention. Crossformer [44] utilizes
channel-dependenn approach to learn cross-variate dependencies. This is achieved through the use of cross-time and cross-dimension attention. iTrrnsformer [31 applies attention and FFN in an inverted way, where attention handles correlations between channels and FFN handles the temporal information. Recently, to address Transformers' potential difficulties in capturing long-range dependencies [53], methodologies based on the linear model and Multi-Layer Perceptron (MLP) structures have emerged. DLinear (53) utilizes the decomposition method introduced in Autoformer and predicts by adding the output of two linear layers for each seasonal and trend element. TiDE [7] proposes an architecture based on MLP residual blocks that combines information from dynamic and static covariates with look-bace window for encoding, followed by decoding. TSMixer [8] performs forecasting by repeatedly mixing time and features using an MLP. RLinear 28 comprises of a single linear layer with RevIN [2] for normalization and de-normalization.
urequency-utilizing models. Using the frequency domain for TSF is well-established approach 3
42 [9]. Conventional approaches leverage frequency information during the preprocessing stage [37
or decompose time series based on frequency filtering [39]. In deep TSF models, research has also explored architectural advancements that are aware of the frequency information. SAAM (34), which applicable to RNNs, performs FFT and autocorrelation on the input signal. WaveFroM [5T] uses discrete wavelet transform (DWT) to project time series into wavelet domains of various scales and performs forecasting through graph convolution and dilated convolution. FEDformer (56) adopts a mixture-of-erte -o-eeeerts strategy to refine the decomposition of seasonal and trend components and introduce sparse attention mechanisms in the frequency domain. TimesNet (48] transforms ID time series into 2D tensors utilizing multi- multi-periodiccity by identifying dominant frequencies through Fourier Transform, modeling temporal variations effectively. FreTS [52] leverages frequency-domain ML.Ps to achieve global signal analysis and compact energy representation, addressing the limitations of
3

---

(a)
Prediction
(b)
(c)
M
E+
b
SA
SA
F
t-L-3
Input
t+2
a.M,
o.)
h
SA
M
Fi- M
feature
1-L-2 Input [el
SA- Matrix
h
Ft
Ft
SA
M
Per-column
b+b+
Input
M
Weighted Sum
Iriput
Look-Back Window (L)
Momentums off
M
Figure 2: (a) Plug-in Spectral Attention (SA) module takes subset of intermediate feature F and returns F' with long-range information beyond the look-back window. The model is trained end-to- end, and gradients flow through the SA module. (b) To capture the long-range dependency, SA stores momentums of feature F generated from the sequential inputs. Multiple momentum parameters og capture dependencies across various ranges. (c) SA module computes F' by attending multiple low-frequency (Ma) ) and high-frequency (F M) components and feature (F) using learnable Spectral Attention Matrix (SA-Matrix)
point-wise mappings and information bottlenecks in conventional MLP-based methods. FITS 50 employs interpolation within the complex frequency domain to construct a concise and robust model. While these models leverage frequency information, they are limited in modeling long-range dependencies, as the frequency conversion is confined to the look-back window. On the other hand, our Spectral Attention is the first achieve long-range dependency modeling beyond the look-back window by incorporating consecutive data streams during model training.
3 Methods
Problem Statement. In multivariate time series forecasting, time series data is given Dy (Z1,... 2T) E RTXN time 7' with N variates. Our goal is, at arbitrary future time T. to predict future S time steps Y; (z++1,.... I+ts } E R To achieve this goal, TSF model utilizes length L fook-back window as imput X, (t.., 213 E R L making prediction P. = f(x,, P PSXNN
Model is trained with the training dataset Dr = Y. S - S). While conventional methods typically randomly sample each X,y Y from Dr to constitute the mini-batch, we utilize sequential sampling incorporate temporal correlations between samples into the learning process.
3.1 Spectral Attention
Spectral Attention (SA) can be applied to every TSF model that satisfies the aforementioned problem statement. This base TSF model is represented by P f(X), and SA can be applied to arbitrary activation within the model. The base model can be reformulated as P [2(F, E) and F, E (X)). F, E are intermediate state and SA module takes an arbitrary subset F as input and transforms into F' of the same size; F' SA(F), P' f2(F", E). The resulting SA plugged model SSA is depicted in Digure2h.
With X; as the base model input, SA takes D-dimensional feature vector F, RR as input. SA updates the exponential moving average (EMA) M, E IR KDD of F, in its internal memory with the K smoothing factors {a1,... ..., 0Kk E R (a1 < <... og as shown in Figure 2p.
M 2+1 a x M k. + (1 ak) x F
(I)
EMA retains the trend features over long-range time periods based on the smoothing factor. It oper-
ates low-pass filter, with the -3db (half) cut-off frequency of freqeut 2 coo-
(1-o)²
2a
effectively preserving the trend over 6,000 period with = 0.999
To represent high-frequency patterns contrasting with the low-pass filtered long-range pattern M,, generated H, E IR xD by subtracting M, from F,.
H. Fi M, K --1

---

Mil
BSA Module
M
Miv2
Foue-1
(Hay)
Fi- M
Irpuit
Mi
feature
Fe-T
(1-a))
(1-
B-
Ft
SA-Matrix
Input
Ft
(Hai)
(1-0
(1-a
B-1
Ma
Per-column
Input
M
MC4
Weighted Sum
M
Figure 3: BSA module takes a sequentially-sampled mini batch (X+::..X++BBI and computes the corresponding EMA momentums MMMeeBB. over time. This is done via single matrix multiplication enabling parallelization. We made the momentum parameter as learnable, allowing the model directly learn the periodicity of the information essential for the future prediction.
SA contains learnable parameters: sa-matrix E (22K+1)xD which learns what frequency the model should attend for each feature. H,, Fú, M2 are concatenated on dimension 0, resulting in B (22 which is then weighted summed with sa-matrix dimension 0, generating output Fj (Figure 2+).
(3) The sa-matrix is initialized so that sof toft (sa-matri))) resembles Gaussian distribution axis + +1 sa-matrix K+ ++++) and makes S S.A an
Fg sum
dim 0) concat x H,, Fr, M,), dim 0), dim 0)
This results in symmetric value on axis (sa-matrix" identity function on initialization (:.' H + M
F)).
F (F)
(4)
SA allows the model to attend to multiple frequencies of its feature signal, enabling to focus on either long-range dependencies or high-frequency patterns as needed and shift the feature F distribution on the frequency domain. By initializing SA as an identity function, the model can be fine-tuned with the already trained base model, allowing efficient implementation.
3.2 Batched Spectral Attention
Batched Spectral Attention (BSA) enables batch training over multiple time steps. The main concept involves unfolding EMA, which facilitates gradients to flow across consecutive samples in mini- batch, akin to BPTT. This enables efficient parallel training and promotes the model to extract long-range information beneficial for future prediction, extending the effective look-back window. The overall flow of BS is depicted Figure
With mini- batch of size B. consecutive samples [ (XI,....++BBI) E R Bx8NN are given as input. Following aforementioned SA setting, BSA takes Fig,-B-11 B-1] = (F.....F E BB B D as input. In the next step. BSA utilizes F1,.+B-1) and the stored Mj E RKXD to generate M0 & b < B) by unfolding the Equation
M 0% M + ax)ak Ft. +.... (1 ax) F Pitb1
(5)
This equation can be transformed to calculate Mμ E R(B+1)xXXD in parallel as follows.
M t B (oouer-triangl(AA") x con psi Fli++-y) dim 0)
(6)
A (ERKx(B4))((BH)) Akp. = (1 (1- (l)) D-9
(7)
A refers to unfolding matrix and 7 refers to indicator function. Mr+ s is stored in BSA A for the next mini-batch input. F/ [-]+BB1] is computed in parallel, similar to Equation 2 and 3 using Ffr,++B1]]- B1,.+B11)- and sa-matrixg (2K+1)xD The lower-triangle function prevents gradients from the past timestep from flowing into future models, aligning with the characteristies of time-series data.
5

---

At the beginning of each training epoch, Mo is initialized to Fo for all a, enhancing stability for subsequent EMA accumulation. Since SA is proposed to address long-range dependencies in training, it lacks sufficient information in the early stages when not enough steps have been seen. Therefore, for the stability of training, we linearly warm up the learning rate for the first 1/(1 maz (smoothinn factors)) timesteps at the beginning of each training epoch. The overall training of both the base model and BSA is conducted according to Algorithm [
In SA, the smoothing factors {ai.-.. ox3 IR were given as scalar values, whereas in BSA, they are expressed by learnable parameters. This is because BSA A can uti- lize additional past information in training beyond the look-back window by incorpo- rating batch-sized time window, allowing to determine the extent of long-range de- pendency required for training. To keep the smoothing factors between O and 1, we ini- tialized learnable parameters by applying inverse sigmoid to the initial smoothing fac- tors and then applied sigmoid function in training.
Algorithm 1 Batched Spectral Attention (1 epoch)
Input: Trained up to (E-1)th epoch
TSF model fs. BS A with sa-matrix, smoothing factors
Train data: +.- = {(Xo, Yo), (Xtr; Yor)
Valid data: Deal {(Xrr+1, Yer+1), (Xeat, Yeal))
mini-batch size: B
Train:
st = O, = B -
initialize Mo in BSA with Xo and fo (:= s2s fio) for ((st,ed)) Yst,eed in Dtr do train phase)
Plstee t f2e -SAA- (xa(XXtt.ed)
L = L (Pus.edd Y(st.ed))
Compute VL. and adjust learning rate
update fo. sa-matrix, smoothing factors s/+ B, eel = min(ed B,tr)
So far, we assume the feature F from the base model as a vector. However, the out- put of the intermediate layers of the model is often represented as a tensor with two or more dimensions. In real practice, we use additional channel dimensions in BSA A to process the activation tensor, which acts as applying multiple BSA modules simultane- ously.
end for
st = br + 1,ed = tr
for X(mt,ed), (ot,ed) in D do (: Validation phase)
P Paated + f20 BSA fio(X((r.ed))
L= LmseePutt.cd (Plut.edd Y(st,ed)
accumulate L and calculate validation loss Loal
stt = B, ed = min(ed + B, val)
end for Output: Trained up to E th epoch fe. sa sa-matrix, {ar, ax.. Lost for E th epoch
3.3 Consecutive dataset split
In the TSF scenario, the entire time series data { (Tena) is divided into train, validation, and test sets in chronological order. Let Tir and T yaul denote the last time in the train and valida- tion data respectively. Model training occurs using data for t in [1, Te-l, while model selection for the best model can utilize data for [Trr41, Teai]. The test set comprises predicting time steps [T Tendl. which are not accessible during training or validation. However, since each training sample consists of the look-back window of size and a prediction window of size S, the training input samples are restricted to X XII.T,--S] s]. Validation samples and test input samples range from X(at..T.aI-S]] and from X X[Toai:Tend -]. respectively. While this approach is plausible for independent data like images, it is unnatural for sequential data, as it leaves unreachable gaps X X St1.--+11....11).. undermining the consecutive characteristics. We filled the missing gaps, making train, validation, and test sets consecutive so that our BSA model could update momentum continuously. For fair evaluation, added samples were not used for either model training, validation, or performance assessment. Detailed explanations of model validation and evaluation are provided in Appendix A.3 The full code is available at htttss//gituub.co//DD.... 1008/BSA 2024
(Xiz,--++1I7.
4 Experiments
We first evaluate BSA using state-of-the-art TSF models and various real-world time series forecasting scenarios in section 4.1 Next, to demonstrate that BSA effectively addresses long-range dependencies and is robust to distribution shift, we perform experiments on synthetic signals of various frequencies in section 4.2 Finally, analyze the BSA's performance variations depending the insertion sites within the base model, examine computation and memory costs, and conduct an ablation study in section 4.3
Datasets. We use eleven real-world public datasets: Weather, Traffic, ECL. ETT (4 sub-datasets; h1, h2, ml, m2), Exchange, PEMS03, EnergyData, and Illness 66 26 29 49 In the IlIness dataset,

---

Table 1: Forecasting results averaged across prediction lengths S E (96, 192,336, 720) and three seeds. Illness and Exchange datasets use different prediction lengths due to short data lengths. Higher performance between the base model and BSA is bolded. The Avg. column shows the mean across all datasets. Red indicates p-value < 0.05 in paired t-test. Abbreviations are as follows, We.; Weather, Tr.: Traffic, Fh1: ETTh1, Hh2: ETTh2, Eml: ETTm1, Em2: ETTm2, Ex.:Exchange, PE.: PEMS03,
En.: EnergyData, II.: Illness. Full results are provided in Appendix B.1
Model
Metric
We.
Tr.
ECL
Eh1
Eh2
Em1
Em2
Ex.
PE.
En.
Avg.
MSE
0.244
0.737
0.227
0.529
0.349
0.462
0.248
0.088
0.436
0.874
2.849
0,640
base
MAE
0.308
0.451
0.325
0.507
0.406
0.447
0.332
0.211
0.502
0.683
1.111
0.480
DLincar
MSE
0.220
0.691
0.217
0.527
0.341
0.441
0.221
0.081
0.384
0.838
2.797
0.614
BSA
MAE
0,281
0.435
0,316
0.507
0.398
0.447
0.310
0.207
0.464
0.667
1.114
0.468
MSE
0.248
0.723
0.231
0.551
0.309
0.488
0.220
0.085
0.992
0.816
2.637
0.664
base
MAE
0.274
0.442
0.319
0.516
0.370
0.457
0.307
0.204
0.740
0.630
1.013
0.479
RLincar
BSA
MSE
0.237
0.665
0.212
0,545
0.304
0,475
0.210
0.085
0.674
0.797
2.605
0.619
MAE
0.267
0.404
0.301
0.517
0.370
0.453
0.303
0.203
0.607
0.622
1.005
0.459
MSE
0.240
0.527
0.195
0.513
0.310
0,460
0.234
0.095
0.259
0.976
5.073
0.807
base
MAR
0.286
0.322
0.285
0.506
0.393
0.450
0.321
0.232
0.354
0.735
1.603
0.499
FreTS
MSE
0.236
0.502
0.192
0.511
0.308
0.431
0.241
0.089
0.229
0.940
4.930
0.783
BSA
MAE
0,292
0.317
0.287
0.505
0.387
0.442
0.326
0.219
0.324
0.710
1.591
0.491
MSE
0.260
0.620
0.203
0.576
0.386
0.508
0.244
0.094
0.232
0.880
2.719
0.611
hase
MAE
0.284
0.333
0.301
0.545
0.427
0.482
0.326
0.219
0.311
0.662
0.942
0.439
TimesNet
MSE
0.252
0.624
0.199
0.569
0,378
0.499
0.229
0.094
0.234
0,861
2.720
0.605
BSA
MAE
0.279
0.335
0.298
0.540
0.418
0.476
0.314
0.219
0.314
0.655
0.938
0.435
MSE
0.256
0.428
0.542
0.321
0,466
0.224
0.262
0.833
2.454
0.551
base
MAR
0.277
0.284
0.272
0.518
0.380
0.454
0.314
0.211
0.345
0.639
0.947
0.422
iTransformrr
BSA
MSE
0.236
0.428
0.176
0.537
0.317
0.466
0.219
0.090
0.198
0.786
2.540
0.545
MAE
0,268
0.288
0.270
0.517
0.378
0.452
0.310
0.211
0.298
0.619
0.957
0.415
base
MSE
0.240
0.565
0.182
0.517
0.323
0.485
0.243
0.220
0.212
1.178
4.946
0.828
MAE
0,292
0.292
0.276
0.520
0.402
0.470
0.342
0.345
0.299
0.817
1.518
0,507
Crossformer
MSE
0.227
0.554
0.186
0.520
0.317
0.468
0.245
0.219
0.201
1.204
4.809
0.814
BSA
MAE
0.284
0.288
0,280
0.522
0.396
0.469
0,340
0.340
0.290
0.823
1.489
0.502
MSE
0.255
0.467
0.198
0.535
0.317
0.473
0,220
0.087
0.361
0.839
2,016
0,524
base
MAE
0,278
0,296
0,286
0,516
0,375
0.456
0,310
0,204
0,413
0.642
0.880
0.423
PatchTST
MSE
0.236
0.467
0.189
0.539
0.314
0.458
0.214
0.086
0.244
0.788
1.974
0.501
BSA
MAE
0.268
0.297
0.285
0.520
0.374
0.451
0.346
0.202
0.341
0.621
0.866
0.412
the look-back window is set to 36, and the forecasting lengths are 24, 36, 48, and 60. For the other datasets, the look-back window is set to 96, and the forecasting lengths are 96, 192, 336. and 720. Train, validation, and test split ratio are 0.6, 0.2, 0.2 for the ETT dataset and 0.7, 0.1, 0.2 for the Weather, Traffic, ECL, Exchange, PEMS03, EnergyData, and Illness datasets. The Weather, Traffie, and ECL datasets settings follow the TimesNet paper 48 Details on datasets are provided in Appendix A.I A.I
Baseline models. As a benchmark, we apply BSA to 7 recent or well-known forecasting models.
(1) Linear based models: DLinear [53] RLinear (28). FreTS (52) (2) Convolution based methods:
TimesNet [48] (3) Transformer based methods: iTrrasforrmer [31] Crossformer [54], PSt (36]
For each dataset, the model structure configuration is based the Time-Series-Library 48 Details
on the base model configurations are provided in Appendix A..2]
Training details. We first train the base model for more than 30 epochs (20 epochs for the Traffic dataset) using Adam 22 to ensure that the validation MSE saturates, while also conducting an extensive hyperparameter search. Then, we fine-tune with the BSA module attached to the pre- treitrained base model. All experiments are based on the average values of three random seeds. We provide further details in Appendix A.3
4.1 Real world datasets
We demonstrate that BSA improves forecasting performance by providing the ability to address long-range dependencies, regardless of the base model architecture. [ the averaged values over prediction lengths for both the base and the BSA applied model across 11 time series datasets and 7 TSF models. We apply the BSA module at the beginning of the base model's activation, with the number of BSA channels matching the number of channels in the data (position 1 in Figure
7

---

FFT Amplltu de
Attention Level
FFT Amplltude
Affennion Level
FFT Amplluude
Attainton
402
(a) SA- Matrix heat map
(b) Weather-Temp."S
(c) Weather-SWDR
(d) ETTh--HULL
Figure 4: This figure illustrates the analysis of the SA-matrix of the DLinear model trained on the st0-step prediction task for the Weather and ETTh1 datasets. Panel (a) shows the heatmap of the SA SA-matrix, and (b)-(d) show the attention and FFT graphs.
7) BSA effectively improves MSE and MAE across all architectures. The average performance high as 7.2%, with linear-based models
gain, in terms of MSE, ranged from low 0.96% demonstrating relatively high performance. Paired t-tests demonstrate statistically significant (p-value 0.05) improvements in most MSE and all MAE across various models. This result emphasizes the model-agnostic versatility of BSA. Furthermore, BSA exhibited overall performance gain across all datasets, with the enhancement being statistically significant in 82% of cases. BSA's higher gain on ETTm compared to ETTh, both derived from the same signal but with different sampling rates, further indicates BSA's effectiveness handling long-range dependencies.
To understand how BSA addresses long-range dependencies, we conduct an internal inspection. In Figure4al present the heat map of the trained SA-Matrix of the DLinear (Temperature( channel, Weather data). The positive x-axis corresponds to the log1o values of the periods preserved by the low-pass filter, derived with the smoothing factor. Negative values correspond high-frequency components. The blue graph in Figure 4b represents the SA-Matrix averaged over the feature dimension, illustrating the frequencies which BSA attends overall. The red graph represents the result of applying the Fast Fourier Transform (FFT) and denoising to the raw signal. The blue graph skewed towards the low-frequency side indicates that the BSA effectively captures the long-rate gange trend of the data. Figure 44 depicts the graph for the SWDR (Short Wave Downward Radiation per unit area, W/m) channel in the same SA-Matrix. While not identical Figure [bb it also exhibits strong attention towards the low-frequency pattern. In contrast, Figure 4d the FFT graph for the HULL (High UseLess Load channel, ETTh1 data), shows that the signal itself lacks long- oongrrange trends, resulting in the symmetric SA-Matrix. This result demonstrates that BSA operates as intended, learning the low-frequency components of the signal for future prediction. We provided other graphs and detailed information the graph plotting method in Appendix B.2
4.2 Synthetic datasets
To further demonstrate that BSA learns long-range dependencics beyond the look-back window, we add sine waves with periods of 100, 300, and 1000 to the natural data while maintaining the mean and standard deviation (Refer to Appendix C for details on synthetic data generation). Figure [ illustrates the performance improvement of BSA the iTransformer model on the ETThl and
(a) ETTh1 MSE
(b) ETTh2 MSE
1000
300
300
40
Gain (96)
Gain (9))
30-
10
13*
26
332
335
700
08
302 336
750
Prediction Length
Prediction Length
Figure 5: Results of the iTransformer model on synthetic (a) ETThl and (b) ETTh2 datasets. The x-axis is the prediction length (96, 192, 336. 720), and the y-axis is the performance improvement (%) compared to the base model. Each color represents the different periods the sine wave added to the natural data. 0 indicates original data and serves the baseline.

---

Alternt Leve!
FFT Arp tude
FFT Ampi tude
Altend on Level
FFT Ampi tude
Aftent on Lewel
FFT Amplituude
(a) LUFL original
(b) LUFL period 100
(c) LUFL period 300
(d) LUFL period 1000
Figure 6: Attention and FFT graphs on LUFL channel of the original and synthetic ETThl data (Trransformer, 720-step prediction). (a) is from the original data and (b)-(d) are from the synthetic data created by adding sine waves with periods of 100, 300, and 1000, respectively. The red arrows on the FFT graphs show the added synthetic signals. Full visualization is provided in Appendix C.1
ETTh2 datasets. The x-axis the prediction length, and each line represents the period of the added sine wave. Performance improvement is calculated as 100x (base MSE BSA MSE) base MSE While the base model with 96-length look-back window is expected to learn the 100-period trend, BSA outperformed it, especially for 96 and 192-step predictions (green line). The yellow line (period 300) shows nearly a 30% performance improvement across all prediction lengths. While the base model fails learn the long-range interactions within a period of 300, BSA captures and utilizes the underlying trend. BSA also learns the 1000-period signal (red line) and demonstrates substantial improvemment improvements, especially in long prediction-length (336, 720) tasks. These results show that BSA effectively learns long-range patterns beyond the look-back window, essential for future prediction.
Figure 6 is generated from LUFI. (Low UseFul Load) channel of ETTh1 data and with added sine waves of periods 100, 300, and 1000. The red arrow shows the added synthetic trend in the FFT graph (red line). As long-range trends are introduced, the attention graph (blue line) shifts from resembling symmetric Gaussian to a low-frequency bias. Longer sine wave periods cause greater shifts, prioritizing long-range information for predictions.
4.3 Analysis and ablation studies
Add Norm
iTransformer MSE
DLinear MSE
Location
MAE
MAE
Multi-Head
0.2357
0.2682
0.2196
0.2815
Attention
Decoder
0.2352
0.2681
5
6
7
3
0.2329
0.2654
0
K
V
Add Norm
0.2508
0.2752
0.2327
0.2994
Query-5
0.2326
0.2671
Key-6
0.2538
0.2762
2 Input Embedding
Value-7 0.2415
0.2702
bascline
0.2556
0.2766
0.2444
0.3084
Table 2: Performance analysis on BSA insertion site. Each
Figure 7: Schematic diagram of the BSA insertion site Transformer.
number corresponds with the insertion site in Figure
The BSA module offers high flexibility as can be applied arbitrary activations of the base model. In Table we analyzed the performance changes by applying BSA at various locations within the model. Each location corresponds to a number in the Transformer architecture depicted in Figure While we uniformly applied BSA to position 1 for the main result Table I. the results in Table suggest the potential for further performance enhancement by applying BSA at appropriate locations. Additionally, while there is variability depending the placement, the performance consistently remains higher compared to the baseline, demonstrating the stability of our method. We provided a full result in Appendix D.I
BSA shows consistent performance improvement across varying look-back window (input) lengths. Saabeeddmmonstrttee BSA's superiority for look-back window lengths of 48, 96, and 192. Notably, while the baseline model's performance drops significantly with shorter inputs, BSA maintains high performance.

---

Table 3: BSA's performance improvement (%, de- noted with *) compared to base model across three input lengths {48, 96, 192] (iTransformer, average over 4 prediction lengths and 3 seeds). The full result, including other models, in Appendix D.2
Table 4: Computational cost increase with the BSA (%), averaged over 3 seeds and 4 prediction lengths on the PEMS03 dataset. TN: TimesNet, iTF: iTransformer, CF: Cross- former, PTST: PStchTST. The full result, in- cluding other datasets, is in Appendix DD3
Diput- length
Weather
PEMS03
(%)
(%)
(S)
(%)
Increase (%)
TN
iTF
CF
PTST
48
12.08
5.21
29.45
17.35
Time
0.21
2.24
-2.04
-0.29
96
7.78
3.03
24.55
13.66
Memory
0.34
2.34
0.17
0.38
192
6.87
3.34
9.63
5.18
Parameter
0.16
4.88
1.96
2.25
We also demonstrate the impact of using BSA on the training time, peak memory, and the number of model parameters. The extra computation required for BSA is constant with data length and linear with a look-back window length (c.f. quadratic for the base model with transformer architecture). Table demonstrates the computational cost of the BSA is quite small, showing less than 5% increase even with the large PEMS03 dataset.
We conduct ablation study the three key compo- nents that constitute BSA (Table "BPTT" refers to whether gradients flow between samples within the mini-batch. Without BPTT, BSA learns similarly to SA. "SFs" denotes whether to use multiple smooth- ing factors. Lastly, "Learn SF" indicates whether the smoothing factor is treated learnable. The results indicate that each component significantly contributes the performance improvement of BSA.
Table 5; Ablation study with 3 components in BSA (Weather data, 720-step prediction, iTransformer).
BPTT SFs Learn SF
MSE
MAE
baseline
0.3551
0.3473
0.3480
0.3452
0.3401
0.3452
0.3320
0.3395
v
0.3263
0.3358
5
Conclusion
Our study addresses the challenges in handling long-range dependencies inherent in time series data by introducing fast and effective Spectral Attention mechanism. By preserving temporal correlations and enabling the flow of gradients between samples, this mechanism facilitates the model in capturing crucial long-range interactions essential for accurate future predictions. Therefore, our research paves the way for fixed-sized input models to effectively handle long-range dependencies extending far beyond the input window. Through extensive experimentation, we demonstrated that our Spectral Attention mechanism enhances performance across various base architectures, with its ability to grasp long-term dependencies being the key factor behind this improvement. BSA effectively tackles long-term fluctuations, complementing the base model's 's capacity to manage intricate yet short-term patterns. This integrated model holds promise for improving real-world application performance. For instance, it could boost weather forecast accuracy by simultaneously capturing minute-by- -iminttee weather changes and seasonal variations. Moreover, when predicting deterioration from a patient's real-time data, it can consider medications with lengthy onset times. Our study has limitations: we did not analyze the impact of BSA placement within the base model in detail. Also, BSA's performance gains may be limited when applied to datasets with only high-frequence information within the look-back window. These issues should be addressed in future research.
Acknowledgement
This research was supported by a grant of the MID-Phd/Medical Scientist Training Program through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea, National Research Foundation of Korea (NRF) grants funded by the Korea gov- ernment (Ministry of Science and ICT, MSIT) 2202RIA33 1077720 and 2022RIA5A7IO ASAA788398 Institute of Information & Communications Technology Planning & Evaluation (ITTP) grants funded by the Korea government (MSIT) (2021-0-01343. Artificial Intelligence Graduate School Program
(Seoul National University). 2022-0-00959 and 202--204-RS--2 -----RS--22---000 2024-00397085: Leading Generative AI Human Resources Development), and the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2024, AI-Bio Research Grant through Seoul National University, Hyundai Motor Company, HUINNO AIM Company through HA-Rod Andd 2325-PredictClimie PrecDietCimicalDeeeeiraaatt
10

---

References
[1] Abien Fred Agarap. Deep learning using rectified linear units (relu).
arXiv preprint
arXiv:1803.08335, 2018.
[2] Nesreen K Ahmed, Amir F Atiy,, Neamat El Gayar, and Hisham El-Shishiny. An empirical comparison of machine learning models for time series forecasting. Econometric reviews, 29(5-6):594-62) 621, 2010.
[3] Peter Bloomficld. Fourier analysis of time series: an introduction. John Wiley & Sons, 2004.
[4] Bernhard E. Boser, Isabelle Guyon, and Vladimir Vapnik. training algorithm for optimal margin classifiers. In Proceedings of the fifth annual workshop on Computational learning theory, pages 144-152, 1992.
[5] George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Sociery. Series (Applied Statisties), 17(2):11-19, 1968.
[6] Luis M Candanedo, Véronique Feldheim, and Dominique Deramaix. Data driven prediction models of energy use of appliances in a low-energy house. Energy and buildings, 140:81-97, 2017.
[7] Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan Mathur, Rajat Sen, and Rose Yu. Long- term forecasting with tide: Time-series dense encoder. arXiv preprint 234402230,, 2023.
[8] Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. Tsmixer: Lightweight mip-mixer model for multivariate time series forecasting. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 459-469, 2023.
[9] Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189-1232, 2001.
[10] Agathe Girard, Carl Rasmussen, Joaquin Q Candela, and Roderick Aurray-Smith Gaussian process priors with uncertain inputs application to multiple-step ahead time series forecasting. Advances in neural information processing systems, 15, 2002.
[11] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1rXiv:1606.00841 2016.
[12] Pradeep Hewage, Ardhendu Behera, Marcello Trovati, Ella Pereira, Morteza Ghahremani, Francesco Palmieri, and Yonghuai Liu. Temporal convolutional neural (ten) network for an effective weather forecasting using time-series data from the local weather station. Soft Computing, 24:16453-16482, 2020.
[13] Geoffrey E Hinton, Simon Osindero, and Whye Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7):1527-1544 2006.
[14] Tin Kam Ho. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, volume pages 278-282. IEEE, 1995.
[15] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98):11335 1997.
[16] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint
2019.
[17] MJC Hu and Halbert Root. An adaptive data processing system for weather forecasting. Journal of Applied Meteorology and Climatoloyy, 3(5):513-523, 1964.
[18] Rob Hyndman and George Athanasopoulos. Forecasting: principles and practice, OTexts, 2018.

---

[19] A. Kahraman, G. Yang, and P. Hou. Wind power forecasting using Istm incorporating fourier transformation based denoising technique. In 20th International Workshop on Large-Scce Scale Integration of Wind Power into Power Systems well as on Transmission Networks for Offshore Wind Power Plants (WIW 2021), volume 2021, pages 94-98, 2021.
[20] Kyoung-jar jae Kim. Financial time series forecasting using support vector machines. Neurocom- puting, 55(1-2):307-31, 2003.
[21] Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
[22] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[23] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001. 044451, 2020.
[24] Diego Krapf, Enzo Marinari, Ralf Metzler, Gleb Oshanin, Xinran Xu, and Alessio Squarcini. Power spectral density of single brownian trajectory: what can and cannot learn from it. New Journal of Physics, 20(2):023029, 2018.
[25] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 95-104, 2018.
[26] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The s1 international ACM SIGIR conference on research & development in information retrieval, pages 95-104, 2018
[27] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang. and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, Advances in neural information processing systems, 32, 2019.
[28] Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on linear mapping. arXiv preprint arXiv:2305. 10721, 2023.
[29] Minhao Liu, Ailing Zeng. Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction, Advances in Neural Information Processing Systems, 35:5816 18-2888,, 2022.
[30] Yeqi Liu, Chuanyang Gong, Ling Yang. and Yingyi Chen. Dstp-rmn: A dual-stage two-phase attention-based recurrent neural network for long-term and multivariate time series prediction. Expert Systems with Applications, 143:1 13082, 2020.
[31] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625. 2023.
[32] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Rethinking the stationarity in time series forecasting. NeurIPS, 2022.
[33] MickOhrberg. Brownian noise - Wikipedia, the free encyclopedia, 2024. [Online; accessed 2-r2..
[34] Fernando Moreno-Pino, Pablo M Olmos, and Antonio Rrdés-Rod Deep autoregressive models with spectral attention. Pattern Recognition, 133:109014, 2023.
[35] Brian K Nelson. Time series analysis using autoregressive integrated moving average (arima) models. Academic emergency medicine, 5(7):739-744, 1998.
[36] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint 2rX 2022.
12

---

[37] Vigneashwara Pandiyan, Rita Drissi-Daoure Sergey Shevchik, Giulio Masinelli, Roland Logé, and Kilian Wasmer. Analysis of time, frequency and time-frequeccy domain features from acoustic emissions during laser powder-bed fusion process. Procedia CIRP, 94:392-397, 2020.
[38] Adam Paszke, S. Gross, Francisco Massa, A. Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Z. Lin, N. Gimelshein, L. Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. NeurIPS, 2019.
[39] Hadi Rezaci, Hamidreza Faaljou, and Gholamreza Mansourfar, Stock price prediction using deep learning and frequency decomposition. Expert Systems with Applications, 169: 14332. 2021.
[40] Frank Rosenblatt. The perceptron, perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957.
[41] Alaa Sagheer and Mostafa Kotb. Time series forecasting of petroleum production using deep Istm recurrent networks. Neurocompuling, 323:203-213, 2019.
[42] Donghwan Song, Adrian Matias Chung Back, and Namhun Kim. Forecasting stock market indices using andingg-bsed fourier transform denoising and time series deep learning models. IEEE Access, 9:83786-83799 2021.
[43] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances neural information processing systems, 27, 2014.
[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[45] Renzhuo Wan, Shuping Mei, Jun Wang. Min Liu, and Fan Yang. Multivariate temporal convolutional network: deep neural networks approach for multivariate time series forecasting. Electronics, 8(8):876, 2019.
[46] Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma. Junchi Yan, and Liang Sun. Transformers in time series: A survey. arXiv preprint arXiv:2202.07115,, 2022.
[47] Paul Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 15000),15 1990.
[48] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long, Timesnet: Temporal 2d-variation modeling for general time series analysis. ICLR, 2023.
[49] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34:22419-22430, 2021.
[50] Zhijian Xu, Ailing Zeng, and Qiang Xu. FITS: Modeling time series with $10k$ parameters. In The Twelfth International Conference on Learning Representations, 2024.
[51] Fuhao Yang, Xin Li, Min Wang, Hongyu Zang, Wei Pang. and Mingzhong Wang. Waveform: Graph enhanced wavelet learning for long sequence forecasting of multivariate time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10754-10761, 2023.
[52] Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Ning An, Defu Lian, Longbing Cao, and Zhendong Niu. Frequency-domain mips are more effective learners in time series forecasting. Advances in Neural Information Processing Systems, 36, 2024.
[53] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121-11128, 2023.
13

---

[54] Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross cross- dimension dependency for multivariate time series forecasting. In The eleventh international conference on learning representations, 2022.
[55] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang. Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference artificial intelligence, volume 35, pages 11106-11115,
2021.
[56] Tian Zhou, Ziqing Ma. Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning, pages 27268-27286. PMLR, 2022.
14

---

A Details on Datasets, Models, and Training
A.1 Details on datasets
Dataset Information. We conducted experiments on 11 real-world datasets assess the performance of baseline models and the proposed BSA method. The Weather dataset [49] includes 21 meteo- rological factors acquired every 10 minutes in 2020 from the Weather Station of the Max Planck Institute for Biogeochemistry. The Traffic dataset [49] records hourly road occupancy rates from 862 sensors San Francisco Bay Area freeways, covering the period from January 2015 to December 2016. The ECL dataset [99 captures the hourly electricity consumption of 321 clients. The ETT dataset [55] contains seven factors related to electricity transformers, spanning from July 2016 to July 2018. It is divided into four sub-datasets: ETTh! and ETTh2 are recorded hourly, while ETTml and ETTm2 are collected every 15 minutes. The Exchange dataset [49 comprises panel data of daily exchange rates from eight countries, ranging from 1990 to 2016. Illness dataset [49 contains weekly data influenza-like illness (IIII) cases recorded by the U.S. Centers for Disease Control and Prevention (CDC) from 2002 to 2021. The dataset tracks the proportion of IIII patients relative to the overall number of patients seen during that period. PEMS03 dataset (29] is sub-dataset of the PEMS dataset, which includes public traffic network data from California, recorded at 5-minute intervals. The Energy Data dataset 66] comprises hourly end-use measurements gathered from 454 residential properties and 140 commercial establishments located in the Pacific Northwest. All these public datasets were downloaded from the referenced sources March 2024.
Dataset split. We adhere to the data processing protocol and train- nain-validation-- split used in Times- Net [48), where the training, validation, and test datasets are sequentially separated in chronological order. Our paper's data split ratios for train, validation, and test set are as follows: (0.7, 1. 0.2) for Weather, Traffic, ECI., Exchange, Ilnness, PEMS03, and EnergyData datasets and (0.6, 0.2, 0.2) for ETT. The details of datasets are provided in Table 6
Forecasting setting. Following the approach in iTransformer (31. the look-back window length is set to (96), while the forecast lengths are [96, 192, 336, 720) for the Weather, Traffic, ECL, ETT, PEMS03, and EnergyData datasets. Forecasting lengths is only (96) for the Exchange dataset since the dataset is too short, which causes biased best model selection with the validation set. For Illness dataset, the look-back window length is (36) and the forecasting lengths are (24, 36, 48, 60).
Table Summary of Datasets. Channel denotes the number of time series variables (channels) for each dataset. Pre-Train is the number of training epochs for baseline model saturation. Finetune is the number fine-tuning epochs for our method. Data Split means the number of time steps in (Train, Validation, Test) data split respectively. Sampling Rate denotes how often the data samples are collected. Type is to show the domain which the data is acquired. We have indicated cases where the data split count matches exactly with that used in TimesNet [48] by marking them with an asterisk (*). For the ETT and the Exchange dataset, the length of the downloaded data differed from
the data length reported in TemesNet.
Dataset
Channel
Pre-Train
Finetune
Data Split
Sampling Rate
Type
* Traffic*
21
40
20
(36792, 5271, 10540)
10min
Weather
862
20
20
(12185, 1757, 3509)
Hourly
Transportation
ECL
321
30
20
(18317, 2633, 5261)
Hourly
Electricity
ETTh1, ETTh2
30
30
(10357, 3485, 3485)
Hourly
Electricity
ETTml. ETTm2
30
20
(41713, 13937, 13937)
15min
Electricity
Exchange
30
30
(5216, 761, 1518)
Daily
Bconomy
Illness
30
20
(692, 120, 226)
Weekly
Health
PRMS03
358
30
20
(18250, 2623, 5242)
5min
Transportation
EnergyData
28
30
20
(13719, 1975, 3948)
Hourly
Energy
A.2 Details on Model Implementations
Model configurations. In this section, we discuss the configurations of the baseline models and specify where the BSA module was integrated into each baseline model. For each dataset, the base model structure configuration was directly replicated from the Time-Series-Libral [48] scripts
15

---

when available. Where configurations were not provided, we adjusted them to align closely with the available examples.
average = 25) for the series decomposition module from Autoformer [49]. We the Individual True so that there are separate linear models for each number of input variables. The BSA module is implemented the very beginning of the forward pass right before series decomposition. The BSA module is implemented for each channel, which is the number of input variables for this case.
DI Diner [53] The only hyperparameter for this baseline model is the
2. iTransformer 31: The hyperparameters for this baseline model (d model = 512, d_f = 512, dropout = 0.1, num heads = 8, encoder layeer___yyrrrr = 3, activation GELU (TT) for the Weather, ECL, PEMS03, and EnergyDati datasets,
follows:
(d_model = 512, d_ff== 512, dropout = 0.1, num sea__e 8, encoder_ = 4, activation = GEL.U)
for the Traffic dataset,
(d_model= = 128, d_ff= 128, dropout = 0.1, num_heads 8, encoder = 2, activation = GELJU for the ETT and Exchange datasets.
The BSA module is implemented at the beginning part of the forward pass right after normalization from the Non-stationary Transformer [32] and right before the input embedding. The BSA module is implemented for each channel, which is the number of input variables for this case.
3. Crossformer [54] The hyperparameters for this baseline model
follows:
(seg_ Jen = 12, win sizessizee = 2, factor = 3, model = 32, d ff = 32, dropout = 0.1. num heads 8, encoder_layeen
activation RELU (1) for the Weather and EnergyData dataset,
(seg_len= 12, win 2, factor = 3, d_model= = 128, d_ff= 128, dropout = 0.1, num head_ 2, encoder_layens = activation RELU) for the Traffic dataset,
[seg_len=122 2
d_model= = 256, d_ ff 512, dropout = 0.1, ham_heade 8,
encoder_layen's = activation RELU) for the ECL and PEMS03 dataset,
= d_ff = 2048, dropout = 0.1, num_heads 8, encoder_layen'' = awtiiaaiiiioonoo RELU for the E and ETTh2 datsets,
(seg_len= 2
(seg len= 12, win win__sizee=sie 2, factor mmddemmo==__________d___dd_1d = 512, d ff 2048, dropout = 0.1, num heads 8, encoder layers activation RELU) for the ETTml and ETTm2 datasets,
(seg_len = 12, win_sizee = 2, factor = 3, d model = 64, d ff = 64, dropout = 0.1, num heads 8. encoder_layery
activation = RELUU for the Exchange dataset.
The BSA module is implemented at the very beginning of the forward pass, right before the input embedding. The BSA module is implemented for each channel, which is the number of input variables for this case.
PatchTST 36 The hyperparameters for this baseline model are as follows:
(patch_len 16, stride = 8, d_ model = 512, d_ff= 2048, dropout = 0.1, num num_heads = 4, encoder_layers 2, activation GELU [T] for the Weather and EnergyData dataset,
(patch_len) 16. stridd 8, model= 512, d ff = 512, dropout 0.1, num heads = 8, encoder_
2, activation for the Traffic dataset,
16 d_modell=:0000 512, d f 2048, dropout = 0.1, num_heade 8, encoder_layers 2, activation for the ECL. and PEMS03 dataset,
strid: d_model== activation = for the ETTh1 datset,
c ff 2048, dropout = 0.1, num _eadseeds = 8, encoder_layers
(patch_len= 16. stristirie= strdd - 512, 20 dropout = 0.1, num send_heds-- = 4, encoder_layers =3, activation = GELU) for the ETTh2, ETTml, and ETTm2 datsets.
The BSA module is implemented at the beginning part of the forward pass right after normalization from the Non-stationary Transformer [32] and right before the patch embedding. The BSA module is implemented for each channel, which is the number of input variables for this case.
TimesNet 48
The hyperparameters for this baseline model are as follows:
(top_k= bop_k=5, 5, num Leeneneeeeeesseeeeeee = 6, embed 'timeF', freq 'h', d model = 32, d_ff= = 32, dropout = 0.1, encoder_ layers = 2) for the Weather, ETTh2, ETTm2, and Energy datasets,
(top_k = num kernelseeeeleeeee
embed = 'timeF", freq = 'h', d d_model= = 512, d [fff== 512, dropout 0.1, encoder_layens = 2) for the Traffic dataset, 6, embed = "timeF", freq = "h', d_model 256, d_ff= 512, dropout 0.1,
(top_k= = nwm__emmeee encoder_layers = 2) for the ECL and PEMS03 dataset,
(top_k k= 5. num kernels 6, embed 'timeF', freq= 'h', d model 16, Iff 32, dropout 0.1. encoder layers = 2) for the ETThl and ETTml datasets,
16

---

(top_k- k 5, num kernels = 6, embed 'timeF", fieq 'h', model = 64, d ff= 64, dropout = 0.1, encoder layers for the Exchange dataset.
The BSA module is implemented at the beginning part of the forward pass right after normalization from the Non-stationary Transformer [32] and right before the input embedding. The BSA module implemented for each channel, which is the number of input variables for this case.
FreTS 122 : The hyperparameters for this baseline model are as follows:
(embed_ size = 128, hidden_sizz size-siiee--ee 256, sparsity_threhholle threshold = 0.01, scale = 0.02) for all datasets. Based on the paper, the channel-inde indeeeddent strategy is selected.
The BSA module is implemented at the very beginning of the forward pass, right before the token embedding. The BSA module is implemented for each channel, which is the number of input variables for this case.
7. RLinear (28 The only hyperparameter for this baseline model is the (dropout 0.1). We set the Individual to True so that there are separate linear models for each number of input variables. The BSA module is implemented at the beginning part the forward pass right after normalization from the RevIN [2I] and right before the linear layer. The BSA module is implemented for each channel, which is the number of input variables for this case.
A.3 Details on Training
Pre-training and Finetuning configurations. To show how our BSA module performs when added to the baseline models, saturated the models by greedy hyperparameter search.
The hyperparameter search space for the base model is as follows: the possible learning rate (0.03,
0.01, 0.003, 0.001, 0.0003), and the weight decay is (0.01, 0.003, 0.001, 0.0003, 0.0001, 0.00003).
The hyperparameter search space for BSA finetuning is follows: the possible learning rate for the SA-Matrix in the BSA module is (0.08, 0.05, 0.03, 0.01, 0.003, 0,001), learning rate for the rest of the model, i.e. original modules, is (0.01, 0.003, 0.001, 0.0003, 0.0001, 0.00003, 0.00001), learning rate for smoothing factor og is (none, 0.03, 0.01, 0.003, 0.001, 0.0001, 0.00001), initialization for smoothing factor a is (I0.9, 0.99, 0.999], [0.9, 0.99, 0.999, 0.999], [0.9, 0.95, 0.992, 0.9991, [0.8, 0.96, 0.992, 0.9984, 0.99968]).
Model selection Hyperparameter search is conducted based the validation set. While models trained using conventional sample shuffling evenly represent the entire time series distribution from which the dataset is sampled, BSA learns data in chronological order. Consequently, the final model tends to favor the distribution of the later part of the data. This can be seen as a mild version of catastrophic forgetting, commonly occurring in continual learning. To mitigate this effect, we assigned higher weights to the later samples during the validation process. The weights are represented by 0.5 + 0.5 x sin ( x vol idæ sample.
). continuously increasing from 0.5 for the first sample to 1.0 for the last val_een
Optimization The whole code is implemented in PyTorch 38). Each experiment was conducted on single NVIDIA GeForee RTX 3090Ti NVIDIA A40 or NVIDIA L40 GPU. The default batch size for baseline model saturation is 64, while for our method wwhic involves fine-tuning after integrating the BSA module is 256. If a baseline model is too heavy and results GPU memory overflow, the batch size is adjusted to fit within the available memory. We used the ADAM [22] optimizer and L2 loss (MSE loss) for the model optimization. The baseline saturation training epoch is set to 40 epochs for the Weather dataset, 20 epochs for the Traffic dataset, and 30 epochs for the rest of the datasets. The finetuning epoch is set to 30 epochs for the ETTh1, ETTh2, and Exchange datasets and 20 epochs for the rest of the datasets.
B Real world dataset experiments
B.1 Full experiment results
The experiments were conducted on total of public real-world datasets and forecasting models. The average results (MSE, MAE) of the experiments conducted with three random seeds 0, 1, and 2 are shown in Table 7 and the standard deviations are shown Table The values reported Table are labeled as Avg Table 7 For the Exchange dataset, only report experiments with prediction
17

---

Ampltuuee
2400
Amplttuee
2400
1600
1600
800
800
2.0
1.0
0.0
.40
3.0
2.00
60
1.0
2.0
30
(a) Denoisins
(b) Post-denoising
(c) Brownian Noise
Figure 8: The illustration compares the pre-denoising and post-denoising results alongside Brownian noise. For (a) and (b), the y-axis represents amplitude, and the x-axis is on negative log scale, indicating that moving toward the right corresponds lower frequencies. Conversely, in (c), the y-axis represents decibels, and the x-axis is on log scale, where moving toward the left corresponds to lower frequencies.
length of (96). Using prediction lengths of (192, 336, 720) causes improper best model selection due insufficient number of validation samples. The empty results indicate that the training is too heavy, and the results are not yet available. We expect to have the results ready by the rebuttal.
B.2 SA-Matrix and FFT visualization
To investigate how the SA-Matrix of BSA predominantly attends to specific frequency bands, employed the heatmaps, the Gaussian kernel density estimate graphs, and the FFT graphs as shown in Figure 9 TO
12 15 and B
The 2D heatmap depicts the learnable parameters of the SA-Matrix, defined as sa-matrixE g(2K+1)xD where represents the number of smoothing factors and D) represents the dimen- sion of the input features. Consequently, the y-axis of the 2D heatmap matches the length of D, and the x-axis corresponds 2K + 1, symmetrically arranged around zero. On the x-axis, positive values indicate the low-frequency regions, while negative values represent the high-frequency regions.
The Gaussian kernel density estimate graphs intuitively reveal which frequency bands the SA-Matrix predominantly attends to. The K smoothing factors were modified according to Equation S and symmetrically arranged around zero, serving as the data points for kemel density estimation (KDE). The weight values from each row of the SA-Matrix were converted to probabilities using the softmax function, and the resulting outputs established mapping of these weight values data points necessary for Gaussian KDE. This mapping facilitated the construction of the Gaussian KDE graph. Subsequently, the overall probability density of the SA-Matrix was estimated by calculating the mean across columns. Consequently, the y-axis of the graphs denotes the attention level of the SA-Matrix, whereas the x-axis indicates the scalar indices, encompassing the range of the smoothing factors. To better represent the variation in weight values across frequency bands, we adjusted the bandwidth of the KDE function to 0.4, based on the product of the Gaussian kernel's covariance factor and the standard deviation of the sampled weights in the matrix.
04 logio (aw)
(8)
The FFT graph is depicted with red line. FFT analysis was conducted for each variable to determine if the fine-tuned SA-Matrix aligns with the frequencies exhibited by the dataset. To compare the low-frequency parts, the negative log-scale was used on the x-axis, showing that progression to the right indicates decreasing frequencies. The Gaussian filter with a fixed standard deviation of 5 was utilized to smooth the signal's ''s amplitude.
The denoising phase was executed on the FFT output to analyze the correlation between the frequency tendencies of the dataset and the manipulated weights the SA-Matrix. Figure depicts Brownian noise, with the x-axis denoting frequency and the y-axis in decibels (dB). Both axes configured
18

---

Table 7: Full average results of three random seeds for prediction lengths S E (96, 192, 336, 720) with a fixed look- -back window T' = [96]. The average values across all prediction lengths are reported in Table [ The Exchange dataset is only done with prediction lengths of S E {96} due to its short data length. The setting for the Illness datasel is T {36). S E {24, (24,36,4 36, 48, 60).
MAR
02300
03323
02900
0.3355
02505
02000
0311T
012300
0.283
03216
02000
0.400
0A2X
05
0.300
04558
03764
0.36
0.103
0.5205
03179
(1305
0.004
0.003
0.3114
03300
03327
03724
03415
(1564
0A40
08350
08210
03112
02883
(1800
118090
01885
BSA
MSR
0:250
0.822
02356
0.405
0/551
0460
0.500
0
0.094
02524
9.00
0.435
0
0.514
0.0244
0.534
04139
0,3145
0.3770
0.531
04636
0.5383
04575
02142
0.065
0,200
02330
02390
03001
0.000
0.508
12000
02096
19000
2000
19559
1.5
1973
Pacc775T
MAR
0-356
0.2051
0.3450
0.276
0.381
0.309
0:306
0.314
000000
0.501
0.300
U.455
UAN
0.
0.50
0.310
1.404
0.433
0-4696
0.53
0/4522
0,310
0.3035
D. 0.39S
D-4256
0.385
0.470
06660
0.555
0.806
0.55
05009
11550
11
0.
MSR
0.2355
03552
02553
044099
04551
04300
05087
04487
01983
04277
0AXI
0400
05352
03359
04345
03972
03RKS
04402
04454
0.584
04731
03503
02204
00067
03513
03300
0
05710
1M11
0608
1915
2300
20033
1998
2015T
MAR
0,2996
0,350
02337
0,2004
02387
02344
03125
03390
0250
04407
044507
03539
0623
0,33
0.3991
04457
0L992
042N
04511
04561
0.5001
044992
0.399
03399
03IKK
02745
ONIN
0825
1-009
1499K
1,527
132M
1488
BSA
Conaommer
M4R
0.360
03215
02266
0.533
03500
05007
05985
0550
01979
0241
01901
0A12
051AR
03607
0-5003
0,330
03535
04185
0433
0500
0
02452
02188
03150
01ETO
0373
00465
1
1880
1200
135H
4058
4063
49571
MAR
0,3066
0.3735
0.291
02715
02871
0.303
03185
0.3966
0.396
03211
0.700
0.519
0
0S53K
DARR
0.
04044
0A4C4
0.424
BASS
0.599
0517%
0.800M
03W67
0344
0.345
0/24/2
0:356
03
0.307
0.295
03600
1992
15007
1589
1921
1815
Nove
MSR
0.2595
0.3521
0.3000
0.5537
0.5537
0.5830
0/0004
005055
0.0909
0.340
424200
0.00%
0.5145
0.5
1. 0.4270
0-4617
0.50
0.5090
0.000
0.2426
092999
0 JAS
0.3433
0:3177
1.2771
1991
1.7785
6333
SONM
50040
MAR:
0.3571
0.2652
0.2725
0.3811
0.399
01500
0.3740
002099
0. 44445
05
01
0/
01
11 0.30M4
0-
0.51KC
0. 26-609
0.044
0.505
1.0006
080045
BSA
0.
iTunshenne
MSR
012063
0.236
035M
02355
0.9988
01170
04331
04201
04283
03427
01
02212
01700
0.427
0480
05377
0
0.500
0,521
0.596
04230
04995
0.56T
044663
OL5SSS
92189
919999
DENSS
0L993
06266
1.7122
07850
3309
20867
2596
2209
2540
MAR:
0,2952
0.5773
0.206
0.3644
0220
0.382
02765
03139
02703
0.445
0
0521
0
04519
0L7V7
0L9942
04571
049966
05105
OL57A5
03145
021
015566
035503
045344
01655
01867
06995
1.037
00110
09923
00000
09669
MAR
0,392
03353
02556
03955
01772
04330
0465
04
01003
01
0229
0181
04316
06905
0561
01891
0551
0,316
0330
03861
0
0
0552
0.395
0225
DAWIS
0221
0
2249
0281
08300
030SS
3308
0832
3305
2002
23705
21722
24509
MAR
0.5447
0,2022
0,5
015988
03357
035TT
03356
0,303
03272
0.999
0L6305
0.585
0500
0.550
04700
0477
1L4
0L6584
85515
0L399
03354
0S66
06311
0655
09973
0900
0880
1966
0998
BSSAA
MSR
0:25
0.5930
0.500
0/8366
010025
0.3013
0.3336
0A554
0.5M
0.720
0.003
0.5314
0160%6
8999
0.215
0,044
0.21
0-3427
0.2
2808
01254
0,0935
1.3744
3,597
36800
2.55%
2000
2:796
TinesNee
MAR
0.5508
0:3840
0,5300
013992
015004
0.31
NO
0,467
USIT
R.5588
R6835
0.5455
0,4522
0.46
0.5852
0.2157
0.3000
292922
3350
0131
066897
05000
000024
05977
659940
01895
099200
MSR
0,293
0.583
0,590
0.511
0500
DASESS
00099
02100
DEES
0.000
04666
0:557
05006
0.5%
04258
04300
0
066062
0.507
01966
0L5R0
0.E99
00944
0,2006
02109
02304
05319
00990
3449
08800
35955
34000
23N0
200%
2718
MAR:
0.3801
0.293
0.300
0.506
0263
0.W
04407
05
04422
0,529
0.1
0.504
022
4700
1.582
1550
15900
1.500
BSS
MAS
03307
02:55
0.530
01575
04654
0.50
05:00
0 01860
0198
04763
OSIR
06455
0.
03306
03385
04074
04528
D.586
0253
03599
02407
01SS
02135
0.300
0
0L230
07221
09995
1576
0939
1998
S0N8S
4308T
1389
49N4
MAR
03367
0255
05169
05/20
05396
013300
015:24
0,3275
0.360
0445
015822
01505
(L3900
(1A85
OLENSS
01334
(L401
0.313
0251
0.574
03456
0.58
0ASSS
0U9C
0759
16000
1808
1.985
1900
1800
0,255
0.2099
0 5059
05088
0.508
0.563T
0.300T
0.39
0.175
D1530
0.5128
0.5755
U-ESS
0.565
0.855F
00.554
R.3400
0.333
0.3055
0,25/6
0.66
0.258
0.390
1.841
1.80
0.996
5.00
5,006
5.04
5.05
5.03
0.338
0,2967
0,3922
055627
03998
01439
03451
0.000
0.421
D5210
06148
0,112
03331
0.432
0.3071
DA300
0.170
0.1530
0,2500
0L3026
0.2035
DA359
DESSS
05688
0615
0.5400
DA4SS
03338
06221
1021
18080
BSA
MSR
0.541
0,3355
0,2066
06010
0822
06833
0.7355
0005
02120
03669
0215
0.519
04955
OSSSS
0.994
0.5366
3
0.3099
0.5004
04330
0.008
0.533
047766
0
0URS1
0C1E3
0.5601
0711
0.554
06137
04299
1.808
0.98
23333
2588
2685
24031
2,009
RLines
MAF:
0.27
0.4555
04158
0485
0,399
0.400
0A 10
0.31
0,3167
0.30C
0,402
DJESS
04218
0.SUP
0.357
0,3007
0.313
0:2055
0.395T
0.634
7
6.74I6
0.647
03055
0.00
1663
09R5S
1.039
1.00
1.055
0.365
0.400
06089
09700
0714
08152
01228
0229
02355
02M6
0.430
05004
(1588
0774
05512
04083
0307
0333
05225
05955
0190
02204
JOWS
LO09S
DREET
10000
00922
0543
1.305
08163
2752
250S
26499
2501
2639
MAF:
(L2991
0,3633
02815
0/2122
011187
03218
03365
031M
04550
01507
055955
0.50
R41ST
04558
(LESH
0.086
(L3992
03304
0.2000
(LAST
0.311
(LOSTO
0L7033
08900
06670
11114
1906
11311
11774
BSA
MSR
0.255
0.297
02119
00007
05517
0.821
0.760
0/5000
0
0.456
0.55
05615
0.5295
0.30S
D
0.5334
0-4444
0-421
0.363
0.366
0.335
1.305
0.87S
20001
2778
25571
2461
2.971
Diincer
MAR
0,308
0.3084
0.0300
0/122
0.500%
0.326
0375
0338
0.170
11598
0.505
04206
0A04
0.006
0ASS
0:500
0.5055
0.4EN0
0,02
0,321
02110
0.5077
0:539
0,0000
0.50%%
059800
0,742
09339
0.029
10734
1998
11882
11318
11909
MSF
0.3614
0,522
0.244
96 0.0891
06153
0.7100
0.662
0.306
0.296
96 04529
04004
5.586
01621
0.539
0,50M
0.30K
0.EX2
04003
0.556
0.4077
0.585
0.2775
0,0650
04463
0:3973
04420
044844
009411
1.4187
0.6737
21 2.755
2,346
2.9956
2.9.60
2.408
193
596
730
Arg
192
536
730
Arg
596
730
Agg
192
3.6
720
Axg
96
596
730
96
192
396
730
Axg
595
720
Aoe
96
96
102
335
730
Avg
195
536
730
Ang
96
46
00
Avg
Wodhe
Tralic
BCL.
BITHI
ETTH2
m37ml
Exchange
PEMSOS
Elsign
Illess
19

---

Table 8: Full standard deviation results of three random seeds with prediction lengths S (96, 192, 336, 720} and a fixed lookback length T' (96). The Exchange dataset is only done with prediction lengths of 5 E {96} due to its short data length. The setting for the Ilnness dataset is T = {36), S E {24, 36, 48, 60).
MAR
00000
01008
(U005
02004
OIROO
(URQ)
(192)
00005
DAR20
OR0OS
01008
(1001
0J00
(10)
02004
BJROLS
(UIOS
ORDS
ORROS
0012K
00114
O0110
BSS
MARE
00012
03056
00008
0.008
0.0014
0:0014
000011
00000
03002
00000
01002
0.0015
0.003
000027
0.003
00005
0:001
000000
0.003
0,0038
0:0017
0,000
00011
0.008
000025
0.000
0:0012
0:0007
000014
00000
DUMIS
00013
00023
000000
IS 00041
PachTST
MAR
0:000
05300
01000
BL30S
D1000
0:00S
0,0227
03005
B:006
010034
U:3020
110000
8:006
110000
0,0035
0100/6
013002
D:00K
1200
DONGS
HOISI
0.00
MSS
0U0E1
010000
0002SS
0000Z
0007
DAXKSS
D0004
QLORE
DUOD9
DUOSO
DUOTH
00024
DOOLE
DUXON
DUOTO
0OXSS
0L006
DU0OL
00000
GOOD
DUOOKS
00009
0040
04097
MAR
0AKEET
00001
0.039
0L00N
000077
GUXDS
DUXON
DU007
DUROUS
OUOESS
0012
0U2077
GOIS
OUTSRI
0OND
GOIZ
(L0380
BSA
MA0F
0190
000203
0001S
010277
(0000
BUXXZ
OUNILS
OUOLZ
0
00130
041445
0J0SS
0019
DIOOX
GANNT
0
0
0m161
SEEES
0012
OOSIO
OUTEST
OAIOT
OUTRO
00338
0.2000
OUTT
0D00
01921
01S1S
Coudonee
0
MAR
01005
00/04
00008
OURLLS
OOISS
0
00004
00125
03004
BOOKS
0JR8S
O
O
07400
0
00299
OUZIC
OUINT
lexe
MSR
0.0808
0.300
12005
BOIES
BUAST
BL3K2S
0.315
01001
BLOCS
01001
BUSETT
000014
0.3156
0,3055
1:0053
0/0017
0.0034
0.0010
0,172
0.004
0.353
U:005]
D/0612
BUSKS
BORSS
0.EES
D J9IS
D. 0.0000
D. 120
MAR
01001
0,3005
010011
013013
0000
01001
11000M
01,025
0:0010
BLO0K
01000
BL3000
11,0015
0:0030
D. 0,0020
0,,009
11:009
0:0000
0.0007
010010
D. 0:0006
0,3013
01003
1100122
BU005
0,0017
0.002N
(00001
BSA
iTuasfeencc
MSF
0.0012
0.0000
00000
0/00//
010011
0U0022
0L0033
010012
000000
000077
QU000
0,0011
OL9GER
DUMOSS
0U0009
000099
RU0000
0,0000
0.0019
0.0039
0U000S
0U000
05739
011422
025522
MAR
0.0003
0
(1000
QIM1T4
DUOMES
0101
DUNTO
QUMINS
QUMEES
OUM
DURIPO
0URUU7
0
GURES
01622
0000
OUMESO
010899
MASE
0.000
00011
0/000
09000
0100S
OIROLS
0
GROOS
GIOOS
OUOOSS
0
0
BIEKS
DUDIO
GIOMS
GIROS
GIOSS
0L2006
GUNET
OBM
0255)
MARE
0.003
00009
(INCE
OLIRESS
0O0T
CLUKES
CUMES
DLOOSS
CLUSKSS
ELIVDO
GINOS
QUIDS
OLOOSS
ORESS
QUESO
GUMSS
(1063
BSAA
MSR
01046
0.0053
00061
DOONG
BAOOS
BOYOS
DOOAS
BLOKSD
0/0441
DIDIRO
BASESS
BU0KST
0/006
DIORKE
LOOKO
BOOKS
DIDIAI
BIOOS
0.0003
R,00S4
0,00/9
BLOOSS
BOXOA
BOOES
0/0150
D.00S
13300
DASOS
DICGOO
04712
ThoesNo
MAR
B0000
0/00
0.0005
BOCKS
B1003
BOOSI
BIDI
BUCKS
COOS
0100 32
B1008
BLOK22
0/007
BOOES
1/01
RIDIS4
0:007
R.000
0,00S
BUCKES
LORKS
BOOKN
ROOD
NOT
lest
MSR
00009
0/027
0,0000
00120
04003
OUAEE
GOIST
QUACO
QLORS
CUOLED
OV08S
BOOSS
GLOSS
00063
DUOOSS
00012
00056
0.000S
060044
06004
0L004
CLO0N
00034
GUODD
DINN
JAUE
GUOSS
04454
MAR
(1810)
CURIS
CURND
AURIN
QURISS
QUIUIT
QUOUS
QURESS
OUBIR
QURINE
QUINI
0URR2
00031
CLOIG
01417
QLOTS
LIGI
016N
BSS
MFST
00005
BUEOS
00009
GULKIS
BAX04
01022
0
DAKS)
BASS
BUSS)
00012
DAKOL
00172
01259
00006
00008
0
00009
GIMI
DUKOLS
GANGZ
GAIST
GOSTS
OOEIS
033H
01226
0292
05142
MAR
0000
0000
(LU0)
0006
QUIOIS
QUXAN
GLOGES
0O91
(UIS)
ROMAS
QUOIT
0U002
GUMPP
0U89T
00112
00003
(U00)
001S
AUMSS
00775
MSR
0000)
03001
0,000
00001
00000
B130S
0.005
BLAKS
UI BLO0SS
BL002
B13025
203
0,00S
0.3044
0.305
B.00
BUON2
0.2045
0,0000
3000
BOO0
RUS50
0.20%6
3222
D.ING
0,000
04410
MAI
0A000
01000
00000
UIRCO
05000
BARKO
DARKO
RAROO
DAICO
DARCII
DANKI
BANKA
2
BARKS
00000
BAROH
DANCH
BANKI
BARCO
nAKK
NUROO
0,011
DARCSS
DNICO
DURCO
DANKN
NURCO
O
BAIOS
BSA
MSR
QUANO
0.000
0,000
0,000
00000
DUALO
0.000
0U00
0000
DUNOI
0L0022
0U000
DUANO
000001
000015
QU0N22
0U009
CLOSS
000016
QUANZ
QU0E0
0.042
RLikes
MAR:
0:00
0.0001
6.000
D.ON
0.000
8.000
DAKNC
CORON
DAMGI
DUMG
0,0001
0.000
0,003
0.0N3
0.00IC
6,0012
0.0020
0.0011
ULKIC
D
DAUS
(000)
00002
00000
(UCO)
(100)
(000)
DUCCIS
(OCO)
DOODS
(0000)
CUICSS
0000
00/02
0001
COCOSS
000014
MANALS
0006)
(104H
MAR:
00000
01000
CUICIS
EURANO
CUODO
0U00
COLOS
CUICES
TUNOS
GOOLS
OO0O
(UMIN
BSA
MARE
00000
0.000
0,000
0,000
0.000
BAOOO
0300
BLOOSS
ELOOO
DOOIN
DOYSI
BO000
0O000
0100
BAOOS
IOOIS
0,005
0,002
0,004
(2000
FOOOS
0,002
BOOO
BOOM
0O00
DOOM
0.
UONI
0,971
Diiaccl
MAR
01000
01000
DIME
0
DRIOS
DRINK
DRWKS
UIKEE
DRINT
OATAA
00002
NUNISS
00004
UNIEY
UOING
0ISSS
MASFF
0,000
0/000
0.0000
000000
01000
01000
0.0011
000037
0/00P
06
20059
0999
0/E31
000'S
0.006
0:0011
0/0006
96 0UAESR
0.082
0OVI
0,002
0,0010
0,0001
0.000
0L000
0/044
0.00S
0.096
0.030
0.303
0 1945
96
192
336
720
Ang
96
103
535
730
Ang
%
192
336
720
Avg
96
192
5.5
730
Ang
96
192
396
720
Avg
96
192
306
730
Avg
192
396
720
AvR
96
96
192
355
700
Ang
96
192
535
Any
24
36
48
00
Ang
Waater
Traliic
BCL
EITHI
ETTH2
ml7ml
mT
Eachange
PEMM53
Ilnoss
20

---

Table 9: Full average results of three random seeds with prediction lengths S E (96, 192, 336, 720) and a fixed lookback length T {96} on synthetic data. The original results are from the public datasets E and ETTh2, as shown in Table 7 Synthetie datasets were created by adding sine waves with periods of 100, 300, and 1000 to the original ETTh datasets. Gains mean performance improvements by using our method.
iTranfoormr
iTranforter BSA
BSA
Gain(S)
base
Gain(%)
MAR
MSE
MSR
MAE
MSE
MAE
MAE
MSE
96
0.4316
0.4453
0.4277
0.4426
0.909
0.605
96
0.2399
0.3278
0.2347
0.3236
2.062
1.260
192
0.4906
0.4857
0.4809
0.4812
1.973
0.931
192
0.2950
0.3641
0.2901
0.3624
1.654
0.467
original
720
336
0.6991
0,5451
0.5217
0.6174
0.6979
0.5401
0,5243
0.6204
0.908
0.172
-0.493
-0.504
original
ETTh2
720
0.4160
0.3316
0.4389
0,3880
0.5271
0.4144
0.4391
0.3884
0.364
1.370
-0.040
-0.103
avg
0.5416
0.5175
0.5367
0.5171
0.991
0.135
0.3206
0.3797
0.3166
0.3784
1.362
0.396
96
0,3108
0.3889
0.2980
0,3800
4.111
2.286
0.1608
0.2738
0.1565
0.2724
2.716
0.530
ETTh!
192
0.3470
0,4172
0.3298
0,4062
4 4.968
2,635
ETTh2
192
181880
0.25993
D. 18199
0.3009
2.198
-D.516
100
720
0.4825
0.5162
0.4784
0.5150
0.844
0.230
720
0,2580
0.3526
0.3156
0,2558
0.3522
0.855
0.137
0,3807
0.4419
0.3695
0.4355
3.474
1,538
0.2042
0,3103
0.2008
0.3102
1.776
0,064
0,3674
31,330
0.3663
35.319
19.272
ETTh
300
336
192
0.5741
0.6278
0,5608
0.5873
0.4156
0.4217
0,4729
0.4788
26.537
33.799
14,763
19.476
ETTh2
300
336
192
0.2971
0.3219
0.3921
0.4107
0,2189
0.2287
0.3356
0.3444
32.010
23,023
18.290
12:178
0,5273
25 068
0,4410
23,582
avg
0.6101
0.5783
0.4330
0,4839
29.183
16.408
avg
0.3126
0,4025
0.2248
0.3398
28.483
15.685
96
0.3370
0,40822
0.3295
0,4033
2.225
1,135
96
0.1880
0,3003
0.1799
0.2927
4.305
2.529
ETTh1
1000
336
730
1. 1,2900
0.8177
0.8721
0.6785
0.6741
0,4392
0.5441
0.6411
0.5591
47.742
33,455
12.487
26. 484
17.591
6.992
ETTh2
1000
720
336
0.6335
1.1044
0.5958
0,4108
0,7933
0.4554
0,4320
0,5046
0.4856
58.760
31,820
18.492
36.389
avg
0.6196
0.4967
23.978
13,050
avg
0.5251
0,4144
27,843
on log scale. Converting the y-axis from decibels to amplitude is akin implementing linear scaling, which reveals a steeper rise in noise at lower frequencies. This noise generally arises in the low-frequency bands of most real-world systems, such as electronic systems and environmental sciences [24]. Figure g (a) shows that noise escalates within the low-frequency regions across vari- ables, leading to the assumption that unique Brownian noise originates in each system. Consequently, a linear approximation was used for the denoising function, which was performed manually. The outcomes of this process are presented in Figure 8 (b).
C Synthetic dataset experiments
We constructed synthetic datasets by adding sine waves with periods of 100, 300, and 1000 each channel (variable) of the ETTh1 and ETTh2 datasets. The scale of the sine wave is set to the standard deviation value of each channel. The phases of the sine waves for each channel are randomly sampled from uniform distribution from O to 2 to make decorrelated sine waves. The full results are reported in Table 9
C.1 SA-Matrix and FFT visualization
As illustrated in Figure 14 the LUFL., MULI., OT, and LULL channels are organized into columns, with each graph positioned within rows that correspond to the periods added to the dataset. Figure [4] (a) shows the SA-Matrix from the iTransformer model fine-tuned with 720 prediction length on the ETThl data that does not include added sine waves. From (b) to (d), the models were trained with synthetic datasets increasingly augmented with sine waves at periods of 100, 300, and 1000. As the number of added periods increases, there is a shift in the SA-Ma Matrix weight distribution toward low-frequency patterns in each channel.
D Analysis and Ablation Studies
D.1 BSA insertion site analysis
Table T0 shows the full results for different BSA module insertion sites in the iTransformer 31 and DLinear [5]
models on the Weather Dataset. Table 2 is from the average value of Table T0 The
BSA module insert position can be found in Figure]
21

---

Table 10: Full results for different BSA module locations for i'Tranformer [1] and DLinear [53 on the Weather dataset. Positions 1 and 4 are the only available options for the DLinear Model.
(Transfonner
MSE
MSE
MAE
MSE
MSC
MAE
MSC
MSE
MAB
MSE
1706
0:3996
0.159%
0:3987
0.1589
0.3027
0.158
0.190
0.1667
0.307
0.1545
0.199
0.1676
0.206n
0.1582
0.2911
0.22003
0.2544
0.3082
0.3466
0:2034
0.3463
0.193
0.3441
0.2149
0.2526
0.2003
0.3451
0.2256
0.2548
0.2110
0.2504
weather
0.22762
0.2952
0.2526
0.3856
0.2510
0.2846
0.2528
0.2857
0.2719
0.2940
0.2519
0.2855
0.2756
0.294%
0.2534
0.2854
0.3551
0.3571
0.3897
0.3580
0.3513
0.3440
Avg
0.2556
0.3683
0.2350
0.7681
0.2588
0,2538
0.2762
0.2415
0.2702
Dlinear
hase
2
MSR
MAE
MSB
MAF
MSE
MAE
MSE
MAF
MSF
MAE
MSE
MAR
MSE
MAE
MSR
MAR
0.1630
0.2363
0.1565
0.2306
0.1539
0.2203
192
0:20911
0.2882
0.1985
0.2724
0.1911
0.260)
weather
356
0126634
0.337]
0.2590
0.3251
0.2375
0.2953
720
0.3421
0.3868
0.3167
0.369/7
0.2977
0.3463
Avg
012444
0.3084
0.2327
0.2994
0.2196
0.2815
D.2 BSA variable input lengths analysis
Table 11: Full result for the performance of the base model and BSA according to changes in Input
length. Experiments were conducted on Weather and PEMS03 data using DLinear, RLinear, and iTransformer models.
Dlineur
RLinear
iTransformer
base
BSA
base
BSA
base
BSA
MSE
MAE
MSE
MAE
MSE
MAE
MSB
MAB
MSE
MAE
MSE
MAE
0.2187
0.2950
0.1616
0.2332
0.1898
0,2322
0.1783
0.2233
0.2054
0.2271
0.1713
0.2124
Weather
192
0.2477
0,3203
0.1994
0.2715
0,2315
0.2654
0,2206
0.2583
0,2490
0.2695
0.2178
0.2541
48
336
0,3086
0.3696
0.2460
0.3094
0,2897
0.3064
0,2768
0,2980
0,3075
0.3099
0.2716
0.2947
720
0.3817
0.4204
0.3085
0,3585
0.3699
0.3566
0.3579
0.3499
0,3831
0.3595
0,3459
0,3442
Avg
0,2892
0.3513
0.2289
0.2932
0,2702
0.2902
0,2584
0.2824
0,2862
0.2915
0,2517
0.2763
96
0.1630
0.2363
0.1519
0,2203
0,1656
0,2108
0.1594
0.2065
0,1706
0.2096
0,1598
0.2037
Weather
192
0.2091
0.2832
0.1912
0.2601
0.2119
0,2512
0,2013
0.2444
0.2203
0.2544
0,2052
0.2466
96
336
0.2634
0.3271
0.2375
0,2993
0.2679
0,2911
0.2823
0,2762
0,2952
0,2526
0.2856
720
0.3421
0,3868
0.2979
0,3463
0,3465
0.3412
0,3315
0.3338
0,3551
0.3473
0.3252
0.3371
Avg
0.2444
0.3084
0.2196
0,2815
0,2480
0.2736
0,2366
0.2667
0,2556
0.2766
0,2357
0.2682
96
0.1514
0,2208
0,1466
0,2138
0.1522
0.1982
0.1471
0.1960
0,1650
0.2100
0,1521
0.2008
Weather
192
0.1952
0,2660
0.1856
0,2530
0.1963
0.2383
0,1898
0,2360
0.2098
0.2510
0.2023
0,2450
192
336
0.2492
0.3100
0.2315
0.2914
0,2502
0,2785
0.2393
0.2733
0,2625
0,2903
0,2428
0,2810
720
0.3324
0.3771
0.2958
0.3436
0.3285
0.3313
0,3111
0.3242
0,3356
0.3410
0.3089
0.3291
Avg
0.2321
0.2934
0.2149
0,2754
0,2318
0,2616
0,2218
0.2574
0,2432
0.2731
0.2265
0,2640
0.5159
0.5565
0.4340
0,5021
1,2342
0,8375
1,0040
0.7564
0.4214
0,4518
0.2934
0.3715
PEMS03
192
0.5673
0.5921
0.4646
0,5287
1.7909
1,0790
1.3785
0.9286
0.4010
0.4454
0,2551
0,3513
48
336
0.5056
0.5437
0.4018
0,4767
1,2898
0.8454
1.0455
0.7490
0.4196
0.4855
0.3084
0.3801
720
0.5524
0,5811
0.4398
0.5047
1.5496
0.9620
1.3303
0.8777
0.4699
0.4848
0.3507
0.100
Avg
0.5353
0.5684
0.4351
0,5031
1.4661
0.9310
1.1896
0.8279
0.4280
0.4576
0,3019
0.3782
0.4405
0.5097
0.4121
0.4875
1.0366
0.7815
0.6789
0.6359
0.2213
0.3226
0.1569
0.2689
PEMS03
192
0.4658
0,5259
0.3876
0.4677
1.0995
0.7937
0.7128
0.6326
0,2687
0.3566
0,18665
0.2949
96
336
0.3973
0.4669
0.3461
0,4317
0.8227
0.6434
0.5861
0.5408
0.2479
0.3503
0.7979
0.2957
720
0.4420
0,5048
0,3922
0.4703
1.0101
0.7432
0.7194
0.6184
0,3093
0,3716
0,2488
0.3330
Avg
0.4364
0.5018
0.3845
0.4643
0.9922
0.7405
0.6743
0.6069
0.2618
0.3453
0.1975
0.2981
96
0.3904
0.4570
0.2660
0.3698
0.4078
0.4618
0.2674
0.3609
0,14663
0.2569
0,1218
0.2323
PEMS03
192
0.3156
0,3932
0.2543
0.3552
0.3290
0.3916
0.2581
0.3394
0,1637
0.2641
0,14
0.2484
192
336
0.3078
0.3903
0.2650
0.3641
0.3276
0.3890
0.2723
0.3495
0.1738
0,2680
0.1623
0.2596
720
0.3588
0.4370
0.3221
0.4110
0.3991
0.4418
0,3466
0.4038
0.2226
0.3000
0,2091
0.2924
Avg
0.3432
0.4194
0.2768
0,3750
0.3659
0.4211
0.2861
0.3634
0.1766
0.2723
0.1596
0.2582
Table [1 shows the full results of the performance of the base model and BSA according to changes in Input length. Experiments were conducted on Weather and PEMS03 data using DLincar [53],
RLinear (8). and iTrrnsffrmer (31 models. Table in the main text shows the summarized results.
D.3 BSA computational cost analysis
Table 12 presents a comprehensive analysis of BSA's computational cost. We measured the model training time (sec/1 step), peak memory usage (GB), and the number of parameters (M) for both the base model and the model with BSA applied. The experiments were conducted using the lightweight Weather dataset with 21 channels and the heavy PEMS03 dataset with 358 channels. Tests were
22

---

Table 12: Full result for the model training time (sec/1 step), peak memory usage (GB), and the number of parameters (M) for both the base model and the model with BSA applied. The experiments were conducted using the lightweight Weather dataset with channels and the heavy PEMS03 dataset with 358 channels. Tests were performed on Timesnet, i'Tranffrmer, Crossformer, and PatchTST models.
TimesNet
iTransformer
Crossformer
PatchTST
Time (see/Istog) step)
BSA
gain(%)
BSA
gain
base
BSA
gain(
base
BSA
gain(
96
0.066
0.070
5.37
0.024
0.029
20.71
0.076
0.079
3.58
0.045
0.046
1.52
192
0.076
0.072
-456
0.024
0.025
3,92
0.072
0.078
0.046
0.048
Weather
336
0.081
0,078
-3.30
0.025
0.027
8.05
0.077
0.080
3.66
0.050
0.052
5.12
720
0.104
0.107
2.47
0.025
0.033
30.74
0.075
0.080
6.50
0.048
0.049
2.88
Avg
0.082
0.082
0.00
0.028
15.86
0.075
0.079
96
1.160
1.147
-1.06
0.077
0,080
3,96
0.163
0.162
-0.95
0.878
0.877
-0.10
192
1.791
1.717
.4.12
0.079
0.082
3.69
0.296
0.290
-1.98
0.889
0.882
-0.74
PEMS03
2.374
2.502
5.40
0.096
-1.26
0.492
0.485
-1.45
0.881
0.883
720
4.397
4.425
0.63
0.115
0,118
2,57
0.965
0.929
-3.78
0.908
01900
40.57
Avg
2.430
2.448
0.21
0.092
0.094
2.24
0.479
0.466
-2.04
0.889
0.886
Memory (GB)
gSin(Sk)
gain
gain(
base
gain(S)
96
0.43
0.43
0.04
0.20
0.25
24.65
0.27
0.28
2.85
1.59
1.60
0.36
192
0.57
0.58
1.18
0.21
0.32
54.08
0.48
01.49
1.11
1.60
1.61
0.47
Weather
336
2.15
0.22
0.29
30.93
0.87
0.88
0.81
1.63
1.63
0.30
720
1.38
1.37
00.07
0.25
0.30
19.57
2.28
2,28
0.24
1.68
1.68
11.25
Avg
0.82
32.31
0.97
0.98
1.62
3.66
3.57
3.68
3.04
7.23
7.25
0.32
25,40
25.52
0,46
192
7,32
7.59
3.74
3.65
3.75
2.76
12.24
12.26
0.22
25.49
25.60
0.44
PEMS03
10.01
-2.46
3.80
3.89
2,30
19.96
19.98
25.61
25.71
720
17.07
16.46
-3.58
4.18
4.23
1.27
42.34
42.36
0.05
25.95
26.02
0.24
Avg
10.11
10.02
0.34
3.80
3.89
2,34
20.44
20.46
0.17
25.61
25.71
0.38
Parameters (M)
base
BSA
gain((%
base
BSA
gain/ %6)
base
BSA
gain(%))
base
BSA
gaint
1.354
1.368
1.05
4.848
4.852
0.08
0.283
0.301
6.41
9.46
9.48
0.15
192
1.363
1.381
1.34
4.897
4.897
0.00
0.291
0.313
7.62
10.05
10.07
0.18
Weather
1.391
1.03
4,971
0.302
0.317
4.67
10.94
10.96
720
1.414
1.433
1.29
5,168
5.172
0.08
0.333
0.347
4.24
13.30
13.32
0.14
Avg
1.377
1.393
1.18
4.971
4,973
0.04
0.302
0.320
5.73
10.94
10.96
0.16
9.71
192
151.6
151.9
0.16
4,883
5.125
4.95
11.45
11.69
2.11
10.05
10,30
2.40
PEMS03
336
151.6
151.9
0.16
4.957
5.199
4.87
12.57
12.81
10.94
11.18
2.21
720
151.7
151.9
0.16
5.154
5,396
4,69
15.58
15.82
1.55
13,30
13.54
1.82
Avg
151.6
151.9
0.16
4.957
5.199
4.88
12.57
12.81
1.96
10.94
11.18
2.25
performed on Timesnet [48]. iTTrasformmer 31. Crossformer S4
and PStchTST 36 models. Linear
models were excluded from the experiments they very lightweight and their training cost not an issue. Table in the main text shows the summarized results.
D.4 Pre/Post BSA activation signal visualization
demonstrates the transition between pre-activation F (left) and post-activation F" (right) during the inference process. The figure shows the spectrum randomly sampled activations from the DLinear model trained with a 720 prediction length on the SWDR (W/) channel of the weather dataset. The x-axis signifies timesteps while the y-axis indicates frequency. To enhance visibility in the low-frequency region, y-axis values below 0.1 are presented in log scale, whereas values above 0.1 are depicted a linear seale. A darkening in the fighfrreuency region is observed from pre-activation post-activation, suggesting that the activations may undergo denoising effect in the nigh-frequency areas as they pass through the SA-Matris
Figure 16
23

---

832
500
432
600
600
630
708
700
700
(mbbar)
VPact (mbar)
VPPGE (mber))
270
100
300
130
270-
200
200
270
300-
500
100
390
500
500
000
800
700
40 (g/
+20C (mim Wmalt
tho (mmm)))
100
200
ITT (MUS)
M45 Idegt
rann (mmm)
miing
SWDR
(molmms))
600
700
238
330
437
500
432
Figure 9: The 2D heatmaps of the SA-Matrix from the DLinear model finetuned with a 720 prediction length the Weather dataset.
24

---

1.00
0.32
0.45
2206
2200
1220
0424
2408
2430
2400
220
2420
0.28
5:08
0:02
(mbbart
6200
4020
6000
4800
1.24
0.43
6.22
18
21re
2030
2448
1000
1500
1.06
0.15
800
3.00
0.02
1.07
0.00
6th (gr
6200
M20C
(mmolmmo
150
(ptmm13)
4006
8WV (rvi))
4050
2.00
3200
0.42
3230
deromanttee Lrand
3.50
0.35
2.00
1866
1630
106
0.08
2.00
0.02
8.00
maa.. Emi55
wid (dlgg
FSR
(m/))
rhinin
1000
0.43
2.00
3200
0.32
3220
2200
3220
0.32
0.2
1100
1808
0,22
660
860
SWTA (Wom)
(moom..3
1206
0.44
12:30
1206
1200
1.30
2400
anvitten call
2400
All-ttn
800
302
100
300
M
1010
2400
2.00
Figure 10: The kernel density estimate graphs of the SA-Matrix from the DLinear model finetuned
with 720 prediction length on the Weather dataset and FFT graphs of the data.
25

---

(deg()
4000
1630
0.42
1.42
2016
-206
1220
0.22
30teiee Level
2408
2400
Alavton
2.2
21
1600
1.50
0.12
666
1.00
0.02
8:00
Virmm (mber)
(mbbrt)
Vindel (mbart
2240
3230
2200
3230
Alteroon Lovel
orrenee toarl
2.2
1868
1620
C.2
1500
0.18
1.00
0.03
0.02
8.00
69 (g1g
H20C
(mmotimo
1h0 (ptm")))
8W (nvi))
1206
0.42
1000
1050
3200
3220
0.35
2406
2408
2428
2.2
100
3.00
8.02
4000
0.02
1630
2.00
1216
6.40
SST Ande fode
2tre
2030
2.00
2428
2.2
0.22
20
0.22
6.12
800
510
SWDR
0.42
2200
3230
2200
3200
Orre Lewes
022
0.22
160
0.32
1000
0.00
6.22
660
07
4000
1016
2408
Minsdon
1000
2.00
Figure 11: The kernel density estimate graphs of the SA-Matrix from the iTTranformee model finetuned with 720 prediction length on the Weather Data and FFT graphs of the data.
26

---

HRPP
4000
MUULL
MUUT
4006
MALLL
4050
345
0.45
1/6
0.45
3200
3230
gooedd Level
1408
1408
2483
Kloniann
1808
1500
6200
4020
6000
0.2
2.00
800
302
800
Figure 12: The kernel density estimate graphs of the SA-Matrix from the DLinear model finetuned with 720 prediction length on the ETThl dataset and FFT graphs of the data.
HUFL
HULL
MUFL
MALL
4200
4006
4050
245
3200
3230
Ar Laate
3.0
21re
0.43
2033
2428
1100
0.11
2.00
6200
8933
0.62
210e
Dendy
2.30
0.35
820
802
100
2.00
Figure 13: The kernel density estimate graphs of the SA-Matrix from the DLinear model finetuned with 720 prediction length on the ETTml dataset and FFT graphs of the data.
27

---

(a]
4300
4300
4330
2.00
4033
Original
3:30
8.2
1877
1933
400
930
990
0.20
890
0.90
3000
-300
3300
1000
+100 +100 Porlod
2000
-2000
3000
0:30
2.0
0.5
0.00
0.00
3.0
0.00
(6)
0.90
0.65
-300 Period
0.90
2400
-200
2405
1902
1800
1620
1953
330
(d)
8331
4033
11000 Period
0.45
0.0
1200
242
1300
0.43
1000
0:30
200
900
0.0
Figure 14: The kernel density estimate graphs of LUFI., MULL,, OT, and LULL channels of the SA-Matrix from the iTransformer model with 720 prediction length on the original and synthetic ETThl data. Row (a) represents the original data, while rows (b) (d) display synthetic datasets. These datasets were generated by adding sine waves with periods of 100, 300, and 1000, respectively.
U.FL
MULL
OT
U.LLL
1000
1030
1000
1000
0.32
024
200e
2400
2426
0.34
1000
1508
0.08
6.00
6.0
100
0.03
0.02
8:00
Figure 15: The kernel density estimate graphs of the SA-Matrix from the i'Tranformer model finetuned with 720 prediction length on the ETTml dataset and FFT graphs of the data.
28

---

1st
1st Posr on
1300
$,000
2.10
$,002
Temcile
Temeslap
26th Pro Adivaaiin of 54 Matx
26th Post Actividiiiiinnn of SA-Macrte
1.500
9.000
Prggeery (HQ
2.030
6030
8260
12th PreAdivaiion of SA-Hatr
72th Acst-Attivaiion of SA-Naciit
0.000
3300
5.000
1.00
0.00
2002
4003
6020
8000
12000
:2202
14083
2046
4000
6000
5000
23000
12200
34600
167th Pre-Activation SA-Marrix
167th Post-lctivatin
1300
0.003
0,001
1.000
2002
6000
8203
12020
12302
14083
0,000
4000
6000
32000
:2000
34000
Tesedes
Taxeding
2901
2.000
0,00
2.000
14999
6.000
12066
Timedta
Timssip
581th Predectioniin tivntinn SA-Maerri
601h Pesiokctiedinn SA-Maltic
3500
0.500
2.000
Timastop
Figure 16: Illustration of the SWDR (W/)) channel of the Pre-Activation (left) and Post-Activation (right) through SA-Matrix from the DLinear model trained with a 720 prediction length on weather dataset.
29

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count toward the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "rrovided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material provided in the appendix. you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer:[Yes)
Justification:
The paper's contributions and scope are well-explained throughout the abstract and introduction, and the entire paper is written to provide consistent and comprehensive conclusion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
30

---

Justification:
Our paper discusses the limitations and future research directions to address them the conclusion section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-ten system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our paper proposes new model architecture and demonstrates its superiority through practical approach with extensive experiments rather than theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our paper provides detailed information to reproduce all experiments. Addi- tionally, the complete experiment code will be provided.
31

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel urchitecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
http:///iitubbcoo//JJLc////1///9 22224
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
hhipss.c//ublic/uuudds wttpee/e/oooeSeeeeooooeeeoe
www.obiiiiiiioooooioiiooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
32

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our paper provides detailed information about the experiments, including data splits, hyperparameters, how they were chosen, and the type of optimizer, in the main text and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail
that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In the main results, we used a paired t-test to demonstrate the superiority of methodology and reported the p-value.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the eтror bar is the standard deviation the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We reported the types and number GPUs used the experiments, as well as the packages used for training.
33

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://eeur/ httpr/ps....ppllc//iiissssoollll
Answer: [Yes]
Justification: Yes, the research conducted in the paper fully conforms to the NeurlPS Code of Ethics in every respect. We have ensured that all ethical guidelines and standards have been meticulously followed throughout the study.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper discusses the potential broader impacts of research in the conclusion.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
34

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: Our model does not pose such risks, and therefore, safeguards for responsible data model release were not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we have properly credited the creators original owners of assets used in the paper, including publicly available models and packages, with accurate citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi sesrsswithhcdee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not include new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
35

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsoureing.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer:[NA]
Justification:
This paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "9QIT5RsGRM", "reviewer": "Reviewer_A63Y", "summary": "This paper introduces a new mechanism called \"Spectral Attention\" designed to address the challenge of long-term dependencies in time series prediction. Traditional models such as linear and Transformer-based predictors face limitations in handling long-term dependencies due to fixed input sizes and the shuffling of samples during training. Spectral Attention enhances model performance by preserving temporal correlations and facilitating long-term information processing. It utilizes low-pass filters to capture long-term trends and supports parallel training across multiple time steps, thus expanding the effective input window. Experimental results demonstrate that this mechanism achieves state-of-the-art prediction performance on multiple real-world datasets, opening new avenues for exploration in time series analysis.", "questions": "1. the use of sequential sampling instead of random sampling, which will cause the model to generalize in the face of new data or different distribution of data, there is a risk of overfitting?\n2. There are some formulas in the paper that do not explain the specific meaning. For example, \"The base model can be reformulated as P=f2(F,E) and F,E=f1(X).\" What is the meaning of \"E\"?\n3. How is the formula \"fSA=f2·SA·f1\" derived in the sentence \"Our Spectral Attention module takes feature F as input and transforms it into F' of the same size, F' =SA(F), modifying the base model as follows: fSA= f2 · SA · f1\"?\n4. why was the experiment not conducted on the common ILI \"dataset\"?\n5. Do spectral attention mechanisms significantly increase the computational cost of model training and inference?", "limitations": "no.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Aiming at the problem of long-term dependence in time series prediction, this paper proposes a novel Spectral Attention mechanism.\n2. This method can be extended to multiple prediction models and achieve performance improvement.", "weaknesses": "1. There are some formulas in the paper that do not explain the specific meaning.\n2. the source code of the paper is not open, can not verify the effect of the experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you very much for the insightful comments and suggestions. We have addressed each of your questions below. Also, please review the global comment and the attached PDF.\n\n---\n\n### **Weaknesses**\n\n**W1** We apologize for not providing sufficient specific meanings for some of the formulas in our paper. We have thoroughly addressed this issue based on your comments and revised our manuscript. Additionally, we have included more detailed diagrams to facilitate understanding of each step of our formulas and to provide a more detailed explanation. Kindly refer to Figures R1 and R2 in the attached PDF.\n\n**W2** We will definitely share a GitHub link containing the complete code in the final version of the manuscript so that all experiments can be fully reproduced.\n\n---\n\n### **Questions**\n\n**Q1** Thank you for an inspiring question. We have pondered the same issue. Our sequential sampling may fit more strongly to the distribution of the latter part of the training sequence rather than the overall distribution. This is a topic that has been extensively researched in the field of continual learning, where data distribution changes over time. To address this, we added a regularization term to prevent the distribution shift and conducted experiments. We found that using well-known methods such as EWC[1], LwF[2], and L2 regularization, actually resulted in decreased performance on the test set.\nThis indicates that the validation set, which follows the training sequence, plays a sufficient role in preventing such overfitting. \n\n[1] Elastic Weight Consolidation, 2017\n\n[2] Learning without Forgetting, 2017\n\n**Q2** We apologize for the insufficient explanation. To aid understanding, we have added detailed diagrams to the manuscript, which can be found in Fig. R1 of the attached PDF. We aimed to illustrate that BSA can be applied to arbitrary activations of the TSF model. For intermediate activations [E, F] of the model, SA can only be applied to a subset, namely F. E represents intermediate activations where BSA is not applied.\n\n**Q3** The notation we used is unconventional. We revised this as in the **Fig. R1 in the attached PDF**.\n\n**Q4** Thank you for your valuable comment. We excluded ILI dataset due to its short sequence length, which didn't suit our work on long-range dependencies beyond the look-back window. Following your advice, we conducted experiments and obtained the following results:\n\n| Illness   |    base    |    |   BSA  |   |\n|:--:|:--:|:--:|:--:|:--:|\n| |     MSE    |     MAE    |     MSE    |     MAE    |\n|    Dlinear   |   2.8488   | 1.1109 | 2.7971 |   1.1135   |\n|    RLinear   |   2.6369   |   1.0133   | 2.6049 | 1.0050 |\n|     FreTS    |   5.0733   |   1.6030   | 4.9304 | 1.5907 |\n|   TimesNet   | 2.7188 |   0.9420   |   2.7196   | 0.9381 |\n| iTransformer | 2.4539 | 0.9469 |   2.5402   |   0.9571   |\n|  Crossformer |   4.9456   |   1.5175   | 4.8094 | 1.4887 |\n|   PatchTST   |   2.0157   |   0.8804   | 1.9738 | 0.8656 |\n\nBSA showed an average performance improvement of 1.0% in MSE and 0.6% in MAE, and achieved SOTA performance with an MSE of 1.97 and an MAE of 0.866 in the Patch TST model.\n\n**Q5** Thank you for pointing out this important aspect. To measure the computational cost of BSA, we conducted comprehensive experiments on the additional cost of peak memory usage, running time, and parameter numbers. The full results can be found in Table E2, E3, E4, and E5 in the global comment. \n\n**Table E5** Total Average Additional Cost of BSA in Percentage(%). We report the average value of Table E2, E3 and E4.\n|| Time   | Memory | Num_Param |\n|:--:|:--:|:--:|:--:|\n| Weather | 6.1212 | 8.6826 | 1.7774 |\n| PEMS03  | 0.0312 | 0.8081 | 2.3110 |\n\nFor the small weather dataset, we observed a 6.1% increase in running time, an 8.7% increase in memory usage, and a 1.8% increase in the number of parameters. In contrast, for the large PEMS dataset, there was only a 0.03% increase in running time, a 0.80% increase in memory usage, and a 2.3% increase in the number of parameters. This demonstrates that our module has minimal cost and excellent scalability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uw2IEmeEum", "reviewer": "Reviewer_pdEA", "summary": "The paper presents a Spectral Attention mechanism to address the challenge of capturing long-range dependencies in time series forecasting. By preserving temporal correlations among samples and leveraging frequency domain information, the proposed method enhances the forecasting performance of various baseline models. Extensive experiments demonstrate the efficacy of Spectral Attention, achieving state-of-the-art results on multiple real-world datasets.", "questions": "1. How does the proposed method specifically improve upon the referenced Spectral Attention Autoregressive Model [1]?\n    * A detailed comparative analysis and, if possible, empirical validation are necessary to substantiate the claims of improvement.\n2. Can you provide a more detailed explanation of the complexity involved in the spectral attention mechanism?\n3. Given that the proposed model operates in the frequency domain, is it capable of handling irregularly sampled data even if the underlying architecture cannot?\n\nI am open to raising my score if the authors adequately address these concerns.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tEfficacy: The proposed Spectral Attention mechanism significantly improves forecasting performance across multiple real-world datasets.\n2.\tVersatility: The method can be integrated into various baseline models, showcasing its adaptability and broad applicability.\n3.\tExperimental validation: The experiments show consistent performance improvements.", "weaknesses": "1. Novelty and Contribution/Missing Related Work\n* While the paper introduces Spectral Attention as a novel method, using frequency domain analysis in time series forecasting is not a new concept. Notably, there is an existing paper [1] with the same title, \"Spectral Attention,\" also focused on time series forecasting, which is not referenced in the manuscript. This referenced paper employs a global/local perspective: the global Spectral Attention provides comprehensive information about the random process the time series are considered part of, thereby addressing some limitations mentioned in the paper, such as the inability to model long-range dependencies. Furthermore, the referenced work includes spectral filtering, differing from the authors' approach by learning a cut-off frequency while filtering each frequency component independently. Their solution integrates easily into pre-existing architectures. Given the significant similarities and the relevance of this other paper as a precursor to the authors' idea, why is it not cited in the manuscript? The authors must include a thorough discussion of [1], acknowledging its relation and relevance to their work.\n* Additionally, recent SSM-based approaches should be covered in the related work section.\n2. Differentiation from Existing Methods\n* The paper would benefit from a clearer distinction between Spectral Attention and existing frequency-based methods such as WaveFroM, FEDformer, or the Spectral Attention Autoregressive Model. The unique advantages and innovations of Spectral Attention compared to these methods should be emphasized more explicitly.\n3. Computational Complexity\n* The paper claims that Spectral Attention adds minimal computational overhead. However, the analysis of computational complexity and memory requirements is somewhat superficial. A detailed comparison of training and inference times, as well as memory usage, with and without Spectral Attention, should be provided to substantiate these claims.\n4. Integration with Different Models\n* While the paper demonstrates that Spectral Attention can be integrated with various TSF models, the integration process lacks detail. Specific guidelines or algorithms for integrating Spectral Attention with different architectures should be included. This would facilitate practitioners in applying the proposed method to their models.\n\n\n[1] Moreno-Pino, Fernando, Pablo M. Olmos, and Antonio Artés-Rodríguez. \"Deep autoregressive models with spectral attention.\" Pattern Recognition 133 (2023): 109014.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you very much for the insightful comments and suggestions. We have addressed your questions below.\n\n---\n\n**W1-a & Q1** Thank you for introducing this significant work. We will add this research to the related work section (we reported a summary in the global comment). While this study is remarkable, we think there are significant differences between their works and our work in terms of scenarios and methods.\n\nFrom the perspective of the model's objective, SAAM aims to enable autoregressive models to better capture the trend of signals within the input sequence (look-back window). In contrast, our BSA is the first attempt to learn temporal correlations across multiple input samples beyond the look-back window.\n\nIn terms of the learning methodology, SAAM can only be applied to autoregressive models. However, many recent state-of-the-art models use non-autoregressive architectures that inherently excel at learning the trend of input signals. BSA, on the other hand, is agnostic to the model structure and can be applied to various types of neural network-based TSF models.\n\nAdditionally, SAAM requires performing FFT and calculating autocorrelation on the input signal, which results in quadratic memory complexity on input length. \nExperimental results show that on the 137-channel solar dataset, SAAM required an additional 18.79% training time. In contrast, BSA does not perform Fourier transforms and maintains a minimal computation cost that is constant with respect to input length. Experiment on the BSA’s computational complexity is provided below in the answer to **W3**.\n\n**W1-b** Thank you for your insightful comment. As you suggested, we will add related works such as the State Space Model[1], Structured State Space Model[2], and Mamba[3] to our related work section. Please review the global comment.\n\n[1] Koller 2009.\n\n[2] Gu, Albert, 2021.\n\n[3] Gu, Albert, 2023.\n\n**W2**  We apologize if it was unclear how our BSA method differs from the WaveForM and FEDformer discussed in the later part of the \"2. Related Works, Frequency-Utilizing Models\".\n\nWaveForM and FEDformer's goal of using Wavelet transform or FFT to the frequency domain is to decompose information from a single input into a multiple-frequency representation.\nTherefore, if we try to find the long-term dependency beyond the current look-back window using Wavelet or FFT decomposition, we need the whole data of multiple inputs previous to the current input, which is intractable.\n\nHowever, our BSA method aims to learn the long-term dependency beyond the look-back window by extracting temporal correlation information between the sequential input samples using multiscale EMA, which is suited for streaming time series inputs.\nTo learn the long-term dependency, BSA does not need all the data from previous sequences; it only needs the momentum values of the selected feature and the current input sequence. This approach is highly efficient and tractable.\n\nTo briefly recap the Related Works, WaveForM, FEDformer, and FreTS all utilize the frequency domain transformation and are limited to utilizing the information only to the length of the look-back window. Our BSA is a model-agnostic module that can allow these frequency-utilizing models to learn long-term dependency over the look-back window, as shown by our experiments with FreTS in the main Table 1.\n\nPlease refer to our response above on **W1-a** for detailed differences with SAAM to avoid repetition.\n\n**W3 & Q2** Thank you for pointing out this important aspect. To measure the computational cost of BSA, we conducted comprehensive experiments on peak memory usage, running time, and number of parameters. We presented the full table in the global comment Table E2~E5. The average results are as follows:\n\n**Table E5** Average Additional Cost of BSA in Percentage(%).\n||Time|Memory|Num_Param|\n|:--:|:--:|:--:|:--:|\n|Weather|6.1212|8.6826|1.7774|\n|PEMS03|0.0312|0.8081|2.3110|\n\nFor the weather dataset, which has a small number of channels (21), we observed a 6.1% increase in running time, an 8.7% increase in memory usage, and a 1.8% increase in the number of parameters. In contrast, for the PEMS dataset, which has a large number of channels (358), there was only a 0.03% increase in running time, a 0.80% increase in memory usage, and a 2.3% increase in the number of parameters. This demonstrates the high scalability and applicability of the BSA.\n\n**W4** We agree that providing a thorough explanation of the integration process is necessary, and we will strengthen this part accordingly. As you pointed out, this is crucial for facilitating practitioners.\n\nOur method can be applied to any TSF model that satisfies the problem statement in the manuscript and can be used simply by plugging it into any arbitrary activation within the model (For any arbitrary activation F within the model, BSA can be used as a plugin by adding F = BSA(F) in the forward function). The integration is completed by subsequently changing the sampling method to sequential sampling.\n\nIn Section 4.3 of the main text, we conducted experiments to determine which part of the model benefits most from the application of the BSA module. We observed performance improvements regardless of where the module was inserted, with the highest performance gains observed when it was applied to the queries or the activations before the decoder.\n\nTo make BSA easy to use, we will provide convenient plugin code along with the experimental code in the final version of the manuscript.\n\n**Q3** Thank you for your insightful comment. Since BSA is applied as a plug-in to the underlying architecture, we hypothesize that BSA would also be constrained if the underlying architecture cannot handle irregular input. However, if the underlying architecture can process irregular input, we believe BSA could also handle irregular input by using momentum proportional to the irregular time intervals. We consider this to be a valuable topic for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0rmoE49mXb", "reviewer": "Reviewer_weYb", "summary": "This paper focuses on the exploration of long-term dependency within a whole time series sequence, to address the challenge of the short-term look-back window, which is interesting. Based on the observation, this paper provides Batched Spectral Attention to enable parallel training across multiple timesteps.", "questions": "See weaknesses.", "limitations": "Not sufficient. I can't find the potential negative societal impact of their work in the Conclusion.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The analysis of attention is sufficient to show the FFT graphs.\n\n* The method is designed as an easy plug-in module, which benefits various base models.", "weaknesses": "* The writing and logical presentation can be strengthened, which is hard to understand, e.g., Fig. 2. (I spent lots of time understanding the formulations and Fig. 2 )\n\n* It is not clear why only the low-frequency components are used, just for long-term dependency, and why the lack of high-frequency components does not affect the short-term vibrations.\n\n* The comparison in Table 1 may be a little unfair since more computation and memory are introduced by the BSA. Thus, please provide the computation and memory comparison in Table 1 to further evaluate the superiority of BSA.\n\n* I am curious will the performance still be improved when applying BSA as the look-up window T increases or decreases? That is to validate the effectiveness of BSA in more scenarios to demonstrate its generalization ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you very much for the insightful comments and suggestions. We have addressed each of your questions below. Please also review the global comment and the attatched PDF since they are used for answering your question.\n\n---\n\n**W1** We sincerely apologize for not providing a clear enough presentation of our model and logical framework. Following your valuable comment, we have thoroughly revised Fig. 2. Additionally, we have updated the main text, adding diagrams and explanations to make our methodology easier to understand. The revised version has been added to the **attached PDF (Fig. R1 and Fig. R2)**. Please kindly review these updates in the PDF file from the global comment.\n\n**W2** Thank you for your comment. Our BSA module leverages both low-frequency and high-frequency components. The Spectral Attention mechanism focuses on specific frequencies during training, targeting the essential information frequency in the data. If and only if the data's prominent information frequency is low-frequency, the BSA module attends more to low-frequency information.\n\nOur BSA module stores the low-frequency component (M) of the feature (F) obtained through EMA. When computing F', a replacement F, attention involves three components: M (low-frequency component), F (identity), and F-M (high-frequency component). These components are used to compute F' through a weighted sum using the SA-Matrix (in practice, multiple momentums are employed, resulting in multiple Ms).\n\nThe learning results indicate that for data with long-term dependencies, the BSA module prioritizes low-frequency components, effectively capturing long-range patterns. This can be seen in Fig. 3 and 4 of the main text.\n\n**W3** Thank you for your insightful comment. As you mentioned, BSA, a plug-in module,  may increase computational complexity. Therefore, we measured the running time, peak memory, and number of parameters across various models. Full results are reported in the global comment Table E2~E5. \n\n**Table E5** Total Average Additional Cost of BSA in Percentage(%). We report the average value of Table E2, E3 and E4.\n|         | Time   | Memory | Num_Param |\n|:--:|:--:|:--:|:--:|\n| Weather | 6.1212 | 8.6826 | 1.7774 |\n| PEMS03  | 0.0312 | 0.8081 | 2.3110 |\n\nEach number represents the increased cost due to BSA as a percentage. Experiments on the lightweight weather dataset with 21 channels and the complex PEMS dataset with 358 channels demonstrate the low computational cost and scalability of our model. Our model has constant complexity with respect to input length, making it highly applicable.\n\n**W4** Thank you for your inspiring comment. To demonstrate that BSA consistently maintains high performance regardless of changes in the look-back window size, we conducted experiments by modifying the original 96 look-back window to 48 and 192. This table shows the percentage performance gain:\n\n|         |     | Dlinear |         | RLinear |         | iTransformer |         |\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|         |     |   MSE   |   MAE   |   MSE   |   MAE   |      MSE     |   MAE   |\n| Weather |  48 | 20.8490 | 16.5494 |  4.3724 |  2.6808 |    12.0786   |  5.2078 |\n|         |  96 | 10.1386 |  8.7108 |  4.5913 |  2.5039 |    7.7836    |  3.0341 |\n|         | 192 |  7.4182 |  6.1351 |  4.3021 |  1.6094 |    6.8710    |  3.3374 |\n|  PEMS03 |  48 | 18.7263 | 11.4883 | 18.8634 | 11.0712 |    29.4545   | 17.3504 |\n|         |  96 | 11.8983 |  7.4829 | 32.0424 | 18.0321 |    24.5469   | 13.6611 |\n|         | 192 | 19.3270 | 10.5795 | 21.7945 | 13.7005 |    9.6336    |  5.1841 |\n\nThe results show that BSA provides constant performance improvements across all input sizes, with particularly notable improvements as the look-back window size decreases. This clearly indicates that BSA enables the base model to learn long-range information beyond the look-back window, achieving high prediction performance even when the look-back window is limited.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable comments and constructive suggestions to strengthen our work. Also, for the positive comments and encouraging remarks: The paper introduces a novel Spectral Attention mechanism, addressing the long-term dependence in time series prediction (A63Y). This method is designed as an easy plug-in module (weYb), that can be applied to various models regardless of the underlying architecture (A63Y, weYb). This methodology demonstrates high efficacy (pdEA). The proposed Spectral Attention mechanism led to significant improvements in forecasting performance across various real-world datasets (A63Y, pdEA). Moreover, this model exhibits versatility (pdEA). The model can be integrated into various baseline models, showcasing its adaptability and broad applicability(A63Y, pdEA, weYb). The paper validated through experiments that BSA consistently delivers performance improvements (A63Y, pdEA).

---

## **Literature reviews**
We cited and discussed the following papers recommended by the reviewers.

**SAAM[1]** This paper proposes the Spectral Attention Autoregressive Model (SAAM), which can be applied as a plug-in to auto-regressive models (e.g., DeepAR) to improve the performance of time series forecasting. It is known that autoregressive models, due to their structure, are less capable of addressing long-range trends compared to structures such as Transformers. SAAM compensates for this by calculating the DFT and correlation matrix for the input signal. SAAM is used only with recurrent models and differs from our BSA in that it operates within the model's look-back window. We will include this paper in the "related work" section and describe its relevance and differences compared to our research.

**SSM[2], S4[3]** Existing recurrent models were computationally inefficient as they required forward passes and backpropagation through time proportional to the length of the input sequence due to their dependence on previous states. S4 enabled parallel computation by efficiently calculating the Linear State-Space Layer, which consists of matrix powers in the computational process of the discretized SSM. This allowed the model to consider long-range dependencies and can be seen as a form that takes advantage of both convolution and recurrent approaches.

**Mamba[4]** Traditional SSMs are linear time-invariant (LTI) models with finite hidden states for the entire sequence, which are computationally very efficient but failed in tasks such as Selective Copying. To improve the model's ability to understand sequence context, the authors introduced additional learnable parameters and proposed Mamba, a time-varying model.

---

## **New Figures (Attached PDF)**
We revised the figure and elaborated on the caption to make it easier to understand the principles of SA and BSA modules.

**SA module** We provided detailed illustrations and explanations of (A) how the SA module is applied to the model in a plug-in manner, (B) how the SA module utilizes sequential input samples and momentum, and (C) how the SA module internally performs spectral attention from features and momentum and updates the momentum.

**BSA module** We revised Fig. 2 in the manuscript to make it clearer. We intuitively modified the sequential input of data and the flow of each component.

---


## **Experiments**
**Table E1**  Additional Datasets Performance Evaluation. We report the average value of all prediction lengths. We can see performance increase in all datasets, especially significant on PEMS03 and Energy-Data datasets.
||||PEMS03|Energy-Data|Solar|
|:-:|:-:|:-:|:-:|:-:|:-:|
|Dlinear|base|MSE|0.4364|0.8737|0.3451|
|||MAE|0.5018|0.6829|0.4189|
||**BSA**| MSE|**0.3845**|**0.8375**|**0.3112**|
|||MAE| **0.4643**| **0.6670**|**0.3949**|
|RLinear|base|MSE|0.9922|0.8163|0.3806|
|||MAE|0.7405|0.6302| 0.3657|
||**BSA**|MSE|**0.6743**|**0.7968**|**0.3462**|
|||MAE|**0.6069**| **0.6221**|**0.3488**|
|FreTS|base|MSE|0.2594 |0.9764 | 0.2439 |
|||MAE|0.3535|0.7349|0.2921|
||**BSA**| MSE |**0.2289**| **0.9399**|**0.2423**|
|||MAE|**0.3236**| **0.7103**|**0.2899**|
| iTransformer | base | MSE | 0.2618|0.8332|**0.2550**|
||| MAE | 0.3453 | 0.6395 | **0.2763**|
||**BSA**|MSE|**0.1975**| **0.7859**|0.2558|
||| MAE|**0.2981**|**0.6189**|0.2775|

**Table E2**  Additional Time Cost of BSA in Percentage(%). We report the average value of all prediction lengths.
||TimesNet|iTransformer|Crossformer|PatchTST|
|:-:|:-:|:-:|:-:|:-:|
|Weather|-0.0033|15.8550|5.5010|3.1320|
|PEMS03|0.2129|2.2388|-2.0413|-0.2854|

**Table E3**  Additional Memory Cost of BSA in Percentage(%). We report the average value of all prediction lengths.
|| TimesNet | iTransformer | Crossformer | PatchTST |
|:-:|:-:|:-:|:-:|:-:|
|Weather|0.8237| 32.3081 |1.2534 | 0.3453 |
|PEMS03|0.3412 | 2.3414 | 0.1708 | 0.3791 |

**Table E4**  Additional Parameter Cost of BSA in Percentage(%). We report the average value of all prediction lengths.
|| TimesNet | iTransformer | Crossformer | PatchTST |
|:-:|:-:|:-:|:-:|:-:|
| weather| 1.1757 | 0.0418 | 5.7333 | 0.1587 |
| PEMS03| 0.1594 | 4.8778 | 1.9611 | 2.2458 |

**Table E5**  Total Average Additional Cost of BSA in Percentage(%). We report the average value of Table E2, E3 and E4.
||Time | Memory| Num_Param |
|:-:|:-:|:-:|:-:|
| Weather | 6.1212 | 8.6826 | 1.7774 |
|PEMS03| 0.0312 | 0.8081 | 2.3110 |

---

## **References**
[1] Moreno-Pino, Fernando, Pablo M. Olmos, and Antonio Artés-Rodríguez. "Deep autoregressive models with spectral attention." Pattern Recognition 133 (2023): 109014.

[2] Koller, Daphne, and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.

[3] Gu, Albert, Karan Goel, and Christopher Re. "Efficiently Modeling Long Sequences with Structured State Spaces." International Conference on Learning Representations (2022).

[4] Gu, Albert, and Tri Dao. "Mamba: Linear-time sequence modeling with selective state spaces." arXiv preprint arXiv:2312.00752 (2023).

Meta-review: The paper received two Borderline Accepts and one Weak Accept after the rebuttal and discussion period. The reviewers stated that the proposed spectral attention mechanism is capable of resolving long-term dependence in time series forecasting and is designed as a simple plug-in module that can be applied to a variety of models regardless of the underlying architecture. The AC did not find any reason to overturn all the positive evaluations of the reviewers. Therefore, considering the submitted paper, the reviewers' comments, the authors' rebuttals, and the discussion, the AC considers it appropriate to rate this paper as acceptable. However, the AC recommends that the reviewers' comments be incorporated into the camera-ready version.
Decision: Accept (poster)