Title: Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting
Venue: NeurIPS 2024
Accepted: yes
Keywords: Time series forecasting

Long-range dependency

Low-pass filter

Spectral attention

Long term trend
Abstract: Sequence modeling faces challenges in capturing long-range dependencies across diverse tasks. Recent linear and transformer-based forecasters have shown superior performance in time series forecasting. However, they are constrained by their inherent inability to effectively address long-range dependencies in time series data, primarily due to using fixed-size inputs for prediction. Furthermore, they typically sacrifice essential temporal correlation among consecutive training samples by shuffling them into mini-batches. To overcome these limitations, we introduce a fast and effective Spectral Attention mechanism, which preserves temporal correlations among samples and facilitates the handling of long-range information while maintaining the base model structure. Spectral Attention preserves long-period trends through a low-pass filter and facilitates gradient to flow between samples. Spectral Attention can be seamlessly integrated into most sequence models, allowing models with fixed-sized look-back windows to capture long-range dependencies over thousands of steps. Through extensive experiments on 11 real-world time series datasets using 7 recent forecasting models, we consistently demonstrate the efficacy of our Spectral Attention mechanism, achieving state-of-the-art results.

Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting

Abstract

Sequence modeling faces challenges in capturing long-range dependencies across diverse tasks. Recent linear and transformer-based forecasters have shown superior performance in time series forecasting. However, they are constrained by their inherent inability to effectively address long-range dependencies in time series data, primarily due to using fixed-size inputs for prediction. Furthermore, they typically sacrifice essential temporal correlation among consecutive training samples by shuffling them into mini-batches. To overcome these limitations, we introduce a fast and effective Spectral Attention mechanism, which preserves temporal correlations among samples and facilitates the handling of long-range information while maintaining the base model structure. Spectral Attention preserves longperiod trends through a low-pass filter and facilitates gradient to flow between samples. Spectral Attention can be seamlessly integrated into most sequence models, allowing models with fixed-sized look-back windows to capture longrange dependencies over thousands of steps. Through extensive experiments on 11 real-world time series datasets using 7 recent forecasting models, we consistently demonstrate the efficacy of our Spectral Attention mechanism, achieving state-ofthe-art results.

1 Introduction

Time series forecasting (TSF) stands as a core task in machine learning, ubiquitous in our lives through applications such as weather forecasting, traffic flow estimation, and financial investment. Over time, TSF techniques have evolved from statistical [5, 10, 18] and machine learning approaches [2, 9, 20] to deep learning models like Recurrent Neural Networks [15, 25, 41] and Convolution based Networks [12, 45]. Following the success of Transformers [44] in various domains, Transformerbased models have also become mainstream in the time series domain [23, 27, 36, 49, 54, 55, 56]. Recently, methodologies based on Multi-layer Perceptron have received renewed attention [7, 8, 28, 52]. However, despite the advancements, long-term dependency modeling in TSF remains challenging [53].

Unlike image models, where data are randomly sampled from the image distribution and are thus independent of each other [16], TSF models sample data from the continuous signal, dependent

a)TimeSeriesData

(b)conventionat

(c)ours

Look-BackWindowPrediction

Decorrelated

PreserveCorrelation

(Input)

(Label)

mini-batch

t~t+3

t+

t+2

t+1

WholeTimeSeriesSequence

Look-BackWindow

Look-BackWindow

on the time variable t as shown in Figure 1a. This leads to a high level of correlation between each training sample, which consists of a fixed-sized look-back window before t (as input) and the subsequent prediction sequence after t (as label). Therefore, the conventional approach of shuffling the consecutive samples into mini-batches deprives the model of utilizing the crucial inherent temporal correlation between the samples. This restricts the model's consideration to only a fixed-size look-back window for sequence modeling, limiting the ability to address long-range dependencies (Figure 1b).

Recent studies pointed out that simply increasing the look-back window leads to substantially detrimental effects such as increased model size and longer training and inference times [53]. This is particularly challenging for transformer forecasters, which exhibit quadratic time/memory complexity [36, 44, 54], and even for efficient models using techniques like Sparse Attention, which have O(nlogn) complexity [23, 27, 55]. Furthermore, if the commonly used look-back window of 96 is extended fivefold, the model can only utilize time steps of less than 500, making it difficult to consider long-range dependencies spanning thousands or the entire dataset. Also, increasing the look-back window may not be beneficial, often leading to decreased performance [53], highlighting the fact that current models are not sufficient in capturing long-range dependencies.

To address this limitation, we propose Spectral Attention, which can be applied to most TSF models and enables the model to utilize long-range temporal correlations in sequentially obtained training data. With the stream of consecutive training samples (Figure 1c), Spectral Attention stores an exponential moving average (EMA) of the activations with various smoothing factors at each time step. This serves as a low-pass filter, inherently embedding long-range information over a thousand steps. Attention is then applied to the stored EMA activations of various smoothing factors (low-pass filters with different frequencies), enabling the model to learn which periodic trends to consider when predicting the future, thereby enhancing its performance. Spectral Attention is even applicable to models such as iTransformer [31], which do not preserve the temporal order of time series data internally.

We further extend Spectral Attention, where computations depend on the activation of the previous timesteps, to Batched Spectral Attention, enabling parallel training across multiple timesteps. This extension makes Spectral Attention faster and more practical and allows for the direct utilization of temporal relationships among consecutive data samples within a mini-batch in the training base model. In Batched Spectral Attention, the EMA is unfolded over time to perform Spectral Attention simultaneously across multiple time steps. This unfolding allows gradients at time t to propagate through the Spectral Attention module to the previous time step within the mini-batch, achieving effects similar to Backpropagation Through Time (BPTT) [47] and extends the model's effective input window.

Our approach preserves the base TSF model architecture and learning objective while enabling the model to leverage long-term trends spanning thousands of steps. By effectively utilizing the temporal correlation of training samples, our method allows gradients to flow back in time beyond the lookback window, extending to the entire mini-batch. Also, our method requires little additional training time and memory. We conducted experiments on 7 recent TSF models and 11 real-world datasets and demonstrated consistent performance enhancement in all architecture, achieving state-of-the-art results. We summarize the main contributions as follows:

We propose Spectral Attention, which addresses long-range dependencies spanning thousands of steps through frequency filtering and attention mechanisms, leveraging the temporal correlation among the consecutive samples.

We propose Batched Spectral Attention, which enables parallel training across multiple timesteps and expends the effective input window, allowing the gradient to flow through time within the mini-batch.

Batched Spectral Attention is applicable to most existing TSF models and practical in real-world scenarios with minimal additional memory and comparable training time. Also, it allows finetuning with a trained TSF model.

Batched Spectral Attention demonstrates consistent model-agnostic performance improvements, particularly showcasing superior performance on datasets with significant long-term trend variations.

2 Related Works

Classic TSF models. Statistical TSF methods, such as ARIMA [35], Holt-Winters [18], and Gaussian Process [10], assume that temporal variations adhere to predefined patterns. However, their practical applicability is largely limited by the complex nature of real-world data. Machine learning approaches, such as Support Vector Machines [4] and Random Forests [14] have proven to be effective even compared to early artificial neural networks [13, 17, 40]. Convolutional network-based methods leverage convolution kernels to capture temporal variations sliding along the temporal dimension [12, 45]. Recurrent neural network (RNN) grasp changes over time via state transitions across different time steps. [25, 41]. However, RNN-based models exhibit limitations in modeling long-range dependencies due to challenges such as vanishing gradients and error accumulation [30, 43]. Recently, transformer and linear-based models have emerged as alternatives, demonstrating superior performance compared to recurrent models [46, 53].

Transformer and Linear based models. Transformer-based models [44] address temporal relationships between time points using the attention. LogSparseTransformer [27], Reformer [23], and Informer [55] have been proposed to make the Transformer architecture efficient, addressing the quadratic time complexity. The Autoformer [49] incorporates series decomposition as an inner block of Transformer and aggregates similar sub-series by utilizing the Auto-Correlation mechanism. PatchTST [36] introduces patching, a channel-independent approach that processes each variable separately and focuses on cross-time attention. Crossformer [54] utilizes a channel-dependent approach to learn cross-variate dependencies. This is achieved through the use of cross-time and cross-dimension attention. iTransformer [31] applies attention and FFN in an inverted way, where attention handles correlations between channels and FFN handles the temporal information. Recently, to address Transformers' potential difficulties in capturing long-range dependencies [53], methodologies based on the linear model and Multi-Layer Perceptron (MLP) structures have emerged. DLinear [53] utilizes the decomposition method introduced in Autoformer and predicts by adding the output of two linear layers for each seasonal and trend element. TiDE [7] proposes an architecture based on MLP residual blocks that combines information from dynamic and static covariates with a look-back window for encoding, followed by decoding. TSMixer [8] performs forecasting by repeatedly mixing time and features using an MLP. RLinear [28] comprises of a single linear layer with RevIN [21] for normalization and de-normalization.

Frequency-utilizing models. Using the frequency domain for TSF is a well-established approach [3, 42, 19]. Conventional approaches leverage frequency information during the preprocessing stage [37] or decompose time series based on frequency filtering [39]. In deep TSF models, research has also explored architectural advancements that are aware of the frequency information. SAAM [34], which is applicable to RNNs, performs FFT and autocorrelation on the input signal. WaveFroM [51] uses discrete wavelet transform (DWT) to project time series into wavelet domains of various scales and performs forecasting through graph convolution and dilated convolution. FEDformer [56] adopts a mixture-of-experts strategy to refine the decomposition of seasonal and trend components and introduce sparse attention mechanisms in the frequency domain. TimesNet [48] transforms 1D time series into 2D tensors utilizing multi-periodicity by identifying dominant frequencies through Fourier Transform, modeling temporal variations effectively. FreTS [52] leverages frequency-domain MLPs to achieve global signal analysis and compact energy representation, addressing the limitations of

(a)

Prediction

(b)

(c)

t+1

t+S

Ft+2

SA

f2

t-L+3

Input

t+2

E

F

SA

SA

t-L+2

Input

t+1

f1

F

Ft

SA

t-L+1

Input

Input

M

t-L+1

Qz

a

Look-BackWindow(L)

MomentumsofF

(c)

MC+1

SA

αiMi +(1 -αi)Fi

F't

Ft-Me

Ft-M

featureF'

SA-

Ft

Ft

F't

Matrix

W

Per-column

W

WeightedSum

2

BW

ofF

point-wise mappings and information bottlenecks in conventional MLP-based methods. FITS [50] employs interpolation within the complex frequency domain to construct a concise and robust model. While these models leverage frequency information, they are limited in modeling long-range dependencies, as the frequency conversion is confined to the look-back window. On the other hand, our Spectral Attention is the first to achieve long-range dependency modeling beyond the look-back window by incorporating consecutive data streams during model training.

3 Methods

Problem Statement. In multivariate time series forecasting, time series data is given D T : { x 1 , ..., x T } ∈ R T × N at time T with N variates. Our goal is, at arbitrary future time t > T , to predict future S time steps Y t = { x t +1 , ..., x t + S } ∈ R S × N . To achieve this goal, TSF model f utilizes length L look-back window as input X t = { x t -L +1 , ..., x t } ∈ R L × N making prediction P t = f ( X t ) , P ∈ R S × N .

Model is trained with the training dataset D T = { ( X t , Y t ) | L ≤ t ≤ T -S } . While conventional methods typically randomly sample each X,Y from D T to constitute the mini-batch, we utilize sequential sampling to incorporate temporal correlations between samples into the learning process.

3.1 Spectral Attention

Spectral Attention ( SA ) can be applied to every TSF model that satisfies the aforementioned problem statement. This base TSF model is represented by P = f ( X ) , and SA can be applied to arbitrary activation F within the model. The base model can be reformulated as P = f 2 ( F, E ) and F, E = f 1 ( X ) . F, E are intermediate state and SA module takes an arbitrary subset F as input and transforms it into F ′ of the same size; F ′ = SA ( F ) , P ′ = f 2 ( F ′ , E ) . The resulting SA plugged model f SA is depicted in Figure 2a.

With X t as the base model input, SA takes D -dimensional feature vector F t ∈ R D as input. SA updates the exponential moving average (EMA) M t ∈ R K × D of F t in its internal memory with the K smoothing factors { α 1 , ..., α K } ∈ R K ( α 1 < · · · < α K ) as shown in Figure 2b.

EMAretains the trend of features over long-range time periods based on the smoothing factor. It operates as a low-pass filter, with the -3db (half) cut-off frequency of freq cut = 1 2 π cos -1 [ 1 -(1 -α ) 2 2 α ] , effectively preserving the trend over 6,000 period with α = 0 . 999 .

To represent high-frequency patterns contrasting with the low-pass filtered long-range pattern M t , we generated H t ∈ R K × D by subtracting M t from F t .

MCB

mini batch

BSAModule

M

M

a1

t+1

M

a1

t+2

M

t+B

a1

F

Ft+B-1

(1-a)

Ft-Mak

Input

t+B

Ft-

M

featureF'

F

（1-a）…（1-a）a

B-2

SA-Matrix

t+B-1

t+1

Input

Ft

(1-a)

（1-a）a（1-a）a

B-1

F

Per-column

Input

W

al

a2

Ma

t+B-1

WeightedSum

a1

Ma1

SA contains learnable parameters: sa-matrix ∈ R (2 K +1) × D , which learns what frequency the model should attend to for each feature. 2 × H t , F t , 2 × M t are concatenated on dimension 0, resulting in R (2 K +1) × D , which is then weighted summed with sa-matrix on dimension 0, generating output F ′ t (Figure 2c).

The sa-matrix is initialized so that softmax ( sa-matrix ) resembles a Gaussian distribution on axis 0. This results in symmetric value on axis 0 ( sa-matrix K +1 -i = sa-matrix K +1+ i ) and makes SA an identity function on initialization ( ∵ H k + M K -k -1 = F ).

SA allows the model to attend to multiple frequencies of its feature signal, enabling it to focus on either long-range dependencies or high-frequency patterns as needed and shift the feature F distribution on the frequency domain. By initializing SA as an identity function, the model can be fine-tuned with the already trained base model, allowing efficient implementation.

3.2 Batched Spectral Attention

Batched Spectral Attention ( BSA ) enables batch training over multiple time steps. The main concept involves unfolding EMA, which facilitates gradients to flow across consecutive samples in a minibatch, akin to BPTT. This enables efficient parallel training and promotes the model to extract long-range information beneficial for future prediction, extending the effective look-back window. The overall flow of BSA is depicted in Figure 3.

With mini-batch of size B , consecutive samples X [ t,t + B -1] = { X t , ...X t + B -1 } ∈ R B × S × N are given as input. Following aforementioned SA setting, BSA takes F [ t,t + B -1] = { F t , ...F t + B -1 } ∈ R B × D as input. In the next step, BSA utilizes F [ t,t + B -1] and the stored M t ∈ R K × D to generate M t + b (0 ≤ b ≤ B ) by unfolding the Equation 1.

This equation can be transformed to calculate M [ t,t + B ] ∈ R ( B +1) × K × D in parallel as follows.

A refers to unfolding matrix and I refers to indicator function. M t + B is stored in BSA for the next mini-batch input. F ′ [ t,t + B -1] is computed in parallel, similar to Equation 2 and 3, using F [ t,t + B -1] , B [ t,t + B -1] , and sa-matrix ∈ R (2 K +1) × D . The lower -triangle function prevents gradients from the past timestep from flowing into future models, aligning with the characteristics of time-series data.

At the beginning of each training epoch, M 0 is initialized to F 0 for all α , enhancing stability for subsequent EMA accumulation. Since SA is proposed to address long-range dependencies in training, it lacks sufficient information in the early stages when not enough steps have been seen. Therefore, for the stability of training, we linearly warm up the learning rate for the first 1 / (1 -max ( smoothing factors )) timesteps at the beginning of each training epoch. The overall training of both the base model and BSA is conducted according to Algorithm 1.

In SA, the smoothing factors { α 1 , ..., α K } ∈ R K were given as scalar values, whereas in BSA , they are expressed by learnable parameters. This is because BSA can utilize additional past information in training beyond the look-back window by incorporating a batch-sized time window, allowing it to determine the extent of long-range dependency required for training. To keep the smoothing factors between 0 and 1, we initialized learnable parameters by applying an inverse sigmoid to the initial smoothing factors and then applied a sigmoid function in training.

So far, we assume the feature F from the base model as a vector. However, the output of the intermediate layers of the model is often represented as a tensor with two or more dimensions. In real practice, we use additional channel dimensions in BSA to process the activation tensor, which acts as applying multiple BSA modules simultaneously.

3.3 Consecutive dataset split

In the TSF scenario, the entire time series data { x 1 , ..., x T end } is divided into train, validation, and test sets in chronological order. Let T tr and T val denote the last time in the train and validation data respectively. Model training occurs using data for t in [1, T tr ], while model selection for the best model can utilize data for t in [ T tr +1 , T val ]. The test set comprises predicting time steps [ T val +1 , T end ], which are not accessible during training or validation. However, since each training sample consists of the look-back window of size L and a prediction window of size S, the training input samples are restricted to X [ L,T tr -S ] . Validation samples and test input samples range from X [ T tr ,T val -S ] , and from X [ T val ,T end -S ] , respectively. While this approach is plausible for independent data like images, it is unnatural for sequential data, as it leaves unreachable gaps ( X [ T tr -S +1 ,T tr -1] , X [ T val -S +1 ,T val -1] ), undermining the consecutive characteristics. We filled the missing gaps, making train, validation, and test sets consecutive so that our BSA model could update momentum continuously. For fair evaluation, added samples were not used for either model training, validation, or performance assessment. Detailed explanations of model validation and evaluation are provided in Appendix A.3. The full code is available at https://github.com/DJLee1208/BSA_2024.

4 Experiments

Wefirst evaluate BSA using state-of-the-art TSF models and various real-world time series forecasting scenarios in section 4.1. Next, to demonstrate that BSA effectively addresses long-range dependencies and is robust to distribution shift, we perform experiments on synthetic signals of various frequencies in section 4.2. Finally, we analyze the BSA's performance variations depending on the insertion sites within the base model, examine computation and memory costs, and conduct an ablation study in section 4.3.

Datasets. We use eleven real-world public datasets: Weather, Traffic, ECL, ETT (4 sub-datasets; h1, h2, m1, m2), Exchange, PEMS03, EnergyData, and Illness [6, 26, 29, 49]. In the Illness dataset,

Algorithm 1 Batched Spectral Attention (1 epoch)

Input:

Trained up to ( E -1 )th epoch

TSF model

θ

,

with sa-matrix

f

BSA

Train data:

,

smoothing factors

0

tr tr

)

, ...,

(

X

=

D

Valid data:

0

tr

(

X

, Y

, Y

)

{

}

D val = { ( X tr +1 , Y tr +1 ) , ..., ( X val , Y val ) }

mini-batch size: B

Train:

st = 0 , ed = B -1

initialize M 0 in BSA with X 0 and f θ ( := f 2 θ · f 1 θ )

for tr

X

P

[

[

]

]

, Y

L

mse

=

L

st,ed

[

f

]

2

θ

·

in

D

BSA

(

Compute update

θ

f

st do

·

1

f

θ

(

{:

X

[

]

)

]

[

st,ed

]

, Y

)

st,ed

[

P

∇L

and adjust learning rate

,

,

sa-matrix

,

smoothing factors ed

=

min

(

ed

+

B,tr

)

+ =

B

end for

st = tr +1 , ed = tr + B

for

[

st,ed

]

[

st,ed

]

val in

X

, Y

D

P

[

st,ed

]

f

L

mse

θ

2

·

P

=

(

L

accumulate

BSA

[

st,ed

L

st

+ =

do

·

{:

Validation phase

1

f

θ

(

X

[

}

st,ed

]

)

]

, Y

[

st,ed

]

)

and calculate validation loss

B

,

ed

end for

Output:

E

th epoch min

(

ed

+

B,val

)

=

Trained up to f θ , sa-matrix , { α 1 , ..., α K } , L val for E th epoch

}

L

val

the look-back window is set to 36, and the forecasting lengths are 24, 36, 48, and 60. For the other datasets, the look-back window is set to 96, and the forecasting lengths are 96, 192, 336, and 720. Train, validation, and test split ratio are 0.6, 0.2, 0.2 for the ETT dataset and 0.7, 0.1, 0.2 for the Weather, Traffic, ECL, Exchange, PEMS03, EnergyData, and Illness datasets. The Weather, Traffic, and ECL datasets settings follow the TimesNet paper [48]. Details on datasets are provided in Appendix A.1.

Baseline models. As a benchmark, we apply BSA to 7 recent or well-known forecasting models. (1) Linear based models: DLinear [53], RLinear [28], FreTS [52] (2) Convolution based methods: TimesNet [48] (3) Transformer based methods: iTransformer [31], Crossformer [54], PatchTST [36]. For each dataset, the model structure configuration is based on the Time-Series-Library [48]. Details on the base model configurations are provided in Appendix A.2.

Training details. We first train the base model for more than 30 epochs (20 epochs for the Traffic dataset) using Adam [22] to ensure that the validation MSE saturates, while also conducting an extensive hyperparameter search. Then, we fine-tune with the BSA module attached to the pre-trained base model. All experiments are based on the average values of three random seeds. We provide further details in Appendix A.3.

4.1 Real world datasets

We demonstrate that BSA improves forecasting performance by providing the ability to address long-range dependencies, regardless of the base model architecture. Table 1 presents the averaged values over prediction lengths for both the base and the BSA applied model across 11 time series datasets and 7 TSF models. We apply the BSA module at the beginning of the base model's activation, with the number of BSA channels matching the number of channels in the data (position 1 in Figure

4

-4

-3

-2

-1

0

1

2

3

4

0

100

200

300

400

500

600

700

Feature

5

4

3

2

1

0

1

2

(a) SA-Matrix heat map

-4

-3

-2

-1

0

1

2

3

4

Attention Level

-4

-3

-2

-1

0

1

2

3

4

Attention Level

(c) Weather-SWDR(W/m)

-4

-3

-2

-1

0

1

2

3

Attention Level

(d) ETTh1-HULL

7). BSA effectively improves MSE and MAE across all architectures. The average performance gain, in terms of MSE, ranged from as low as 0.96% to as high as 7.2%, with linear-based models demonstrating relatively high performance. Paired t-tests demonstrate statistically significant (p-value < 0.05) improvements in most MSE and all MAE across various models. This result emphasizes the model-agnostic versatility of BSA. Furthermore, BSA exhibited overall performance gain across all datasets, with the enhancement being statistically significant in 82% of cases. BSA's higher gain on ETTm compared to ETTh, both derived from the same signal but with different sampling rates, further indicates BSA's effectiveness in handling long-range dependencies.

To understand how BSA addresses long-range dependencies, we conduct an internal inspection. In Figure 4a, we present the heat map of the trained SA-Matrix of the DLinear (Temperature(°C) channel, Weather data). The positive x-axis corresponds to the log 10 values of the periods preserved by the low-pass filter, derived with the smoothing factor. Negative values correspond to high-frequency components. The blue graph in Figure 4b represents the SA-Matrix averaged over the feature dimension, illustrating the frequencies to which BSA attends overall. The red graph represents the result of applying the Fast Fourier Transform (FFT) and denoising to the raw signal. The blue graph skewed towards the low-frequency side indicates that the BSA effectively captures the long-range trend of the data. Figure 4c depicts the graph for the SWDR (Short Wave Downward Radiation per unit area, W/m) channel in the same SA-Matrix. While not identical to Figure 4b, it also exhibits strong attention towards the low-frequency pattern. In contrast, Figure 4d, the FFT graph for the HULL (High UseLess Load channel, ETTh1 data), shows that the signal itself lacks long-range trends, resulting in the symmetric SA-Matrix. This result demonstrates that BSA operates as intended, learning the low-frequency components of the signal for future prediction. We provided other graphs and detailed information on the graph plotting method in Appendix B.2.

4.2 Synthetic datasets

To further demonstrate that BSA learns long-range dependencies beyond the look-back window, we add sine waves with periods of 100, 300, and 1000 to the natural data while maintaining the mean and standard deviation (Refer to Appendix C for details on synthetic data generation). Figure 5 illustrates the performance improvement of BSA over the iTransformer model on the ETTh1 and

(a) ETTh1 MSE

(b) ETTh2 MSE

Prediction Length

Prediction Length

Gain (%)

Gain (%)

-4

-3

-2

-1

0

1

2

3

4

Attention Level

(a) LUFL original

-4

-3

-2

-1

0

1

2

3

4

Attention Level

(b) LUFL period 100

-4

-3

-2

-1

0

1

2

3

4

Attention Level

(c) LUFL period 300

-4

-3

-2

-1

0

1

2

3

4

Attention Level

(d) LUFL period 1000

+

ETTh2 datasets. The x-axis is the prediction length, and each line represents the period of the added sine wave. Performance improvement is calculated as 100 × (base MSE - BSA MSE) / base MSE. While the base model with a 96-length look-back window is expected to learn the 100-period trend, BSA outperformed it, especially for 96 and 192-step predictions (green line). The yellow line (period 300) shows nearly a 30% performance improvement across all prediction lengths. While the base model fails to learn the long-range interactions within a period of 300, BSA captures and utilizes the underlying trend. BSA also learns the 1000-period signal (red line) and demonstrates substantial improvements, especially in long prediction-length (336, 720) tasks. These results show that BSA effectively learns long-range patterns beyond the look-back window, essential for future prediction.

Figure 6 is generated from LUFL (Low UseFul Load) channel of ETTh1 data and with added sine waves of periods 100, 300, and 1000. The red arrow shows the added synthetic trend in the FFT graph (red line). As long-range trends are introduced, the attention graph (blue line) shifts from resembling symmetric Gaussian to a low-frequency bias. Longer sine wave periods cause greater shifts, prioritizing long-range information for predictions.

4.3 Analysis and ablation studies

Add & Norm

Multi-Head

4

Attention

Decoder

个

个

个

5

6

7

3

Q

K

V

Add&Norm

FFN

2

InputEmbedding

1

The BSA module offers high flexibility as it can be applied to arbitrary activations of the base model. In Table 2, we analyzed the performance changes by applying BSA at various locations within the model. Each location corresponds to a number in the Transformer architecture depicted in Figure 7. While we uniformly applied BSA to position 1 for the main result Table 1, the results in Table 2 suggest the potential for further performance enhancement by applying BSA at appropriate locations. Additionally, while there is variability depending on the placement, the performance consistently remains higher compared to the baseline, demonstrating the stability of our method. We provided a full result in Appendix D.1.

BSA shows consistent performance improvement across varying look-back window (input) lengths. Table 3 demonstrates BSA's superiority for look-back window lengths of 48, 96, and 192. Notably, while the baseline model's performance drops significantly with shorter inputs, BSA maintains high performance.

We also demonstrate the impact of using BSA on the training time, peak memory, and the number of model parameters. The extra computation required for BSA is constant with data length and linear with a look-back window length (c.f. quadratic for the base model with transformer architecture). Table 4 demonstrates the computational cost of the BSA is quite small, showing less than a 5% increase even with the large PEMS03 dataset.

Weconduct an ablation study on the three key components that constitute BSA (Table 5). 'BPTT' refers to whether gradients can flow between samples within the mini-batch. Without BPTT, BSA learns similarly to SA. 'SFs' denotes whether to use multiple smoothing factors. Lastly, 'Learn SF' indicates whether the smoothing factor is treated as learnable. The results indicate that each component significantly contributes to the performance improvement of BSA.

5 Conclusion

Our study addresses the challenges in handling long-range dependencies inherent in time series data by introducing a fast and effective Spectral Attention mechanism. By preserving temporal correlations and enabling the flow of gradients between samples, this mechanism facilitates the model in capturing crucial long-range interactions essential for accurate future predictions. Therefore, our research paves the way for fixed-sized input models to effectively handle long-range dependencies extending far beyond the input window. Through extensive experimentation, we demonstrated that our Spectral Attention mechanism enhances performance across various base architectures, with its ability to grasp long-term dependencies being the key factor behind this improvement. BSA effectively tackles long-term fluctuations, complementing the base model's capacity to manage intricate yet short-term patterns. This integrated model holds promise for improving real-world application performance. For instance, it could boost weather forecast accuracy by simultaneously capturing minute-by-minute weather changes and seasonal variations. Moreover, when predicting deterioration from a patient's real-time data, it can consider medications with lengthy onset times. Our study has limitations: we did not analyze the impact of BSA placement within the base model in detail. Also, BSA's performance gains may be limited when applied to datasets with only high-frequency information within the look-back window. These issues should be addressed in future research.

References

Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375 , 2018.

Nesreen K Ahmed, Amir F Atiya, Neamat El Gayar, and Hisham El-Shishiny. An empirical comparison of machine learning models for time series forecasting. Econometric reviews , 29(5-6):594-621, 2010.

Peter Bloomfield. Fourier analysis of time series: an introduction . John Wiley & Sons, 2004.

Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik. A training algorithm for optimal margin classifiers. In Proceedings of the fifth annual workshop on Computational learning theory , pages 144-152, 1992.

George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics) , 17(2):91-109, 1968.

Luis M Candanedo, Véronique Feldheim, and Dominique Deramaix. Data driven prediction models of energy use of appliances in a low-energy house. Energy and buildings , 140:81-97, 2017.

Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan Mathur, Rajat Sen, and Rose Yu. Longterm forecasting with tide: Time-series dense encoder. arXiv preprint arXiv:2304.08424 , 2023.

Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. Tsmixer: Lightweight mlp-mixer model for multivariate time series forecasting. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 459-469, 2023.

Jerome H Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics , pages 1189-1232, 2001.

Agathe Girard, Carl Rasmussen, Joaquin Q Candela, and Roderick Murray-Smith. Gaussian process priors with uncertain inputs application to multiple-step ahead time series forecasting. Advances in neural information processing systems , 15, 2002.

Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Pradeep Hewage, Ardhendu Behera, Marcello Trovati, Ella Pereira, Morteza Ghahremani, Francesco Palmieri, and Yonghuai Liu. Temporal convolutional neural (tcn) network for an effective weather forecasting using time-series data from the local weather station. Soft Computing , 24:16453-16482, 2020.

Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural computation , 18(7):1527-1554, 2006.

Tin Kam Ho. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition , volume 1, pages 278-282. IEEE, 1995.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335 , 2019.

MJC Hu and Halbert E Root. An adaptive data processing system for weather forecasting. Journal of Applied Meteorology and Climatology , 3(5):513-523, 1964.

Rob J Hyndman and George Athanasopoulos. Forecasting: principles and practice . OTexts, 2018.

A. Kahraman, G. Yang, and P. Hou. Wind power forecasting using lstm incorporating fourier transformation based denoising technique. In 20th International Workshop on Large-Scale Integration of Wind Power into Power Systems as well as on Transmission Networks for Offshore Wind Power Plants (WIW 2021) , volume 2021, pages 94-98, 2021.

Kyoung-jae Kim. Financial time series forecasting using support vector machines. Neurocomputing , 55(1-2):307-319, 2003.

Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations , 2021.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020.

Diego Krapf, Enzo Marinari, Ralf Metzler, Gleb Oshanin, Xinran Xu, and Alessio Squarcini. Power spectral density of a single brownian trajectory: what one can and cannot learn from it. New Journal of Physics , 20(2):023029, 2018.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 95-104, 2018.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval , pages 95-104, 2018.

Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems , 32, 2019.

Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on linear mapping. arXiv preprint arXiv:2305.10721 , 2023.

Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems , 35:5816-5828, 2022.

Yeqi Liu, Chuanyang Gong, Ling Yang, and Yingyi Chen. Dstp-rnn: A dual-stage two-phase attention-based recurrent neural network for long-term and multivariate time series prediction. Expert Systems with Applications , 143:113082, 2020.

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625 , 2023.

Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Rethinking the stationarity in time series forecasting. NeurIPS , 2022.

MickOhrberg. Brownian noise - Wikipedia, the free encyclopedia, 2024. [Online; accessed 9-April-2024].

Fernando Moreno-Pino, Pablo M Olmos, and Antonio Artés-Rodríguez. Deep autoregressive models with spectral attention. Pattern Recognition , 133:109014, 2023.

Brian K Nelson. Time series analysis using autoregressive integrated moving average (arima) models. Academic emergency medicine , 5(7):739-744, 1998.

Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730 , 2022.

Vigneashwara Pandiyan, Rita Drissi-Daoudi, Sergey Shevchik, Giulio Masinelli, Roland Logé, and Kilian Wasmer. Analysis of time, frequency and time-frequency domain features from acoustic emissions during laser powder-bed fusion process. Procedia CIRP , 94:392-397, 2020.

Adam Paszke, S. Gross, Francisco Massa, A. Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Z. Lin, N. Gimelshein, L. Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. NeurIPS , 2019.

Hadi Rezaei, Hamidreza Faaljou, and Gholamreza Mansourfar. Stock price prediction using deep learning and frequency decomposition. Expert Systems with Applications , 169:114332, 2021.

Frank Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para . Cornell Aeronautical Laboratory, 1957.

Alaa Sagheer and Mostafa Kotb. Time series forecasting of petroleum production using deep lstm recurrent networks. Neurocomputing , 323:203-213, 2019.

Donghwan Song, Adrian Matias Chung Baek, and Namhun Kim. Forecasting stock market indices using padding-based fourier transform denoising and time series deep learning models. IEEE Access , 9:83786-83796, 2021.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems , 27, 2014.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Renzhuo Wan, Shuping Mei, Jun Wang, Min Liu, and Fan Yang. Multivariate temporal convolutional network: A deep neural networks approach for multivariate time series forecasting. Electronics , 8(8):876, 2019.

Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. arXiv preprint arXiv:2202.07125 , 2022.

Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE , 78(10):1550-1560, 1990.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. ICLR , 2023.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems , 34:22419-22430, 2021.

Zhijian Xu, Ailing Zeng, and Qiang Xu. FITS: Modeling time series with $10k$ parameters. In The Twelfth International Conference on Learning Representations , 2024.

Fuhao Yang, Xin Li, Min Wang, Hongyu Zang, Wei Pang, and Mingzhong Wang. Waveform: Graph enhanced wavelet learning for long sequence forecasting of multivariate time series. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 10754-10761, 2023.

Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Ning An, Defu Lian, Longbing Cao, and Zhendong Niu. Frequency-domain mlps are more effective learners in time series forecasting. Advances in Neural Information Processing Systems , 36, 2024.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence , volume 37, pages 11121-11128, 2023.

Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The eleventh international conference on learning representations , 2022.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 11106-11115, 2021.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning , pages 27268-27286. PMLR, 2022.

A Details on Datasets, Models, and Training

A.1 Details on datasets

Dataset Information. We conducted experiments on 11 real-world datasets to assess the performance of baseline models and the proposed BSA method. The Weather dataset [49] includes 21 meteorological factors acquired every 10 minutes in 2020 from the Weather Station of the Max Planck Institute for Biogeochemistry. The Traffic dataset [49] records hourly road occupancy rates from 862 sensors on San Francisco Bay Area freeways, covering the period from January 2015 to December 2016. The ECL dataset [49] captures the hourly electricity consumption of 321 clients. The ETT dataset [55] contains seven factors related to electricity transformers, spanning from July 2016 to July 2018. It is divided into four sub-datasets: ETTh1 and ETTh2 are recorded hourly, while ETTm1 and ETTm2 are collected every 15 minutes. The Exchange dataset [49] comprises panel data of daily exchange rates from eight countries, ranging from 1990 to 2016. Illness dataset [49] contains weekly data on influenza-like illness (ILI) cases recorded by the U.S. Centers for Disease Control and Prevention (CDC) from 2002 to 2021. The dataset tracks the proportion of ILI patients relative to the overall number of patients seen during that period. PEMS03 dataset [29] is a sub-dataset of the PEMS dataset, which includes public traffic network data from California, recorded at 5-minute intervals. The EnergyData dataset [6] comprises hourly end-use measurements gathered from 454 residential properties and 140 commercial establishments located in the Pacific Northwest. All these public datasets were downloaded from the referenced sources in March 2024.

Dataset split. We adhere to the data processing protocol and train-validation-test split used in TimesNet [48], where the training, validation, and test datasets are sequentially separated in chronological order. Our paper's data split ratios for train, validation, and test set are as follows: (0.7, 0.1, 0.2) for Weather, Traffic, ECL, Exchange, Ilnness, PEMS03, and EnergyData datasets and (0.6, 0.2, 0.2) for ETT. The details of datasets are provided in Table 6.

Forecasting setting. Following the approach in iTransformer [31], the look-back window length is set to {96}, while the forecast lengths are {96, 192, 336, 720} for the Weather, Traffic, ECL, ETT, PEMS03, and EnergyData datasets. Forecasting lengths is only {96} for the Exchange dataset since the dataset is too short, which causes biased best model selection with the validation set. For Illness dataset, the look-back window length is {36} and the forecasting lengths are {24, 36, 48, 60}.

A.2 Details on Model Implementations

Model configurations. In this section, we discuss the configurations of the baseline models and specify where the BSA module was integrated into each baseline model. For each dataset, the base model structure configuration was directly replicated from the Time-Series-Library [48] scripts

when available. Where configurations were not provided, we adjusted them to align closely with the available examples.

1. DLinear [53]: The only hyperparameter for this baseline model is the (moving_average = 25) for the series decomposition module from Autoformer [49]. We set the Individual to True so that there are separate linear models for each number of input variables. The BSA module is implemented at the very beginning of the forward pass right before series decomposition. The BSA module is implemented for each channel, which is the number of input variables for this case.

2. iTransformer [31]: The hyperparameters for this baseline model are as follows:

(d_model = 512, d_ff = 512, dropout = 0.1, num_heads = 8, encoder_layers = 3, activation = GELU [11]) for the Weather, ECL, PEMS03, and EnergyData datasets,

(d_model = 512, d_ff = 512, dropout = 0.1, num_heads = 8, encoder_layers = 4, activation = GELU) for the Traffic dataset,

(d_model = 128, d_ff = 128, dropout = 0.1, num_heads = 8, encoder_layers = 2, activation = GELU) for the ETT and Exchange datasets.

The BSA module is implemented at the beginning part of the forward pass right after normalization from the Non-stationary Transformer [32] and right before the input embedding. The BSA module is implemented for each channel, which is the number of input variables for this case.

3. Crossformer [54]: The hyperparameters for this baseline model are as follows:

(seg_len = 12, win_size = 2, factor = 3, d_model = 32, d_ff = 32, dropout = 0.1, num_heads = 8, encoder_layers = 2, activation = RELU [1]) for the Weather and EnergyData dataset,

(seg_len = 12, win_size = 2, factor = 3, d_model = 128, d_ff = 128, dropout = 0.1, num_heads = 2, encoder_layers = 2, activation = RELU) for the Traffic dataset,

(seg_len = 12, win_size = 2, factor = 3, d_model = 256, d_ff = 512, dropout = 0.1, num_heads = 8, encoder_layers = 2, activation = RELU) for the ECL and PEMS03 dataset,

(seg_len = 12, win_size = 2, factor = 3, d_model = 512, d_ff = 2048, dropout = 0.1, num_heads = 8, encoder_layers = 2, activation = RELU) for the ETTh1 and ETTh2 datsets,

(seg_len = 12, win_size = 2, factor = 1, d_model = 512, d_ff = 2048, dropout = 0.1, num_heads = 8, encoder_layers = 2, activation = RELU) for the ETTm1 and ETTm2 datasets,

(seg_len = 12, win_size = 2, factor = 3, d_model = 64, d_ff = 64, dropout = 0.1, num_heads = 8, encoder_layers = 2, activation = RELU) for the Exchange dataset.

The BSA module is implemented at the very beginning of the forward pass, right before the input embedding. The BSA module is implemented for each channel, which is the number of input variables for this case.

4. PatchTST [36]: The hyperparameters for this baseline model are as follows:

(patch_len = 16, stride = 8, d_model = 512, d_ff = 2048, dropout = 0.1, num_heads = 4, encoder_layers = 2, activation = GELU [11]) for the Weather and EnergyData dataset,

(patch_len = 16, stride = 8, d_model = 512, d_ff = 512, dropout = 0.1, num_heads = 8, encoder_layers = 2, activation = GELU) for the Traffic dataset,

(patch_len = 16, stride = 8, d_model = 512, d_ff = 2048, dropout = 0.1, num_heads = 8, encoder_layers = 2, activation = GELU) for the ECL and PEMS03 dataset,

(patch_len = 16, stride = 8, d_model = 512, d_ff = 2048, dropout = 0.1, num_heads = 8, encoder_layers = 1, activation = GELU) for the ETTh1 datset,

(patch_len = 16, stride = 8, d_model = 512, d_ff = 2048, dropout = 0.1, num_heads = 4, encoder_layers = 3, activation = GELU) for the ETTh2, ETTm1, and ETTm2 datsets.

The BSA module is implemented at the beginning part of the forward pass right after normalization from the Non-stationary Transformer [32] and right before the patch embedding. The BSA module is implemented for each channel, which is the number of input variables for this case.

5. TimesNet [48]: The hyperparameters for this baseline model are as follows:

(top_k = 5, num_kernels = 6, embed = 'timeF', freq = 'h', d_model = 32, d_ff = 32, dropout = 0.1, encoder_layers = 2) for the Weather, ETTh2, ETTm2, and EnergyData datasets,

(top_k = 5, num_kernels = 6, embed = 'timeF', freq = 'h', d_model = 512, d_ff = 512, dropout = 0.1, encoder_layers = 2) for the Traffic dataset,

(top_k = 5, num_kernels = 6, embed = 'timeF', freq = 'h', d_model = 256, d_ff = 512, dropout = 0.1, encoder_layers = 2) for the ECL and PEMS03 dataset,

(top_k = 5, num_kernels = 6, embed = 'timeF', freq = 'h', d_model = 16, d_ff = 32, dropout = 0.1, encoder_layers = 2) for the ETTh1 and ETTm1 datasets,

(top_k = 5, num_kernels = 6, embed = 'timeF', freq = 'h', d_model = 64, d_ff = 64, dropout = 0.1, encoder_layers = 2) for the Exchange dataset.

The BSA module is implemented at the beginning part of the forward pass right after normalization from the Non-stationary Transformer [32] and right before the input embedding. The BSA module is implemented for each channel, which is the number of input variables for this case.

6. FreTS [52]: The hyperparameters for this baseline model are as follows:

(embed_size = 128, hidden_size = 256, sparsity_threshold = 0.01, scale = 0.02) for all datasets. Based on the paper, the channel-independent strategy is selected.

The BSA module is implemented at the very beginning of the forward pass, right before the token embedding. The BSA module is implemented for each channel, which is the number of input variables for this case.

7. RLinear [28]: The only hyperparameter for this baseline model is the (dropout = 0.1). We set the Individual to True so that there are separate linear models for each number of input variables. The BSA module is implemented at the beginning part of the forward pass right after normalization from the RevIN [21] and right before the linear layer. The BSA module is implemented for each channel, which is the number of input variables for this case.

A.3 Details on Training

Pre-training and Finetuning configurations. To show how our BSA module performs when added to the baseline models, we saturated the models by greedy hyperparameter search.

The hyperparameter search space for the base model is as follows: the possible learning rate is (0.03, 0.01, 0.003, 0.001, 0.0003), and the weight decay is (0.01, 0.003, 0.001, 0.0003, 0.0001, 0.00003).

The hyperparameter search space for BSA finetuning is as follows: the possible learning rate for the SA-Matrix in the BSA module is (0.08, 0.05, 0.03, 0.01, 0.003, 0.001), learning rate for the rest of the model, i.e. original modules, is (0.01, 0.003, 0.001, 0.0003, 0.0001, 0.00003, 0.00001), learning rate for smoothing factor α k is (none, 0.03, 0.01, 0.003, 0.001, 0.0001, 0.00001), initialization for smoothing factor α k is ([0.9, 0.99, 0.999], [0.9, 0.99, 0.999, 0.999], [0.9, 0.95, 0.992, 0.999], [0.8, 0.96, 0.992, 0.9984, 0.99968]).

Model selection Hyperparameter search is conducted based on the validation set. While models trained using conventional sample shuffling evenly represent the entire time series distribution from which the dataset is sampled, BSA learns data in chronological order. Consequently, the final model tends to favor the distribution of the later part of the data. This can be seen as a mild version of catastrophic forgetting, commonly occurring in continual learning. To mitigate this effect, we assigned higher weights to the later samples during the validation process. The weights are represented by 0 . 5 + 0 . 5 × sin ( π 2 × val _ idx val _ len ) , continuously increasing from 0.5 for the first sample to 1.0 for the last sample.

Optimization The whole code is implemented in PyTorch [38]. Each experiment was conducted on a single NVIDIA GeForce RTX 3090Ti or NVIDIA A40 or NVIDIA L40 GPU. The default batch size for baseline model saturation is 64, while for our method-which involves fine-tuning after integrating the BSA module-it is 256. If a baseline model is too heavy and results in GPU memory overflow, the batch size is adjusted to fit within the available memory. We used the ADAM [22] optimizer and L2 loss (MSE loss) for the model optimization. The baseline saturation training epoch is set to 40 epochs for the Weather dataset, 20 epochs for the Traffic dataset, and 30 epochs for the rest of the datasets. The finetuning epoch is set to 30 epochs for the ETTh1, ETTh2, and Exchange datasets and 20 epochs for the rest of the datasets.

B Real world dataset experiments

B.1 Full experiment results

The experiments were conducted on a total of 11 public real-world datasets and 7 forecasting models. The average results (MSE, MAE) of the experiments conducted with three random seeds 0, 1, and 2 are shown in Table 7, and the standard deviations are shown in Table 8. The values reported in Table 1 are labeled as Avg in Table 7. For the Exchange dataset, we only report experiments with a prediction

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

800

1600

2400

3200

4000

Amplitude

(a) Pre-Denoising

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

800

1600

2400

3200

4000

Amplitude

(b) Post-denoising

-80

-70

-60

-50

-40

-30

-20

100

1000

10000

Intensity (dB)

Frequency (Hz)

length of {96}. Using prediction lengths of {192, 336, 720} causes improper best model selection due to an insufficient number of validation samples. The empty results indicate that the training is too heavy, and the results are not yet available. We expect to have the results ready by the rebuttal.

B.2 SA-Matrix and FFT visualization

To investigate how the SA-Matrix of BSA predominantly attends to specific frequency bands, we employed the heatmaps, the Gaussian kernel density estimate graphs, and the FFT graphs as shown in Figure 9, 10, 11, 12, 15 and 13.

The 2D heatmap depicts the learnable parameters of the SA-Matrix, defined as sa-matrix ∈ R (2 K +1) × D , where K represents the number of smoothing factors and D represents the dimension of the input features. Consequently, the y-axis of the 2D heatmap matches the length of D , and the x-axis corresponds to 2 K +1 , symmetrically arranged around zero. On the x-axis, positive values indicate the low-frequency regions, while negative values represent the high-frequency regions.

The Gaussian kernel density estimate graphs intuitively reveal which frequency bands the SA-Matrix predominantly attends to. The K smoothing factors were modified according to Equation 8 and symmetrically arranged around zero, serving as the data points for kernel density estimation (KDE). The weight values from each row of the SA-Matrix were converted to probabilities using the softmax function, and the resulting outputs established a mapping of these weight values to data points necessary for Gaussian KDE. This mapping facilitated the construction of the Gaussian KDE graph. Subsequently, the overall probability density of the SA-Matrix was estimated by calculating the mean across columns. Consequently, the y-axis of the graphs denotes the attention level of the SA-Matrix, whereas the x-axis indicates the scalar indices, encompassing the range of the smoothing factors. To better represent the variation in weight values across frequency bands, we adjusted the bandwidth of the KDE function to 0.4, based on the product of the Gaussian kernel's covariance factor and the standard deviation of the sampled weights in the matrix.

The FFT graph is depicted with a red line. FFT analysis was conducted for each variable to determine if the fine-tuned SA-Matrix aligns with the frequencies exhibited by the dataset. To compare the low-frequency parts, the negative log-scale was used on the x-axis, showing that progression to the right indicates decreasing frequencies. The Gaussian filter with a fixed standard deviation of 5 was utilized to smooth the signal's amplitude.

The denoising phase was executed on the FFT output to analyze the correlation between the frequency tendencies of the dataset and the manipulated weights in the SA-Matrix. Figure 8(c) depicts Brownian noise, with the x-axis denoting frequency and the y-axis in decibels (dB). Both axes are configured

Dlinear

RLinear

FreTS

TimesNet

iTransformer

Crossformer

PatchTST

base

BSA

base

BSA

base

BSA

base

BSA

base

BSA

base

BSA

base

BSA

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

MSE

MAE

Weather

96

0.0002

0.0005

0.0000

0.0000

0.0002

0.0003

0.0000

0.0000

0.0001

0.0002

0.0003

0.0005

0.0008

0.0002

0.0015

0.0011

0.0015

0.0017

0.0008

0.0011

0.0024

0.0012

0.0029

0.0027

0.0004

0.0005

0.0012

0.0013

192

0.0003

0.0010

0.0001

0.0000

0.0000

0.0000

0.0000

0.0000

0.0001

0.0002

0.0003

0.0002

0.0019

0.0014

0.0040

0.0019

0.0014

0.0013

0.0019

0.0015

0.0014

0.0022

0.0018

0.0019

0.0000

0.0007

0.0006

0.0003

336

0.0008

0.0014

0.0002

0.0002

0.0000

0.0001

0.0000

0.0000

0.0006

0.0009

0.0002

0.0004

0.0010

0.0002

0.0034

0.0026

0.0003

0.0005

0.0008

0.0010

0.0026

0.0028

0.0019

0.0016

0.0001

0.0004

0.0007

0.0001

720

0.0064

0.0082

0.0003

0.0005

0.0000

0.0000

0.0005

0.0002

0.0043

0.0075

0.0058

0.0059

0.0072

0.0059

0.0038

0.0033

0.0007

0.0004

0.0013

0.0010

0.0027

0.0023

0.0004

0.0014

0.0005

0.0002

0.0001

0.0005

Avg

0.0019

0.0028

0.0002

0.0002

0.0001

0.0001

0.0001

0.0001

0.0013

0.0022

0.0017

0.0017

0.0027

0.0019

0.0032

0.0022

0.0010

0.0010

0.0012

0.0012

0.0023

0.0021

0.0018

0.0019

0.0003

0.0004

0.0006

0.0006

Traffic

96

0.0000

0.0000

0.0001

0.0000

0.0002

0.0001

0.0001

0.0000

0.0008

0.0002

0.0015

0.0002

0.0060

0.0015

0.0053

0.0023

0.0003

0.0003

0.0012

0.0005

0.0208

0.0088

0.0203

0.0099

0.0021

0.0003

0.0008

0.0001

192

0.0000

0.0000

0.0000

0.0000

0.0000

0.0000

0.0000

0.0000

0.0001

0.0002

0.0008

0.0001

0.0082

0.0059

0.0081

0.0049

0.0011

0.0007

0.0009

0.0011

0.0020

0.0004

0.0018

0.0004

0.0009

0.0001

0.0008

0.0008

336

0.0001

0.0000

0.0000

0.0000

0.0001

0.0002

0.0000

0.0000

0.0001

0.0002

0.0004

0.0005

0.0120

0.0033

0.0042

0.0030

0.0009

0.0009

0.0005

0.0013

0.0227

0.0058

0.0027

0.0007

0.0025

0.0005

0.0014

0.0010

720

0.0001

0.0000

0.0001

0.0000

0.0000

0.0000

0.0000

0.0000

0.0008

0.0001

0.0008

0.0002

0.0032

0.0030

0.0022

0.0015

0.0006

0.0003

0.0023

0.0005

0.0005

0.0002

0.0034

0.0005

0.0012

0.0007

0.0014

0.0006

Avg

0.0000

0.0000

0.0000

0.0000

0.0001

0.0001

0.0001

0.0000

0.0005

0.0002

0.0009

0.0002

0.0073

0.0034

0.0050

0.0029

0.0007

0.0006

0.0012

0.0008

0.0115

0.0038

0.0071

0.0029

0.0017

0.0004

0.0011

0.0006

ECL

96

0.0001

0.0000

0.0000

0.0000

0.0000

0.0000

0.0000

0.0000

0.0005

0.0004

0.0002

0.0001

0.0021

0.0013

0.0029

0.0024

0.0009

0.0009

0.0006

0.0005

0.0051

0.0047

0.0052

0.0051

0.0005

0.0011

0.0005

0.0006

192

0.0003

0.0002

0.0000

0.0000

0.0000

0.0000

0.0000

0.0000

0.0004

0.0004

0.0004

0.0004

0.0119

0.0084

0.0076

0.0055

0.0008

0.0005

0.0013

0.0011

0.0057

0.0054

0.0084

0.0082

0.0014

0.0011

0.0004

0.0001

336

0.0001

0.0001

0.0000

0.0000

0.0000

0.0000

0.0000

0.0000

0.0007

0.0004

0.0001

0.0001

0.0167

0.0118

0.0124

0.0088

0.0028

0.0039

0.0020

0.0029

0.0013

0.0015

0.0038

0.0058

0.0007

0.0013

0.0002

0.0004

720

0.0001

0.0001

0.0001

0.0001

0.0003

0.0002

0.0002

0.0002

0.0009

0.0008

0.0034

0.0025

0.0070

0.0060

0.0057

0.0041

0.0038

0.0050

0.0001

0.0002

0.0254

0.0169

0.0204

0.0148

0.0009

0.0020

0.0007

0.0007

Avg

0.0001

0.0001

0.0000

0.0000

0.0001

0.0001

0.0000

0.0001

0.0006

0.0005

0.0010

0.0008

0.0094

0.0069

0.0071

0.0052

0.0021

0.0026

0.0010

0.0012

0.0094

0.0071

0.0095

0.0085

0.0008

0.0014

0.0005

0.0005

ETTH1

96

0.0002

0.0002

0.0001

0.0002

0.0001

0.0002

0.0001

0.0001

0.0053

0.0019

0.0048

0.0015

0.0050

0.0039

0.0048

0.0017

0.0006

0.0014

0.0002

0.0007

0.0005

0.0006

0.0006

0.0003

0.0004

0.0006

0.0010

0.0012

192

0.0006

0.0004

0.0002

0.0001

0.0000

0.0001

0.0004

0.0003

0.0015

0.0004

0.0008

0.0002

0.0063

0.0067

0.0052

0.0054

0.0013

0.0006

0.0003

0.0006

0.0012

0.0009

0.0013

0.0011

0.0047

0.0033

0.0024

0.0010

336

0.0011

0.0006

0.0003

0.0002

0.0001

0.0001

0.0000

0.0000

0.0029

0.0005

0.0024

0.0007

0.0083

0.0058

0.0069

0.0048

0.0024

0.0009

0.0004

0.0004

0.0023

0.0015

0.0008

0.0009

0.0032

0.0013

0.0035

0.0021

720

0.0007

0.0009

0.0004

0.0006

0.0003

0.0001

0.0001

0.0000

0.0026

0.0023

0.0022

0.0019

0.0120

0.0038

0.0111

0.0035

0.0054

0.0046

0.0038

0.0024

0.0125

0.0071

0.0012

0.0007

0.0009

0.0001

0.0039

0.0021

Avg

0.0007

0.0005

0.0002

0.0003

0.0001

0.0001

0.0002

0.0001

0.0031

0.0013

0.0025

0.0011

0.0079

0.0051

0.0070

0.0039

0.0024

0.0019

0.0012

0.0010

0.0041

0.0025

0.0010

0.0007

0.0023

0.0013

0.0027

0.0016

ETTH2

96

0.0116

0.0108

0.0095

0.0081

0.0001

0.0001

0.0001

0.0001

0.0097

0.0095

0.0051

0.0053

0.0064

0.0051

0.0041

0.0036

0.0011

0.0009

0.0003

0.0006

0.0028

0.0014

0.0032

0.0013

0.0060

0.0027

0.0038

0.0020

192

0.0152

0.0098

0.0145

0.0088

0.0006

0.0004

0.0003

0.0001

0.0009

0.0070

0.0014

0.0049

0.0268

0.0132

0.0268

0.0154

0.0022

0.0012

0.0007

0.0004

0.0012

0.0007

0.0009

0.0009

0.0014

0.0015

0.0005

0.0005

336

0.0191

0.0065

0.0160

0.0062

0.0007

0.0006

0.0003

0.0003

0.0018

0.0047

0.0057

0.0053

0.0152

0.0098

0.0189

0.0107

0.0006

0.0008

0.0020

0.0021

0.0048

0.0097

0.0038

0.0056

0.0021

0.0006

0.0011

0.0021

720

0.0231

0.0141

0.0222

0.0137

0.0003

0.0001

0.0017

0.0012

0.0242

0.0151

0.0051

0.0008

0.0096

0.0048

0.0338

0.0163

0.0005

0.0010

0.0010

0.0003

0.0367

0.0340

0.0130

0.0116

0.0019

0.0010

0.0007

0.0003

Avg

0.0173

0.0103

0.0156

0.0092

0.0004

0.0003

0.0006

0.0004

0.0092

0.0090

0.0044

0.0041

0.0145

0.0082

0.0209

0.0115

0.0011

0.0010

0.0010

0.0009

0.0114

0.0115

0.0052

0.0048

0.0028

0.0015

0.0015

0.0012

ETTm1

96

0.0003

0.0001

0.0000

0.0000

0.0003

0.0001

0.0006

0.0002

0.0015

0.0005

0.0007

0.0004

0.0055

0.0018

0.0067

0.0025

0.0006

0.0006

0.0027

0.0015

0.0014

0.0014

0.0047

0.0031

0.0069

0.0032

0.0010

0.0007

192

0.0002

0.0001

0.0000

0.0000

0.0003

0.0002

0.0003

0.0001

0.0008

0.0003

0.0009

0.0002

0.0087

0.0062

0.0075

0.0051

0.0063

0.0043

0.0050

0.0030

0.0079

0.0041

0.0047

0.0016

0.0037

0.0017

0.0025

0.0010

336

0.0006

0.0002

0.0001

0.0001

0.0000

0.0000

0.0002

0.0000

0.0005

0.0007

0.0012

0.0009

0.0052

0.0049

0.0036

0.0027

0.0024

0.0008

0.0029

0.0020

0.0186

0.0125

0.0345

0.0211

0.0024

0.0021

0.0026

0.0028

720

0.0011

0.0006

0.0001

0.0000

0.0006

0.0003

0.0001

0.0001

0.0044

0.0012

0.0011

0.0009

0.0058

0.0029

0.0062

0.0031

0.0032

0.0018

0.0011

0.0009

0.0055

0.0024

0.0035

0.0026

0.0041

0.0034

0.0008

0.0011

Avg

0.0006

0.0003

0.0001

0.0000

0.0003

0.0002

0.0003

0.0001

0.0018

0.0007

0.0010

0.0006

0.0063

0.0039

0.0060

0.0033

0.0031

0.0019

0.0029

0.0019

0.0083

0.0051

0.0119

0.0071

0.0043

0.0026

0.0017

0.0014

ETTm2

96

0.0038

0.0040

0.0005

0.0004

0.0001

0.0002

0.0001

0.0001

0.0010

0.0009

0.0004

0.0003

0.0015

0.0017

0.0004

0.0005

0.0009

0.0010

0.0006

0.0006

0.0048

0.0083

0.0023

0.0038

0.0004

0.0004

0.0001

0.0001

192

0.0069

0.0032

0.0015

0.0009

0.0000

0.0001

0.0000

0.0000

0.0002

0.0011

0.0013

0.0004

0.0050

0.0025

0.0052

0.0030

0.0005

0.0002

0.0007

0.0005

0.0061

0.0080

0.0008

0.0004

0.0010

0.0006

0.0011

0.0011

336

0.0062

0.0057

0.0041

0.0022

0.0002

0.0002

0.0001

0.0000

0.0052

0.0037

0.0046

0.0030

0.0212

0.0114

0.0181

0.0098

0.0002

0.0008

0.0013

0.0009

0.0017

0.0140

0.0057

0.0077

0.0005

0.0006

0.0004

0.0003

720

0.0238

0.0160

0.0044

0.0025

0.0005

0.0003

0.0000

0.0000

0.0323

0.0112

0.0172

0.0062

0.0236

0.0154

0.0015

0.0011

0.0010

0.0009

0.0009

0.0008

0.0034

0.0038

0.0056

0.0039

0.0004

0.0009

0.0009

0.0004

Avg

0.0102

0.0072

0.0026

0.0015

0.0002

0.0002

0.0001

0.0000

0.0097

0.0043

0.0059

0.0025

0.0128

0.0078

0.0063

0.0036

0.0007

0.0007

0.0009

0.0007

0.0040

0.0085

0.0036

0.0040

0.0006

0.0006

0.0006

0.0005

Exchange

96

0.0025

0.0023

0.0002

0.0003

0.0017

0.0012

0.0015

0.0011

0.0043

0.0077

0.0016

0.0031

0.0005

0.0004

0.0004

0.0004

0.0020

0.0018

0.0020

0.0019

0.0172

0.0133

0.0161

0.0123

0.0033

0.0033

0.0026

0.0025

PEMS03

96

0.0003

0.0004

0.0004

0.0005

0.0017

0.0020

0.0002

0.0001

0.0009

0.0011

0.0008

0.0008

0.0054

0.0035

0.0019

0.0007

0.0021

0.0011

0.0009

0.0006

0.0034

0.0015

0.0039

0.0037

0.0012

0.0016

0.0006

0.0007

192

0.0010

0.0006

0.0000

0.0000

0.0011

0.0011

0.0024

0.0006

0.0002

0.0001

0.0022

0.0014

0.0034

0.0026

0.0063

0.0048

0.0035

0.0020

0.0019

0.0013

0.0508

0.0400

0.0235

0.0207

0.0001

0.0002

0.0012

0.0010

336

0.0001

0.0004

0.0001

0.0000

0.0012

0.0002

0.0009

0.0006

0.0004

0.0001

0.0019

0.0012

0.0042

0.0030

0.0014

0.0005

0.0022

0.0011

0.0019

0.0014

0.0061

0.0066

0.0121

0.0104

0.0007

0.0003

0.0007

0.0007

720

0.0003

0.0004

0.0002

0.0001

0.0015

0.0008

0.0028

0.0017

0.0001

0.0002

0.0011

0.0007

0.0004

0.0012

0.0024

0.0019

0.0045

0.0015

0.0029

0.0017

0.0612

0.0425

0.0610

0.0458

0.0010

0.0006

0.0014

0.0008

Avg

0.0004

0.0004

0.0002

0.0001

0.0014

0.0010

0.0016

0.0008

0.0004

0.0004

0.0015

0.0010

0.0034

0.0026

0.0030

0.0020

0.0031

0.0014

0.0019

0.0012

0.0304

0.0227

0.0251

0.0201

0.0008

0.0007

0.0010

0.0008

EnergyData

96

0.0030

0.0027

0.0002

0.0002

0.0002

0.0001

0.0002

0.0001

0.0078

0.0065

0.0022

0.0017

0.0026

0.0018

0.0023

0.0017

0.0036

0.0027

0.0039

0.0024

0.0485

0.0262

0.0101

0.0068

0.0006

0.0018

0.0013

0.0005

192

0.0044

0.0030

0.0006

0.0005

0.0003

0.0001

0.0001

0.0000

0.0530

0.0288

0.0157

0.0081

0.0020

0.0010

0.0017

0.0005

0.0028

0.0028

0.0006

0.0005

0.0665

0.0250

0.0780

0.0276

0.0009

0.0016

0.0017

0.0006

336

0.0078

0.0041

0.0027

0.0010

0.0006

0.0003

0.0003

0.0001

0.0096

0.0041

0.0096

0.0032

0.0071

0.0028

0.0091

0.0030

0.0006

0.0010

0.0030

0.0015

0.0285

0.0051

0.0272

0.0049

0.0056

0.0010

0.0018

0.0008

720

0.0147

0.0046

0.0100

0.0042

0.0006

0.0003

0.0004

0.0001

0.0135

0.0072

0.0577

0.0232

0.0407

0.0108

0.0159

0.0033

0.0071

0.0022

0.0084

0.0026

0.0638

0.0184

0.0393

0.0120

0.0027

0.0013

0.0048

0.0008

Avg

0.0075

0.0036

0.0034

0.0015

0.0004

0.0002

0.0002

0.0001

0.0210

0.0117

0.0213

0.0091

0.0131

0.0041

0.0073

0.0021

0.0035

0.0022

0.0040

0.0017

0.0518

0.0187

0.0386

0.0128

0.0024

0.0014

0.0024

0.0007

Illness

24

0.0366

0.0186

0.0018

0.0011

0.0561

0.0168

0.0388

0.0131

0.8467

0.1525

0.3844

0.0673

1.3113

0.1001

1.3169

0.1003

0.6023

0.0524

0.5739

0.0528

0.1945

0.0299

0.2660

0.0507

0.0887

0.0072

0.0794

0.0098

36

0.0730

0.0477

0.0624

0.0409

0.0414

0.0104

0.0395

0.0096

0.3923

0.0615

0.5034

0.0926

0.3605

0.0246

0.4506

0.0295

0.2656

0.0474

0.2402

0.0386

0.0960

0.0217

0.0774

0.0169

0.0629

0.0305

0.0313

0.0128

48

0.2656

0.0754

0.1651

0.0482

0.0304

0.0065

0.0320

0.0056

0.4186

0.0755

0.1296

0.0311

0.0553

0.0273

0.0506

0.0250

0.0527

0.0039

0.1422

0.0168

0.1000

0.0161

0.0906

0.0263

0.0402

0.0181

0.0223

0.0099

60

0.0427

0.0115

0.0791

0.0162

0.0414

0.0123

0.0264

0.0092

0.1066

0.0203

0.2392

0.0413

0.0543

0.0281

0.0668

0.0305

0.0197

0.0041

0.0565

0.0123

0.0908

0.0181

0.0921

0.0182

0.1070

0.0293

0.0436

0.0114

Avg

0.1045

0.0383

0.0771

0.0266

0.0424

0.0115

0.0342

0.0094

0.4410

0.0775

0.3142

0.0581

0.4454

0.0450

0.4712

0.0463

0.2351

0.0269

0.2532

0.0301

0.1204

0.0214

0.1315

0.0280

0.0747

0.0213

0.0441

0.0110

on a log scale. Converting the y-axis from decibels to amplitude is akin to implementing linear scaling, which reveals a steeper rise in noise at lower frequencies. This noise generally arises in the low-frequency bands of most real-world systems, such as electronic systems and environmental sciences [24]. Figure 8 (a) shows that noise escalates within the low-frequency regions across variables, leading to the assumption that unique Brownian noise originates in each system. Consequently, a linear approximation was used for the denoising function, which was performed manually. The outcomes of this process are presented in Figure 8 (b).

C Synthetic dataset experiments

We constructed synthetic datasets by adding sine waves with periods of 100, 300, and 1000 to each channel (variable) of the ETTh1 and ETTh2 datasets. The scale of the sine wave is set to the standard deviation value of each channel. The phases of the sine waves for each channel are randomly sampled from a uniform distribution from 0 to 2 π to make decorrelated sine waves. The full results are reported in Table 9.

C.1 SA-Matrix and FFT visualization

As illustrated in Figure 14, the LUFL, MULL, OT, and LULL channels are organized into columns, with each graph positioned within rows that correspond to the periods added to the dataset. Figure 14 (a) shows the SA-Matrix from the iTransformer model fine-tuned with a 720 prediction length on the ETTh1 data that does not include added sine waves. From (b) to (d), the models were trained with synthetic datasets increasingly augmented with sine waves at periods of 100, 300, and 1000. As the number of added periods increases, there is a shift in the SA-Matrix weight distribution toward low-frequency patterns in each channel.

D Analysis and Ablation Studies

D.1 BSA insertion site analysis

Table 10 shows the full results for different BSA module insertion sites in the iTransformer [31] and DLinear [53] models on the Weather Dataset. Table 2 is from the average value of Table 10. The BSA module insert position can be found in Figure 7.

D.2 BSA variable input lengths analysis

Table 11 shows the full results of the performance of the base model and BSA according to changes in Input length. Experiments were conducted on Weather and PEMS03 data using DLinear [53], RLinear [28], and iTransformer [31] models. Table 3 in the main text shows the summarized results.

D.3 BSA computational cost analysis

Table 12 presents a comprehensive analysis of BSA's computational cost. We measured the model training time (sec/1step), peak memory usage (GB), and the number of parameters (M) for both the base model and the model with BSA applied. The experiments were conducted using the lightweight Weather dataset with 21 channels and the heavy PEMS03 dataset with 358 channels. Tests were

performed on Timesnet [48], iTransformer [31], Crossformer [54], and PatchTST [36] models. Linear models were excluded from the experiments as they are very lightweight and their training cost is not an issue. Table 4 in the main text shows the summarized results.

D.4 Pre/Post BSA activation signal visualization

Figure 16 demonstrates the transition between pre-activation F (left) and post-activation F ′ (right) during the inference process. The figure shows the spectrum of randomly sampled activations from the DLinear model trained with a 720 prediction length on the SWDR (W/m) channel of the weather dataset. The x-axis signifies timesteps while the y-axis indicates frequency. To enhance visibility in the low-frequency region, y-axis values below 0.1 are presented in a log scale, whereas values above 0.1 are depicted in a linear scale. A darkening in the high-frequency region is observed from pre-activation to post-activation, suggesting that the activations may undergo a denoising effect in the high-frequency areas as they pass through the SA-Matrix.

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

p (mbar)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

T (degC)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

Tpot (K)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

Tdew (degC)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

rh (%)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

VPmax (mbar)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

VPact (mbar)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

VPdef (mbar)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

sh (g/kg)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

H2OC (mmol/mol)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

rho (g/m**3)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

wv (m/s)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

max. wv (m/s)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

wd (deg)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

rain (mm)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

raining (s)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

SWDR (W/m )

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

PAR ( mol/m /s)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

max. PAR ( mol/m /s)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

Tlog (degC)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

0

100

200

300

400

500

600

700

Feature

OT

4

2

0

2

4

5

4

3

2

1

0

1

2

5

4

3

2

1

0

1

2

5

4

3

2

1

0

1

2

6

5

4

3

2

1

0

1

5

4

3

2

1

0

1

2

5

4

3

2

1

0

1

2

6

4

2

0

2

5

4

3

2

1

0

1

2

5

4

3

2

1

0

1

2

5

4

3

2

1

0

1

2

5

4

3

2

1

0

1

2

6

4

2

0

2

6

5

4

3

2

1

0

1

2

6

4

2

0

2

4

2

0

2

6

4

2

0

2

6

4

2

0

2

6

4

2

0

2

5

4

3

2

1

0

1

2

4

2

0

2

4

-4

-3

-2

-1

0

1

2

3

4

0.00

0.25

0.50

0.75

1.00

Attention Level

p (mbar)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.08

0.16

0.24

0.32

Attention Level

T (degC)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.08

0.16

0.24

0.32

Attention Level

Tpot (K)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

Attention Level

Tdew (degC)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.06

0.12

0.18

0.24

Attention Level

rh (%)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.08

0.16

0.24

0.32

Attention Level

VPmax (mbar)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

VPact (mbar)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.08

0.16

0.24

0.32

Attention Level

VPdef (mbar)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

sh (g/kg)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

H2OC (mmol/mol)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.08

0.16

0.24

0.32

Attention Level

rho (g/m**3)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

0.60

Attention Level

wv (m/s)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.05

0.10

0.15

0.20

Attention Level

max. wv (m/s)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.08

0.16

0.24

0.32

Attention Level

wd (deg)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

rain (mm)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.08

0.16

0.24

0.32

Attention Level

raining (s)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.06

0.12

0.18

0.24

Attention Level

SWDR (W/m )

-4

-3

-2

-1

0

1

2

3

4

0.00

0.06

0.12

0.18

0.24

Attention Level

PAR ( mol/m /s)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.08

0.16

0.24

0.32

Attention Level

max. PAR ( mol/m /s)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.08

0.16

0.24

0.32

Attention Level

Tlog (degC)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

OT

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

Learnable_matrix/itransformer_adam/finetune-720/iTransformer_momentum1/seed0

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

p (mbar)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

T (degC)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

Tpot (K)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.08

0.16

0.24

0.32

Attention Level

Tdew (degC)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

rh (%)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

VPmax (mbar)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

VPact (mbar)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

Attention Level

VPdef (mbar)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

sh (g/kg)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

H2OC (mmol/mol)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

rho (g/m**3)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

wv (m/s)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

max. wv (m/s)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

wd (deg)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

rain (mm)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

raining (s)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

SWDR (W/m )

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

PAR ( mol/m /s)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.10

0.20

0.30

0.40

Attention Level

max. PAR ( mol/m /s)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

Attention Level

Tlog (degC)

-4

-3

-2

-1

0

1

2

3

4

0.00

0.08

0.16

0.24

0.32

Attention Level

OT

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

Learnable_matrix/dl_adam/finetune-720/DLinear_momentum1/seed0

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

Attention Level

HUFL

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

Attention Level

HULL

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

Attention Level

MUFL

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

Attention Level

MULL

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

Attention Level

LUFL

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

Attention Level

LULL

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

Attention Level

OT

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

Learnable_matrix/dl_adam/finetune-720/DLinear_momentum1/seed0

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

Density

HUFL

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

0.60

Density

HULL

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

Density

MUFL

-4

-3

-2

-1

0

1

2

3

4

0.00

0.20

0.40

0.60

Density

MULL

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

0.60

Density

LUFL

-4

-3

-2

-1

0

1

2

3

4

0.00

0.20

0.40

0.60

Density

LULL

-4

-3

-2

-1

0

1

2

3

4

0.00

0.15

0.30

0.45

0.60

Density

OT

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

-4

-3

-2

-1

0

1

2

3

4

0.00

0.08

0.16

0.24

0.32

Attention Level

LUFL

-4

-3

-2

-1

0

1

2

3

4

0.00

0.08

0.16

0.24

0.32

Attention Level

MULL

-4

-3

-2

-1

0

1

2

3

4

0.00

0.08

0.16

0.24

0.32

Attention Level

OT

-4

-3

-2

-1

0

1

2

3

4

0.00

0.08

0.16

0.24

0.32

Attention Level

LULL

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

0

800

1600

2400

3200

4000

2000

4000

6000

8000

10000

12000

14000

Timestep

0.000

0.001

0.010

0.100

0.500

Frequency [Hz]

1st Pre-Activation of SA-Matrix

2000

4000

6000

8000

10000

12000

14000

Timestep

0.000

0.001

0.010

0.100

0.500

Frequency [Hz]

1st Post-Activation of SA-Matrix

2000

4000

6000

8000

10000

12000

14000

Timestep

0.000

0.001

0.010

0.100

0.500

Frequency [Hz]

26th Pre-Activation of SA-Matrix

2000

4000

6000

8000

10000

12000

14000

Timestep

0.000

0.001

0.010

0.100

0.500

Frequency [Hz]

26th Post-Activation of SA-Matrix

2000

4000

6000

8000

10000

12000

14000

Timestep

0.000

0.001

0.010

0.100

0.500

Frequency [Hz]

72th Pre-Activation of SA-Matrix

2000

4000

6000

8000

10000

12000

14000

Timestep

0.000

0.001

0.010

0.100

0.500

Frequency [Hz]

72th Post-Activation of SA-Matrix

2000

4000

6000

8000

10000

12000

14000

Timestep

0.000

0.001

0.010

0.100

0.500

Frequency [Hz]

167th Pre-Activation of SA-Matrix

2000

4000

6000

8000

10000

12000

14000

Timestep

0.000

0.001

0.010

0.100

0.500

Frequency [Hz]

167th Post-Activation of SA-Matrix

2000

4000

6000

8000

10000

12000

14000

Timestep

0.000

0.001

0.010

0.100

0.500

Frequency [Hz]

2000

4000

6000

8000

10000

12000

14000

Timestep

0.000

0.001

0.010

0.100

0.500

Frequency [Hz]

406th Post-Activation of SA-Matrix

2000

4000

6000

8000

10000

12000

14000

Timestep

0.000

0.001

0.010

0.100

0.500

Frequency [Hz]

681th Pre-Activation of SA-Matrix

2000

4000

6000

8000

10000

12000

14000

Timestep

0.000

0.001

0.010

0.100

0.500

Frequency [Hz]

681th Post-Activation of SA-Matrix

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count toward the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material provided in the appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer:[Yes]

Justification: The paper's contributions and scope are well-explained throughout the abstract and introduction, and the entire paper is written to provide a consistent and comprehensive conclusion.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Our paper discusses the limitations and future research directions to address them in the conclusion section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper proposes a new model architecture and demonstrates its superiority through a practical approach with extensive experiments rather than theoretical proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Our paper provides detailed information to reproduce all experiments. Additionally, the complete experiment code will be provided.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: https://github.com/DJLee1208/BSA_2024

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Our paper provides detailed information about the experiments, including data splits, hyperparameters, how they were chosen, and the type of optimizer, in the main text and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In the main results, we used a paired t-test to demonstrate the superiority of our methodology and reported the p-value.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We reported the types and number of GPUs used in the experiments, as well as the packages used for training.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, the research conducted in the paper fully conforms to the NeurIPS Code of Ethics in every respect. We have ensured that all ethical guidelines and standards have been meticulously followed throughout the study.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Our paper discusses the potential broader impacts of our research in the conclusion.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: Our model does not pose such risks, and therefore, safeguards for responsible data or model release were not applicable.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes, we have properly credited the creators or original owners of assets used in the paper, including publicly available models and packages, with accurate citations.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not include new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer:[NA]

Justification: This paper does not involve research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper focuses on the exploration of long-term dependency within a whole time series sequence, to address the challenge of the short-term look-back window, which is interesting. Based on the observation, this paper provides Batched Spectral Attention to enable parallel training across multiple timesteps.

STRENGTHS:
* The analysis of attention is sufficient to show the FFT graphs.

* The method is designed as an easy plug-in module, which benefits various base models.

WEAKNESSES:
* The writing and logical presentation can be strengthened, which is hard to understand, e.g., Fig. 2. (I spent lots of time understanding the formulations and Fig. 2 )

* It is not clear why only the low-frequency components are used, just for long-term dependency, and why the lack of high-frequency components does not affect the short-term vibrations.

* The comparison in Table 1 may be a little unfair since more computation and memory are introduced by the BSA. Thus, please provide the computation and memory comparison in Table 1 to further evaluate the superiority of BSA.

* I am curious will the performance still be improved when applying BSA as the look-up window T increases or decreases? That is to validate the effectiveness of BSA in more scenarios to demonstrate its generalization ability.

QUESTIONS:
See weaknesses.

LIMITATIONS:
Not sufficient. I can't find the potential negative societal impact of their work in the Conclusion.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a Spectral Attention mechanism to address the challenge of capturing long-range dependencies in time series forecasting. By preserving temporal correlations among samples and leveraging frequency domain information, the proposed method enhances the forecasting performance of various baseline models. Extensive experiments demonstrate the efficacy of Spectral Attention, achieving state-of-the-art results on multiple real-world datasets.\n\nSTRENGTHS:\n1.\tEfficacy: The proposed Spectral Attention mechanism significantly improves forecasting performance across multiple real-world datasets.\n2.\tVersatility: The method can be integrated into various baseline models, showcasing its adaptability and broad applicability.\n3.\tExperimental validation: The experiments show consistent performance improvements.\n\nWEAKNESSES:\n1. Novelty and Contribution/Missing Related Work\n* While the paper introduces Spectral Attention as a novel method, using frequency domain analysis in time series forecasting is not a new concept. Notably, there is an existing paper [1] with the same title, \"Spectral Attention,\" also focused on time series forecasting, which is not referenced in the manuscript. This referenced paper employs a global/local perspective: the global Spectral Attention provides comprehensive information about the random process the time series are considered part of, thereby addressing some limitations mentioned in the paper, such as the inability to model long-range dependencies. Furthermore, the referenced work includes spectral filtering, differing from the authors' approach by learning a cut-off frequency while filtering each frequency component independently. Their solution integrates easily into pre-existing architectures. Given the significant similarities and the relevance of this other paper as a precursor to the authors' idea, why is it not cited in the manuscript? The authors must include a thorough discussion of [1], acknowledging its relation and relevance to their work.\n* Additionally, recent SSM-based approaches should be covered in the related work section.\n2. Differentiation from Existing Methods\n* The paper would benefit from a clearer distinction between Spectral Attention and existing frequency-based methods such as WaveFroM, FEDformer, or the Spectral Attention Autoregressive Model. The unique advantages and innovations of Spectral Attention compared to these methods should be emphasized more explicitly.\n3. Computational Complexity\n* The paper claims that Spectral Attention adds minimal computational overhead. However, the analysis of computational complexity and memory requirements is somewhat superficial. A detailed comparison of training and inference times, as well as memory usage, with and without Spectral Attention, should be provided to substantiate these claims.\n4. Integration with Different Models\n* While the paper demonstrates that Spectral Attention can be integrated with various TSF models, the integration process lacks detail. Specific guidelines or algorithms for integrating Spectral Attention with different architectures should be included. This would facilitate practitioners in applying the proposed method to their models.\n\n\n[1] Moreno-Pino, Fernando, Pablo M. Olmos, and Antonio Artés-Rodríguez. \"Deep autoregressive models with spectral attention.\" Pattern Recognition 133 (2023): 109014.\n\nQUESTIONS:\n1. How does the proposed method specifically improve upon the referenced Spectral Attention Autoregressive Model [1]?\n    * A detailed comparative analysis and, if possible, empirical validation are necessary to substantiate the claims of improvement.\n2. Can you provide a more detailed explanation of the complexity involved in the spectral attention mechanism?\n3. Given that the proposed model operates in the frequency domain, is it capable of handling irregularly sampled data even if the underlying architecture cannot?\n\nI am open to raising my score if the authors adequately address these concerns.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new mechanism called \"Spectral Attention\" designed to address the challenge of long-term dependencies in time series prediction. Traditional models such as linear and Transformer-based predictors face limitations in handling long-term dependencies due to fixed input sizes and the shuffling of samples during training. Spectral Attention enhances model performance by preserving temporal correlations and facilitating long-term information processing. It utilizes low-pass filters to capture long-term trends and supports parallel training across multiple time steps, thus expanding the effective input window. Experimental results demonstrate that this mechanism achieves state-of-the-art prediction performance on multiple real-world datasets, opening new avenues for exploration in time series analysis.\n\nSTRENGTHS:\n1. Aiming at the problem of long-term dependence in time series prediction, this paper proposes a novel Spectral Attention mechanism.\n2. This method can be extended to multiple prediction models and achieve performance improvement.\n\nWEAKNESSES:\n1. There are some formulas in the paper that do not explain the specific meaning.\n2. the source code of the paper is not open, can not verify the effect of the experiment.\n\nQUESTIONS:\n1. the use of sequential sampling instead of random sampling, which will cause the model to generalize in the face of new data or different distribution of data, there is a risk of overfitting?\n2. There are some formulas in the paper that do not explain the specific meaning. For example, \"The base model can be reformulated as P=f2(F,E) and F,E=f1(X).\" What is the meaning of \"E\"?\n3. How is the formula \"fSA=f2·SA·f1\" derived in the sentence \"Our Spectral Attention module takes feature F as input and transforms it into F' of the same size, F' =SA(F), modifying the base model as follows: fSA= f2 · SA · f1\"?\n4. why was the experiment not conducted on the common ILI \"dataset\"?\n5. Do spectral attention mechanisms significantly increase the computational cost of model training and inference?\n\nLIMITATIONS:\nno.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}