Title: FilterNet: Harnessing Frequency Filters for Time Series Forecasting
Venue: NeurIPS 2024
Accepted: yes
Keywords: time series forecasting

learning in the frequency domain
Abstract: Given the ubiquitous presence of time series data across various domains, precise forecasting of time series holds significant importance and finds widespread real-world applications such as energy, weather, healthcare, etc. While numerous forecasters have been proposed using different network architectures, the Transformer-based models have state-of-the-art performance in time series forecasting. However, forecasters based on Transformers are still suffering from vulnerability to high-frequency signals, efficiency in computation, and bottleneck in full-spectrum utilization, which essentially are the cornerstones for accurately predicting time series with thousands of points. In this paper, we explore a novel perspective of enlightening signal processing for deep time series forecasting. Inspired by the filtering process, we introduce one simple yet effective network, namely FilterNet, built upon our proposed learnable frequency filters to extract key informative temporal patterns by selectively passing or attenuating certain components of time series signals. Concretely, we propose two kinds of learnable filters in the FilterNet: (i) Plain shaping filter, that adopts a universal frequency kernel for signal filtering and temporal modeling; (ii) Contextual shaping filter, that utilizes filtered frequencies examined in terms of its compatibility with input signals for
dependency learning. Equipped with the two filters, FilterNet can approximately surrogate the linear and attention mappings widely adopted in time series literature, while enjoying superb abilities in handling high-frequency noises and utilizing the whole frequency spectrum that is beneficial for forecasting. Finally, we conduct extensive experiments on eight time series forecasting benchmarks, and experimental results have demonstrated our superior performance in terms of both effectiveness and efficiency compared with state-of-the-art methods. Our code is available at$^1$.

FilterNet: Harnessing Frequency Filters for Time Series Forecasting

Abstract

Given the ubiquitous presence of time series data across various domains, precise forecasting of time series holds significant importance and finds widespread real-world applications such as energy, weather, healthcare, etc. While numerous forecasters have been proposed using different network architectures, the Transformer-based models have state-of-the-art performance in time series forecasting. However, forecasters based on Transformers are still suffering from vulnerability to high-frequency signals, efficiency in computation, and bottleneck in full-spectrum utilization, which essentially are the cornerstones for accurately predicting time series with thousands of points. In this paper, we explore a novel perspective of enlightening signal processing for deep time series forecasting. Inspired by the filtering process, we introduce one simple yet effective network, namely FilterNet , built upon our proposed learnable frequency filters to extract key informative temporal patterns by selectively passing or attenuating certain components of time series signals. Concretely, we propose two kinds of learnable filters in the FilterNet: (i) Plain shaping filter, that adopts a universal frequency kernel for signal filtering and temporal modeling; (ii) Contextual shaping filter, that utilizes filtered frequencies examined in terms of its compatibility with input signals for dependency learning. Equipped with the two filters, FilterNet can approximately surrogate the linear and attention mappings widely adopted in time series literature, while enjoying superb abilities in handling high-frequency noises and utilizing the whole frequency spectrum that is beneficial for forecasting. Finally, we conduct extensive experiments on eight time series forecasting benchmarks, and experimental results have demonstrated our superior performance in terms of both effectiveness and efficiency compared with state-of-the-art methods. Our code is available at 1 .

1 Introduction

Time series forecasting has been playing a pivotal role across a multitude of contemporary applications, spanning diverse domains such as climate analysis [1], energy production [2], traffic flow patterns [3], financial markets [4], and various industrial systems [5]. The ubiquity and profound significance of time series data has recently garnered substantial research efforts, culminating in a plethora of deep learning forecasting models [6] that have significantly enhanced the domain of time series forecasting.

Previously, leveraging different kinds of deep neural networks derives a series of time series forecasting methods, such as Recurrent Neural Network-based methods including DeepAR [7], LSTNet [8],

0

10

20

30

40

50

Frequency

0

10

20

30

40

Amplitude

Middle-frequency

High-frequency

Low-frequency



































(a) The spectrum of input signal (b) iTransformer with MSE=1.1e-01



































(c) FilterNet with MSE=2.7e-05

Convolution Neural Network-based methods including TCN [9], SCINet [10], etc. Recently, however, with the continuing advancement of deep learning, two branches of methods that received particularly more attention have been steering the development of time series forecasting, i.e., Multilayer Perceptron (MLP)-based methods, such as N-BEATS [11], DLinear [12], and FreTS [13], and Transformerbased methods, such as Informer [14], Autoformer [15], PatchTST [16], and iTransformer [17]. While MLP-based models are capable of providing accurate forecasts, Transformer-based models continue to achieve state-of-the-art time series forecasting performance.

However, forecasters based on Transformers are still suffering from vulnerability to high-frequency signals, efficiency in computation, and bottleneck in full-spectrum utilization, which essentially are the cornerstones for accurately predicting time series composed of thousands of timesteps. In designing a very simple simulation experiment on the synthetic data only composed of a low-, middle- and high-frequency signal respectively (see Figure 1(a)), we find the state-of-the-art iTransformer [17] model performs much worse in forecasting (Figure 1(b) and Figure 1(c)). This observation shows that state-of-the-art Transformer-based model cannot utilize the full spectrum information, even for a naive signal of three different frequency components. In contrast, in the field of signal processing , a frequency filter enjoys many good properties such as frequency selectivity, signal conditioning, and multi-rate processing. These could have great potential in advancing the model's ability to extract key informative frequency patterns in time series forecasting.

Thus, inspired by the filtering process [18] in signal processing, in this paper, we introduce one simple yet effective framework, namely FilterNet , for effective time series forecasting. Specifically, we start by proposing two kinds of learnable filters as the key units in our framework: (i) Plain shaping filter, which makes the naive but universal frequency kernel learnable for signal filtering and temporal modeling, and (ii) Contextual shaping filter, which utilizes filtered frequencies examined in terms of its compatibility with input signals for dependency learning. The plain shaping filter is more likely to be adopted in predefined conditions and efficient in handling simple time series structures, while the contextual filter can adaptively weight the filtering process based on the changing conditions of input and thus have more flexibility in facing more complex situations. Besides, these two filters as the built-in functions of the FilterNet can also approximately surrogate the widely adopted linear mappings and attention mappings in time series literature [12, 14, 17]. This also illustrates the effectiveness of our FilterNet in forecasting by selectively passing or attenuating certain signal components while capturing the core time series structure with adequate learning expressiveness. Moreover, since filters are better fit in the stationary frequency domain, we let filters wrapped by two reversible transformations, i.e., instance normalization [19] and fast Fourier transform [20] to reduce the influence of non-stationarity and accomplish the domain transformation of time series respectively. In summary, our contributions can be listed as follows:

In studying state-of-the-art deep Transformer-based time series forecasting models, an interesting observation from a simple simulation experiment motivates us to explore a novel perspective of enlightening signal processing techniques for deep time series forecasting.

Inspired by the filtering process in signal processing, we introduce a simple yet effective network, FilterNet , built upon our proposed two learnable frequency filters to extract key informative temporal patterns by selectively passing or attenuating certain components of time series signals, thereby enhancing the forecasting performance.

We conduct extensive experiments on eight time series forecasting benchmarks, and the results have demonstrated that our model achieves superior performance compared with state-of-the-art forecasting algorithms in terms of effectiveness and efficiency.

2 Related Work

Deep Learning-based Time Series Forecasting In recent years, deep learning-based methods have gained prominence in time series forecasting due to their ability to capture nonlinear and complex correlations [21]. These methods have employed various network architectures to learn temporal dependencies, such as RNN [8, 7], TCN [9, 10], etc. Notably, MLP- and Transformerbased methods have achieved competitive performance, emerging as state-of-the-art approaches. N-HiTS [22] integrates multi-rate input sampling and hierarchical interpolation with MLPs to enhance univariate forecasting. DLinear [12] introduces a simple approach using a single-layer linear model to capture temporal relationships between input and output time series data. RLinear [23] utilizes linear mapping to model periodic features, demonstrating robustness across diverse periods with increasing input length. In contrast to the simple structure of MLPs, Transformer's advanced attention mechanisms [24] empower the models [14, 15, 25, 26] to capture intricate dependencies and longrange interactions. PatchTST [16] segments time series into patches as input tokens to the Transformer and maintaining channel independence. iTransformer [17] inverts the Transformer's structure by treating independent series as variate tokens to capture multivariate correlations through attention.

Time Series Modeling with Frequency Learning In recent developments, frequency technology has been increasingly integrated into deep learning models, significantly improving state-of-theart accuracy and efficiency in time series analysis [27]. These models leverage the benefits of frequency technology, such as high efficiency [28, 29] and energy compaction [13], to enhance forecasting capabilities. Concretely, Autoformer [15] introduces the auto-correlation mechanism, improving self-attention implemented with Fast Fourier Transforms (FFT). FEDformer [25] enhances attention with a FFT-based frequency approach, determining attentive weights from query and key spectrums and conducting weighted summation in the frequency domain. DEPTS [30] utilizes Discrete Fourier Transform (DFT) to extract periodic patterns and contribute them to forecasting. FiLM [31] employs Fourier analysis to retain historical information while filtering out noisy signals. FreTS [13] introduces a frequency-domain Multi-Layer Perceptrons (MLPs) to learn channel and temporal dependencies. FourierGNN [29] transfers the operations of graph neural networks (GNNs) from the time domain to the frequency domain. FITS [32] applies a low pass filter to the input data followed by complex-valued linear mapping in the frequency domain.

Unlike these methods, in this paper we propose a simple yet effective model FilterNet developed from a signal processing perspective, and apply all-pass frequency filters to design the network directly, rather than incorporating them into other network architectures, such as Transformers, MLPs, or GNNs, or utilizing them as low-pass filters, as done in FITS [32] and FiLM [31].

3 Preliminaries

Frequency Filters Frequency filters [33] are mathematical operators designed to modify the spectral content of signals. Specifically, given an input time series signal x [ n ] with its corresponding Fourier transform X [ k ] , a frequency filter H [ k ] is applied to the signal to produce an output signal y [ n ] with its corresponding Fourier transform Y [ k ] = X [ k ] H [ k ] . The frequency filter H [ k ] alters the amplitude and phase of specific frequency components in the input time series signal x [ n ] according to its frequency response characteristics , thereby shaping the spectral content of the output signal.

According to the Convolution Theorem [34], the point-wise multiplication in the frequency domain corresponds to the circular convolution operation between two corresponding signals in the time domain. Consequently, the frequency filter process can be expressed in the time domain as:

where h [ n ] is the inverse Fourier transform of H [ k ] and ⊛ denotes the circular convolution operation. This formulation underscores the intrinsic connections between the frequency filter process and the circular convolution in the time domain, and it indicates that the frequency filter process can efficiently perform circular convolution operations. In time series forecasting, Transformer-based methods have achieved state-of-the-art performance, largely due to the self-attention mechanism [14, 15, 25, 17], which can be interpreted as a form of global circular convolution [35]. This perspective opens up the possibility of integrating frequency filter technologies, which are well-known for their ability to isolate and enhance specific signal components, to further improve time series forecasting models.

Frequency Filter

Block

Input

Output

FFT

IFFT

FFT

IFFT

Plain

Shaping Filter

Contextual

Shaping Filter

FFN

Inverse Instance

Normalization

Instance

Normalization

①

Plain Shaping Filter

②

Contextual Shaping Filter

𝗫

𝗭

ෝ

𝗬

𝗭

𝗭

𝓕(

z

)

𝓕(

z

)

𝓕(

z

)

𝓗

𝓗(𝓕(

z

))

𝓕(

z

)𝓗

𝓕(

z

)𝓗(𝓕(

z

))

𝓢=

𝓢

𝓢

𝓢=

𝐒

𝐒

𝐒

𝓕

𝓕

𝐏

𝜑

𝜑

𝜙

𝜙

𝓕

𝓕

-1

-1

4 Methodology

As aforementioned, frequency filters enjoy numerous advantageous properties for time series forecasting, functioning equivalently to circular convolution operations in the time domain. Therefore, we design the time series forecaster from the perspective of frequency filters. In this regard, we propose FilterNet , a forecasting architecture grounded in frequency filters. First, we introduce the overall architecture of FilterNet in Section 4.1, which primarily comprises the basic blocks and the frequency filter block. Second, we delve into the details of two types of frequency filter blocks: the plain shaping filter presented in Section 4.2 and the contextual shaping filter discussed in Section 4.3.

4.1 Overview

The overall architecture of FilterNet is depicted in Figure 2, which mainly consists of the instance normalization part, the frequency filter block, and the feed-forward network. Specifically, for a given time series input X = [ X 1: L 1 , X 1: L 2 , ..., X 1: L N ] ∈ R N × L with the number of variables N and the lookback window length L , where X 1: L N ∈ R L denotes the N -th variable, we employ FilterNet to predict the future τ time steps Y = [ X L +1: L + τ 1 , X L +1: L + τ 2 , ..., X L +1: L + τ N ] ∈ R N × τ . We provide further analysis about the architecture design of FilterNet in Appendix A.

Instance Normalization Non-stationarity is widely existing in time series data and poses a crucial challenge for accurate forecasting [19, 36]. Considering that time series data are typically collected over a long duration, these non-stationary sequences inevitably expose forecasting models to distribution shifts over time. Such shifts can result in performance degradation during testing due to the covariate shift or the conditional shift [37]. To address this problem, we utilize an instance normalization method, denoted as Norm , on the time series input X , which can be formulated as:

where Mean L denotes the operation that calculates the mean value along the time dimension, and Std L represents the operation that calculates the standard deviation along the time dimension.

Correspondingly, the inverse instance normalization, denoted as InverseNorm , is formulated as:

Filter

Z

S

Filter

Filter

Filter

S

Z

*

*

*

*

*

*

(a) Plain Shaping Filter

S

Z

FFT

𝓕

Dense

𝝹

σ

FFT

𝓦

𝓦

𝓩

𝓢

𝓐

2

1

𝓕

-1

𝓔

𝓔

𝓢 =𝓐⨀𝓔

(b) Contextual Shaping Filter

Frequency Filter Block Previous representative works primarily leverage MLP architectures (e.g., DLinear [12], RLinear [23]) or Transformer architectures (e.g., PatchTST [16], iTransformer [17]) to model the temporal dependencies among time series data. As mentioned earlier, time series forecasters can be implemented through performing a frequency filter process in the frequency domain, and thus we propose to directly apply the frequency filter in the frequency domain, denoted as FilterBlock , to replace the aforementioned methods for modeling corresponding temporal dependencies, such as:

where F is Fourier transform, F -1 is inverse Fourier transform and H filter is the frequency filter.

Inspired by MLP that randomly initializes a learnable weight parameters and Transformer that learns the data-dependent attention scores from data (further explanations are provided in Appendix B), we introduce two types of frequency filters, i.e., p l ai n shaping filter ( PaiFilter ) and con tex tual shaping filter ( TexFilter ). PaiFilter applies a random initialized learnable weight H ϕ to instantiate the frequency filter H filter , and then the frequency filter process is reformulated as:

TexFilter learns a data-dependent frequency filter H φ ( F ( Z )) from the input data by using a neural network H φ () , and then the corresponding frequency filter process is reformulated as:

Feed-forward Network The frequency filter block has captured temporal dependencies among time series data, and then we employ a feed-forward network ( FFN ) to project them back onto the time series data and make predictions for the future τ time steps. As the output P of FFN are instance-normalized values, we conduct an inverse instance normalization operation ( InverseNorm ) on them and obtain the final predictions ˆ Y . The entire process can be formulated as follows:

4.2 Plain Shaping Filter

PaiFilter instantiates the frequency filter by randomly initializing learnable parameters and then performing multiplication with the input time series. In general, for multivariate time series data, the channel-independence strategy in channel modeling has proven to be more effective compared to the channel-mixing strategy [12, 16]. Following this principle, we also adopt the channel-independence strategy for designing the frequency filter. Specifically, we propose two types of plain shaping filters: the universal type, where parameters are shared across different channels, and the individual type, where parameters are unique to each channel, as illustrated in Figure 3(a).

Given the time series input Z ∈ R N × L and the plain shaping filter H ϕ , we apply PaiFilter by:

where F is Fourier transform, F -1 is inverse Fourier transform, ⊙ L denotes the element-wise product along L dimension, H ( Uni ) ϕ ∈ C 1 × L is the universal plain shaping filter, H ( Ind ) ϕ ∈ C N × L is the individual plain shaping filter, and S ∈ R N × L is the output of PaiFilter . We further compare and analyze the two types of PaiFilter in Section 5.3.

4.3 Contextual Shaping Filter

In contrast to PaiFilter , which randomly initializes the parameters of frequency filters and fixes them after training, TexFilter learns the parameters generated from the input data, allowing for better adaptation to the data. Consequently, we devise a neural network H φ that flexibly adjusts the frequency filter in response to the input data, as depicted in Figure 3(b).

Given the time series input Z ∈ R N × L and its corresponding Fourier transform denoted as Z = F ( Z ) ∈ C N × L , the network H φ is utilized to derive the contextual shaping filter, expressed as H φ : C N × L ↦→ C N × D . First, it embeds the raw data by a linear dense operation κ : C L ↦→ C D to improve the capability of modeling complex data. Then, it applies a series of complex-value multiplication with K learnable parameters W 1: K ∈ C 1 × D yielding σ ( κ ( Z ) ⊙W 1: K ) where σ is the activation function, and finally outputs H φ ( Z ) . Then we apply TexFilter by:

where ⊙ D denotes the element-wise product along D dimension and S ∈ R N × D is the output. The contextual shaping filter can adaptively weight the filtering process based on the changing conditions of input and thus have more flexibility in facing more complex situations.

5 Experiments

In this section, we extensively experiment with eight real-world time series benchmarks to assess the performance of our proposed FilterNet. Furthermore, we conduct thorough analytical experiments concerning the frequency filters to validate the effectiveness of our proposed framework.

5.1 Experimental Setup

Datasets We conduct empirical analyses on diverse datasets spanning multiple domains, including traffic, energy, and weather, among others. Specifically, we utilize datasets such as ETT datasets [14], Exchange [38], Traffic [15], Electricity [15], and Weather [15], consistent with prior studies on long time series forecasting [16, 17, 25]. We preprocess all datasets according to the methods outlined in [16, 17], and normalize them with the standard normalization method. We split the datasets into training, validation, and test sets in a 7:2:1 ratio. More dataset details are in Appendix C.1.

Baselines We compare our proposed FilterNet with the representative and state-of-the-art models to evaluate their effectiveness for time series forecasting. We choose the baseline methods from four categories: (1) Frequency-based models, including FreTS [13] and FITS [32]; (2) TCN-based models, such as MICN [39] and TimesNet [40]; (3) MLP-based models, namely DLinear [12] and RLinear [23]; and (4) Transformer-based models, which include Informer [14], Autoformer [15], Pyraformer [26], FEDformer [25], PatchTST [16], and the more recent iTransformer [17] for comparison. Further details about the baselines can be found in Appendix C.2.

Implementation Details All experiments are implemented using Pytorch 1.8 [41] and conducted on a single NVIDIA RTX 3080 10GB GPU. We employ MSE (Mean Squared Error) as the loss function and present MAE (Mean Absolute Errors) and MSE (Mean Squared Errors) results as the evaluation metrics. For further implementation details, please refer to Appendix C.3.

5.2 Main Results

We present the forecasting results of our FilterNet compared to several representative baselines on eight benchmarks with various prediction lengths in Table 1. Additional results with different lookback window lengths are reported in Appendix F and G.3. Table 1 demonstrates that our model consistently outperforms other baselines across different benchmarks. The average improvement of FilterNet over all baseline models is statistically significant at the confidence of 95%. Specifically, PaiFilter performs well on small datasets (e.g., ETTh1), while TexFilter excels on large datasets (e.g., Electricity) due to the ability to model the more complex and contextual correlations present in larger datasets. Also, the prediction performance of iTransformer [17], which achieves the best results on the Traffic dataset (862 variables) but not on smaller datasets, suggests that simpler structures may be more suitable for smaller datasets, while larger datasets require more contextual structures due to their complex relationships. Compared with FITS [32] built on low-pass filters, our model outperforms it validating an all-pass filter is more effective. Since PaiFilter is simple yet effective, the following FilterNet in the experimental section refer to PaiFilter unless otherwise stated.

5.3 Model Analysis

In this part, we conduct experiments to delve into a thorough exploration of frequency filters, including their modeling capabilities, comparisons among different types of frequency filters, and the various factors impacting their performance. Detailed experimental settings are provided in Appendix C.5.

Modeling Capability of Frequency Filters Despite the simplicity of frequency filter architecture, Table 1 demonstrates that this architecture can also achieve competitive performance. Hence, in this part, we perform experiments to explore the modeling capability of frequency filters. Particularly, given the significance of trend and seasonal signals in time series forecasting, we investigate the efficacy of simple filters in modeling these aspects. We generate a trend signal and a multi-period signal with noise, and then we leverage the frequency filters (i.e., PaiFilter ) to perform training on the two signals respectively. Subsequently, we produce prediction values based on the trained frequency filters. Specifically, Figure 4(a) and 4(b) show that the filter can effectively model trend and periodic signals respectively even compared with state-of-the-art iTransformer [17] when the data contains noise. These results illustrate that the filter has excellent and robust modeling capabilities for trend and periodic signals which are important components for time series. This can also explain the effectiveness of FilterNet since the filters can perform well in such settings.

















































































(a) Prediction on trend signals with noises

(b) Prediction on multi-periodic signals with noises

Shared vs. Unique Filters Among Channels To analyze the different channel strategies of filters, we further conduct experiments on the ETTh and Exchange datasets. Specifically, we compare forecasting performance under different prediction lengths between two different types of frequency filters, i.e., H ( Uni ) ϕ and H ( Ind ) ϕ . In H ( Uni ) ϕ , filters are shared across different channels, whereas H ( Ind ) ϕ signifies filters unique to each channel. The evaluation results are presented in Table 2. It demonstrates that filters shared among different channels consistently outperform across all prediction lengths. In addition, we visualize the prediction values predicted on the ETTh1 dataset by the two different types of filters, as illustrated in Figure 11 (see Appendix G.1). The visualization reveals that the prediction values generated by filters shared among different channels exhibit a better fit than the unique filters. Therefore, the strategy of channel sharing seems to be better suited for time series forecasting and filter designs, which is also validated in DLinear [12] and PatchTST [16].

Visualization of Prediction We present a prediction showcase on ETTh1 dataset, as shown in Figure 5. We select iTransformer [17], PatchTST [16] as the representative compared methods. Comparing with these different state-of-the-art models, we can observe FilterNet delivers the most accurate predictions of future series variations, which has demonstrated superior performance. In addition, we include more visualization cases and please refer to Appendix G.3.

0

25

50

75

100

125

150

175

200

Time

1.8

1.6

1.4

1.2

1.0

0.8

Values

Prediction

GroundTruth

InputData

(a) FilterNet









































(b) iTransformer









































(c) PatchTST

Visualization of Frequency Filters To provide a comprehensive overview of the frequency response characteristics of frequency filters, we conduct visualization experiments on the Weather, ETTh, and Traffic datasets with the lookback window length of 96 and the prediction length of 96. The frequency response characteristics of learned filters are visualized in Figure 7. From Figures 7(a) and 7(b), we can observe that compared with Transformer-based approaches (e.g., iTransformer [17], PatchTST [16]) tend to attenuate high-frequency components and preserve low-frequency information, FilterNet exhibits a more nuanced and adaptive filtering behavior that can be capable of attending to all frequency components. Figure 7(c) demonstrates that the main patterns of the Traffic dataset primarily resides in the low-frequency range. This observation also explains why iTransformer performs well on the Traffic dataset, despite its low-frequency nature. Overall, Figure 7 demonstrates

0

10

20

30

40

50

Frequency

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Amplitude

(a) Weather

0

10

20

30

40

50

Frequency

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Amplitude

(b) ETTh1

0

10

20

30

40

50

Frequency

0

2

4

6

8

10

Amplitude

(c) Traffic

0

10

20

30

40

50

Frequency

0

1

2

3

4

5

Amplitude

(a)

96

- →

96

0

10

20

30

40

50

Frequency

0

1

2

3

4

5

6

Amplitude

(b)

96

- →

192

0

10

20

30

40

50

Frequency

0

1

2

3

4

5

6

Amplitude

(c)

96

- →

336

that FilterNet possesses comprehensive processing capabilities. Moreover, visualization experiments conducted on the ETTm1 dataset across various prediction lengths, as shown in Figure 8, further illustrate the extensive processing abilities of FilterNet. Additional results conducted on different lookback window lengths and prediction lengths can be found in Appendix G.2.

Efficiency Analysis The complexity of FilterNet is O (Log L ) where L is the input length. To comprehensively assess efficiency, we evaluate it based on two dimensions: memory usage and training time. Specifically, we choose two different sizes of datasets: the Exchange (8 variables, 7588 timestamps) and Electricity datasets (321 variables, 26304 timestamps). We compare the efficiency of our FilterNet with the representative Transformer- and MLP-based methods under the same settings (lookback window length of 96 and prediction length of 96), and the results are shown in Figure 6. It highlights that FilterNet surpasses other Transformer models, regardless of dataset size. While our approach exhibits similar efficiency to DLinear, our effective results outperform its performance. In Appendix E, we further conduct ablation studies to validate the rationale of FilterNet designs.

FilterNet

0.5GB, 1.5s

iTransformer

PatchTST

Transformer

DLinear

0.5GB, 0.9s

Autoformer

FreTS

Pyraformer

Exchange (8 Variables)

0.5GB

1.3GB

Memory Footprint

FilterNet

iTransformer

0.8GB, 21.0s

PatchTST

1.3GB, 168.4s

Transformer

0.8GB, 29.2s

DLinear

Autoformer

0.8GB, 37.8s

Informer

Pyraformer

1.0GB, 35.2s

0.5GB

1.3GB

Memory Footprint

Electricity (321 Variables)

6 Conclusion Remarks

In this paper, we explore an interesting direction from a signal processing perspective and make a new attempt to apply frequency filters directly for time series forecasting. We propose a simple yet effective architecture, FilterNet , built upon our proposed two kinds of frequency filters to accomplish the forecasting. Our comprehensive empirical experiments on eight benchmarks have validated the superiority of our proposed method in terms of effectiveness and efficiency. We also include many careful and in-depth model analyses of FilterNet and the internal filters, which demonstrate many good properties. We hope this work can facilitate more future research integrating signal processing techniques or filtering processes with deep learning on time series modeling and accurate forecasting.

Acknowledgments and Disclosure of Funding

The work was supported in part by the National Natural Science Foundation of China under Grant 92367110, the Shanghai Baiyulan TalentPlan Pujiang Project under Grant 23PJ1413800, and Shanxi Scholarship Council of China (2024-61).

References

Haixu Wu, Hang Zhou, Mingsheng Long, and Jianmin Wang. Interpretable weather forecasting for worldwide stations with a unified deep model. Nat. Mac. Intell. , 5(6):602-611, 2023.

Kun Yi, Qi Zhang, Hui He, Kaize Shi, Liang Hu, Ning An, and Zhendong Niu. Deep coupling network for multivariate time series forecasting. ACM Trans. Inf. Syst. , 42(5):127:1-127:28, 2024.

Hui He, Qi Zhang, Simeng Bai, Kun Yi, and Zhendong Niu. CATN: cross attentive tree-aware network for multivariate time series forecasting. In AAAI , pages 4030-4038. AAAI Press, 2022.

Adebiyi A Ariyo, Adewumi O Adewumi, and Charles K Ayo. Stock price prediction using the arima model. In 2014 UKSim-AMSS 16th international conference on computer modelling and simulation , pages 106-112. IEEE, 2014.

Nijat Mehdiyev, Johannes Lahann, Andreas Emrich, David Enke, Peter Fettke, and Peter Loos. Time series classification using deep learning for process planning: A case from the process industry. Procedia Computer Science , 114:242-249, 2017.

Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A , 379(2194):20200209, 2021.

David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting , 36(3):1181-1191, 2020.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In SIGIR , pages 95-104, 2018.

Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. CoRR , abs/1803.01271, 2018.

Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems , 35:5816-5828, 2022.

Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437 , 2019.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? In AAAI , pages 11121-11128. AAAI Press, 2023.

Kun Yi, Qi Zhang, Wei Fan, Shoujin Wang, Pengyang Wang, Hui He, Ning An, Defu Lian, Longbing Cao, and Zhendong Niu. Frequency-domain mlps are more effective learners in time series forecasting. In NeurIPS , 2023.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI , pages 11106-11115, 2021.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In NeurIPS , pages 2241922430, 2021.

Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations , 2023.

Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. In ICLR . OpenReview.net, 2024.

Richard Asselin. Frequency filter for time integrations. Monthly Weather Review , 100(6):487490, 1972.

Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations , 2021.

Pierre Duhamel and Martin Vetterli. Fast fourier transforms: a tutorial review and a state of the art. Signal processing , 19(4):259-299, 1990.

Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , 379(2194):20200209, feb 2021.

Cristian Challu, Kin G. Olivares, Boris N. Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. CoRR , abs/2201.12886, 2022.

Zhe Li, Shiyi Qi, Yiduo Li, and Zenglin Xu. Revisiting long-term time series forecasting: An investigation on linear mapping. CoRR , abs/2305.10721, 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS , pages 5998-6008, 2017.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML , 2022.

Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In ICLR , 2021.

Kun Yi, Qi Zhang, Longbing Cao, Shoujin Wang, Guodong Long, Liang Hu, Hui He, Zhendong Niu, Wei Fan, and Hui Xiong. A survey on deep learning based time series analysis with frequency transformation. CoRR , abs/2302.02173, 2023.

Wei Fan, Kun Yi, Hangting Ye, Zhiyuan Ning, Qi Zhang, and Ning An. Deep frequency derivative learning for non-stationary time series forecasting. In IJCAI , 2024.

Kun Yi, Qi Zhang, Wei Fan, Hui He, Liang Hu, Pengyang Wang, Ning An, Longbing Cao, and Zhendong Niu. Fouriergnn: Rethinking multivariate time series forecasting from a pure graph perspective. In NeurIPS , 2023.

Wei Fan, Shun Zheng, Xiaohan Yi, Wei Cao, Yanjie Fu, Jiang Bian, and Tie-Yan Liu. DEPTS: deep expansion learning for periodic time series forecasting. In ICLR . OpenReview.net, 2022.

Tian Zhou, Ziqing Ma, Xue Wang, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, and Rong Jin. Film: Frequency improved legendre memory model for long-term time series forecasting. In NeurIPS , 2022.

Zhijian Xu, Ailing Zeng, and Qiang Xu. FITS: modeling time series with 10k parameters. CoRR , abs/2307.03756, 2023.

Fausto Pedro García Márquez and Noor Zaman. Digital filters and signal processing . BoDBooks on Demand, 2013.

RTMAC Lu. Algorithms for discrete Fourier transform and convolution . Springer, 1989.

John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro. Adaptive fourier neural operators: Efficient token mixers for transformers. CoRR , abs/2111.13587, 2021.

Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems , 35:9881-9893, 2022.

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Deeptime: Deep time-index meta-learning for non-stationary time-series forecasting. arXiv preprint arXiv:2207.06046 , 2022.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In SIGIR , pages 95-104, 2018.

Huiqiang Wang, Jian Peng, Feihu Huang, Jince Wang, Junhui Chen, and Yifei Xiao. MICN: multi-scale local and global context modeling for long-term series forecasting. In ICLR . OpenReview.net, 2023.

Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Temporal 2d-variation modeling for general time series analysis. In ICLR . OpenReview.net, 2023.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS , pages 8024-8035, 2019.

A More Analysis about the Architecture of FilterNet

The Necessity of Instance Normalization Block From the frequency perspective, the mean value is equal to zero frequency component. Specifically, given a signal x [ n ] with a length of N , we can obtain its corresponding discrete Fourier transform X [ k ] by:

where j is the imaginary unit. We set k as 0 and then,

According the above equation, we can find that the mean value 1 N ∑ N -1 n =0 x [ n ] in the time domain is equal to the zero frequency component X [0] in the frequency domain. Similarly, we can also view the standard deviation from the frequency perspective, and it is related to the power spectral density.

According to these analysis, the instance normalization is analogous to a form of data preprocessing. Given that filters are primarily crafted to discern particular patterns within input data, while instance normalization aims to normalize each instance in a dataset, a function distinct from the conventional role of filters, we treat instance normalization as a distinct block within our FilterNet architecture.

Frequency Filter Block Recently, Transformer- and MLP-based methods have emerged as the two main paradigms for time series forecasting, exhibiting competitive performance compared to other model architectures. Building on prior work that conceptualizes self-attention and MLP architectures as forms of global convolution [35, 13], it becomes apparent that frequency filters hold promise for time series forecasting tasks. Just as self-attention mechanisms capture global dependencies and MLPs learn to convolve features across the entire input space, frequency filters offer a means to extract and emphasize specific temporal patterns and trends from time series data. By applying frequency filters to time series data, we can learn recurring patterns, trends, and periodic behaviors that are essential for forecasting future time series data and making accurate predictions.

Feed-forward Network Incorporating a feed-forward network within the FilterNet architecture is essential for enhancing the model's capacity to capture complex relationships and non-linear patterns within the data. While frequency filters excel at extracting specific frequency components and temporal patterns from time series data, they may not fully capture the intricate dependencies and higher-order interactions present in real-world datasets [17]. By integrating a feed-forward network, the model gains the ability to learn hierarchical representations and abstract patterns from the input data, allowing it to capture more nuanced relationships and make more accurate predictions. This combination of frequency filters and a feed-forward network leverages the strengths of both approaches, enabling the model to effectively process and analyze time series data across different frequency bands while accommodating the diverse and often nonlinear nature of temporal dynamics. Overall, the inclusion of a feed-forward network enriches the expressive power of FilterNet, leading to improved forecasting performance and robustness across various domains.

B Explanations about the Design of Two Filters

Self-attention mechanism is a highly data-dependent operation that both derives its parameters from data and subsequently applies these parameters back to the data. Concretely, given the input data X , the self-attention can be formulated as:

where Q (queries), K (keys), and V (values) are linear transformations of the input data X, as:

where W Q , W K , and W V are learned weight matrices. Since the values Q , K , and V are derived directly from the input data X , the attention scores SA are inherently dependent on the data.

Unlike self-attention mechanism that dynamically adapts to the input data during inference, MLPs maintain a consistent architecture regardless of the dataset characteristics. Specifically, for the input data X , MLP can be formulated as:

where W are the learned weights and b are the learned biases. Once trained, the weights W and biases b remain static, meaning that they do not dynamically change in response to new data inputs.

MLPs are straightforward, less data-dependent models that apply fixed transformations to the input data, making them suitable for tasks with static relationships between the input data. In contrast, self-attention mechanisms are highly data-dependent, dynamically computing attention scores based on the input data to capture complex, context-specific dependencies, making them ideal for tasks requiring an understanding of sequential or structured data.

Inspired by the two paradigms, FilterNet designs two corresponding types of filters: plain shaping filters and contextual shaping filters. Plain shaping filters offer stability and efficiency, making them suitable for tasks with static relationships. In contrast, contextual shaping filters provide the flexibility to capture dynamic dependencies, excelling in tasks that require context-sensitive analysis. This dual approach allows FilterNet to effectively handle a wide range of data types and forecasting scenarios, combining the best aspects of both paradigms to achieve superior performance.

C Experimental Details

C.1 Datasets

We evaluate the performance of our proposed FilterNet on eight popular datasets, including Exchange, Weather, Traffic, Electricity, and ETT datasets. In detail, the Exchange 2 dataset collects daily exchange rates of eight different countries including Singapore, Australia, British, Canada, Switzerland, China, Japan, and New Zealand ranging from 1990 to 2016. The Weather 3 dataset, including 21 meteorological indicators such as air temperature and humidity, is collected every 10 minutes from the Weather Station of the Max Planck Institute for Biogeochemistry in 2020. The Traffic [15] dataset contains hourly traffic data measured by 862 sensors on San Francisco Bay area freeways, which has been collected since January 1, 2015. The Electricity 4 dataset collects the hourly electricity consumption of 321 clients from 2012 to 2014. The ETT 5 (Electricity Transformer Temperature) datasets contain two visions of the sub-dataset: ETTh and ETTm, collected from electricity transformers every 15 minutes and 1 hour between July 2016 and July 2018. Thus, in total we have 4 ETT datasets (ETTm1, ETTm2, ETTh1, and ETTh2) recording 7 features such as load and oil temperature. The details about these datasets are summarized in Table 3.

C.2 Baselines

We choose twelve well-acknowledged and state-of-the-art models for comparison to evaluate the effectiveness of our proposed FilterNet for time series forecasting, including Frequency-based models, TCN-based models, MLP-based models, and Transformer-based models. We introduce these models as follows:

FreTS [13] introduces a novel direction of applying MLPs in the frequency domain to effectively capture the underlying patterns of time series, benefiting from global view and energy compaction. The official implementation is available at https://github.com/aikunyi/FreTS . To ensure fair and objective comparison, the results showed in Table 4 are obtained using instance normalization instead of min-max normalization in the original code.

FITS [32] performs time series analysis through interpolation in the complex frequency domain, enjoying low cost with 10k parameters. The official implementation is available at https://github. com/VEWOXIC/FITS . Because its original paper doesn't provide the forecasting results with the fixed lookback length L = 96 , we report the performance of FITS with lookback length L = 96 under five runs in Table 1.

MICN [39] employs multi-scale branch structure to model different potential patterns separately with linear complexity. It combines local features and global correlations to capture the overall view of time series. The official implementation is available at https://github.com/wanghq21/MICN . The experimental results showed in Table 4 follow its original paper.

TimesNet [40] transforms 1D time series into a set of 2D tensors based on multiple periods to analyse temporal variations. The above transformation allows the 2D-variations to be easily captured by 2D kernels with encoding the intraperiod- and interperiod-variations into the columns and rows of the 2D tensors respectively. The official implementation is available at https://github.com/thuml/ TimesNet . The results showed in Table 1 follow iTransformer [17] and the results showed in Table 5 follow RLinear [23].

DLinear [12] utilizes a simple yet effective one-layer linear model to capture temporal relationships between input and output sequences. The official implementation is available at https://github. com/cure-lab/LTSF-Linear . We report the performance of DLinear with lookback length L ∈ { 96 , 336 } under five runs in Table 1 and 5.

RLinear [23] uses linear mapping to model periodic features in multivariate time series with robustness for diverse periods when increasing the input length. It applies RevIN (reversible normalization) and CI (Channel Independent) to improve overall forecasting performance by simplifying learning about periodicity. The official implementation is available at https://github.com/plumprc/ RTSF . The experimental results showed in Table 1 follow iTransformer [17].

Informer [14] enhances Transformer with KL-divergence based ProbSparse attention for O ( L log L ) complexity, efficiently encoding dependencies among variables and introducing a novel architecture with a DMS forecasting strategy. The official implementation is available at https://github.com/ zhouhaoyi/Informer2020 and the experimental results showed in Table 4 follow FEDformer [25].

Autoformer [15] employs a deep decomposition architecture with auto-correlation mechanism to extract seasonal and trend components from input series, embedding the series decomposition block as an inner operator. The official implementation is available at https://github.com/thuml/ Autoformer and the experimental results showed in Table 4 follow FEDformer [25].

Pyraformer [26] introduces pyramidal attention module (PAM) with an O ( L ) time and memory complexity where the inter-scale tree structure summarizes features at different resolutions and the intra-scale neighboring connections model the temporal dependencies of different ranges. The official implementation is available at https://github.com/ant-research/Pyraformer and the experimental results showed in Table 4 follow DLinear [12].

FEDformer [25] implements sparse attention with low-rank approximation in frequency domain, enjoying linear computational complexity and memory cost. And it proposes mixture of experts decomposition to control the distribution shifting. The official implementation is available at https: //github.com/MAZiqing/FEDformer and the experimental results showed in Table 1 follow iTransformer [17].

PatchTST [16] divides time series data into subseries-level patches to extract local semantic information and adopts channel-independence strategy where each channel shares the same embedding and Transformer weights across all the series. The official implementation is available at https://github.com/yuqinie98/PatchTST . The experimental results showed in Table 1 follow iTransformer [17]. And because iTransformer doesn't provide the prediction results with lookback length L = 336 , we report the performance of PatchTST with lookback length L = 336 under five runs in Table 5.

iTransformer [17] inverts the structure of Transformer without modifying any existing modules by encoding individual series into variate tokens. These tokens are utilized by the attention mechanism to capture multivariate correlations and FFNs are adopted for each variate token to learn nonlinear representations. The official implementation is available at https://github.com/thuml/ iTransformer . The experimental results showed in Table 1 follow its original paper. And because iTransformer doesn't provide the prediction results with lookback length L = 336 , we report the performance of iTransformer with lookback length L = 336 under five runs in Table 5.

C.3 Implementation Details

The architecture of our FilterNet is very simple and has two main hyperparameters, i.e., the bandwidth of filters and the hidden size of FFN. As shown in Figure 9, the bandwidth of the filters corresponds to the lookback window length, so we select the lookback window length as the bandwidth accordingly. For the hidden size of FFN, we carefully tune the size over {64, 128, 256, 512}. Following the previous methods [16, 32], we use RevIN [19] as our instance normalization block. Besides, we carefully tune the hyperparameters including the batch size and learning rate on the validation set, and we choose the settings with the best performance. We tune the batch size over {4, 8, 16, 32} and tune the learning rate over {0.01, 0.05, 0.001, 0.005, 0.0001, 0.0005}.

C.4 Experimental Settings for Simulation Experiments

To evaluate the Transformer's modeling ability for different frequency spectrums, we generate a signal consisting of three different frequency components, i.e., low-frequency, middle-frequency, and high-frequency. We then apply both iTransformer and FilterNet to this signal, respectively, and compare the forecasting results with the ground truth. The results are presented in Figure 1.

C.5 Experimental Settings for Filters Analysis

Modeling capability of frequency filters We generate two signals: a trend signal with Gaussian noise and a multi-periodic signal with Gaussian noise. We then apply PaiFilter to these signals with a lookback window length of 96 and a prediction length of 96. The results are displayed in Figure 4.

Visualization of Frequency Filters Given a filter H ∈ R 1 × L , where L is its bandwidth, we visualize the frequency response characteristic of the filter by plotting the values in R 1 × L . First, we perform a Fourier transform on these values to obtain the spectrum, which includes the frequency and its corresponding amplitude. Finally, we visualize the spectrum, as shown in Figures 7, 8, and 12.

D Study of the Bandwidth of Frequency Filters

The bandwidth parameter (i.e., L in Equation (8) and D in Equation (9)) holds significant importance in the functionality of filters. In this part, we conduct experiments on the Weather dataset to delve into the impact of bandwidth on forecasting performance. We explore a range of bandwidth values within the set { 96 , 128 , 192 , 256 , 320 , 386 , 448 , 512 } while keeping the lookback window length and prediction length constant. Specifically, we conduct experiments to evaluate the impact under three different combinations of lookback window length and prediction length, i.e., 96 - → 96 , 96 - → 192 , and 192 - → 192 , and the results are represented in Figure 9. We observe clear trends in the relationship between bandwidth settings and lookback window length. Figure 9(a) and Figure 9(b) show that increasing the bandwidth results in minimal changes in forecasting performance. Figure 9(c) demonstrates that while forecasting performance fluctuates with increasing bandwidth, it is optimal when the bandwidth equals the lookback window length. These results indicate that using the lookback window length as the bandwidth is sufficient since the filters can effectively model the data at this setting, and it also results in lower complexity.

E Ablation Study

To validate the rationale behind the architectural design of our FilterNet, we conduct ablation studies on the ETTm1, ETTh1, and Electricity datasets. We evaluate the impact on the model's performance



































(a)

96

- →

96































(b)

96

- →

192





































(c)

192

- →

192

0.40

W/O Norm

W/O Filter

W/O FFN

0.35

0.36

0.37

0.38

0.39

MSE

Orignal

W/O

(a) ETTh1

W/O Norm

W/O Filter

W/O FFN

0.310

0.315

0.320

0.325

0.330

0.335

0.340

MSE

Orignal

W/O

(b) ETTm1

W/O Norm

W/O Filter

W/O FFN

0.150

0.155

0.160

0.165

0.170

0.175

0.180

MSE

Orignal

W/O

(c) Electricity

by eliminating the particular component of the FilterNet architecture. The evaluation results are present in Figure 10. In the figure, W/ONorm indicates that instance normalization and inverse instance normalization have been removed from FilterNet. W/OFilter signifies the removal of the filter block, and W/OFFN denotes the exclusion of the feed-forward network. The experiments are conducted with lookback window length of 96 and output length of 96. From the figure, we can conclude that each block is indispensable, as the removal of any component results in a noticeable decrease in performance. This highlights the critical role each block plays in the overall architecture of FilterNet, contributing to its effectiveness in time series forecasting.

F Additional Results

Table 4 compares the performance of various methods with our FilterNet, demonstrating that our model consistently outperforms the others. To further assess the performance of FilterNet under different lookback window lengths, we conducted experiments on the ETTh1, ETTm1, Exchange, Weather, and Electricity datasets with the lookback window length of 336. The results, shown in Table 5, indicate that our model achieves the best performance across these datasets.

0

25

50

75

100

125

150

175

200

1.1

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

GroundTruth

Prediction

(a) Forecasting with channel-shared filters

0

25

50

75

100

125

150

175

200

1.1

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

GroundTruth

Prediction

(b) Forecasting with channel-unique filters

0

25

50

75

100

125

150

175

Frequency

0

1

2

3

4

5

6

Amplitude

(a)

336

- →

96

on Electricity

0

25

50

75

100

125

150

175

Frequency

0

1

2

3

4

5

6

7

Amplitude

(b)

336

- →

192

on Electricity

0

25

50

75

100

125

150

175

Frequency

0

2

4

6

8

Amplitude

(c)

336

- →

336

on Electricity

G Visualizations

G.1 Visualization of Channel-shared vs Channel-unique Filters

To further compare the channel-shared and channel-unique filters, we visualize the prediction values by the corresponding filters. The results are shown in Figure 11. The figure demonstrates that the values predicted by channel-shared filters closely align with the ground truth compared to those predicted by channel-unique filters. This observation is consistent with the findings presented in Table 2, indicating the superiority of channel-shared filters.

G.2 Visualization of Frequency Filters

We further conduct visualization experiments to explore the learnable filters under different lookback window lengths and prediction lengths. The experiments are performed on the Electricity dataset, and the results are illustrated in Figure 12. These figures illustrate that FilterNet possesses full spectrum learning capability, as the learnable filters exhibit values across the entire spectrum. Besides, we observe that the frequency primarily concentrates in the low and middle ranges which explains that some works based on low-pass filters can also achieve good performance.

G.3 Visualizations of Predictions

To further offer an evident comparison of our model with the state-of-the-art models, we present supplementary prediction showcases on ETTm1 dataset, and the results are shown in 13. We choose the following representative models, including iTransformer [17], PatchTST [16], and DLinear [12], as the baselines. Comparing with these different types of state-of-the-art models, FilterNet delivers the most accurate predictions of future series variations, demonstrating superior performance.

0

25

50

75

100

125

150

175

200

Time

1.2

1.1

1.0

0.9

0.8

0.7

Values

Prediction

GroundTruth

InputData

(a) FilterNet









































(b) iTransformer

0

25

50

75

100

125

150

175

200

Time

1.2

1.1

1.0

0.9

0.8

0.7

Values

Prediction

GroundTruth

InputData

(c) DLinear









































NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims are clearly written in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed the limitation of our method in the conclusion remarks.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: No theoretical proofs.

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We included the implementation details in the main paper and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will include the source data and code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please see implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in the appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The average improvement of FilterNet over all baseline models is statistically significant at the confidence of 95%.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see implementation details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper is with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed the broader impact of time series forecasting in both abstract and introduction.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not have this risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We included it in implementation details.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: N/A.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: N/A.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: N/A.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FilterNet, a novel neural network architecture for time series forecasting that leverages learnable frequency filters inspired by signal processing techniques. This paper proposes two types of filters within FilterNet: a plain shaping filter that uses a universal frequency kernel for signal filtering and temporal modeling, and a contextual shaping filter that examines filtered frequencies for compatibility with input signals to learn dependencies. This paper demonstrates that FilterNet can effectively handle high-frequency noise and utilize the full frequency spectrum, addressing key limitations of existing Transformer-based forecasters. This paper shows through extensive experiments on eight benchmarks that FilterNet achieves superior performance in both effectiveness and efficiency compared to state-of-the-art methods in time series forecasting.\n\nSTRENGTHS:\n1. This paper studies an interesting research direction, which is to adapt the advanced techniques in the signal processing domain into the time series forecasting, rather than following the existing transformer-based time series works. This is the original and novel contribution to the time series community.\n2. The main model part is overall descent. The simple but effective architectures built upon the frequency filters are proposed. I notice there is not many redundant components in the basic model, compared with existing transformer models. It can somehow demonstrates the effectiveness of their proposed two filters.\n3. The experiments are solid and convincing. Several experimental results are reported on existing benchmark datasets. Also, some experiments on synthetic periodic and trend signals with noises shown in experiments make sense. The efficiency analysis and visualization is intuitive.\n\nWEAKNESSES:\n1. The paper mainly introduces two filters. The connection and difference between the two filters is not highlighted, which makes me curious about which one is better.\n2. As far as I know, there exist many frequency filters in signal processing. The motivation for these two filters is not very clear.\n3. In experiment part, the results of TexFilter on ETTm1, ETTm2, ETTh1 and Exchange are neither the best nor the second best. Does it show the filter cannot win in many cases of forecasting? Are there any analysis or explanation about it?\n\nQUESTIONS:\n1. Can the authors elaborate on the connection and difference of the two filters?\n2. Can the authors give more details about proposed filters?\n3. Can the authors explain the lower performance of TexFilter in some cases?\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores a novel direction that leverages frequency filters directly for time series forecasting, and proposes a simple yet effective framework, namely FilterNet, from a signal processing perspective. It designs two types of learnable frequency filters corresponding to the linear and attention mapping widely used in time series modeling, i.e., PaiFilter and TexFilter. Abundant analysis experiments verify the competitive modeling capability of frequency filters, and extensive experiments on eight real-world datasets demonstrate FilterNet achieves good performances in both accuracy and efficiency over state-of-the-art methods.\n\nSTRENGTHS:\n1.\tThis paper is well-written overall, and the contribution and novelty of the proposed work are clearly presented.\n2.\tThis paper introduces a novel and straightforward framework based on the frequency filters for time series forecasting. It offers a new direction by incorporating more signal processing theory into time series analysis.\n3.\tThe frequency filter has a simple architecture but demonstrates strong temporal modeling capabilities for time series. Results in Tables 1, 4, and 5 show that the two types of proposed frequency filters are suitable for different datasets and settings, which provides valuable insights for future model design.\n4.\tThis paper seems to have solid and extensive experiments. The experiments are conducted on eight real-world different datasets, and abundant evaluation analysis and visualization experiments are presented.\n\nWEAKNESSES:\n1.\tThe visualization setting for Figure 1 appears to be missing in Appendix C.4, and I recommend adding some references about the spectral bias of neural networks, such as [1].\n2.\tSome typo errors. In line 282, the complexity of FilterNet should be O(LogL), not O(L LogL).\n\nQUESTIONS:\n1.\tIn ICML 2024, some works also adopt simple architectures for time series forecasting, such as SparseTSF [2]. Could you further compare the performance of FilterNet with SparseTSF?\n2.\tThe recent literature has seen papers based on foundation models for time series forecasting. Conversely, some works are designed with simple architectures, yet achieve better performance than foundation models. Why can simple models (e.g., FilterNet) also achieve competitive performance?\n[1] On the Spectral Bias of Neural Networks, in ICML 2019\n[2] SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameters, in ICML 2024\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-08"}

comments: SUMMARY:
The paper proposes to use Fourier transforms with parametrized filters
as layers for time series forecasting models. Filters can either be
learnable parameters directly (called "plain shaping filter") or be
a function in the inputs, gates (called "contextual shaping filter").
In experiments the authors show that their method outperforms
many baselines from the literature.

STRENGTHS:
s1. two simple, but promising ideas: i) use Fourier transforms to
  represent time series and ii) represent the filters by gates.
s2. well written paper
s3. standard experiment with improvements over several baselines

WEAKNESSES:
w1. Empirical comparison with results from a closely related baseline, FiLM
  are not shown, and FiLM reports better results for all but one dataset.
- FiLM reports better results for ETTm2, Exchange, and Traffic. The proposed
  FilterNet wins for Weather. Why do you not report those results?
  If FiLM overall is the better performing model, why is FilterNet still
  interesting? How is it simpler than FiLM? Is it maybe faster to train?

w2. Conceptual differences to existing layers using Fourier transforms
  are not discussed in detail.
- So far, we just learn that other architectures also use "other network architectures,
  such as Transformers". But a more detailed delineation would be useful,
  esp. as FiLM is not using Transformers.

minor comments:
- reference [30], FiLM, is missing its venue, NeurIPS.

QUESTIONS:
Q1. Why do you not report the better results of FiLM?
Q2. If FiLM overall is the better performing model, why is FilterNet still
  interesting? How is it simpler than FiLM?
Q3. Is it maybe faster to train?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-11

comments: SUMMARY:
This paper investigates an interesting direction from a signal processing perspective, making a new attempt to apply frequency filters directly for time series forecasting. A simple yet effective architecture called FilterNet is proposed, utilizing two types of frequency filters to achieve forecasting. Comprehensive empirical experiments on eight benchmarks confirm the superiority of this paper's proposed method in terms of effectiveness and efficiency.

STRENGTHS:
1. The paper is in good writing shape and easy to follow.
2. The proposed method seems technically solid.
3. Extensive experiments on many datasets have been conducted to verify the effectiveness of the approach.

WEAKNESSES:
1. The marginal improvement over existing approaches, as shown in Table 1, raises doubts about the compelling rationale for adopting the proposed method.
2. The discussion on related work appears lacking. It is common knowledge that frequency filters are widely used in traditional time series analysis methods and signal processing. Therefore, a detailed exploration of the connections and distinctions between them is warranted in the related work section.
3. The experiments have space to improve. Including more recent baselines for comparison, particularly deep learning methods that involve frequency processing, would add value. Additionally, a thorough investigation into the impact of hyperparameters is missing from the study.

QUESTIONS:
Please address the questions in the weaknesses.

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-30