Title: Frequency-domain MLPs are More Effective Learners in Time Series Forecasting
Venue: NeurIPS 2023
Accepted: yes
Keywords: time series forecasting

multi-layer perceptrons

frequency domain
Abstract: Time series forecasting has played the key role in different industrial, including finance, traffic, energy, and healthcare domains. While existing literatures have designed many sophisticated architectures based on RNNs, GNNs, or Transformers, another kind of approaches based on multi-layer perceptrons (MLPs) are proposed with simple structure, low complexity, and superior performance. However, most MLP-based forecasting methods suffer from the point-wise mappings and information bottleneck, which largely hinders the forecasting performance. To overcome this problem, we explore a novel direction of applying MLPs in the frequency domain for time series forecasting. We investigate the learned patterns of frequency-domain MLPs and discover their two inherent characteristic benefiting forecasting, (i) global view: frequency spectrum makes MLPs own a complete view for signals and learn global dependencies more easily, and (ii) energy compaction: frequency-domain MLPs concentrate on smaller key part of frequency components with compact signal energy. Then, we propose FreTS, a simple yet effective architecture built upon Frequency-domain MLPs for Time Series forecasting. FreTS mainly involves two stages, (i) Domain Conversion, that transforms time-domain signals into complex numbers of frequency domain; (ii) Frequency Learning, that performs our redesigned MLPs for the learning of real and imaginary part of frequency components. The above stages operated on both inter-series and intra-series scales further contribute to channel-wise and time-wise dependency learning. Extensive experiments on 13 real-world benchmarks (including 7 benchmarks for short-term forecasting and 6 benchmarks for long-term forecasting) demonstrate our consistent superiority over state-of-the-art methods. Code is available at this repository: https://github.com/aikunyi/FreTS.

Frequency-domain MLPs are More Effective Learners in Time Series Forecasting

Abstract

Time series forecasting has played the key role in different industrial, including finance, traffic, energy, and healthcare domains. While existing literatures have designed many sophisticated architectures based on RNNs, GNNs, or Transformers, another kind of approaches based on multi-layer perceptrons (MLPs) are proposed with simple structure, low complexity, and superior performance. However, most MLP-based forecasting methods suffer from the point-wise mappings and information bottleneck , which largely hinders the forecasting performance. To overcome this problem, we explore a novel direction of applying MLPs in the frequency domain for time series forecasting. We investigate the learned patterns of frequency-domain MLPs and discover their two inherent characteristic benefiting forecasting, (i) global view : frequency spectrum makes MLPs own a complete view for signals and learn global dependencies more easily, and (ii) energy compaction : frequency-domain MLPs concentrate on smaller key part of frequency components with compact signal energy. Then, we propose FreTS, a simple yet effective architecture built upon Fre quency-domain MLPs for T ime S eries forecasting. FreTS mainly involves two stages, (i) Domain Conversion, that transforms time-domain signals into complex numbers of frequency domain; (ii) Frequency Learning, that performs our redesigned MLPs for the learning of real and imaginary part of frequency components. The above stages operated on both inter-series and intra-series scales further contribute to channel-wise and time-wise dependency learning. Extensive experiments on 13 real-world benchmarks (including 7 benchmarks for short-term forecasting and 6 benchmarks for long-term forecasting) demonstrate our consistent superiority over state-of-the-art methods. Code is available at this repository: https://github.com/aikunyi/FreTS .

1 Introduction

Time series forecasting has been a critical role in a variety of real-world industries, such as climate condition estimation [1, 2], traffic state prediction [3, 4], economic analysis [5, 6], etc. In the early stage, many traditional statistical forecasting methods have been proposed, such as exponential smoothing [7] and auto-regressive moving averages (ARMA) [8]. Recently, the emerging development of deep learning has fostered many deep forecasting models, including Recurrent Neural Network-based methods (e.g., DeepAR [9], LSTNet [10]), Convolution Neural Network-based methods (e.g., TCN [11], SCINet [12]), Transformer-based methods (e.g., Informer [13], Autoformer [14]), and Graph Neural Network-based methods (e.g., MTGNN [15], StemGNN [16], AGCRN [17]), etc.

0.8

0.10

20

0.6

20

0.05

00°0

40 -

0,4

40-

=0.0

60

0.260

=0.1

=0.1

80 -

0.0

80

=0.2

0

20

40

60

80

20

40

60

80

0

0

0.10

0.3

20 -

20

0.05

0.2

40

40

0.1

60

0.00

60

0.0

08

=0.05

-08

0.1

100 -

100

0.2

=0.10

120

120

0.3

20

40

09

80

100

120

0

20

40

60

80

100

120

(a)

Left

: time domain.

Right

: frequency domain.

(b)

Left

: time domain.

Right

: frequency domain.

While these deep models have achieved promising forecasting performance in certain scenarios, their sophisticated network architectures would usually bring up expensive computation burden in training or inference stage. Besides, the robustness of these models could be easily influenced with a large amount of parameters, especially when the available training data is limited [13, 18]. Therefore, the methods based on multi-layer perceptrons (MLPs) have been recently introduced with simple structure, low complexity, and superior forecasting performance, such as N-BEATS [19], LightTS [20], DLinear [21], etc. However, these MLP-based methods rely on point-wise mappings to capture temporal mappings, which cannot handle global dependencies of time series. Moreover, they would suffer from the information bottleneck with regard to the volatile and redundant local momenta of time series, which largely hinders their performance for time series forecasting.

To overcome the above problems, we explore a novel direction of applying MLPs in the frequency domain for time series forecasting. We investigate the learned patterns of frequency-domain MLPs in forecasting and have discovered their two key advantages: (i) global view : operating on spectral components acquired from series transformation, frequency-domain MLPs can capture a more complete view of signals, making it easier to learn global spatial/temporal dependencies. (ii) energy compaction : frequency-domain MLPs concentrate on the smaller key part of frequency components with the compact signal energy, and thus can facilitate preserving clearer patterns while filtering out influence of noises. Experimentally, we have observed that frequency-domain MLPs capture much more obvious global periodic patterns than the time-domain MLPs from Figure 1(a), which highlights their ability to recognize global signals. Also, from Figure 1(b), we easily note a much more clear diagonal dependency in the learned weights of frequency-domain MLPs, compared with the more scattered dependency learned by time-domain MLPs. This illustrates the great potential of frequency-domain MLPs to identify most important features and key patterns while handling complicated and noisy information.

To fully utilize these advantages, we propose FreTS, a simple yet effective architecture of Fre quencydomain MLPs for T ime S eries forecasting. The core idea of FreTS is to learn the time series forecasting mappings in the frequency domain . Specifically, FreTS mainly involves two stages: (i) Domain Conversion: the original time-domain series signals are first transformed into frequencydomain spectrum on top of Discrete Fourier Transform (DFT) [22], where the spectrum is composed of several complex numbers as frequency components, including the real coefficients and the imaginary coefficients . (ii) Frequency Learning: given the real/imaginary coefficients, we redesign the frequencydomain MLPs originally for the complex numbers by separately considering the real mappings and imaginary mappings. The respective real/imaginary parts of output learned by two distinct MLPs are then stacked in order to recover from frequency components to the final forecasting. Also, FreTS performs above two stages on both inter-series and intra-series scales, which further contributes to the channel-wise and time-wise dependencies in the frequency domain for better forecasting performance. We conduct extensive experiments on 13 benchmarks under different settings, covering 7 benchmarks for short-term forecasting and 6 benchmarks for long-term forecasting, which demonstrate our consistent superiority compared with state-of-the-art methods.

2 Related Work

Forecasting in the Time Domain Traditionally, statistical methods have been proposed for forecasting in the time domain, including (ARMA) [8], VAR [23], and ARIMA [24]. Recently, deep learning

based methods have been widely used in time series forecasting due to their capability of extracting nonlinear and complex correlations [25, 26]. These methods have learned the dependencies in the time domain with RNNs (e.g., deepAR [9], LSTNet [10]) and CNNs (e.g., TCN [11], SCINet [12]). In addition, GNN-based models have been proposed with good forecasting performance because of their good abilities to model series-wise dependencies among variables in the time domain, such as TAMP-S2GCNets [4], AGCRN [17], MTGNN [15], and GraphWaveNet [27]. Besides, Transformerbased forecasting methods have been introduced due to their attention mechanisms for long-range dependency modeling ability in the time domain, such as Reformer [18] and Informer [13].

Forecasting in the Frequency Domain Several recent time series forecasting methods have extracted knowledge of the frequency domain for forecasting [28]. Specifically, SFM [29] decomposes the hidden state of LSTM into frequencies by Discrete Fourier Transform (DFT). StemGNN [16] performs graph convolutions based on Graph Fourier Transform (GFT) and computes series correlations based on Discrete Fourier Transform. Autoformer [14] replaces self-attention by proposing the auto-correlation mechanism implemented with Fast Fourier Transforms (FFT). FEDformer [30] proposes a DFT-based frequency enhanced attention, which obtains the attentive weights by the spectrums of queries and keys, and calculates the weighted sum in the frequency domain. CoST [31] uses DFT to map the intermediate features to frequency domain to enables interactions in representation. FiLM [32] utilizes Fourier analysis to preserve historical information and remove noisy signals. Unlike these efforts that leverage frequency techniques to improve upon the original architecture such as Transformer and GNN, in this paper, we propose a new frequency learning architecture that learns both channel-wise and time-wise dependencies in the frequency domain.

MLP-based Forecasting Models Several studies have explored the use of MLP-based networks in time series forecasting. N-BEATS [19] utilizes stacked MLP layers together with doubly residual learning to process the input data to iteratively forecast the future. DEPTS [33] applies Fourier transform to extract periods and MLPs for periodicity dependencies for univariate forecasting. LightTS [20] uses lightweight sampling-oriented MLP structures to reduce complexity and computation time while maintaining accuracy. N-HiTS [34] combines multi-rate input sampling and hierarchical interpolation with MLPs for univariate forecasting. LTSF-Linear [35] proposes a set of embarrassingly simple one-layer linear model to learn temporal relationships between input and output sequences. These studies demonstrate the effectiveness of MLP-based networks in time series forecasting tasks, and inspire the development of our frequency-domain MLPs in this paper.

3 FreTS

In this section, we elaborate on our proposed novel approach, FreTS, based on our redesigned MLPs in the frequency domain for time series forecasting. First, we present the detailed frequency learning architecture of FreTS in Section 3.1, which mainly includes two-fold frequency learners with domain conversions. Then, we detailedly introduce our redesigned frequency-domain MLPs adopted by above frequency learners in Section 3.2. Besides, we also theoretically analyze their superior nature of global view and energy compaction, as aforementioned in Section 1.

Problem Definition Let [ X 1 , X 2 , · · · , X T ] ∈ R N × T stand for the regularly sampled multi-variate time series dataset with N series and T timestamps, where X t ∈ R N denotes the multi-variate values of N distinct series at timestamp t . We consider a time series lookback window of lengthL at timestamp t as the model input, namely X t = [ X t -L +1 , X t -L +2 , · · · , X t ] ∈ R N × L ; also, we consider a horizon window of lengthτ at timestamp t as the prediction target, denoted as Y t = [ X t +1 , X t +2 , · · · , X t + τ ] ∈ R N × τ . Then the time series forecasting formulation is to use historical observations X t to predict future values ˆ Y t and the typical forecasting model f θ parameterized by θ is to produce forecasting results by ˆ Y t = f θ ( X t ) .

3.1 Frequency Learning Architecture

The frequency learning architecture of FreTS is depicted in Figure 2, which mainly involves Domain Conversion/Inversion stages, Frequency-domain MLPs, and the corresponding two learners, i.e., the Frequency Channel Learner and the Frequency Temporal Learner. Besides, before taken to learners, we concretely apply a dimension extension block on model input to enhance the model capability. Specifically, the input lookback window X t ∈ R N × L is multiplied with a learnable

0

0

NxL

NxLxd

FFN

Nxτ

Input

Output

Frequency Channel Learner

Domain

Conversion

Domain

Inversion

Dimension

Extension

Frequency-domain

MLP

L

N

NxLxd

Concat

Nxd

Nxd

NxLxd

Concat

L

N

time dimensions

Channel dimensions

Domain

Conversion

Domain

Inversion

Frequency Temporal Learner

Frequency-domain

MLP

N

L

L

N

Frequency

Frequency

Amplitude

Amplitude

L

N

τ

𝓦

chan

𝓑

chan

𝓦

𝓑

temp

temp

Complex Number

Real Part

Imaginary Part

𝓦

𝓑

Complex number weight

Complex number bias

Imaginary Part

Real Part

𝓦

𝓑

𝓗

𝓗

𝓗𝓦+𝓑

𝐗

𝐇

𝐙

𝓢

𝐒

𝓏

𝓏

chan

temp

Operations in the frequency domain

෠ 𝐘

Shared with N channels

Shared with L timestamps

FreTS

weight vector ϕ d ∈ R 1 × d to obtain a more expressive hidden representation H t ∈ R N × L × d , yielding H t = X t × ϕ d to bring more semantic information, inspired by word embeddings [36].

Domain Conversion/Inversion The use of Fourier transform enables the decomposition of a time series signal into its constituent frequencies. This is particularly advantageous for time series analysis since it benefits to identify periodic or trend patterns in the data, which are often important in forecasting tasks. As aforementioned in Figure 1(a), learning in the frequency spectrum helps capture a greater number of periodic patterns. In view of this, we convert the input H into the frequency domain H by:

where f is the frequency variable, v is the integral variable, and j is the imaginary unit, which is defined as the square root of -1; ∫ ∞ -∞ H ( v ) cos(2 πfv )d v is the real part of H and is abbreviated as Re ( H ) ; ∫ ∞ -∞ H ( v ) sin(2 πfv )d v is the imaginary part and is abbreviated as Im ( H ) . Then we can rewrite H in Equation (1) as: H = Re ( H ) + jIm ( H ) . Note that in FreTS we operate domain conversion on both the channel dimension and time dimension, respectively. Once completing the learning in the frequency domain, we can convert H back into the the time domain using the following inverse conversion formulation:

where we take frequency f as the integral variable. In fact, the frequency spectrum is expressed as a combination of cos and sin waves in H with different frequencies and amplitudes inferring different periodic properties in time series signals. Thus examining the frequency spectrum can better discern the prominent frequencies and periodic patterns in time series. In the following sections, we use DomainConversion to stand for Equation (1), and DomainInversion for Equation (2) for brevity.

Frequency Channel Learner Considering channel dependencies for time series forecasting is important because it allows the model to capture interactions and correlations between different variables, leading to a more accurate predictions. The frequency channel learner enables communications between different channels; it operates on each timestamp by sharing the same weights between L timestamps to learn channel dependencies. Concretely, the frequency channel learner takes H t ∈ R N × L × d as input. Given the l -th timestamp H : , ( l ) t ∈ R N × d , we perform the frequency channel learner by:

where H : , ( l ) chan ∈ C N 2 × d is the frequency components of H : , ( l ) t ; DomainConversion ( chan ) and DomainInversion ( chan ) indicates such operations are performed along the channel dimension. FreMLP are frequency-domain MLPs proposed in Section 3.2, which takes W chan = ( W chan r + j W chan i ) ∈ C d × d as the complex number weight matrix with W chan r ∈ R d × d and W chan i ∈ R d × d , and B chan = ( B chan r + j B chan i ) ∈ C d as the biases with B chan r ∈ R d and B chan i ∈ R d . And Z : , ( l ) chan ∈ C N 2 × d is the output of FreMLP , also in the frequency domain, which is conversed back to time domain as Z : , ( l ) ∈ R N × d . Finally, we ensemble Z : , ( l ) of L timestamps into a whole and output Z t ∈ R N × L × d .

Frequency Temporal Learner The frequency temporal learner aims to learn the temporal patterns in the frequency domain; also, it is constructed based on frequency-domain MLPs conducting on each channel and it shares the weights between N channels. Specifically, it takes the frequency channel learner output Z t ∈ R N × L × d as input and for the n -th channel Z ( n ) , : t ∈ R L × d , we apply the frequency temporal learner by:

where Z ( n ) , : temp ∈ C L 2 × d is the corresponding frequency spectrum of Z ( n ) , : t ; DomainConversion ( temp ) and DomainInversion ( temp ) indicates the calculations are applied along the time dimension. W temp = ( W temp r + j W temp i ) ∈ C d × d is the complex number weight matrix with W temp r ∈ R d × d and W temp i ∈ R d × d , and B temp = ( B temp r + j B temp i ) ∈ C d are the complex number biases with B temp r ∈ R d and B temp i ∈ R d . S ( n ) , : temp ∈ C L 2 × d is the output of FreMLP and is converted back to the time domain as S ( n ) , : ∈ R L × d . Finally, we incorporate all channels and output S t ∈ R N × L × d .

Projection Finally, we use the learned channel and temporal dependencies to make predictions for the future τ timestamps ˆ Y t ∈ R N × τ by a two-layer feed forward network (FFN) with one forward step which can avoid error accumulation, formulated as follows:

where S t ∈ R N × L × d is the output of the frequency temporal learner, σ is the activation function, ϕ 1 ∈ R ( L ∗ d ) × d h , ϕ 2 ∈ R d h × τ are the weights, b 1 ∈ R d h , b 2 ∈ R τ are the biases, and d h is the inner-layer dimension size.

3.2 Frequency-domain MLPs

As shown in Figure 3, we elaborate our novel frequency-domain MLPs in FreTS that are redesigned for the complex numbers of frequency components, in order to effectively capture the time series key patterns with global view and energy compaction , as aforementioned in Section 1.

Definition 1 ( Frequency-domain MLPs ) . Formally, for a complex number input H ∈ C m × d , given a complex number weight matrix W ∈ C d × d and a complex number bias B ∈ C d , then the frequency-domain MLPs can be formulated as:

where Y ℓ ∈ C m × d is the final output, ℓ denotes the ℓ -th layer, and σ is the activation function.

As both H and W are complex numbers, according to the rule of multiplication of complex numbers (details can be seen in Appendix C), we further extend the Equation (6) to:

where W ℓ = W ℓ r + j W ℓ i and B ℓ = B ℓ r + j B ℓ i . According to the equation, we implement the MLPs in the frequency domain (abbreviated as FreMLP ) by the separate computation of the real and imaginary parts of frequency components. Then, we stack them to form a complex number to acquire the final results. The specific implementation process is shown in Figure 3.

Theorem 1. Suppose that H is the representation of raw time series and H is the corresponding frequency components of the spectrum, then the energy of a time series in the time domain is equal to the energy of its representation in the frequency domain. Formally, we can express this with above notations by:

where H ( f ) = ∫ ∞ -∞ H ( v ) e -j 2 πfv d v , v is the time/channel dimension, f is the frequency dimension.

We include the proof in Appendix D.1. The theorem implies that if most of the energy of a time series is concentrated in a small number of frequency components, then the time series can be accurately represented using only those components. Accordingly, discarding the others would not significantly affect the signal's energy. As shown in Figure 1(b), in the frequency domain, the energy concentrates on the smaller part of frequency components, thus learning in the frequency spectrum can facilitate preserving clearer patterns.

Theorem 2. Given the time series input H and its corresponding frequency domain conversion H , the operations of frequency-domain MLP on H can be represented as global convolutions on H in the time domain. This can be given by:

where ∗ is a circular convolution, W and B are the complex number weight and bias, W and B are the weight and bias in the time domain, and F is DFT .

The proof is shown in Appendix D.2. Therefore, the operations of FreMLP , i.e., HW + B , are equal to the operations ( H ∗ W + B ) in the time domain. This implies that the operations of frequency-domain MLPs can be viewed as global convolutions in the time domain.

4 Experiments

To evaluate the performance of FreTS, we conduct extensive experiments on thirteen real-world time series benchmarks, covering short-term forecasting and long-term forecasting settings to compare with corresponding state-of-the-art methods.

Datasets Our empirical results are performed on various domains of datasets, including traffic, energy, web, traffic, electrocardiogram, and healthcare, etc. Specifically, for the task of short-term forecasting , we adopt Solar 2 , Wiki [37], Traffic [37], Electricity 3 , ECG [16], METR-LA [38], and COVID-19 [4] datasets, following previous forecasting literature [16]. For the task of longterm forecasting , we adopt Weather [14], Exchange [10], Traffic [14], Electricity [14], and ETT datasets [13], following previous long time series forecasting works [13, 14, 30, 39]. We preprocess all datasets following [16, 13, 14] and normalize them with the min-max normalization. We split the datasets into training, validation, and test sets by the ratio of 7:2:1 except for the COVID-19 datasets with 6:2:2. More dataset details are in Appendix B.1.

Baselines We compare our FreTS with the representative and state-of-the-art models for both shortterm and long-term forecasting to evaluate their effectiveness. For short-term forecasting, we compre FreTS against VAR [23], SFM [29], LSTNet [10], TCN [11], GraphWaveNet [27], DeepGLO [37], StemGNN [16], MTGNN [15], and AGCRN [17] for comparison. We also include TAMP-S2GCNets [4], DCRNN [38] and STGCN [40], which require pre-defined graph structures, for comparison. For long-term forecasting, we include Informer [13], Autoformer [14], Reformer [18], FEDformer [30], LTSF-Linear [35], and the more recent PatchTST [39] for comparison. Additional details about the baselines can be found in Appendix B.2.

Implementation Details Our model is implemented with Pytorch 1.8 [41], and all experiments are conducted on a single NVIDIA RTX 3080 10GB GPU. We take MSE (Mean Squared Error) as the loss function and report MAE (Mean Absolute Errors) and RMSE (Root Mean Squared Errors) results as the evaluation metrics. For additional implementation details, please refer to Appendix B.3.

4.1 Main Results

Short-Term Time Series Forecasting Table 1 presents the forecasting accuracy of our FreTS compared to thirteen baselines on six datasets, with an input length of 12 and a prediction length of 12. The best results are highlighted in bold and the second-best results are underlined. From the table, we observe that FreTS outperforms all baselines on MAE and RMSE across all datasets, and on average it makes improvement of 9.4% on MAE and 11.6% on RMSE. We credit this to the fact that FreTS explicitly models both channel and temporal dependencies, and it flexibly unifies channel and temporal modeling in the frequency domain, which can effectively capture the key patterns with the global view and energy compaction. We further report the complete benchmarks of short-term forecasting under different steps on different datasets (including METR-LA dataset) in Appendix F.1.

Long-term Time Series Forecasting Table 2 showcases the long-term forecasting results of FreTS compared to six representative baselines on six benchmarks with various prediction lengths. For the traffic dataset, we select 48 as the lookback window size L with the prediction lengths τ ∈ { 48 , 96 , 192 , 336 } . For the other datasets, the input lookback window length is set to 96 and the prediction length is set to τ ∈ { 96 , 192 , 336 , 720 } . The results demonstrate that FreTS outperforms all baselines on all datasets. Quantitatively, compared with the best results of Transformer-based models, FreTS has an average decrease of more than 20% in MAE and RMSE. Compared with more recent LSTF-Linear [35] and the SOTA PathchTST [39], FreTS can still outperform them in general. In addition, we provide further comparison of FreTS and other baselines and report performance under different lookback window sizes in Appendix F.2. Combining Tables 1 and 2, we can conclude that FreTS achieves competitive performance in both short-term and long-term forecasting task.

4.2 Model Analysis

Frequency Channel and Temporal Learners We analyze the effects of frequency channel and temporal learners in Table 3 in both short-term and long-term experimental settings. We consider two variants: FreCL : we remove the frequency temporal learner from FreTS, and FreTL : we remove the frequency channel learner from FreTS. From the comparison, we observe that the frequency chan-

nel learner plays a more important role in short-term forecasting. In long-term forecasting, we note that the frequency temporal learner is more effective than the frequency channel learner. In Appendix E.1, we also conduct the experiments and report performance on other datasets. Interestingly, we find out the channel learner would lead to the worse performance in some long-term forecasting cases. A potential explanation is that the channel independent strategy [39] brings more benefit to forecasting.

FreMLP vs. MLP We further study the effectiveness of FreMLP in time series forecasting. We use FreMLP to replace the original MLP component in the existing SOTA MLP-based models (i.e., DLinear and NLinear [35]), and compare their performances with the original DLinear and NLinear under the same experimental settings. The experimental results are presented in Table 4. From the table, we easily observe that for any prediction length, the performance of both DLinear and NLinear models has been improved after replacing the corresponding MLP component with our FreMLP . Quantitatively, incorporating FreMLP into the DLinear model brings an average improvement of 6.4% in MAE and 11.4% in RMSE on the Exchange dataset, and 4.9% in MAE and 3.5% in RMSE on the Weather dataset. A similar improvement has also been achieved on the two datasets with regard to NLinear, according to Table 4. These results confirm the effectiveness of FreMLP compared to MLP again and we include more implementation details and analysis in Appendix B.5.

4.3 Efficiency Analysis

The complexity of our proposed FreTS is O ( N log N + L log L ) . We perform efficiency comparisons with some state-of-the-art GNN-based methods and Transformer-based models under different numbers of variables N and prediction lengths τ , respectively. On the Wiki dataset, we conduct experiments over N ∈ { 1000 , 2000 , 3000 , 4000 , 5000 } under the same lookback window size of 12

















































































































































Figure 4: Efficiency analysis (model parameters and training time) on the Wiki and Exchange dataset. (a) The efficiency comparison under different number of variables: the number of variables is enlarged from 1000 to 5000 with the input window size as 12 and the prediction length as 12 on Wiki dataset. (b) The efficiency comparison under the prediction lengths: we conduct experiments with prediction lengths prolonged from 96 to 480 under the same window size of 96 on the Exchange dataset.

and prediction length of 12, as shown in Figure 4(a). From the figure, we can find that: (1) The amount of FreTS parameters is agnostic to N . (2) Compared with AGCRN, FreTS incurs an average 30% reduction of the number of parameters and 20% reduction of training time. On the Exchange dataset, we conduct experiments on different prediction lengths τ ∈ { 96 , 192 , 336 , 480 } with the same input length of 96. The results are shown in Figure 4(b). It demonstrates: (1) Compared with Transformer-based methods (FEDformer [30], Autoformer [14], and Informer [13]), FreTS reduces the number of parameters by at least 3 times. (2) The training time of FreTS is averagely 3 times faster than Informer, 5 times faster than Autoformer, and more than 10 times faster than FEDformer. These show our great potential in real-world deployment.

4.4 Visualization Analysis

In Figure 5, we visualize the learned weights W in FreMLP on the Traffic dataset with a lookback window size of 48 and prediction length of 192. As the weights W are complex numbers, we provide visualizations of the real part W r (presented in (a)) and the imaginary part W i (presented in (b)) separately. From the figure, we can observe that both the real and imaginary parts play a crucial role in learning process: the weight coefficients of the real or imaginary part exhibit energy aggregation characteristics (clear diagonal pat-

(a) The real part

W

r

(b) The imaginary part

W

i

0

0.6

20

0.3

20

0.4

40

0.2

40

- 0.2

60

0.1

09

- 0.0

0.0

0.2

08

0.1

08

0.4

100

0.2

100

0.6

120

0.3

120

8°0-

0

20

40

60

80

100

120

0

20

40

60

08

100

120

terns) which can facilitate to learn the significant features. In Appendix E.2, we further conduct a detailed analysis on the effects of the real and imaginary parts in different contexts of forecasting, and the effects of the two parts in the FreMLP. We examine their individual contributions and investigate how they influence the final performance. Additional visualizations of the weights on different datasets with various settings, as well as visualizations of global periodic patterns, can be found in Appendix G.1 and Appendix G.2, respectively.

5 Conclusion Remarks

In this paper, we explore a novel direction and make a new attempt to apply frequency-domain MLPs for time series forecasting. We have redesigned MLPs in the frequency domain that can effectively capture the underlying patterns of time series with global view and energy compaction. We then verify this design by a simple yet effective architecture, FreTS, built upon the frequency-domain MLPs for time series forecasting. Our comprehensive empirical experiments on seven benchmarks of short-term forecasting and six benchmarks of long-term forecasting have validated the superiority of our proposed methods. Simple MLPs have several advantages and lay the foundation of modern deep learning, which have great potential for satisfied performance with high efficiency. We hope this work can facilitate more future research of MLPs on time series modeling.

Acknowledgments and Disclosure of Funding

The work was supported in part by the National Key Research and Development Program of China under Grant 2020AAA0104903 and 2019YFB1406300, and National Natural Science Foundation of China under Grant 62072039 and 62272048.

References

Edward N Lorenz. Empirical orthogonal functions and statistical weather prediction , volume 1. Massachusetts Institute of Technology, Department of Meteorology Cambridge, 1956.

Yu Zheng, Xiuwen Yi, Ming Li, Ruiyuan Li, Zhangqing Shan, Eric Chang, and Tianrui Li. Forecasting fine-grained air quality based on big data. In KDD , pages 2267-2276, 2015.

Hui He, Qi Zhang, Simeng Bai, Kun Yi, and Zhendong Niu. CATN: cross attentive tree-aware network for multivariate time series forecasting. In AAAI , pages 4030-4038. AAAI Press, 2022.

Yuzhou Chen, Ignacio Segovia-Dominguez, Baris Coskunuzer, and Yulia Gel. TAMP-s2GCNets: Coupling time-aware multipersistence knowledge representation with spatio-supra graph convolutional networks for time-series forecasting. In International Conference on Learning Representations , 2022.

Benjamin F King. Market and industry factors in stock price behavior. the Journal of Business , 39(1):139-190, 1966.

Adebiyi A Ariyo, Adewumi O Adewumi, and Charles K Ayo. Stock price prediction using the arima model. In 2014 UKSim-AMSS 16th international conference on computer modelling and simulation , pages 106-112. IEEE, 2014.

Charles C Holt. Forecasting trends and seasonal by exponentially weighted moving averages. ONR Memorandum , 52(2), 1957.

Peter Whittle. Prediction and regulation by linear least-square methods . English Universities Press, 1963.

David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting , 36(3):1181-1191, 2020.

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In SIGIR , pages 95-104, 2018.

Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. CoRR , abs/1803.01271, 2018.

Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma, and Qiang Xu. Scinet: time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems , 35:5816-5828, 2022.

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI , pages 11106-11115, 2021.

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In NeurIPS , pages 2241922430, 2021.

Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In KDD , pages 753-763, 2020.

Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. Spectral temporal graph neural network for multivariate time-series forecasting. In NeurIPS , 2020.

Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. In NeurIPS , 2020.

Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR , 2020.

Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437 , 2019.

Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures. arXiv preprint arXiv:2207.01186 , 2022.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205.13504 , 2022.

Duraisamy Sundararajan. The discrete Fourier transform: theory, algorithms and applications . World Scientific, 2001.

Mark W. Watson. Vector autoregressions and cointegration. Working Paper Series, Macroeconomic Issues , 4, 1993.

Dimitros Asteriou and Stephen G Hall. Arima models and the box-jenkins methodology. Applied Econometrics , 2(2):265-286, 2011.

Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , 379(2194):20200209, feb 2021.

José Torres, Dalil Hadjout, Abderrazak Sebaa, Francisco Martínez-Álvarez, and Alicia Troncoso. Deep learning for time series forecasting: A survey. Big Data , 9, 12 2020.

Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. In IJCAI , pages 1907-1913, 2019.

Kun Yi, Qi Zhang, Longbing Cao, Shoujin Wang, Guodong Long, Liang Hu, Hui He, Zhendong Niu, Wei Fan, and Hui Xiong. A survey on deep learning based time series analysis with frequency transformation. CoRR , abs/2302.02173, 2023.

Liheng Zhang, Charu C. Aggarwal, and Guo-Jun Qi. Stock price prediction via discovering multi-frequency trading patterns. In KDD , pages 2141-2149, 2017.

Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML , 2022.

Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting. In ICLR . OpenReview.net, 2022.

Tian Zhou, Ziqing Ma, Xue Wang, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin, and Rong Jin. Film: Frequency improved legendre memory model for long-term time series forecasting. 2022.

Wei Fan, Shun Zheng, Xiaohan Yi, Wei Cao, Yanjie Fu, Jiang Bian, and Tie-Yan Liu. DEPTS: deep expansion learning for periodic time series forecasting. In ICLR . OpenReview.net, 2022.

Cristian Challu, Kin G. Olivares, Boris N. Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hierarchical interpolation for time series forecasting. CoRR , abs/2201.12886, 2022.

Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? 2023.

Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In ICLR (Workshop Poster) , 2013.

Rajat Sen, Hsiang-Fu Yu, and Inderjit S. Dhillon. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. In NeurIPS , pages 4838-4847, 2019.

Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In ICLR (Poster) , 2018.

Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations , 2023.

Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In IJCAI , pages 3634-3640, 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS , pages 8024-8035, 2019.

A Notations

B Experimental Details

B.1 Datasets

We adopt thirteen real-world benchmarks in the experiments to evaluate the accuracy of short-term and long-term forecasting. The details of the datasets are as follows:

Solar 4 : It is about the solar power collected by National Renewable Energy Laboratory. We choose the power plant data points in Florida as the data set which contains 593 points. The data is collected from 01/01/2006 to 31/12/2016 with the sampling interval of every 1 hour.

Wiki [37] : It contains a number of daily views of different Wikipedia articles and is collected from 1/7/2015 to 31/12/2016. It consists of approximately 145 k time series and we randomly choose 5 k from them as our experimental data set.

Traffic [37] : It contains hourly traffic data from 963 San Francisco freeway car lanes for short-term forecasting settings while it contains 862 car lanes for long-term forecasting. It is collected since 01/01/2015 with a sampling interval of every 1 hour.

ECG 5 : It is about Electrocardiogram(ECG) from the UCR time-series classification archive. It contains 140 nodes and each node has a length of 5000.

Electricity 6 : It contains electricity consumption of 370 clients for short-term forecasting while it contains electricity consumption of 321 clients for long-term forecasting. It is collected since 01/01/2011. The data sampling interval is every 15 minutes.

COVID-19 [4] : It is about COVID-19 hospitalization in the U.S. state of California (CA) from 01/02/2020 to 31/12/2020 provided by the Johns Hopkins University with the sampling interval of every day.

METR-LA 7 : It contains traffic information collected from loop detectors in the highway of Los Angeles County. It contains 207 sensors which are from 01/03/2012 to 30/06/2012 and the data sampling interval is every 5 minutes.

Exchange 8 : It contains the collection of the daily exchange rates of eight foreign countries including Australia, British, Canada, Switzerland, China, Japan, New Zealand, and Singapore ranging from 1990 to 2016 and the data sampling interval is every 1 day.

Weather 9 : It collects 21 meteorological indicators, such as humidity and air temperature, from the Weather Station of the Max Planck Biogeochemistry Institute in Germany in 2020. The data sampling interval is every 10 minutes.

ETT 10 : It is collected from two different electric transformers labeled with 1 and 2, and each of them contains 2 different resolutions (15 minutes and 1 hour) denoted with m and h. We use ETTh1 and ETTm1 as our long-term forecasting benchmarks.

B.2 Baselines

We adopt eighteen representative and state-of-the-art baselines for comparison including LSTM-based models, GNN-based models, and Transformer-based models. We introduce these models as follows:

VAR [23]: VAR is a classic linear autoregressive model. We use the Statsmodels library ( https: //www.statsmodels.org ) which is a Python package that provides statistical computations to realize the V AR.

DeepGLO [37]: DeepGLO models the relationships among variables by matrix factorization and employs a temporal convolution neural network to introduce non-linear relationships. We download the source code from: https://github.com/rajatsen91/deepglo . We use the recommended configuration as our experimental settings for Wiki, Electricity, and Traffic datasets. For the COVID19 dataset, the vertical and horizontal batch size is set to 64, the rank of the global model is set to 64, the number of channels is set to [32, 32, 32, 1], and the period is set to 7.

LSTNet [10]: LSTNet uses a CNN to capture inter-variable relationships and an RNN to discover long-term patterns. We download the source code from: https://github.com/laiguokun/ LSTNet . In our experiment, we use the recommended configuration where the number of CNN hidden units is 100, the kernel size of the CNN layers is 4, the dropout is 0.2, the RNN hidden units is 100, the number of RNN hidden layers is 1, the learning rate is 0.001 and the optimizer is Adam.

TCN [11]: TCN is a causal convolution model for regression prediction. We download the source code from: https://github.com/locuslab/TCN . We utilize the same configuration as the polyphonic music task exampled in the open source code where the dropout is 0.25, the kernel size is 5, the number of hidden units is 150, the number of levels is 4 and the optimizer is Adam.

Informer [13]: Informer leverages an efficient self-attention mechanism to encode the dependencies among variables. We download the source code from: https://github.com/zhouhaoyi/ Informer2020 . We use the recommended configuration as the experimental settings where the dropout is 0.05, the number of encoder layers is 2, the number of decoder layers is 1, the learning rate is 0.0001, and the optimizer is Adam.

Reformer [18]: Reformer combines the modeling capacity of a Transformer with an architecture that can be executed efficiently on long sequences and with small memory use. We download the source

code from: https://github.com/thuml/Autoformer . We use the recommended configuration as the experimental settings.

Autoformer [14]: Autoformer proposes a decomposition architecture by embedding the series decomposition block as an inner operator, which can progressively aggregate the long-term trend part from intermediate prediction. We download the source code from: https://github.com/thuml/ Autoformer . We use the recommended configuration as the experimental settings.

FEDformer [30]: FEDformer proposes an attention mechanism with low-rank approximation in frequency and a mixture of expert decomposition to control the distribution shifting. We download the source code from: https://github.com/MAZiqing/FEDformer . We use FEB-f as the Frequency Enhanced Block and select the random mode with 64 as the experimental mode.

SFM [29]: On the basis of the LSTM model, SFM introduces a series of different frequency components in the cell states. We download the source code from: https://github.com/z331565360/ State-Frequency-Memory-stock-prediction . We follow the recommended configuration as the experimental settings where the learning rate is 0.01, the frequency dimension is 10, the hidden dimension is 10 and the optimizer is RMSProp.

StemGNN [16]: StemGNN leverages GFT and DFT to capture dependencies among variables in the frequency domain. We download the source code from: https://github.com/microsoft/ StemGNN . We use the recommended configuration of stemGNN as our experiment setting where the optimizer is RMSProp, the learning rate is 0.0001, the number of stacked layers is 5, and the dropout rate is 0.5.

MTGNN [15]: MTGNN proposes an effective method to exploit the inherent dependency relationships among multiple time series. We download the source code from: https://github.com/ nnzhan/MTGNN . Because the experimental datasets have no static features, we set the parameter load_static_feature to false. We construct the graph by the adaptive adjacency matrix and add the graph convolution layer. Regarding other parameters, we follow the recommended settings.

GraphWaveNet [27]: GraphWaveNet introduces an adaptive dependency matrix learning to capture the hidden spatial dependency. We download the source code from: https://github.com/ nnzhan/Graph-WaveNet . Since our datasets have no prior defined graph structures, we use only adaptive adjacent matrix. We add a graph convolutional layer and randomly initialize the adjacent matrix. We adopt the recommended setting as its experimental configuration where the learning rate is 0.001, the dropout is 0.3, the number of epochs is 50, and the optimizer is Adam.

AGCRN [17]: AGCRN proposes a data-adaptive graph generation module for discovering spatial correlations from data. We download the source code from: https://github.com/LeiBAI/AGCRN . We follow the recommended settings where the embedding dimension is 10, the learning rate is 0.003, and the optimizer is Adam.

TAMP-S2GCNets [4]: TAMP-S2GCNets explores the utility of MP to enhance knowledge representation mechanisms within the time-aware DL paradigm. We download the source code from: https: //www.dropbox.com/sh/n0ajd5l0tdeyb80/AABGn-ejfV1YtRwjf_L0AOsNa?dl=0 . TAMPS2GCNets require a pre-defined graph topology and we use the California State topology provided by the source code as input. We adopt the recommended settings as the experimental configuration for COVID-19.

DCRNN [38]: DCRNN uses bidirectional graph random walk to model spatial dependency and recurrent neural network to capture the temporal dynamics. We download the source code from: https://github.com/liyaguang/DCRNN . We use the recommended configuration as our experimental settings with the batch size is 64, the learning rate is 0 . 01 , the input dimension is 2 and the optimizer is Adam. DCRNN requires a pre-defined graph structure and we use the adjacency matrix as the pre-defined structure provided by the METR-LA dataset.

STGCN [40]: STGCN integrates graph convolution and gated temporal convolution through spatialtemporal convolutional blocks. We download the source code from: https://github.com/ VeritasYin/STGCN_IJCAI-18 . We follow the recommended settings as our experimental configuration where the batch size is 50, the learning rate is 0 . 001 and the optimizer is Adam. STGCN requires a pre-defined graph structure and we leverage the adjacency matrix as the pre-defined structure provided by the METR-LA dataset.

LTSF-Linear [35]: LTSF-Linear proposes a set of embarrassingly simple one-layer linear models to

learn temporal relationships between input and output sequences. We download the source code from: https://github.com/cure-lab/LTSF-Linear . We use it as our long-term forecasting baseline and follow the recommended settings as experimental configuration.

PatchTST [39]: PatchTST proposes an effective design of Transformer-based models for time series forecasting tasks by introducing two key components: patching and channel-independent structure. We download the source code from: https://github.com/PatchTST . We use it as our long-term forecasting baseline and adhere to the recommended settings as the experimental configuration.

B.3 Implementation Details

By default, both the frequency channel and temporal learners contain one layer of FreMLP with the embedding size d of 128, and the hidden size d h is set to 256 . For short-term forecasting, the batch size is set to 32 for Solar, METR-LA, ECG, COVID-19, and Electricity datasets. And for Wiki and Traffic datasets, the batch size is set to 4. For the long-term forecasting, except for the lookback window size, we follow most of the experimental settings of LTSF-Linear [35]. The lookback window size is set to 96 which is recommended by FEDformer [30] and Autoformer [14]. In Appendix F.2, we also use 192 and 336 as the lookback window size to conduct experiments and the results demonstrate that FreTS outperforms other baselines as well. For the longer prediction lengths (e.g., 336, 720), we use the channel independence strategy and contain only the frequency temporal learner in our model. For some datasets, we carefully tune the hyperparameters including the batch size and learning rate on the validation set, and we choose the settings with the best performance. We tune the batch size over {4, 8, 16, 32}.

B.4 Visualization Settings

The Visualization Method for Global View . We follow the visualization methods in LTSFLinear [35] to visualize the weights learned in the time domain on the input (corresponding to the left side of Figure 1(a)). For the visualization of the weights learned on the frequency spectrum, we first transform the input into the frequency domain and select the real part of the input frequency spectrum to replace the original input. Then, we learn the weights and visualize them in the same manner as in the time domain. The right side of Figure 1(a) shows the weights learned on the Traffic dataset with a lookback window of 96 and a prediction length of 96, Figure 9 displays the weights learned on the Traffic dataset with a lookback window of 72 and a prediction length of 336, and Figure 10 is the weights learned on the Electricity dataset with a lookback window of 96 and a prediction length of 96.

The Visualization Method for Energy Compaction . Since the learned weights W = W r + j W i ∈ C d × d of the frequency-domain MLPs are complex numbers, we visualize the corresponding real part W r and imaginary part W i , respectively. We normalize them by the calculation of 1 / max( W ) ∗ W and visualize the normalization values. The right side of Figure 1(b) is the real part of W learned on the Traffic dataset with a lookback window of 48 and a prediction length of 192. To visualize the corresponding weights learned in the time domain, we replace the frequency spectrum of input Z temp ∈ C N × L × d with the original time domain input H t ∈ R N × L × d and perform calculations in the time domain with a weight W ∈ R d × d , as depicted in the left side of Figure 1(b).

B.5 Ablation Experimental Settings

DLinear decomposes a raw data input into a trend component and a seasonal component, and two onelayer linear layers are applied to each component. In the ablation study part, we replace the two linear layers with two different frequency-domain MLPs (corresponding to DLinear (FreMLP) in Table 4), and compare their accuracy using the same experimental settings recommended in LTSF-Linear [35]. NLinear subtracts the input by the last value of the sequence. Then, the input goes through a linear layer, and the subtracted part is added back before making the final prediction. We replace the linear layer with a frequency-domain MLP (corresponding to NLinear (FreMLP) in Table 4), and compare their accuracy using the same experimental settings recommended in LTSF-Linear [35].

C Complex Multiplication

For two complex number values Z 1 = ( a + jb ) and Z 2 = ( c + jd ) , where a and c is the real part of Z 1 and Z 2 respectively, b and d is the imaginary part of Z 1 and Z 2 respectively. Then the multiplication of Z 1 and Z 2 is calculated by:

where j 2 = -1

D Proof

D.1 Proof of Theorem 1

Theorem 1. Suppose that H is the representation of raw time series and H is the corresponding frequency components of the spectrum, then the energy of a time series in the time domain is equal to the energy of its representation in the frequency domain. Formally, we can express this with above notations by:

where H ( f ) = ∫ ∞ -∞ H ( v ) e -j 2 πfv d v , v is the time/channel dimension, f is the frequency dimension.

Proof. Given the representation of raw time series H ∈ R N × L × d , let us consider performing integration in either the N dimension (channel dimension) or the L dimension (temporal dimension), denoted as the integral over v , then

where H ∗ ( v ) is the conjugate of H ( v ) . According to IDFT, H ∗ ( v ) = ∫ ∞ -∞ H ∗ ( f ) e -j 2 πfv d f , we can obtain

Proved.

Therefore, the energy of a time series in the time domain is equal to the energy of its representation in the frequency domain.

D.2 Proof of Theorem 2

Theorem 2. Given the time series input H and its corresponding frequency domain conversion H , the operations of frequency-domain MLP on H can be represented as global convolutions on H in the time domain. This can be given by:

where ∗ is a circular convolution, W and B are the complex number weight and bias, W and B are the weight and bias in the time domain, and F is DFT .

Proof. Suppose that we conduct operations in the N (i.e., channel dimension) or L (i.e., temporal dimension) dimension, then

According to convolution theorem, H ( v ) ∗ W ( v ) = ∫ ∞ -∞ ( H ( τ ) W ( v -τ ))d τ , then

Let x = v -τ , then

Accordingly, ( H ( v ) ∗ W ( v )) in the time domain is equal to ( H ( f ) W ( f )) in the frequency domain. Therefore, the operations of FreMLP ( HW + B ) in the channel (i.e., v = N ) or temporal dimension (i.e., v = L ), are equal to the operations ( H ∗ W + B ) in the time domain. This implies that frequency-domain MLPs can be viewed as global convolutions in the time domain. Proved.

E Further Analysis

E.1 Ablation Study

In this section, we further analyze the effects of the frequency channel and temporal learners with different prediction lengths on ETTm1 and ETTh1 datasets. The results are shown in Table 6. It demonstrates that with the prediction length increasing, the frequency temporal learner shows more effective than the channel learner. Especially, when the prediction length is longer (e.g., 336, 720), the channel learner will lead to worse performance. The reason is that when the prediction lengths become longer, the model with the channel learner is likely to overfit data during training. Thus for long-term forecasting with longer prediction lengths, the channel independence strategy may be more effective, as described in PatchTST [39].

E.2 Impacts of Real/Imaginary Parts

To investigate the effects of real and imaginary parts, we conduct experiments on Exchange and ETTh1 datasets under different prediction lengths L ∈ { 96 , 192 } with the lookback window of 96 . Furthermore, we analyze the effects of W r and W i in the weights W = W r + j W i of FreMLP . In this experiment, we only use the frequency temporal learner in our model. The results are shown in

Table 7. In the table, Input real indicates that we only feed the real part of the input into the network, and Input imag indicates that we only feed the imaginary part of the input into the network. W ( W r ) denotes that we set W i to 0 and W ( W i ) denotes that we set W r to 0. From the table, we can observe that both the real part and imaginary part of input are indispensable and the real part is more important to the imaginary part, and the real part of W plays a more significant role for the model performances.

E.3 Parameter Sensitivity

We further perform extensive experiments on the ECG dataset to evaluate the sensitivity of the input length L and the embedding dimension size d . (1) Input length : We tune over the input length with the value { 6 , 12 , 18 , 24 , 30 , 36 , 42 , 50 , 60 } on the ECG dataset and the prediction length is 12, and the result is shown in Figure 6(a). From the figure, we can find that with the input length increasing, the performance first becomes better because the long input length may contain more pattern information, and then it decreases due to data redundancy or overfitting. (2) Embedding size : We choose the embedding size over the set { 32 , 64 , 128 , 256 , 512 } on the ECG dataset. The results are shown in Figure 6(b). It shows that the performance first increases and then decreases with the increase of the embedding size because a large embedding size improves the fitting ability of our FreTS but may easily lead to overfitting especially when the embedding size is too large.

(a) Input window length

(b) Embedding size

0.0545

RMSE

0.0800

MAE

0.0540

0.0795

0.0535

E

S

RM

0.0530

0.0790

MAE

E

0.0785

0.0525

0.0780

0.0520

0.0775

10

20

30

input length T

RMSE

0.05290

MAE

0.0790

0.05285

0.05280

0.0788

RMS

0.05275

E

MA

0.05270

0.0786

0.05265

0.0784

0.05260

200

400

embedding size

F Additional Results

F.1 Multi-Step Forecasting

To further evaluate the performance of our FreTS in multi-step forecasting, we conduct more experiments on METR-LA and COVID-19 datasets with the input length of 12 and the prediction lengths of {3, 6, 9, 12}, and the results are shown in Tables 8 and 9, respectively. In this experiment, we only select the state-of-the-art (i.e., GNN-based and Transformer-based) models as the baselines since they perform better than other models, such as RNN and TCN. Among these baselines, STGCN, DCRNN, and TAMP-S2GCNets require pre-defined graph structures. The results demonstrate that

FreTS outperforms other baselines, including those models with pre-defined graph structures, at all steps. This further confirms that FreTS has strong capabilities in capturing channel-wise and time-wise dependencies.

F.2 Long-Term Forecasting under Varying Lookback Window

In Table 10, we present the long-term forecasting results of our FreTS and other baselines (PatchTST [39], LTSF-linear [35], FEDformer [30], Autoformer [14], Informer [13], and Reformer [18]) under different lookback window lengths L ∈ { 96 , 192 , 336 } on the Exchange dataset. The prediction lengths are { 96 , 192 , 336 , 720 } . From the table, we can observe that our FreTS outperforms all baselines in all settings and achieves significant improvements than FEDformer [30], Autoformer [14], Informer [13], and Reformer [18]. It verifies the effectiveness of our FreTS in learning informative representation under different lookback window.

G Visualizations

G.1 Weight Visualizations for Energy Compaction

We further visualize the weights W = W r + j W i in the frequency temporal learner under different settings, including different lookback window sizes and prediction lengths, on the Traffic and Electricity datasets. The results are illustrated in Figures 7 and 8. These figures demonstrate that

the weight coefficients of the real or imaginary part exhibit energy aggregation characteristics (clear diagonal patterns) which can facilitate frequency-domain MLPs in learning the significant features.

(a)

W

r

under I/O=48/192

(b)

W

r

under I/O=48/336

(c)

W

r

under I/O=72/336

(d)

W

i

under I/O=48/192

(e)

W

i

under I/O=48/336

(f)

W

i

under I/O=72/336

20

0.3

20

0.4

20

0.3

40

0.2

40

0.2

40

0.2

0.1

0.1

60

60

0.0

60

0.0

0.0

80

80

80

0.1

0.2

0.1

100

0.2

100

100

0.2

0.4

120

0.3

120

120

0.3

20

40

60

80

100

120

D

20

40

60

80

100

120

0

20

40

60

80

100

120

0.6

20

0.6

20

0.6

20

0.4

0.4

0.4

40

40

40

0.2

0.2

0.2

0.0

60

0.0

60

60

0.2

0.0

0.2

80

0.4

80

0.2

80

0.4

100

0.6

100

0.4

100

0.6

120

-0.8

120

0.6

120

0.8

20

40

60

80

100

120

20

40

60

80

100

120

20

60

80

100

120

G.2 Weight Visualizations for Global View

To verify the characteristics of a global view of learning in the frequency domain, we perform additional experiments on the Traffic and Electricity datasets and compare the weights learned on the input in the time domain with those learned on the input frequency spectrum. The results are presented in Figures 9 and 10. The left side of the figures displays the weights learned on the input in the time domain, while the right side shows those learned on the real part of the input frequency spectrum. From the figures, we can observe that the patterns learned on the input frequency spectrum exhibit more obvious periodic patterns compared to the time domain. This is attributed to the global view characteristics of the frequency domain. Furthermore, we visualize the predictions of FreTS on

0

20

40

60

80

100

120

0

20

40

60

80

100

120

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

(a)

W

r

under I/O=96/96

0

20

40

60

80

100

120

0

20

40

60

80

100

120

1.5

1.0

0.5

0.0

0.5

1.0

(b)

W

r

under I/O=96/336

0

20

40

60

80

100

120

0

20

40

60

80

100

120

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

(c)

W

r

under I/O=96/720

0

20

40

60

80

100

120

0

20

40

60

80

100

120

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

(d)

W

i

under I/O=96/96

0

20

40

60

80

100

120

0

20

40

60

80

100

120

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

(e)

W

i

under I/O=96/336

0

20

40

60

80

100

120

0

20

40

60

80

100

120

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

(f)

W

i

under I/O=96/720

the Traffic and Electricity datasets, as depicted in Figures 11 and 12, which show that FreTS exhibit a good ability to fit cyclic patterns. In summary, these results demonstrate that FreTS has a strong capability to capture the global periodic patterns, which benefits from the global view characteristics of the frequency domain.

0

50

100

150

200

250

300

0

25

50

0.0

0.2

0.4

0.6

0.8

(a) Learned on the input

0

50

100

150

200

250

300

0

25

50

0.15

0.10

0.05

0.00

0.05

0.10

0.15

(b) Learned on the frequency spectrum

0

20

40

60

80

0

20

40

60

80

0.02

0.00

0.02

0.04

0.06

0.08

0.10

(a) Learned on the input

0

20

40

60

80

0

20

40

60

80

0.100

0.075

0.050

0.025

0.000

0.025

0.050

0.075

0.100

(b) Learned on the frequency spectrum

0

20

40

60

80

0.00

0.02

0.04

0.06

0.08

GroundTruth

Prediction

(a) I/O=48/48

0

20

40

60

80

100

120

140

0.00

0.02

0.04

0.06

0.08

GroundTruth

Prediction

(b) I/O=48/96

0

50

100

150

200

250

0.00

0.02

0.04

0.06

0.08

GroundTruth

Prediction

(c) I/O=48/192

0

50

100

150

200

250

300

350

400

0.00

0.02

0.04

0.06

0.08

GroundTruth

Prediction

(d) I/O=48/336

0

25

50

75

100

125

150

175

200

0.45

0.50

0.55

0.60

0.65

0.70

0.75

0.80

GroundTruth

Prediction

(a) I/O=96/96

0

50

100

150

200

250

300

0.40

0.45

0.50

0.55

0.60

GroundTruth

Prediction

(b) I/O=96/192

0

100

200

300

400

0.40

0.45

0.50

0.55

0.60

GroundTruth

Prediction

(c) I/O=96/336

0

100

200

300

400

500

600

700

800

0.40

0.45

0.50

0.55

0.60

0.65

GroundTruth

Prediction

(d) I/O=96/720

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents FreTS, a framework that addresses both channel-wise and time-wise dependency learning in the frequency domain for time series prediction. FreTS introduces a specially designed frequency-domain MLP structure that processes the real and imaginary parts of the frequency components interactively. The experimental results on 13 benchmarks demonstrate the superior performance of FreTS compared to other state-of-the-art methods.\n\nSTRENGTHS:\n- The two-stage framework and the frequency-domain MLP module introduced in FreTS exhibit some novelty. \n- The experimental results across 13 benchmarks show the potential of the proposed method.\n\nWEAKNESSES:\n- The authors failed to motivate their method both theoretically and in relation to prior work.  1) The authors did not clearly define and explain the terms \"point-wise mappings\" and \"information bottleneck\", not elaborate on how existing methods suffer from these issues, and not clarify how the proposed method addresses and alleviates these problems.  2) While the authors claim that their method has a \"global view\", it did not highlight how it differs from previous MLP-based and Transformer-based methods in capturing long-range dependencies. 3) The claim of the advantage of “energy compaction” is not supported by experimental evidence.\n\n&nbsp;\n- The related work section lacks logical organization and fails to provide a comprehensive summary of previous methods. \n\n&nbsp;\n- The method is not well presented. 1) For the Frequency Temporal Learner, the validity behind performing DFT along the channel dimension needs further discussion. It is more likely to be based on engineering considerations than a theoretical grounding. 2) The rationale behind the design of the proposed frequency-domain MLP is unclear regarding why it calculates the new real and imaginary components as Eq. (7).  \n\n&nbsp;\n- Additional experiments are needed to highlight the benefits of the proposed approach, e.g. the learned global periodic patterns, and the robustness towards noise. Including these experiments will strengthen the empirical evidence for the effectiveness of the proposed method.\n\n&nbsp;\n- Certain phrases or sentences lacks clarity, and need improvements. E.g.,  “stacked MLP layers”, “learning in the frequency spectrum helps capture a greater number of periodic patterns”, .etc.\n\n&nbsp;\n\nQUESTIONS:\n- Are the “Domain Conversion” and “Domain Inversion” in Fig. 2 differentiable? How is the network trained end-to-end?\n- What are the meanings of the axes in Fig. 1? Whether the results are from a time series or a dataset?\n\nLIMITATIONS:\nThe authors are encouraged to discuss the computational complexity after introducing frequency transform and inverse transform into the framework, and its performance on irregular time series.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates time series forecasting in the frequency domain. By utilizing MLPs in the frequency domain, the proposed FreTS effectively captures the patterns of time series with a global view and energy compaction. Frequency learning is applied to both inter-series and intra-series scales, allowing FreTS to capture channel-wise and time-wise dependencies. Extensive empirical experiments demonstrate the effectiveness of FreTS in both short-term and long-term forecasting tasks.\n\nSTRENGTHS:\n1) This paper redesigns MLPs in the frequency domain to effectively capture both time-wise and channel wise correlations. The use of simple MLPs ensures high efficiency and helps mitigate overfiting issues compared to existing sophisticated deep models.\n2) This paper is well-organized, with comprehensive discussion and experiments.\n\nWEAKNESSES:\n1) The authors mention that learning time series in the frequency domain has the nature of energy compaction, as the energy concentrates on the smaller part of frequency components. However, the proposed FreTS still retains all the frequency components when performing MLP in the frequency domain, thereby not fully leveraging this advantageous characteristic.\n2) Frequency domain modeling is advantageous for capturing temporal dependencies due to the inherent periodic patterns in time series. However, the suitability of frequency domain modeling for exploring channel-wise correlations requires further discussion and verification. Moreover, conducting channel-wise modeling for each time point is inefficient, and should be performed at a coarser granularity, such as segment-wise or series-wise.\n3) The dimension extension block takes an input $X_t \\in \\mathbb{R}^{N \\times L}$ and produces an output $H_t \\in \\mathbb{R}^{N \\times L \\times d}$. However, if the number of variables $N$ or the input length $L$ is large, the memory usage of $H_t$ can become very high.\n\nQUESTIONS:\n1) Short-term forecasting comparison shown in Table 1 is suggested to include N-BEATS[1] and N-HITS[2] as baselines as well.\n2) Figure 4(b) demonstrates that the parameter number of FreTS remains constant as the prediction length increases. However, the projection block described in subsection 3.1 includes learnable parameters $\\phi_2 \\in \\mathbb{R}^{d_h \\times \\tau}$ and $b_2 \\in \\mathbb{R}^\\tau$, and the number of these parameters will increase with the prediction length $\\tau$.\n3) From which learner (frequency channel or frequency temporal), and at which layer are the weights used for obtaining the visualizations?\n\nReference\n\n[1] Oreshkin B N, Carpov D, Chapados N, et al. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting[C]//International Conference on Learning Representations. 2019.\n[2] Challu C, Olivares K G, Oreshkin B N, et al. NHITS: Neural Hierarchical Interpolation for Time Series Forecasting[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(6): 6989-6997.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
In this paper, the authors investigate the problem of time-series forecasting. Since the frequency domain can preserve the information from a global view and enjoy the advantage of energy compaction, the authors propose the FreTS model, which is composed of the Frequency Channel Learner and the Frequency Temporal Learner. The authors further prove the reasonableness of the frequency-domain MLPs and evaluate the proposed idea on several datasets.

STRENGTHS:
1.	The authors investigate the time-series forecasting problem from the frequency domain and provide convincing reasons for the advantage of the frequency domain.
2.	The authors devise the frequency channel learner and frequency temporal learner, which address different challenges in a unified framework.
3.	The authors investigate the proposed method on several datasets and achieve the ideal performance.

WEAKNESSES:
1.	To capture the dependencies among channels, the authors propose the frequency channel learner, which applies the FreMLP on the variables from each timestamp, e.g. $H_t^:$. However, Since it is strange to consider $H_t^:$ as a time series, I think it might be not a good idea to employ FreMLP to capture these dependencies. In my opinion, I think the CNNs, the attention mechanism, and GNN might be good choices. It is suggested that the author should try different methods to better capture these dependencies.
2.	The authors address the frequency-domain MLP from time-series forecasting, but some time-series data might be not seasonal and contain some monotonic tendency, can the proposed method address this problem?

QUESTIONS:
Please refer weaknesses

LIMITATIONS:
Please refer weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors argue that MLP-based forecasting methods suffer from point-wise mappings and information bottlenecks and explore an interesting direction of applying MLPs in the frequency domain for time series forecasting. They further analyze the inherent characteristics of frequency-domain MLPs and propose the FreTS model via stacking frequency-domain MLPs for time series forecasting. The paper provides theoretical guarantees and extensive empirical evaluation to analyze the advantages of frequency-domain MLPs and show the superiority of FreTS, verifying the authors’ arguments and the effectiveness of frequency-domain MLPs.\n\nSTRENGTHS:\n1. The authors study an interesting neural network of frequency-domain MLPs and analyze the advantaged effectiveness of frequency-domain MLPs compared MLPs in the time domain. This potentially inspires the MTS forecasting community to pay more attention to frequency analysis.\n2. Frequency-domain MLPs are theoretically proved to be energy compacting and equivalent to global convolutions. FreTS based on frequency-domain MLPs is straightforward in architecture and theoretically sound.\n3. The experimental results are comprehensive and impressive. Extensive results on 13 datasets validate the superiority of FreTS and the effectiveness of frequency-domain MLPs for both short-term forecasting and long-term forecasting. Model analysis and efficiency analysis are also provided to investigate the advantages of FreMLP over MLP and the higher efficiency of FreTS over SOTA baselines.\n4. The visualization analysis provides quite interesting patterns and clearly shows the distinct characteristics between the time domain (MLPs) and the frequency domain (frequency-domain MLPs). Especially, the visualizations of the learned weights in the frequency domain show highly concentrated values in the diagonal, verifying the energy compacting and learning efficiency of frequency-domain MLPs.\n5. The paper is well-organized and easy to follow, and the main contributions are quite clear and solid. In my opinion, it studies an interesting topic and provides a new perspective on incorporating frequency analysis into time series analysis.\n\nWEAKNESSES:\n1. The experimental results are convincing, but only one MLP-based baseline is compared in the experiments. Additional MLP-based baselines help to verify the advantages of FreMLPs over MLPs.\n2. The superiority performance of FreTS verifies the effectiveness of (one layer) frequency-domain MLPs, but the authors did not investigate how well multi-layers of frequency-domain MLPs perform in MTS forecasting.\n\nQUESTIONS:\n1. The learned patterns on the frequency spectrum show that FreTS can capture more obvious periodic patterns. But when there are no periodic patterns in MTS data, can FreTS still perform well? That is, whether FreTS is suitable for non-periodic MTS data.\n2. Why FreTS contains only one layer of FreMLP in either frequency channel learner and frequency temporal learner? How well does FreTS work when stacking with multiple FreMLPs?\n\n===\n\nI have read the rebuttal and would like to keep my rating.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper studies time series forecasting problem under the deep learning paradigm. Authors propose a new network architecture with MLPs in the frequency domain to capture both inter-series and intra-series correlations.

STRENGTHS:
1.	The paper is well-written and easy to follow.
2.	Learning the spatio-temporal correlations in the frequency with MLPs seems reasonable and shows lightweight in the experiment part. 
3.	Experiments with diverse datasets and solid comparison methods demonstrate the effectiveness of the proposed method.

WEAKNESSES:
1.	As mentioned in the related work part, there are already several works in the frequency domain for time series forecasting. It should be clearly discussed what’s the difference of this work.
2.	Since the efficiency is a core merit of the proposed method, it would be better to analyze the theoretical computation complexity compared with other method. 
3.	While the frequency method may improve the efficiency, it is not clear whether these kind of methods would mainly focus on the low-frequency part but ignoring the high-frequency part, which is more important for time series forecasting.

QUESTIONS:
Please check the weakness

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-22