Frequency-domain MLPs are More Effective Learners in Time Series Forecasting
Authors: Kun Yi
Qi Zhang
Wei Fan
Shoujin Wang
Pengyang Wang
Hui He
Ning An
Defu Lian
Longbing Cao
Zhendong Niu
Venue: NeurIPS 2023 poster (2023)
Keywords: time series forecasting
multi-layer perceptrons
frequency domain

Abstract: Time series forecasting has played the key role in different industrial, including finance, traffic, energy, and healthcare domains. While existing literatures have designed many sophisticated architectures based on RNNs, GNNs, or Transformers, another kind of approaches based on multi-layer perceptrons (MLPs) are proposed with simple structure, low complexity, and superior performance. However, most MLP-based forecasting methods suffer from the point-wise mappings and information bottleneck, which largely hinders the forecasting performance. To overcome this problem, we explore a novel direction of applying MLPs in the frequency domain for time series forecasting. We investigate the learned patterns of frequency-domain MLPs and discover their two inherent characteristic benefiting forecasting, (i) global view: frequency spectrum makes MLPs own a complete view for signals and learn global dependencies more easily, and (ii) energy compaction: frequency-domain MLPs concentrate on smaller key part of frequency components with compact signal energy. Then, we propose FreTS, a simple yet effective architecture built upon Frequency-domain MLPs for Time Series forecasting. FreTS mainly involves two stages, (i) Domain Conversion, that transforms time-domain signals into complex numbers of frequency domain; (ii) Frequency Learning, that performs our redesigned MLPs for the learning of real and imaginary part of frequency components. The above stages operated on both inter-series and intra-series scales further contribute to channel-wise and time-wise dependency learning. Extensive experiments on 13 real-world benchmarks (including 7 benchmarks for short-term forecasting and 6 benchmarks for long-term forecasting) demonstrate our consistent superiority over state-of-the-art methods. Code is available at this repository: https://github.com/aikunyi/FreTS.

Frequency-domain MLPs are More Effective Learners in Time Series Forecasting
Kun Yi Qi Zhang", Wei Fan Shoujin Wang', Pengyang Wang", Hui He Defu Lian", Ning An², Longbing Cao Zhendong Niu
Beijing Institute of Technology. 'Tongii University, 3 University of Oxford d University of Technology Sydney, UUiiversity of Macau, USTC
"HeFei University of Technology, Maaquarie University
fyikun, hehui617, zniu} @ bit.eddu.cn cn, zhangqi ce@@cs@tnn loonjji.edu.c. weifan weifan.oxford oxford @ @eaaillacoomm pywang@unn Berang@@u.edu..n.e edu.mo, liandefu @ ustc. uutc.edu..co edu.cn, ning g g.an @ acm.org, longbing cacbbingg.aooooooo @ mq.edu.au
Abstract
Time series forecasting has played the key role in different industrial, including finance, traffic, energy, and healthcare domains. While existing literatures have de- signed many sophisticated architectures based on RNNs, GNNs, or Transformers, another kind of approaches based on multi-layer perceptrons (MLPs) are pro- posed with simple structure, low complexity, and superior performance. However, most MLP-based forecasting methods suffer from the point-wise mappings and information bottleneck, which largely hinders the forecasting performance. To overcome this problem, we explore a novel direction of applying MLPs in the frequency domain for time series forecasting. We investigate the learned patterns of frequency-domain MIPs and discover their two inherent characteristic benefiting forecasting, (i) global view: frequency spectrum makes MLPs for signals and learn global dependencies more easily, and (ii) energy compaction: trequency-domain MLPs concentrate on smaller key part of frequency components with compact signal energy. Then, we propose FreTS, simple yet effective archi- tecture built upon Frequency-domain MIPs for Time Series forecasting. FreTS mainly involves two stages, (i) Domain Conversion, that transforms time-domain signals into complex numbers frequency domain; (ii) Frequency Learning, that performs our redesigned MLPs for the learning of real and imaginary part of fre- quency components. The above stages operated on both inter-series and intra-series scales further contribute to channel-wise and time-wise dependency learning. Ex- tensive experiments on 13 real-world benchmarks (including benchmarks for short-term forecasting and 6 benchmarks for long-term forecasting) demonstrate our consistent superiority over state-of-the-art methods. Code available at this repository: htttp::/ggttaiii com/ aikunyi/FreeS
complete view
1 Introduction
Time series forecasting has been a critical role in a variety of real-world industries, such as climate condition estimation D 2) traffic state prediction 3 4, economic analysis [), etc. In the early stage, many traditional statistical forecasting methods have been proposed, such as exponential smoothing [7] and auto-regressive moving averages (ARMA) 8 Recently, the emerging devel- opment of deep learning has fostered many deep forecasting models, including Recurrent Neural Network-based methods (c.g., DeepAR (9) LSTNet (TO), Convolution Neural Network-bar meth-
ods (e.g., TCN [I]. SCINet (12), Transformer-bar methods (e.g.. Informer [13), Autoformer 14)..
and Graph Neural Network-basse methods (e.g.. MTGNN [5] StemGNN
AGCRN 17 etc.
"Corresponingg author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Left: time domain. Right: frequency domain. (b) Left: time domain. Right: frequency domain.
Figure 1: Visualizations of the learned patterns of MLPs in the time domain and the frequency domain (see Appendix [.4).. (a) global view: the patterns learned in the frequency domain exhibits more obvious global periodic patterns than the time domain; (b) energy compaction: learning in the frequency domain can identify clearer diagonal dependencies and key patterns than the time domain.
While these deep models have achieved promising forecasting performance certain scenarios, their sophisticated network architectures would usually bring up expensive computation burden in training or inference stage. Besides, the robustness of these models could be easily influenced with large amount of parameters, especially when the available training data is limited [I3] [8]
Therefore, the methods based on multi-layer perceptrons (MLPs) have been recently introduced with simple structure, low complexity, and superior forecasting performance, such N-BEATS (T9), LighTTS [20]. DLinear [2I], etc. However, these M-ssed methods rely on point-wise mappings capture temporal mappings, which cannot handle global dependencies of time series. Moreover, they would suffer from the information bottleneck with regard to the volatile and redundant local momenta of time series, which largely hinders their performance for time series forecasting.
To overcome the above problems, we explore a novel direction of applying MLPs in the frequency domain for time series forecasting. We investigate the learned patterns of frequency-domad MLPs in forecasting and have discovered their two key advantages: (i) global view: operating on spectral components acquired from series transformation, frequency-domate MLPs capture more complete view of signals, making it easier to learn global spatial/temporal dependencies. (ii) energy compaction: dommin MLPs concentrate the smaller key part of frequency components with the compact signal energy, and thus can facilitate preserving clearer patterns while filtering influence of noises. Experimentally, much more obvious global periodic patterns than the time-domain MLPs from Figure T(a) which highlights their ability recognize global signals. Also, from Figure [(b] we easily note much more clear diagonal dependency in the learned weights of frequency-domaii MLPs, compared with the more scattered dependency learned by time-domain MLPs. This illustrates the great potential of prrruency-domainn MLPs to identify most important features and key patterns while handling complicated and noisy information.
have observed that frequency-domain MLPs capture
To fully utilize these advantages, we propose FreTSS, a simple yet effective architecture Frequency- domain MLPs for Time Series forecasting. The core idea of FreTS is to learn the time series forecasting mappings in the frequency domain. Specifically, FreTS mainly involves two stages: (i) Domain Conversion: the original time-domain series signals are first transformed into frequency- domain spectrum on top of Discrete Fourier Transform (DFT) 22 where the spectrum is composed of several complex numbers as frequency components, including the real coefficients and the imaginary coefficients. (ii) Frequency Learning: given the real/imaginary coefficients, we redesign the frequency- domain MLPs originally for the complex numbers by separately considering the real mappings and imaginary mappings. The respective real/imaginary parts of output learned by two distinct MLPs are then stacked in order to recover from frequency components to the final forecasting. Also, FreTS performs above two stages on both inter-series and intra-series scales, which further contributes to the channel-wist and time-wise dependencies in the frequency domain for better forecasting performance. We conduct extensive experiments on 13 benchmarks under different settings, covering benchmarks for short-term forecasting and 6 benchmarks for long-term forecasting, which demonstrate our consistent superiority compared with state-of-the-ar methods.
2 Related Work
Forecasting in the Time Domain Traditionally, statistical methods have been proposed for forecast- ing in the time domain, including (ARMA) (8). VAR (23), and ARIMA (24). Recently, deep learning

---

based methods have been widely used in time series forecasting due to their capability of extracting
nonlinear and complex correlations 25 26]. These methods have learned the dependencies in the
time domain with RNNs (e.g., deepAR 99. LSTNet [0]] and CNNs (e.g., TCN [1I], SCINet [12). In addition, GNN-based models have been proposed with good forecasting performance because of their good abilities to model series-wise dependencies among variables in the time domain, such as TAMP-S2GCNets [4], AGCRN [7]. MTGNN [1S), and GraphWaveNet [27]. Besides, Transformer- based forecasting methods have been introduced due to their attention mechanisms for long-range
dependency modeling ability in the time domain, such as Reformer [8] and Informer 13
Forecasting in the Frequency Domain Several recent time series forecasting methods have ex- tracted knowledge of the frequency domain for forecasting 28]. Specifically, SFM (2) decomposes the hidden state of LSTM into frequencies by Discrete Fourier Transform (DFT). StemGNN [16]
performs graph convolutions based on Graph Fourier Transform (GFT) and computes series corre- lations based on Discrete Fourier Transform. Autoformer [4] replaces self-attention by proposing the auto-correlation mechanism implemented with Fast Fourier Transforms (FFT), FEDformer [30] proposes a DFT-based frequency enhanced attention, which obtains the attentive weights by the spectrums queries and keys, and calculates the weighted sum in the frequency domain. CoST [31 uses DFT to map the intermediate features frequency domain enables interactions in representa- tion. FiLM [32] utilizes Fourier analysis to preserve historical information and remove noisy signals. Unlike these efforts that leverage frequency techniques improve upon the original architecture such as Transformer and GNN, in this paper, we propose new frequency learning architecture that learns both channel-wise and time-wise dependencies in the frequency domain.
MLP-based Forecasting Models Several studies have explored the use of MLP-based networks in time series forecasting. N-BEATS (19] utilizes stacked MLP layers together with doubly residual learning to process the input data to iteratively forecast the future. DEPTS [33 applies Fourier transform to extract periods and MLPs for periodicity dependencies for univariate forecasting. LightTS [20) uses lightweight sampling-oriented MLP structures to reduce complexity and com- putation time while maintaining accuracy. N-HiTS 34 combines multi-rate input sampling and hierarchical interpolation with MLPs for univariate forecasting. LTSF-Linear [35] proposes a set of embarrassingly simple one-layer linear model learn temporal relationships between input and output sequences. These studies demonstrate the effectiveness of MLP-based networks in time series forecasting tasks, and inspire the development of our rrequencyddomain MLPs in this paper.
3 FreTS
In this section, we elaborate on our proposed novel approach, FreTS,, based on our redesigned MLPs in the frequency domain for time series forecasting. First, present the detailed frequency learning architecture of FreTS in Section which mainly includes two-fold frequency learners with domain frequencyddomain conversions. Then, we detailedly introduce our redesigned frequency domain MLPs adopted by above frequency learners in Section 3.2] Besides, we also theoretically analyze their superior nature of global view and energy compaction, aforementioned in Section
Problem Definition Let [X1, ...... Xx] E NxT stand for the regularly sampled multi-variate time series dataset with N series and T'' timestamps, where X, E N denotes the multi-variate values of N distinct series at timestamp t. We consider a time series lookback window of length-L L at timestamp & as the model input, namely X, [X,- Xi-L-2. Xx)] E R NxI, also, consider a horizon window of length-r at timestamp t as the prediction target, denoted as Y, = [X++1, X.. X E Rxx Then the time series forecasting formulation is to historical observations X, to predict future values Y, and the typical forecasting model fo parameterized by to produce forecasting results by Y, = fo(.).
3.1 Frequency Learning Architecture
The frequency learning architecture of FreTS is depicted in Figure 2 which mainly involves Domain Conversion/Inversion stages, Frequency-domait MLPs, and the corresponding two learners, i.e., the Frequency Channel Learner and the Frequency Temporal Learner. Besides, before taken to learners, we concretely apply a dimension extension block on model input to enhance the model capability. Specifically, the input lookback window X, E J is multiplied with learnable
3

---

Prequency Channel Learner
Frequency Temporal Learner
FreTS
Re.al
Prequecc domain
Red
Frequescy do man Mip
Wiled
Extension
Dimersion
Conversion
MPP
Inversion
Concent
Conversion
Domain
inversion
Domain
Input
Output
Share Lemoterrps
Stared Mchanneb
wiigh
Gorgleshurbe Inggiaayy
Imaghary Part
Trequero
Figure 2: The framework overview of FreTS: the Frequency Channel Learner focuses on modeling inter-series dependencies with frequency-domann MLPs operating on the channel dimensions; the Frequency Temporal Learner is to capture the temporal dependencies by performing frequeney- domain MLPs on the time dimensions.
weight vector da E R xd to obtain a more expressive hidden representation H, E NxIxd yielding H,- X x du to bring more semantic information, inspired by word embeddings [6].
Domain Conversion/Ivversionn The use of Fourier transform enables the decomposition of a time series signal into its constituent frequencies. This is particularly advantageous for time series analysis since benefits to identify periodic or trend patterns in the data, which are often important in forecasting tasks. As aforementioned in Figure [(a] learning in the frequency spectrum helps capture a greater number of periodic patterns. In view of this, we convert the input H into the frequency domain H by:
H((f)= = J (((v)--2mffe
H(f)
H(v) cos(2n + H(v) sin (1)
where f is the frequency variable, v is the integral variable, and is the imaginary unit, which defined the square root of -1; 1 H(c) fo) is the real part of H. and is abbreviated as ()); _ H(v) (2x fv)de is the imaginary part and is abbreviated as Im). Then we can rewrite H. in Equation as: H = Re(H) + jmm(H). Note that in FreTS we operate domain conversion on both the channel dimension and time dimension, respectively. Once completing the learning in the frequency domain, we can convert H back into the the time domain using the following inverse conversion formulation:
H(v) = H (l f (Re(H(()) + (e)
where take frequency f as the integral variable. In fact, the frequency spectrum is expressed as combination of cos and sin waves in H. with different frequencies and amplitudes inferring different periodic properties in time series signals. Thus examining the frequency spectrum can better discern the prominent frequencies and periodic patterns in time series. In the following sections, we use DomainConversioo to stand for Equation ( and Domainnnversion for Equation (2) for brevity.
Frequency Channel Learner Considering channel dependencies for time series forecasting is important because it allows the model to capture interactions and correlations between different variables, leading to a more accurate predictions. The frequency channel learner enables commu- nications between different channels; it operates each timestamp by sharing the same weights between timestamps to learn channel dependencies. Concretely, the frequency channel leamer takes H, E N Lxd as input. Given the I-th timestamp H: e Nxd, we perform the frequency channel learner by:
H Hall) DomnainConversioulc....) (H: chan
z-(()
FreMLP chan Wehhe Behany
(3)
chan
Z ()(
DomainInversiontchon)) (1) (hhm)

---

chan E C3 is the frequency components of H;:), DomsinConnvessoon and Domainlnverrion(ceen indicates such operations are performed along the channel dimension. FreMLP are frequency-domain MLPs proposed in Section 3.2 which takes Wecha = (Wehan j jWrhann E Cddd as the complex number weight matrix with W E Rddd and W. E Rddd and Behon= = (Byphn + jBahan) E Cd the biases with Behen E TRd and Bokan E Rdd. And 2:10 E &C#xe xd is the output of FreMLP, also in the frequency domain, which is conversed back to time domain Z:(0) RNxd Finally, ensemble Z)(()) of L timestamps into whole and output Z, E N
where Hom
Frequency Temporal Learner The frequency temporal learner aims to learn the temporal patterns in the frequency domain; also, constructed based on frequency-domain MLPs conducting on each channel and it shares the weights between N channels. Specifically, it takes the frequency channel learner output Z₁ E R Xxxd as input and for the n-th channel Z,
IR Lxd, we apply the
frequency temporal learner by:
(n),; DomaannConversion (temp) (Z, (1)))
temp
temp
((a)
FreMLP(Z(()) (n),
(emmp W Blemp)
(()). Domainlnversion (S(m)
where temp E C d is the corresponding frequency spectrum of z m DomnannConversoontenn and Domainnnversoon(emm) indicates the calculations are applied along the time dimension. Weemp (Wzemp- jW E Cdxd is the complex number weight matrix with Weemp E Rdxd and Weeemp e Rddd and Blomp = (Brrmp + -j E Cd are the complex number biases with Bremp Rd and 5 Btemp Rd. s(m)i E C is the output of FreMLP and is converted back the time domain as S(n):: E LLxd Finally, we incorporate all channels and output S, E xx
Projection Finally, we use the learned channel and temporal dependencies to make predictions for the future timestamps Y, E Nxx by two-layer feed forward network (FFN) with one forward step which can avoid error accumulation, formulated follows:
Y, = o (S.01 + bb))2 + +b
(5)
where S, E NxLxd is the output of the frequency temporal learner, a is the activation function, σi E [(Lrd)xd,, d2 E Rddax x are the weights, by E RRd,, b2 E RR are the biases, and dy, is the inner-layer dimension size.
3.2 Frequency-domain MLPs
As shown in Figure we elaborate our novel requenccy-doaai MLPs in FreTS that are redesigned for the complex numbers of frequency components, in order to effectively capture the time series key patterns with global view and energy compaction, as aforementioned in Section []]
Definition 1 (Frequency- -omain MLPs). Formally, for a complex number input H. E Cm given a complex number weight matrix W Cdxd and a complex number bias B E Cd, then the frequency-den domain MLPs can be formulated as:
yd o WW + +B²)
=
where yd E C the final output, denotes the f-th layer, and a is the activation function.
As both H and W are complex numbers, according to the rule of multiplication of complex numbers (details can be seen in Appendix C. we further extend the Equation (6) to:
y' o
Im()*=1)wW + B3)) + jo (Re()* ( +Im("* B,) (7)
where - W jWw, and B = B + j. According to the equation, we implement the MLPs in the frequency domain (abbreviated as FreMLPP by the separate computation of the real and imaginary parts of frequency components. Then, we stack them to form complex number to acquire the final results. The specific implementation process is shown in Figure 3
5

---

Theorem 1. Suppose that H is the representation of raw time series and H. is the corresponding frequency components of the spectrum, then the energy of a time series in the time domain is equal to the energy of its representation in the frequency domain. Formally, we can express this with above notations by:
H (HASAPSSS
(
where H(f)= H(u)e - /2 du, is the time/channel dimension, f is the frequency dimension.
We include the proof in Appendix D.I The theorem implies that if most of the energy of a time series is concentrated in a small number of frequency components, then the time series can be accurately represented using only those components. Accordingly, discarding the others would not significantly affect the signal's is energy. As shown in Figure [(b] in the frequency domain, the energy concentrates on the smaller part of frequency components, thus learning in the frequency spectrum can facilitate preserving clearer patterns.
Real part
Imaginary part
Br
Bi
W
W
W
Theorem 2. Given the time series input H and its corre- sponding frequency domain conversion H. the operations of frequency-domain MLP convolutions on H in the time domain. This can be given by:
can be represented as global
Figure 3: One layer of the frequency- domain MLPs.
HW + B = F(H +
(9)
where * a circular convolution, W and are the complex number weight and bias, W and B are the weight and bias in the time domain, and F is DFT.
The proof is shown in Appendix D.2 Therefore, the operations of FreMLP, i.e., HW + B, are equal to the operations (H W +B) in the time domain. This implies that the operations of requency-domaii MLPs can be viewed as global convolutions in the time domain.
4 Experiments
To evaluate the performance of FreTS, we conduct extensive experiments on thirteen real-world time series benchmarks, covering short-term forecasting and long-term forecasting settings to compare with corresponding state-of-the methods.
Datasets Our empirical results are performed on various domains of datasets, including traffic, energy, web, traffic, clectrocardiogram, and healtheare, etc. Specifically, for the task of short-term forecasting, we adopt Solar 2 Wiki [37], Traffic [37], Electricity |] ECG [16), METR-LA (38), and COVID-19 4 datasets, following previous forecasting literature [16. For the task of long- term forecasting, we adopt Weather [4], Exchange [10]. Traffic 14], Electricity [4], and ETT datasets [13]. following previous long time series forecasting works [13 14 30 33). We preprocess all datasets following [16] 14) and normalize them with the min-max normalization. We split the datasets into training, validation, and test sets by the ratio of 7:2:1 except for the COVID-19 datasets
with 6:2:2. More dataset details are in Appendix B.I
Baselines We compare our FreTS with the representative and state-of-the-arr models for both short- term and long-term forecasting to evaluate their effectiveness. For short-term forecasting, we compre FreTS against VAR (3). SFM (9], LSTNet [00, TCN [], GraphWaveNG [27], DeepGLO [7).
StemGNN [6). MTGNN 115. and AGCRN 17 for comparison. We also include
4. DCRNN 38 and STGCN 40), which require pre-defined graph structures, for comparison. For long-term forecasting, we include Informer [33. Autoformer [14], Reformer [18), FEDformer 30)
LTSF-Linear [35), and the more recent PatchTST [39 for comparison. Additional details about the
baselines can be found in Appendix B.2
ics uci http:://www..aa.aaaee.c..ettttta

---

Implementation Details Our model is implemented with Pytorch 1.8 [41. and all experiments are conducted on a single NVIDIA RTX 3080 10GB GPU. We take MSE (Mean Squared Error) as the loss function and report MAE (Mean Absolute Errors) and RMSE (Root Mean Squared Errors) results as the evaluation metrics. For additional implementation details, please refer to Appendix [3.
4.1 Main Results
Table 1: Short-term forecasting comparison. The best results are in bold, and the second best results are underlined, Full benchmarks of short-term forecasting are in Appendix F.T
Models
Solar
Wiki
Traffic
ECG
Electricity
COVID-19
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
VAR
0.184
0.234
0.052
0.094
0.535
1.133
0.120
0.170
0.101
0.163
0.226
0.326
SFM
0.161
0.283
0.081
0.156
0.029
0.044
0.095
0.135
0.086
0.129
0.205
0.308
L.T
0.148
(12000
0,054
0.090
0.026
0.057
0.079
0.115
0.075
0.138
0.248
0.3115
TCN
0.176
0.222
0.094
0.142
0.052
0.067
0.078
0.107
0.057
0.083
0.317
0.354
DeepGLO
0.178
0.400
0.110
0.113
0.025
0.037
0.110
0.163
0.090
0.131
0.169
0.253
Reformer
0.234
0.292
0.047
0.083
0.029
0.042
0.062
0.090
0.078
0.129
0.152
0.209
Informer
0,151
0.199
0.051
0.086
0.020
0.033
0.056
0.085
0.074
0.123
0.200
0.259
Autoformer
0.150
0.193
0.069
0.103
0.029
0.043
0.055
0,081
0.056
0.083
0.159
0.211
FEDformer
0.139
0,182
0.068
0.098
0.025
0.038
0.055
0.080
0.055
0.081
0.160
0.219
Graph
0.183
0.238
0.061
0.105
0.013
0.034
0.093
0.142
0.094
0.140
0.201
0.255
StemGNN
0.176
0.222
0.190
0.255
0.080
0.135
0,100
0.130
0.070
0.101
0.421
0,508
MIGNN
0.151
0,207
0.101
0.140
0.013
0.030
0.090
0.139
0.077
0.113
0.394
0.488
AGCRN
0.123
0.214
0.044
0.079
0.084
0.166
0.055
0.080
0.074
0.116
0.254
0,309
FreTS (Ours)
0.120
0.162
0.041
0.074
0.011
0.023
0.053
0.078
0.050
0.076
0.123
0.167
Table 2: Long-term forecasting comparison. We set the lookback window size L as 96 and the prediction length as E {96, 192, 336, 720} except for traffic dataset whose prediction length is set as T E (48, 96, 192, 336}. The best results are in bold and the second best are underlined. Full
results of long-term forecasting are included in Appendix F2
Models Metrics
FreTS
PatcTTTT
LISE-Lincar MAE
FEDromeer MAE
Auteformer MAE
Informer MAE
Reformer
MAE
RMSE
MAE
RMSE
RMSE
RMSE
RMSE
RMSE
MAE
RMSE
Weatber
96
0.032
0.071
0.034
0.074
0.040
0.081
0.050
0.088
0.064
0.104
0.101
0.139
0.108
0.152
192
0.040
0.081
0.042
0.084
0.048
0.051
0.092
0.103
0.097
0.147
0.201
336
0.046
0.090
0.049
0.094
0.056
0.098
0.057
0.100
0.059
0.101
0.115
0.155
0.154
0.203
720
0.055
0.099
0.056
0.102
0.065
0.064
0.109
0.110
0.132
0.175
0.173
0.228
Exchange
96
0.037
0,051
0.039
0.052
0.038
0.052
0.050
0.067
0.050
0.066
0,066
0.084
0.126
0,146
192
0.050
0.067
0.055
0.074
0.053
0.069
0.064
0.082
0.063
0.083
0.068
0.088
0.147
0.169
336
0.062
0.082
0.071
0.093
0.064
0.085
0.080
0.105
0.075
0.101
0.093
0.127
0.157
0.189
720
0.088
0.110
0.166
0.092
0.151
0.183
0.150
0.181
0.117
0.166
0.201
48
0.018
0.036
0.016
0.032
0.020
0,039
0.022
0.036
0.026
0.042
0,023
0.039
0.035
0,053
Traffic
96
0.020
0.038
0.018
0.035
0.022
0,042
0.023
0.044
0.033
0.050
0.030
0.047
0.035
0.054
192
0.019
0,038
0.020
0.039
0.020
0.040
0.022
0.042
0.035
0.053
0.034
0.053
0,035
0.054
336
0.020
0.039
0.021
0.040
0.021
0.041
0.021
0.040
0.032
0.050
0.035
0.054
0.035
0.055
ETTml ETThl Eleetricity
96
0.039
0.065
0,041
0.067
0.045
0.075
0.049
0.072
0.051
0.075
0.094
0.124
0.095
0.125
192
0.040
0.064
0,042
0,066
0.043
0.070
0.049
0.072
0.072
0.099
0.105
0.138
0.121
0.152
336
0.046
0.072
0.043
0.067
0.044
0.071
0.051
0.075
0.084
0.115
0,112
0.144
0.122
0.152
720
0.052
0.079
0.055
0.081
0.054
0.080
0.055
0.077
0.088
0.119
0.116
0. 148
0.120
0.151
96
0.061
0.087
0.065
0.091
0.063
0.089
0.072
0.096
0.079
0.105
0.093
0.121
0.113
0.143
192
0.065
0.091
0.069
0.094
0.067
0.094
0.076
0.100
0.086
0.114
0.103
0.137
0,120
0.148
336
0.070
0.096
0,073
0.099
0.070
0,097
0.080
0.105
0.088
0.119
0,112
0.145
0.124
0.155
720
0.082
0.108
0.087
0.113
0.082
0.108
0.090
0.116
0.102
0.136
0.125
0.157
0.126
0.155
96
0.052
0.077
0.055
0.082
0.055
0.080
0.063
0.087
0.081
0.109
0.070
0.096
0.065
0.089
192
0.057
0.083
0.059
0.085
0.060
0.087
0.068
0.093
0.083
0.112
0.082
0.107
0.081
0.108
336
0.062
0.089
0,064
0.091
0.065
0,093
0.075
0.102
0.091
0.125
0,090
0.119
0.100
0.128
720
0.069
0.0%
0.070
0.097
0.072
0.099
0.081
0.108
0.093
0.126
0.115
0.149
0.132
0.163
Short-Term Time Series Forecasting Table D presents the forecasting accuracy of our FreTS compared thirteen baselines on six datasets, with an input length of 12 and prediction length of 12. The best results are highlighted in bold and the second-best best results are underlined. From the table, we observe that FreTS outperforms all baselines on MAE and RMSE across all datasets, and on average it makes improvement of 9.4% on MAE and 1 11.6% on R.SE. We credit this to the fact that FreTS explicitly models both channel and temporal dependencies, and it flexibly unifies channel and temporal modeling in the frequency domain, which can effectively capture the key patterns with the global view and energy compaction. We further report the complete benchmarks of short-term forecasting under different steps on different datasets (including METR-LA dataset) in Appendix F.
7

---

Long-term Time Series Forecasting 2 showeases the long-term forecasting results of FreTS compared six representative baselines on six benchmarks with various prediction lengths. For the traffic dataset, we select 48 as the lookback window size I. with the prediction lengths T E {48, 96 336). For the other datasets, the input lookback window length is set to 96 and the prediction length is set to E {96, 192, 336, 720}. The results demonstrate that Fre'TSS outperforms all baselines on all datasets. Quantitatively, compared with the best results of Transformer-basee models, Fre has average decrease of more than 20% in MAE and RMSE. Compared with more recent Linr-Linear [35] and the SOTA PSt [3], FreTS can still outperform them in general. In addition, provide further comparison of Fre and other baselines and report performance under different lookback window sizes in Appendix F.2 Combining Tables a 2
we can conclude that FreTS achieves competitive performance in both short- term and long- Long-term forecasting task.
4.2 Model Analysis
Frequency Channel and Temporal Learners We analyze the effects of frequency channel and temporal learners in Table in both short-term and long-term experimental settings.
Table 3: Ablation studies of frequency channel and temporal learners in both short-term and long-term forecasting. 'I/O' indicates lookback window sizes/prediction lengths.
Tasks
Short-term
Long term
We consider two variants: FreCL:
Dataset
Electricity
METR-LA
Exchange
Weather
PO
12/12
12/12
96/336
96/336
we remove the frequency temporal learner from FreTS, and FreTL: we remove the frequency channel learner from FreTS. From the comparison, we observe that the frequency chan-
Merries
MAE
RMSE
MAR
RMSE
MAA
RMSE
MAE
RMSE
FreCL FroTL
0.054
0.080
0.086
0.168
0.067
0.086
0.051
0.094
0.058
0.086
0.085
0.167
0.065
0.085
0.047
0.091
FreTS
0.050
0.076
0.080
0.166
0.062
0.082
0,046
0.090
nel learner plays a more important role in short-term forecasting. In long-term forecasting, we note that the frequency temporal learner is more effective than the frequency channel learner. In Appendix E. we also conduct the experiments and report performance on other datasets. Interestingly, we find out the channel learner would lead to the worse performance in some long-term forecasting cases. A potential explanation is that the channel independent strategy [39 brings more benefit to forecasting.
FreMLP vs. MLP We further study the effectiveness of FreMLP in time series forecasting. We use FreMLP to replace the original MLP component in the existing SOTA MLP-based models (i.c., DLinear and NLinear (35)), and compare their performances with the original DLinear and NLinear under the same experimental settings. The experimental results are presented in Table 4 From the table, we easily observe that for any prediction length, the performance of both DLinear and NLinear models has been improved after replacing the corresponding MLP component with FrcMLP. Quantitatively, incorporating FreMLP into the DLinear model brings an average improvement of 6.4% in MAE and 11.4% in RMSE on the Exchange dataset, and 4.9% in MAE and 3.5% in RMSE on the Weather dataset. A similar improvement has also been achieved on the two datasets with regard to NLinear, according to Table These results confirm the effectiveness of FreMLP compared to MIP again and we include more implementation details and analysis in Appendix B.5
Table 4: Ablation study the Exchange and Weather datasets with lookback window size of 96 and the prediction length T E {96, 192, 336, 720). DLinear (FreMLP)/Linee) Linea (FreMLP) means that we replace the MLPs in DLinear/NL iineear with FreMLP. The best results are in bold.
Danasets
Exchange
Weather
Lengthe
316
730
335
720
Metrics
MAB
RMSE
MAB
RMSE
MAE
RMSE
MAB
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSI
MAE
RMSE
DLineer
0:037
0.051
0.054
0.072
0.071
0.085
0.095
0.119
0.41
0.081
0017
0.089
0.056
0.098
0.065
0.106
Dliiner (FreMLP)
0.036
0.049
6.053
0.071
0.063
0.871
0.086
0.101
0.038
0.078
0.645
0.086
0.055
0.097
0.861
0.100
NLinter
0,037
0.051
0.051
0.699
0.059
0.083
0.115
0.146
0.077
0.081
0015
0.089
0.052
0.098
0.088
0.106
NLinea (breMLP
0.136
80050
0,049
0J667
0.067
0
0.109
0.139
(U035
0.176
0J043
0.084
0,050
0,094
0.057
0.103
4.3 Efficiency Analysis
The complexity of our proposed FreTS is O(N log N Log L). We perform efficiency comparisons with some state-of-the-art GNN-based methods and maasformer-basedd models under different numbers of variables N and prediction lengths т, respectively. On the Wiki dataset, we conduct experiments over N e {1000, 2000, 3000, 4000, 5000} under the same lookback window size of 12
8

---

16- Mur-eerrr
Panmsrrag
150
124
234 (a) Parameters (left) and training time (right) under (b) Parameters (left) and training time (right) under different prediction lengths
1003
20.0
1006
8300
3000
2000
8006
5603
Proditionnongtio
100
125
24
different variable numbers
Figure 4: Efficiency analysis (model parameters and training time) on the Wiki and Exchange dataset. (a) The efficiency comparison under different number variables: the number of variables is enlarged from 1000 to 5000 with the input window size 12 and the prediction length as 12 on Wiki dataset. (b) The efficiency comparison under the prediction lengths: we conduct experiments with prediction lengths prolonged from 96 to 480 under the window size of 96 the Exchange dataset.
and prediction length of 12, as shown in Figure ((a) From the figure, we can find that: (1) The amount of Fre'TSS parameters is agnostic to N. (2) Compared with AGCRN, FreTTS incurs an average 30% reduction of the number of parameters and 20% reduction of training time. On the Exchange dataset, we conduct experiments on different prediction lengths
{96, 192, 336, 480} with the same input length of 96. The results are shown in Figure 4bb) It demonstrates: (1) Compared with
Transformer-basee methods (FEDformer 30), Autoformer 14), and Informer [3]), FreTS reduces the number of parameters by at least 3 times. (2) The training time of FreTS is averagely times faster than Informer, 5 times faster than Autoformer, and more than 10 times faster than FEDformer. These show our great potential in real-world deployment.
4.4 Visualization Analysis
In Figure [] we visualize the learned weights W FreMLP on the Traffic dataset with lookback window size of 48 and prediction length of 192. As the weights W are complex numbers, we pro- vide visualizations of the real part Wr (pre- sented in (a)) and the imaginary part W; (presented in (b)) separately. From the fig- ure, we can observe that both the real and imaginary parts play a crucial role in learn- ing process: the weight coefficients of the real or imaginary part exhibit energy aggre- gation characteristics (clear diagonal pat- terns) which can facilitate to learn the significant features. In Appendix E.2
(a) The real part Wr (b) The imaginary part W
Figure 5: Visualizing learned weights of FreMLP on the Traffic dataset. W, represents the real part of W, and W; represents the imaginary part.
we further conduct a detailed analysis the effects of the real and imaginary parts in different contexts of forecasting, and the effects of the two parts in the FreMLP. We examine their individual contributions and investigate how they influence the final performance. Additional visualizations of the weights on different datasets with various settings, as well as visualizations of global periodic patterns, can be found in Appendix G. and Appendix G.2 respectively.
Conclusion Remarks
In this paper, we explore a novel direction and make a new attempt to apply frequency-domat MLPs for time series forecasting. We have redesigned MLPs in the frequency domain that can effectively capture the underlying patterns of time series with global view and energy compaction. We then verify this design by simple yet effective architecture, FreTS, built upon the frequency- MLPs for time series forecasting. Our comprehensive empirical experiments short-term forecasting and six benchmarks of long-term forecasting have validated the superiority of our proposed methods. Simple MI.ss have several advantages and lay the foundation of modern deep learning, which have great potential for satisfied performance with high efficiency. We hope this work can facilitate more future research of MLPs time series modeling.
seven benchmarks of

---

Acknowledgments and Disclosure of Funding
The work was supported in part by the National Key Research and Development Program of China under Grant 2020AAA0104903 and 2019YFB 1406300, and National Natural Science Foundation of China under Grant 62072039 and 62272048.
References
[1] Edward N Lorenz. Empirical orthogonal functions and statistical weather prediction, volume 1. Massachusetts Institute Technology, Department Metcorology Cambridge, 1956.
[2] Yu Zheng, Xiuwen Yi, Ming Li, Ruiyuan Li, Zhangqing Shan, Eric Chang, and Tianrui Li. Forecasting fine-grained air quality based on big data. In KDD, pages 2267-2276, 2015.
[3] Hui He, Qi Zhang, Simeng Bai, Kun Yi, and Zhendong Niu. CATN: cross attentive tree-aware network for multivariate time series forecasting. AAAI, pages 4030-4038. AAAI Press, 2022.
[4] Yuzhou Chen, Ignacio Domninuu, Baris Coskunuzer, and Yulia Gel. TAMP-s2GCNets: Coupling time-aware multipersistence knowledge representation with spatio-supra graph con- volutional networks for time-series forecasting. In International Conference on Learning Representations, 2022.
[5] Benjamin F King. Market and industry factors in stock price behavior. the Journal of Business, 39(1):139-190, 1966.
[6] Adebiyi Ariyo, Adewumi O Adewumi, and Charles Ayo. Stock price prediction using the arima model. In 2014 UKSim-AMSS 16th international conference on computer modelling and simulation, pages 106-112. IEEE, 2014.
[7] Charles C Holt. Forecasting trends and seasonal by exponentially weighted moving averages. ONR Memorandum, 52(2), 1957.
[8] Peter Whittle. Prediction and regulation by linear least-square methods. English Universities Press, 1963.
[9] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181-118) 2020.
[10] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In SIGIR, pages 95-104, 2018.
[11] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolu- tional and recurrent networks for sequence modeling. CORR, 0bss/803.01177 2018.
[12] Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma. and Qiang Xu. Scinet: time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems, 35:5816-588 5828, 2022.
[13] Haoyi Zhou, Shanghang Zhang, Jieqi Peng. Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In AAAI, pages 11106-1111 11115, 2021.
[14] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In NeurIPS, pages 22419- 22430, 2021.
(15) Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In KDD, pages 753-763, 2020.
[16] Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. Spectral temporal graph neural network for multivariate time-series forecasting. In NewrIPS, 2020.
10

---

[17] Lei Bai, Lina Yao. Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. In NeurIPS, 2020.
[18] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2020.
[19] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv: 10437, 2019.
[20] Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li. Less more: Fast multivariate time series forecasting with light sampling-oriented oriented mlp structures. arXiv preprint arX1v:2207.01188 2022.
[21] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? arXiv preprint arXiv:2205.1335 2022.
[22] Duraisamy Sundararajan. The discrete Fourier transform: theory, algorithms and applications. World Scientific, 2001.
[23] Mark W. Watson. Vector autoregressions and cointegration. Working Paper Series, Macroeco- nomic Issues, 4, 1993.
[24] Dimitros Asteriou and Stephen G Hall. Arima models and the box-jenkins methodology. Applied Econometrics, 2(2):265-2888 2011.
[25] Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: survey. Philosoph- ical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 379(2194):2220000, feb 2021.
[26] José Torres, Dalil Hadjout, Abderrazak Sebaa, Francisco Martínez- Alvarezz, and Alicia Troncoso. Deep learning for time series forecasting: A survey. Big Data, 9, 12 2020.
[27] Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. In IJCAI, pages 1907-1913, 2019.
[28) Kun Yi, Qi Zhang, Longbing Cao, Shoujin Wang, Guodong Long, Liang Hu, Hui He, Zhendong Niu, Wei Fan, and Hui Xiong. survey on deep learning based time series analysis with frequency transformation. CoRR, abs/2302.02173, 2023.
[29] Liheng Zhang, Charu C. Aggarwal, and Guo-Jun Qi. Stock price prediction via discovering multi- frequncy trading patterns. In KDD, pages 2141-2149, 2017
[30] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang. Liang Sun, and Rong Jin. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In ICML, 2022.
[31] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven C. H. Hoi. Cost: Contrastive learning of disentangled seasonal-trens representations for time series forecasting. In ICLR. OpenReview.net, 2022.
[32] Tian Zhou, Ziqing Ma, Xue Wang, Qingsong Wen, Liang Sun, Tao Yao. Wotao Yin, and Rong Jin. Film: Frequency improved legendre memory model for long-term time series forecasting. 2022.
[33] Wei Fan, Shun Zheng, Xiaohan Yi, Wei Cao, Yanjie Fu, Jiang Bian, and Tie- Yan Liu. DEPTS: deep expansion learning for periodic time series forecasting. In ICLR. OpenReview.net, 2022.
[34] Cristian Challu, Kin G. Olivares, Boris N. Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski. N-hits: Neural hicrarchical interpolation for time series forecasting. CoRR, abs/2201. 12886, 2022.
[35] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting? 2023.

---

[36] Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In ICLR (Workshop Poster), 2013.
[37] Rajat Sen, Hsiang-Fu Yu, and Inderjit S. Dhillon. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. In NeurIPS, pages 3838-4847, 2019.
[38] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In ICLR (Poster), 2018.
[39] Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, 2023.
[40] Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: deep learning framework for traffic forecasting. In ISCAI, pages 3634-3640, 2018.
[41] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, pages 8024 8035, 2019.
12

---

A
Notations
Table 5: Notation.
multivariate time series with lookback window of at timestamps t, X₁ R.xI.
X
the multivariate values of N distinct series timestamp XG E RN X
the prediction target with horizon window of length т
Y
at timestamps I, Y, E R-NxT
H,
the hidden representation of X, H. E RNxxxd
Ze
the output of the frequency channel learner, Ze E RNNxL
S
the output of the frequency temporal learner, S, E NxLxd
Hechen
the domain conversion of He on channel dimensions, Hehan E NxLxd
ehhan
the FreMLP output of Hchans Zchan N
Z zeemp
the domain conversion of Z, temporal dimensions, Ztemp E NxLxd
Stemp
the FreMLP output of Ziemp Stemp E @NxLxd
the complex number weight matrix of FreMLP in the frequency channel learner, Wichan E Cdxd the complex number bias of FreMLP in the frequency channel learner, Behon E d
Wehan
Bchon
the complex number weight matrix of FreMLP in the frequency temporal learner, Wytemp E Cdxd the complex number bias of FreMLP in the frequency temporal learner, Blemp Cd
Witemp
Btemp
B Experimental Details
B.1 Datasets
We adopt thirteen real-world benchmarks in the experiments evaluate the accuracy of short-term and long-term forecasting, The details of the datasets are as follows:
Solar It is about the solar power collected by National Renewable Energy Laboratory. We choose the power plant data points in Florida as the data set which contains 593 points. The data is collected from 01/01/2006 to 31/12/2016 with the sampling interval of every hour.
Wiki 37 It contains number of daily views of different Wikipedia articles and is collected from 1/7/2015 to 31/12/2016. It consists of approximately 145k time series and we randomly choose 5k: from them as our experimental data set.
Traffic (37]
It contains hourly traffic data from 963 San Francisco freeway car lanes for short-term forecasting settings while it contains 862 car lanes for long-term forecasting. It is collected since 01/01/2015 with a sampling interval of every 1 hour.
E It is about Electrocardiggram(ECG) from the UCR time-series classification archive. It contains 140 nodes and each node has a length of 5000.
13

---

Electricity It contains electricity consumption of 370 clients for short-term forecasting while contains electricity consumption of 321 clients for long-term forecasting. collected since 01/01/2011. 1. The data sampling interval is every 15 minutes.
It about COVID-19 hospitalization in the U.S. state of California (CA) from 01/02/2020 to 31/12/2020 provided by the Johns Hopkins University with the sampling interval of every day.
COVID-19 A
METR-LA" It contains traffic information collected from loop detectors in the highway of Los Angeles County. It contains 207 sensors which are from 01/03/2012 to 30/06/2012 and the data sampling interval is every 5 minutes.
Exchange It contains the collection of the daily exchange rates of eight foreign countries including Australia, British, Canada, Switzerland, China, Japan, New Zealand, and Singapore ranging from 1990 to 2016 and the data sampling interval is every 1 day.
8
Weather It collects 21 meteorological indicators, such as humidity and air temperature, from the Weather Station of the Max Planck Biogeochemistry Institute in Germany in 2020. The data sampling interval is every 10 minutes.
ETT It is collected from two different electric transformers labeled with and 2, and each of them contains 2 different resolutions (15 minutes and 1 hour) denoted with and h. We use ETThl and ETTml as our long-term forecasting benchmarks.
B.2 Baselines
We adopt eighteen representative and state-of-the-art baselines for comparison including LSTM-based models, GNN-based models, and Transformer-basee models. We introduce these models as follows:
VAR 23
VAR is classic linear autoregressive model. We use the Statsmodels library (https //wWw stwdeelelsssssss.......o org) which is a Python package that provides statistical computations to realize the VAR.
DeepGLO 37: DeepGL.O models the relationships among variables by matrix factorization and employs temporal convolution neural network to introduce non-linear relationships. We download the source code from: hittps://gi thub. com wttp://cc//aaaaa//a///aaa jrajaasteen/ddeeggg
We use the recommended configuration our experimental settings for Wiki, Electricity, and Traffic datasets. For the COVID- 19 dataset, the vertical and horizontal batch size is set to 64, the rank of the global model set to 64, the number of channels is set to [32, 32, 32, 1]. and the period is set to 7.
LSTNet [O]: LSTNet uses a CNN to capture inter-variable relationships and an RNN discover com/ lai aigukunn/ LSTNet. our experiment, we use the recommended configuration where the number of CNN hidden units is 100, the kernel size of the CNN layers is 4, the dropout is 0.2, the RNN hidden units is 100, the number of RNN hidden layers 1, the learning rate is 0.001 and the optimizer' is Adam.
long-term patterns. We download the source code from: https://gitth:
TCN II
TCN is a causal convolution model for regression prediction. We download the source code from: htwpt:ttthh/eet /ttps://giiee thub. com/ locus l lab/TCN We utilize the same configuration as the polyphonic music task exampled in the open source code where the dropout is 0.25, the kernel size is 5, the number of hidden units is 150. the number of levels and the optimizer Adam.
Informer leverages an efficient self-attention mechanism to encode the dependen- cies among variables. We download the source code from: https://githi:: cwwb /houhaoyi/ Informer2020 We use the recommended configuration as the experimental settings where the dropout is 0.05, the number of encoder layers is 2, the number of decoder layers is 1, the learning rate is 0.0001, and the optimizer is Adam.
Informer 13:
Reformer 18] Reformer combines the modeling capacity of a Transformer with an architecture that can be exccuted efficiently on long sequences and with small memory use. We download the source
14

---

code from: as the experimental settings.
httpaa//oo//mmm///ttfftttooo We use the recommended configuration
Autoformer proposes a decomposition architecture by embedding the series decomposition block as inner operator, which can progressively aggregate the long-term trend part from intermediate prediction. We download the source code from: Autoformes We use the recommended configuration as the experimental settings.
Autoformer 14]
co//tuum//
FEDformer 30]: FEDformer proposes an attention mechanism with low-rank approximation in frequency and mixture of expert decomposition to control the distribution shifting. We download the source code from: https: / / gi thub. com com/ Mingiiigg///ia/iii FEDformer ormer We use FEB-f as the Frequency Enhanced Block and select the random mode with 64 as the experimental mode.
SFM 29
On the basis of the LSTT model, SFM introduces a series of different frequency compo- nents in the cell states, We download the source code from: https:/ thub. com/ 2tmm//331565566/ State-Frequency-Menoe wwwporr--sstooc-pdectttteeeeeeee
stock prediction We follow the recommended configuration as
the experimental settings where the learning rate is 0.01, the frequency dimension is 10, the hidden dimension is 10 and the optimizer is RMSProp.
StemGNN [T]] StemGNN leverages GFT and DFT to capture dependencies among variables in the frequency domain. We download the source code from: ltttps://github. com/micc/eie crosooft/ StemGNN We use the recommended configuration of stemGNN as our experiment setting where the optimizer RMSProp, the learning rate is 0.0001, the number of stacked layers 5, and the dropout rate is 0.5.
MTGNN 15: MTGNN proposes effective method exploit the inherent dependency relation- ships among multiple time series. We download the source code from: hhttp:://ii com/ nnzhan/MTGNN Because the experimental datasets have no static features, we set the parameter load_static_feall to false. We construct the graph by the adaptive adjacency matrix and add the graph convolution layer. Regarding other parameters, we follow the recommended settings.
introduces an adaptive dependency matrix learning to cap- ture the hidden spatial dependency. We download the source code from: https://githu: com/ Serrhan/Graph-Weee NaveNet) Since our datasets have no prior defined graph structures, we use only adaptive adjacent matrix. We add graph convolutional layer and randomly initialize the adjacent matrix. We adopt the recommended setting as its experimental configuration where the learning rate 0.001, the dropout is 0.3, the number of epochs is 50, and the optimizer is Adam.
GraphWaveNer 277]: Graph
AGCRN 17:: AGCRN proposes a data-adaptive graph generation module for discovering spatial correlations from data. We download the source code from: stttps///hh //tgttubtbbbttt// cwm/ Boom/LeBBI//AGe BAGG//GCNN We follow the recommended settings where the embedding dimension is 10, the learning rate is 0.003, and the optimizer is Adam.
TAMP-S2GCNets (4: WAMP-S2GCNets explores the utility of MP to enhance knowledge represen- tation mechanisms within the time-aware DL paradigm. We download the source code from: https AABGn eeBBGneejEVVYRvviili LOA0sNaadl=00 LL
//wwbodropbox. com/ sh/
TAMP.
S2GCNets require a pre-defined graph topology and we use the California State topology provided by the source code as input. We adopt the recommended settings as the experimental configuration for COVID-19.
DCRNN 3 DCRNN uses bidirectional graph random walk to model spatial dependency and recurrent neural network to capture the temporal dynamics. We download the source code from: https://gi thub. com/ htt///lyaguanggn /DCRNN We use the recommended configuration as our experi- mental settings with the batch size is 64, the learning rate is 0.01, the input dimension is 2 and the optimizer Adam. DCRNN requires a pre-defined graph structure and we use the adjacency matrix as the pre-defined structure provided by the METR-LA dataset.
STGCN 40) STGCN integrates graph convolution and gated temporal convolution through spatial- temporal convolutional blocks. We download the source code from: stttp:://iithu thub. com/ VeritasYin/STGon IJCAI-18 We follow the recommended settings as our experimental config- uration where the batch size is 50, the learning rate is 0.001 and the optimizer is Adam. STGCN requires a pre-defined graph structure and we leverage the adjacency matrix as the pre-defined structure provided by the METR-LA dataset.
LTSF-Linear [35] LTSF-Linear proposes set of embarrassingly simple one-layer linear models
15

---

learn temporal relationships between imput and output sequences. We download the source code from: We use as our long-term forecasting baseline
https://gat gi thub. com 1atre-laaaLLT----L- and follow the recommended settings as experimental configuration.
PatchTST [9 PatchTST proposes an effective design of Transformerrbasee models for time series forecasting tasks by introducing two key components: patching and
structure. com/PatchTST] We use our long-term
We download the source code from: https:// forecasting baseline and adhere to the recommended settings as the experimental configuration.
B.3 Implementation Details
By default, both the frequency channel and temporal learners contain one layer of FreMLP with the embedding size of 128, and the hidden size d,, is set to 256. For short-term forecasting, the batch size is set to 32 for Solar, METR-LA LA. ECG, COVID-19, 19, and Electricity datasets. And for Wiki and Traffic datasets, the batch size is set to 4. For the long-term forecasting, except for the lookback window size, follow most of the experimental settings of LTSF-Linear [35]. The lookback window size is set to 96 which is recommended by FEDformer 30 and Autoformer [14). In Appendix F2 we also use 192 and 336 the lookback window size to conduct experiments and the results demonstrate that Fre'TS outperforms other baselines as well. For the longer prediction lengths (e.g.. 336, 720), we use the channel independence strategy and contain only the frequency temporal learner in our model. For some datasets, we carefully tune the hyperparameters including the batch size and learning rate the validation set, and choose the settings with the best performance. We tune the batch size over (4, 8, 16, 32).
B.4 Visualization Settings
The Visualization Method for Global View. We follow the visualization methods in LTSF- Linear (35] to visualize the weights learned in the time domain on the input (corresponding to the left side of Figure a)). For the visualization of the weights learned the frequency spectrum, we first transform the input into the frequency domain and select the real part of the input frequency spectrum replace the original input. Then, we learn the weights and visualize them in the same manner as in the time domain. The right side of Figure a) shows the weights learned on the Traffic dataset with lookback window of 96 and a prediction length of 96, Figure 9 displays the weights learned on the Traffic dataset with a lookback window of 72 and a prediction length of 336, and Figure T0 is the weights learned on the Electricity dataset with lookback window of 96 and prediction length of 96.
The Visualization Method for Energy Compaction. Since the learned weights W W, + jW; E Cdxd of the frequency-domain MLPs are complex numbers, we visualize the corresponding real part W, and imaginary part W., respectively. We normalize them by the calculation of 1/ max(W) and visualize the normalization values. The right side of Figure [[b) is the real part of W learned on the Traffic dataset with lookback window of 48 and a prediction length of 192. To visualize the corresponding weights learned in the time domain, we replace the frequency spectrum of input Ztemp C xxxxx with the original time domain input H, R Nx Lxd and perform calculations in the time domain with weight W e Edxd as depicted in the left side of Figure b))
W
b).
B.5 Ablation Experimental Settings
DLinear decomposes raw data input into a trend component and seasonal component, and two one- layer linear layers are applied to each component. In the ablation study part, we replace the two linear layers with two different frequency-domate domain MLPs (corresponding to DLinear (FreMLP) in Table 4) and compare their accuracy using the same experimental settings recommended in LTSF-Linear [35]. NLinear subtracts the input by the last value of the sequence. Then, the input goes through linear layer, and the subtracted part is added back before making the final prediction. We replace the linear layer with a frequency-dome their accuracy using the same experimental settings recommended in LTSF-Linear [35]
MIPP (corresponding to NLinear (FreMI.P) in Table 4), and compare
16

---

C Complex Multiplication
For two complex number values ž (a + jb) and 22 = (c + ja), where a and is the real part of 2 and Z2 respectively, b and d is the imaginary part of 2 and Z2 respectively, Then the multiplication of 21 and Z2 calculated by:
2 = (a + jb] + jd) = ac + jd + jad + jbc = (ac - bd) + j(ad + bc)
(10)
where , -1.
D Proof
D.I Proof of Theorem
Theorem 1. Suppose that H is the representation of raw time series and H. the corresponding frequency components of the spectrum, then the energy of a time series in the time domain is equal to the energy of its representation in the frequeney domain. Formally, we can express this with above notations by:
Ho
df
(11)
where H() = - H(v)e- f2 fo v is the time/channel dimension, f is the frequency dimension.
Proof. Given the representation of time series H NxLxd let consider performing integration in either the dimension (channel dimension) or the L dimension (temporal dimension), denoted as the integral over v, then
H(u)
H(v)
where H' is the conjugate of H(U). According to IDFT, H 1 H (' R2f can obtain
(f
H())
H(v)
H" (f)e -Rf
H (f)l
H(u)e (2x)
Proved.
Therefore, the energy of time series in the time domain is equal the energy of its representation in the frequency domain.
D.2 Proof of Theorem 2
Theorem 2. Given the time series input H and its corresponding frequency domain conversion H. the operations of frequency-domatt MLP on H. can be represented as global convolutions on H in the time domain. This can be given by:
HW + B
(12)
where a circular convolution, W and are the complex number weight and bias, W and B are the weight and bias in the time domain, and F is DFT.
17

---

Proof. Suppose that we conduct operations in the N (i.e., channel dimension) or L (i.e., temporal dimension) dimension, then
F(H(O) W())) / (H(v) W(u)) -
de
According to convolution theorem, H(v))W(v)) + = - (e then
F(H(O) W
(H(r) -)) f2= "
W(U =)e 22 du
Let v ---, then
F(H(U) * W()))
- dzH()))
W(r)e j2mfa -j2ST dz)))d
S W(x)e -2a dr
H(r)e- -2ntttt
H(NW()
Accordingly, (v) (w)) in the time domain is equal to (H(f)W(f)) in the frequency domain. Therefore, the operations of FreMLP (HW + B) in the channel (i.e.. v = N) or temporal dimension (i.e., L). are equal the operations (H W + B) in the time domain. This implies that domminn MLPs can be viewed as global convolutions in the time domain. Proved.
E Further Analysis
E.1
Ablation Study
In this section, we further analyze the effects of the frequency channel and temporal learners with different prediction lengths on ETTml and ETTh1 datasets. The results are shown in Table O demonstrates that with the prediction length increasing, the frequency temporal learner shows more effective than the channel learner. Especially, when the prediction length is longer (e.g., 336, 720), the channel learner will lead to worse performance. The reason is that when the prediction lengths become longer, the model with the channel learner is likely to overfit data during training. Thus for long-term forecasting with longer prediction lengths, the channel independence strategy may be more effective, as described in PatchTST 39
Table 6: Ablation studies of the frequency channel and temporal learners in long-term forecasting. '1/O' indicates lookback window sizes/prediction lengths.
Daiaset
ETTml
ETTbl
MD
96/96
96/192
96/336
96/720
96/96
9K/192
9K/336
96/720
Metrics
MAE
RMSE
MAE
RMSE
MAB
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
FreCL
0.053
0.078
0.059
0.085
0.067
0.095
0.097
0.125
0.063
0.089
0.067
0.093
0.071
0.097
0.087
0,115
0.053
0.091
FreTS
0.052
0.077
0.057
0
0.061
0.092
0.071
0.099
0.063
0.089
0.066
0.092
0.072
0.098
0.086
0.113
E.2 Impacts of Real//maginary Parts
To investigate the effects of real and imaginary parts, we conduct experiments on Exchange and ETTh datasets under different prediction lengths L E (96, 192) with the lookback window of 96. Furthermore, we analyze the effects of W, and W; in the weights W WW jW; of FreMI.P. In this experiment, only use the frequency temporal learner in our model. The results are shown in
18

---

Table 7 In the table, Inputreat indicates that we only feed the real part of the input into the network, and Inputinag indicates that we only feed the imaginary part of the imput into the network. w(W,) denotes that we set W; to O and W(W)) denotes that we set W, to O. From the table, we can observe that both the real part and imaginary part of input are indispensable and the real part is more important to the imaginary part, and the real part of W plays a more significant role for the model performances.
Table 7: Investigation the impacts of real/imaainary parts
Dataset
Exchange
ETTh1
I/O
96/96
96/192
96/96
96/192
Metrics
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
Inputreat
0.048
0.062
0.058
0.074
0.080
0.111
0.083
0.113
Inputimag
0.143
0.185
0.143
0.184
0.130
0.156
0.130
0.156
W(W,)
0.039
0.053
0.051
0.067
0.063
0.089
0.067
0.093
W(W)
0.143
0.184
0.142
0.184
0.116
0.138
0.117
0.139
FreTS
0.037
0.051
0.050
0.067
0.061
0.087
0.065
0.091
E.3 Parameter Sensitivity
We further perform extensive experiments the ECG dataset evaluate the sensitivity of the input length I. and the embedding dimension size d. (1) Input length: We tune over the input length with the value {6, 12, 18, 24, 30, 36, 42, 50, 60} on the ECG dataset and the prediction length is 12, and the result is shown in Figure 6 a). From the figure, we can find that with the input length increasing, the performance first becomes better because the long input length may contain more pattern information, and then it decreases due to data redundancy or overfitting. (2) Embedding size: We choose the embedding size over the set {32, 64, 128, 256, 512) on the ECG dataset. The results are shown in Figure b)). It shows that the performance first increases and then decreases with the increase of the embedding size because a large embedding size improves the fitting ability of our FreTS but may easily lead to overfitting especially when the embedding size is too large.
0,0800
1981
0.05290
0.0540
0.0795
0.0790
0.0535
0.05280
BMSF
0.0790
PASE
0.05.0
0,05275
0.0525
0.0785
0.0786
0.05270
0.0780
0.05265
0.0520
0.0/84
10
20
30
200
400
inpeut length
wnibedding wioe
(a) Input window length
(b) Embedding size
Figure 6: The parameter sensitivity analyses of FreTT.
F Additional Results
F1 Multi-Step Forecasting
To further evaluate the performance of our FreTS in multi-step forecasting, we conduct more experiments on METR-LA and COVID-19 19 datasets with the input length of 12 and the prediction lengths of [3, 6, 9, 12), and the results are shown Tables annd respectively. In this experiment, we only select the sate-oof-the-are (i.e., GNN-based and Transformer-based) models as the baselines since they perform better than other models, such as RNN and TCN. Among these baselines, STGCN, DCRNN, and SMM-e2s require pre-defined graph structures. The results demonstrate that
19

---

FreTS outperforms other baselines, including those models with pre-defined graph structures, at all steps. This further confirms that FreTS has strong capabilities in capturing channel-wise and time-wise dependencies.
Table 8: Multi-step short-term forecasting results comparison on the METR-LA dataset with the input length of 12 and the prediction length of 7 E {3, 6, 9, 12). We highlight the best results in bold and the second best results are underline.
Length
6
9
12
Metries
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
Reformer
0.086
0.154
0.097
0.176
0.107
0.193
0.118
0.206
Informer
0.082
0.156
0.094
0.176
0.108
0.193
0.125
0.214
Autoformer
0.087
0.149
0.091
0.162
0.106
0.178
0.099
0.184
FEDformer
0.064
0.127
0.073
0.145
0.079
0.160
0.086
0.175
DCRNN
0.160
0.204
0.191
0.243
0.216
0.269
0.241
0.291
STGCN
0.058
0.133
0.080
0.177
0.102
0.209
0.128
0.238
Graph WaveNet
0.180
0.366
0.184
0.375
0.196
0.382
0.202
0,386
MTGNN
0.135
0.294
0.144
0.307
0.149
0.328
0.153
0.316
StemGNN
0.052
0.115
0.069
0.141
0.080
0.162
0.093
0.175
AGCRN
0.062
0.131
0.086
0.165
0.099
0.188
0.109
0.204
FreTS
0.050
0.113
0.066
0.140
0.076
0.158
0.080
0.166
Table 9: Multi-step short-term forecasting results comparison on the COVID-19 19 dataset with the input length of 12 and the prediction length of 7 E {3,6,9, 12]. We highlight the best results in bold and the second best results are underline.
Length
12
Metrics
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
Reformer
0.212
0.282
0.139
0.186
0.148
0.197
0.152
0.209
Informer
0,234
0.312
0.190
0.245
0.184
0.242
0,200
0.259
Autoformer
0.212
0.280
0.144
0.191
0.152
0.201
0.159
0.211
FEDformer
0.246
0.328
0.169
0.242
0.175
0.247
0.160
0.219
GaveeWaveNet
0.092
0.129
0.133
0.179
0.171
0.225
0.201
0.255
StemGNN
0,247
0.318
0.344
0.429
0.359
0.442
0.421
0.508
AGCRN
0.130
0.172
0.171
0.218
0.224
0.277
0.254
0.309
MTGNN
0.276
0.379
0.446
0.513
0.484
0.548
0.394
0.488
TAMP-S2GCNets
0.140
0.190
0.150
0.200
0.170
0.230
0.180
0.230
FreTS
0.071
0.103
0.093
0.131
0.109
0.148
0.124
0.164
F.2
Long-Term Forecasting under Varying Lookback Window
In Table 10 we present the long-term forecasting results of our FreTS and other baselines
(PatchTST 39 LTSF-linear (35) FEDformer (30], Autoformer (14) Informer [13], and Re-
former (88) under different lookback window lengths T. E (96, 192, 336} on the Exchange dataset. The prediction lengths are {96, 192, 336, 720). From the table, we can observe that our FreTS outperforms all baselines in all settings and achieves significant improvements than FEDformer (30), Autoformer [14] Informer [ and Reformer [18]. It verifies the effectiveness of FreTS in leariing informative representation under different lookback window.
G Visualizations
G.1
Weight Visualizations for Energy Compaction
We further visualize the weights W = W, + ji, in the frequency temporal learner under different settings, including different lookback window sizes and prediction lengths, on the Traffic and Electricity datasets. The results are illustrated in Figures and g These figures demonstrate that
20

---

Table 10: Long-tern forecasting results comparison with different lookback window lengths L {96, 192, 336]. The prediction lengths are as T E {96, 192, 336, 720). The best results are in bold and the second best results underlined.
Models Metrics
FreTS
PatchTST
LTSF-Linear RMSE
FEDformer RMSE
Autoformer
Informer MAE
Reformer
MAE
RMSE
MAE
MAE
RMSE
RMSE
RMSE
96
0.037
0,051
0.039
0.052
0.038
0.052
0.050
0.067
0.050
0.066
0.066
0.084
0.126
0.146
192
0.050
0.067
0.055
0.074
0.053
0.069
0.064
0.082
0.063
0.083
0.068
0.088
0.147
0.169
336
0,062
0.082
0.071
0.093
0.064
0.085
0,080
0,105
0.075
0.101
0.093
0.127
0.157
0.189
0.088
0.110
0.132
0.166
0.092
0.116
0.151
0.183
0.150
0.181
0.117
0.170
0.166
0.201
96
0.036
0,050
0.037
0.051
0.038
0.051
0.067
0,086
0.066
0.085
0.109
0.131
0.123
0.143
192
192
0.051
0.068
0.052
0.070
0.053
0.070
0
0.101
0.080
0.102
0.144
0.172
0.139
0.161
336
0,066
0.087
0.072
0.097
0.073
0.0%
0,093
0.122
0.099
0.129
0.141
0.177
0.155
0.181
720
0.088
0.110
0.099
0.128
0.098
0.1
0.190
0,222
0.191
0.224
0.173
0.210
0.159
0.193
0.038
0.052
0.039
0.053
0.040
0.055
0.088
0.113
0.068
0.1
0.137
0.169
0.128
0.148
336
192
0.053
0.070
0.055
0.071
0.055
0.072
0.103
0.133
0.104
0.133
0.161
0.195
0.138
0.159
336
0.071
0.092
0,074
0.099
0.077
100
0.123
0.155
0.127
0.159
0.156
0.193
0.156
0.179
720
0.082
0.108
0,100
0.129
0.087
0.110
0.210
0,242
0.211
0.244
0.173
0.210
0.168
0.205
the weight coefficients of the real or imaginary part exhibit energy aggregation characteristics (clear diagonal patterns) which can facilitate frequency-domain MLPs in learning the significant features.
84
42
40
C1
50
02
130
30
(a) Wr under 1/0=48/192
(b) W, under 1/0-48/336
(c) W, under 1/0=72/336
16
(d) W: under 1/0=48/192
(c) W: under 1/0=48/336
(f) W: under 1/0=72/336
Figure 7: The visualizations of the weights W in the frequency temporal learner on the Traffic dataset. 'I/O' denotes lookback window sizes/prediction lengths. W, and W. are the real and imaginary parts of W. respectively.
G.2 Weight Visualizations for Global View
To verify the characteristics of global view of learning in the frequency domain, we perform additional experiments on the Traffic and Electricity datasets and compare the weights learned on the input in the time domain with those learned on the input frequency spectrum. The results are presented in Figures and 10 The left side of the figures displays the weights learned on the input the time domain, while the right side shows those learned on the real part of the input frequency spectrum. From the figures, we can observe that the patterns learned on the input frequency spectrum exhibit more obvious periodic patterns compared to the time domain. This attributed to the global view characteristies of the frequency domain. Furthermore, visualize the predictions of Fre on
21

---

20
20
30
64-
50
90
0.5
166
(a) W, under 1/0-96/96
(b) W, under 1/0-96/336
(c) W, under 1/0-96/720
450
830
42
.22
382
153
-01
329
131
91
724
190
139
(d) W; under 1/0-6//96
(e) W/ under 1/0=96/336
(f) W; under 1/=-6//200
Figure 8: The visualizations of the weights W in the frequency temporal learner the Electricity dataset. '1/O' denotes lookback window sizes/prediction lengths.
W, and W; are the real and
imaginary parts of W, respectively.
the Traffic and Electricity datasets, as depicted in Figures) 11 and 12 which show that FreTS exhibit good ability to fit cyclic patterns. In summary, these results demonstrate that FreTS has strong capability to capture the global periodic patterns, which benefits from the global view characteristics of the frequency domain.
0.15
0.05
0.4
25
100
150
200
250
300
0.2
50
100
150
200
250
3.00
-0.6
0.30
0.15
(a) Learned on the imput
(b) Learned on the frequency spectrum
Figure 9: Visualization the weights (L x 7) on the Traffic dataset with lookback window size of 72 and prediction length of 336.
22

---

0.075
20
0.08
20
0.050
0.06
0.025
0.02
0.050
0.00
-0075
-0.00
(a) Learned on the input
(b) Learned on the frequency spectrum
Figure 10: Visualization of the weights (L x 7) on the Electricity dataset with lookback window size of 96 and prediction length 96.
Geuundrr. Prekitte
0:22
Preciter
0.34
0.24
0.02
0.02
60
100
(a) 1/O=48/48
(b) 1/O=48/96
Greandrrch
0.00
0.76
0.56
0.24
0.50
0.50
120
(c) 1/0=48/192
(d) 1/0346/336
Figure 11: Visualizations of predictions (forecast vs. actual) on the Traffic dataset. 'I/O' denotes lookback window sizes/prediction lengths.
23

---

0.80
Ground Frcch
GroundTruth
Predittion
0.52
0.75
0.65
0.55
0.59
75
102
150
175
55
120
1/0=96/96
(b) V/=96/192
0.65
Grourdlachh
0.50
0.00
0.50
0.40
0.40
(c) 1/0=96/336
(d) 1/O=96/720
Figure 12: Visualizations of predictions (forecast vs. actual) on the Electricity dataset. '1/O' denotes lookback window sizes/prediction lengths.
24


Reviews:
[{"review_id": "BZqRfDJnDU", "reviewer": "Reviewer_WVuT", "summary": "The paper studies time series forecasting problem under the deep learning paradigm. Authors propose a new network architecture with MLPs in the frequency domain to capture both inter-series and intra-series correlations. ", "questions": "Please check the weakness", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper is well-written and easy to follow.\n2.\tLearning the spatio-temporal correlations in the frequency with MLPs seems reasonable and shows lightweight in the experiment part. \n3.\tExperiments with diverse datasets and solid comparison methods demonstrate the effectiveness of the proposed method.\n", "weaknesses": "1.\tAs mentioned in the related work part, there are already several works in the frequency domain for time series forecasting. It should be clearly discussed what’s the difference of this work.\n2.\tSince the efficiency is a core merit of the proposed method, it would be better to analyze the theoretical computation complexity compared with other method. \n3.\tWhile the frequency method may improve the efficiency, it is not clear whether these kind of methods would mainly focus on the low-frequency part but ignoring the high-frequency part, which is more important for time series forecasting.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review and the positive comments regarding our paper. We would like to respond to your comments as follows.\n\n**W1. As mentioned in the related work part, there are already several works in the frequency domain for time series forecasting. It should be clearly discussed what’s the difference of this work.**\n\nA1. Thank you for your suggestion. In the Related Work section, we have presented various existing frequency-based models that incorporate frequency techniques into neural networks. These models include SFM, which combines DFT with LSTM; StemGNN, which incorporates DFT with GNN; and Autoformer and FEDformer, which use DFT with self-attention in the Transformer architecture. These models are considered **frequency-enhanced architectures** as they leverage frequency techniques to improve upon the original architecture, such as Transformer. However, our work differs from these approaches since we propose a **frequency learning architecture** that learns channel-wise and time-wise dependencies in the frequency domain through a novel frequency-domain MLP network.\n\nTo clearly distinguish our work from the previous frequency-based models, we will clarify the comprehensive discussion highlighting the key differences and distinctions in the Related Work section.\n\n\n**W2. Since the efficiency is a core merit of the proposed method, it would be better to analyze the theoretical computation complexity compared with other method.**\n\nA2. In the following table, we analyze the theoretical complexity compared with other representative SOTA models:\n\n|Long-term| |Short-term| |\n|:---|:---|:---|:---|\n|Model|Complexity|Model|Complexity|\n|FEDformer|$\\mathcal{O}(L)$ | AGCRN| $\\mathcal{O}(N^2)$|\n|Autoformer|$\\mathcal{O}(L\\operatorname{log}L)$|MTGNN|$\\mathcal{O}(N^2)$|\n|PatchTST| $\\mathcal{O}((L/P)^2)$| StemGNN| $\\mathcal{O}(N^3)$|\n|||FreTS| $\\mathcal{O}(N\\operatorname{log}N+L\\operatorname{log}L)$ | \n\nFrom the table, we can see that our complexity is log-linear. We will add the theoretical computation complexity to the final version.\n\n**W3. While the frequency method may improve the efficiency, it is not clear whether these kind of methods would mainly focus on the low-frequency part but ignoring the high-frequency part, which is more important for time series forecasting.**\n\nA3. Note that, **after frequency transformation, we do not discard any frequency components nor specifically focus on the low-frequency components but ignoring the high-frequency components**. Instead, we preserve the entire frequency components and feed them into the frequency channel/temporal learners. Note that all components, including high-frequency ones or low-frequency ones, are adaptively highlighted or downweighed specific to different data characteristics.\n\n1. The core operation of our FreTS is the frequency-domain MLP (FreMLP) that performs multiplications in the frequency domain and is equal to global convolutions in the time domain (refer to **Appendix D**). The efficiency of our FreMLP arises from two primary factors:\n- First, frequency transformation often leads to a sparse frequency spectrum specific to data characteristics, resulting in a significant portion of the frequency components, including both low-frequency and high-frequency ones, approaching zero. This enables FreTS to adaptively ignore negligible components and highlight significant components based on the characteristics of the data.\n- Second, according to the convolution theorem [1], the Fourier transform of a convolution between two sequences is equal to the pointwise multiplication of their respective Fourier transforms. This theorem enables us to efficiently conduct convolutions in the frequency domain, leading to improved computational efficiency.  \n\nIn summary, both the sparse frequency spectrum and the point-wise product in the frequency domain contribute to the efficient FreTS (FreMLP) model.\n\n2. In addition to efficiency, our FreTS model also benefits from frequency techniques to improve forecasting effectiveness.\n- First, the calculation of a frequency spectrum involves summing all signals across time, resulting in each spectrum element in the frequency domain attending to all timestamps in the time domain. This characteristic indicates that a spectrum provides a global view of the entire sequence of time series, which is advantageous for capturing global patterns, such as global periodic patterns, which are crucial for effective time series forecasting. This is demonstrated in **Figures 1(a), 9, 10, 11, and 12**.\n- Furthermore, frequency transformation also exhibits a characteristic of *energy compaction*, whereby the essential features of signals are typically represented by a subset of frequency coefficients that have significantly smaller magnitudes compared to the original signal. This characteristic helps reduce data redundancy and facilitates identifying more important features and clearer patterns (as shown in **Figures 1(b), 5, 7, and 8**).\n\nReference:\n\n[1]. S. S. Soliman and MD Srinath. Continuous and discrete signals and systems. Prentice Hall, (1990)\n\nWe'll clarify the above in the final version and hope we have addressed your comments. "}, {"review_id": "Xnf9kF8YDk", "reviewer": "Reviewer_aNLj", "summary": "The authors argue that MLP-based forecasting methods suffer from point-wise mappings and information bottlenecks and explore an interesting direction of applying MLPs in the frequency domain for time series forecasting. They further analyze the inherent characteristics of frequency-domain MLPs and propose the FreTS model via stacking frequency-domain MLPs for time series forecasting. The paper provides theoretical guarantees and extensive empirical evaluation to analyze the advantages of frequency-domain MLPs and show the superiority of FreTS, verifying the authors’ arguments and the effectiveness of frequency-domain MLPs.", "questions": "1. The learned patterns on the frequency spectrum show that FreTS can capture more obvious periodic patterns. But when there are no periodic patterns in MTS data, can FreTS still perform well? That is, whether FreTS is suitable for non-periodic MTS data.\n2. Why FreTS contains only one layer of FreMLP in either frequency channel learner and frequency temporal learner? How well does FreTS work when stacking with multiple FreMLPs?\n\n===\n\nI have read the rebuttal and would like to keep my rating.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors study an interesting neural network of frequency-domain MLPs and analyze the advantaged effectiveness of frequency-domain MLPs compared MLPs in the time domain. This potentially inspires the MTS forecasting community to pay more attention to frequency analysis.\n2. Frequency-domain MLPs are theoretically proved to be energy compacting and equivalent to global convolutions. FreTS based on frequency-domain MLPs is straightforward in architecture and theoretically sound.\n3. The experimental results are comprehensive and impressive. Extensive results on 13 datasets validate the superiority of FreTS and the effectiveness of frequency-domain MLPs for both short-term forecasting and long-term forecasting. Model analysis and efficiency analysis are also provided to investigate the advantages of FreMLP over MLP and the higher efficiency of FreTS over SOTA baselines.\n4. The visualization analysis provides quite interesting patterns and clearly shows the distinct characteristics between the time domain (MLPs) and the frequency domain (frequency-domain MLPs). Especially, the visualizations of the learned weights in the frequency domain show highly concentrated values in the diagonal, verifying the energy compacting and learning efficiency of frequency-domain MLPs.\n5. The paper is well-organized and easy to follow, and the main contributions are quite clear and solid. In my opinion, it studies an interesting topic and provides a new perspective on incorporating frequency analysis into time series analysis.", "weaknesses": "1. The experimental results are convincing, but only one MLP-based baseline is compared in the experiments. Additional MLP-based baselines help to verify the advantages of FreMLPs over MLPs.\n2. The superiority performance of FreTS verifies the effectiveness of (one layer) frequency-domain MLPs, but the authors did not investigate how well multi-layers of frequency-domain MLPs perform in MTS forecasting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your constructive comments and suggestions. We provide a point-by-point response to your comments below.\n\n**W1. The experimental results are convincing, but only one MLP-based baseline is compared in the experiments. Additional MLP-based baselines help to verify the advantages of FreMLPs over MLPs.**\n\nA1. Thank you for your suggestion. We have incorporated additional experiments to compare FreTS with two state-of-the-art MLP-based baselines, namely N-BEATS [1] and N-HITS [2], in the context of short-term forecasting. The results of these comparisons are provided below.\n\n| | Solar| |Wiki | |Traffic| |ECG| |Electricity| |COVID-19| |METR-LA| |\n|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|\n| | MAE| RMSE|MAE | RMSE|MAE| RMSE|MAE| RMSE|MAE| RMSE|MAE|RMSE |MAE| RMSE|\n|N-BEATS|0.137|0.182|0.047|0.087|0.016|0.031|0.056|0.088|0.057|0.085|0.162|0.192|0.087|0.166|\n|N-HITS|0.129 |0.174 |0.046|0.083|0.018|0.035|0.055|0.086|0.060|0.089|0.155|0.183|0.090|0.173\n|FreTS|0.120|0.162|0.041|0.074|0.011|0.023|0.053|0.078|0.050|0.076|0.123|0.167|0.080|0.166|\n\nThe results show that FreTS surpasses N-BEATS and N-HITS across all datasets, thereby demonstrating that frequency-domain MLP is more effective than time-domain MLP. We will include these results in the final version.\n\nAdditionally, we have conducted a comparison between FreTS and another MLP-based baseline LightTS [3] on long-term forecasting settings. The results of this comparison are presented in the table below:\n\n| Dataset  | |ETTh1 | |  | | ETTm1| |  | |\n|:---------|:---|:---|:---|:---|:---|:---|:---|:---|:---|\n| Models  |Metric |96| 192| 336 | 720|96| 192| 336 | 720|\n| LightTS|MAE|0.063 | 0.066| 0.072| 0.085| 0.058| 0.061|0.065 | 0.073|\n| |RMSE | 0.090|0.094 |0.099 | 0.112| 0.084| 0.088| 0.092| 0.101|\n| FreTS|MAE| 0.061| 0.065| 0.070| 0.082| 0.052| 0.057| 0.062| 0.069|\n| | RMSE| 0.087| 0.091| 0.096| 0.108| 0.077| 0.083| 0.089| 0.096|\n\nThe results obtained from both the short-term forecasting and long-term forecasting experiments clearly demonstrate that FreTS outperforms the time domain MLP-based baselines. This indicates the superiority of the proposed frequency-domain MLPs over their time-domain counterparts in terms of forecasting performance.\n\nWe will add the above MLP-based baselines in the experiment and included the results in the final version.\n\n[1] N-BEATS: Neural basis expansion analysis for interpretable time series forecasting, ICLR. 2019. \n[2] NHITS: Neural Hierarchical Interpolation for Time Series Forecasting, AAAI. 2023.\n[3] Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structures. arXiv, 2022.\n\n**W2. The superiority performance of FreTS verifies the effectiveness of (one layer) frequency-domain MLPs, but the authors did not investigate how well multi-layers of frequency-domain MLPs perform in MTS forecasting.**\n\nA2. Theoretically, FreMLP shows an inherent characteristic of energy compaction, as shown in **Theorem 1** . It means FreMLP can effectively extract the key patterns while reducing other redundant information, leading to improved generalization and better performance. Intuitively, a one-layer FreMLP is powerful enough and performs well to learn channel-wise or time-wise dependencies, while stacking multiple FreMLP may brings more parameters which reducing learning efficiency. To verify this, we conduct additional experiments on Exchange dataset under the input length of 96 and the prediction length of 96. The results are shown in the following table:\n\n| Metric  | one layer | two layers| three layers | four layers|\n|:--------|:---------|:---|:---|:---|\n| MAE  |  0.037       |  0.037| 0.037|0.038|\n| RMSE |   0.051      | 0.051|0.051|0.052|\n\nFrom the results, we can observe that FreTS achieves the same performance when utilizing \\{1,2,3\\} layer temporal learner. The results validate our claim that a one-layer FreMLP is adequate for constructing the channel learner and temporal learner to enable the acquisition of channel-wise and time-wise dependencies, respectively.\n\nWe will supplement the experiments of hyperparameter analysis in the appendix of our final version.\n\n**Q1. The learned patterns on the frequency spectrum show that FreTS can capture more obvious periodic patterns. But when there are no periodic patterns in MTS data, can FreTS still perform well? That is, whether FreTS is suitable for non-periodic MTS data.**\n\nA3. **Please refer to general response about FreTS on non-periodic data**.\n\n**Q2. Why FreTS contains only one layer of FreMLP in either frequency channel learner and frequency temporal learner? How well does FreTS work when stacking with multiple FreMLPs?**\n\nA4. Note that during our experiments,  we conducted careful hyperparameter tuning for FreTS. Specifically, we focused on tuning the number of layers in both the frequency channel learner and the frequency temporal learner. we found that using a one-layer channel learner and a one-layer temporal learner yielded the best forecasting performance for FreTS. \n\nIn addition, we have provided the corresponding results of FreTS for different numbers of layers, specifically {1, 2, 3, 4}, in the above table in the response to **W2**. From the results, we can observe that FreTS achieves the best performance when adopting one-layer learner. \n\nWe'll update the paper to address the above aspects and hope we have addressed your comments.\n"}, {"review_id": "2e7RclBwEB", "reviewer": "Reviewer_BMVc", "summary": "In this paper, the authors investigate the problem of time-series forecasting. Since the frequency domain can preserve the information from a global view and enjoy the advantage of energy compaction, the authors propose the FreTS model, which is composed of the Frequency Channel Learner and the Frequency Temporal Learner. The authors further prove the reasonableness of the frequency-domain MLPs and evaluate the proposed idea on several datasets.", "questions": "Please refer weaknesses", "limitations": "Please refer weaknesses", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tThe authors investigate the time-series forecasting problem from the frequency domain and provide convincing reasons for the advantage of the frequency domain.\n2.\tThe authors devise the frequency channel learner and frequency temporal learner, which address different challenges in a unified framework.\n3.\tThe authors investigate the proposed method on several datasets and achieve the ideal performance.\n", "weaknesses": "1.\tTo capture the dependencies among channels, the authors propose the frequency channel learner, which applies the FreMLP on the variables from each timestamp, e.g. $H_t^:$. However, Since it is strange to consider $H_t^:$ as a time series, I think it might be not a good idea to employ FreMLP to capture these dependencies. In my opinion, I think the CNNs, the attention mechanism, and GNN might be good choices. It is suggested that the author should try different methods to better capture these dependencies.\n2.\tThe authors address the frequency-domain MLP from time-series forecasting, but some time-series data might be not seasonal and contain some monotonic tendency, can the proposed method address this problem?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review and the positive comments on our work. We address each of them as follows.\n\n**W1. To capture the dependencies among channels, we propose the frequency channel learner, which applies the FreMLP on the variables from each timestamp, e.g., $\\mathbf{H}_t^{:}$. However, since it is strange to consider $\\mathbf{H}_t^{:}$ as a time series, I think it might be not a good idea to employ FreMLP to capture these dependencies. In my opinion, I think the CNNs, the attention mechanism, and GNN might be good choices. It is suggested that the author should try different methods to better capture these dependencies.**\n\nA1. Theoretically, the frequency channel learner is equivalent to applying global convolutions on the variables for each timestamp. Empirically, we have performed visualization experiments to verify the channel-wise learning capability of our frequency channel learner.\n**Please refer to general response about Frequency channel learner**.\n\n**W2. The authors address the frequency-domain MLP from time-series forecasting, but some time-series data might be not seasonal and contain some monotonic tendency, can the proposed method address this problem?**\n\nA2. Yes, FreTS can learn the monotonic tendency, we have analyzed the reasons and conducted experiments to verify this. **Please refer to the general response about FreTS on non-periodic data**.\n\n\nWe'll clarify the above in the final version and hope that we have addressed all your concerns. Thanks."}, {"review_id": "XJ6NNiKCgz", "reviewer": "Reviewer_u93d", "summary": "This paper investigates time series forecasting in the frequency domain. By utilizing MLPs in the frequency domain, the proposed FreTS effectively captures the patterns of time series with a global view and energy compaction. Frequency learning is applied to both inter-series and intra-series scales, allowing FreTS to capture channel-wise and time-wise dependencies. Extensive empirical experiments demonstrate the effectiveness of FreTS in both short-term and long-term forecasting tasks.\n", "questions": "1) Short-term forecasting comparison shown in Table 1 is suggested to include N-BEATS[1] and N-HITS[2] as baselines as well.\n2) Figure 4(b) demonstrates that the parameter number of FreTS remains constant as the prediction length increases. However, the projection block described in subsection 3.1 includes learnable parameters $\\phi_2 \\in \\mathbb{R}^{d_h \\times \\tau}$ and $b_2 \\in \\mathbb{R}^\\tau$, and the number of these parameters will increase with the prediction length $\\tau$.\n3) From which learner (frequency channel or frequency temporal), and at which layer are the weights used for obtaining the visualizations?\n\nReference\n\n[1] Oreshkin B N, Carpov D, Chapados N, et al. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting[C]//International Conference on Learning Representations. 2019.\n[2] Challu C, Olivares K G, Oreshkin B N, et al. NHITS: Neural Hierarchical Interpolation for Time Series Forecasting[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(6): 6989-6997.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This paper redesigns MLPs in the frequency domain to effectively capture both time-wise and channel wise correlations. The use of simple MLPs ensures high efficiency and helps mitigate overfiting issues compared to existing sophisticated deep models.\n2) This paper is well-organized, with comprehensive discussion and experiments.", "weaknesses": "1) The authors mention that learning time series in the frequency domain has the nature of energy compaction, as the energy concentrates on the smaller part of frequency components. However, the proposed FreTS still retains all the frequency components when performing MLP in the frequency domain, thereby not fully leveraging this advantageous characteristic.\n2) Frequency domain modeling is advantageous for capturing temporal dependencies due to the inherent periodic patterns in time series. However, the suitability of frequency domain modeling for exploring channel-wise correlations requires further discussion and verification. Moreover, conducting channel-wise modeling for each time point is inefficient, and should be performed at a coarser granularity, such as segment-wise or series-wise.\n3) The dimension extension block takes an input $X_t \\in \\mathbb{R}^{N \\times L}$ and produces an output $H_t \\in \\mathbb{R}^{N \\times L \\times d}$. However, if the number of variables $N$ or the input length $L$ is large, the memory usage of $H_t$ can become very high.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review and the positive comments regarding our paper. Below, we address your comments.\n\n**W1. ...still retains all the frequency components..., thereby not fully leveraging this advantageous characteristic.**\n\nWe preserve the entire frequency components and feed them into the frequency channel/temporal learners to avoid any information loss, which is necessary and rational. Note that all components, such as high-frequency ones or low-frequency ones, are adaptively highlighted or downweighed according to different data characteristics.\nAccordingly, *energy compaction* is achieved by adaptively learning in the frequency domain instead of manually discarding some components; it means that the information is concentrated on a smaller portion of components than in the time domain. This may be attributed to the fact that the frequency spectrum has discriminative frequency components specified for different data characteristics.\n\n**W2**\n1. Frequency domain modeling is theoretically and empirically suitable for exploring channel-wise correlations.\n  + Theoretically, the frequency channel learner is equivalent to applying global convolutions on the variables for each timestamp (see **Appendix D.2 Proof of Theorem 2**).\n  + Empirically, we have supplemented visualization experiments to verify the channel-wise learning capability of our frequency channel learner. For more details, **please refer to the general response about Frequency channel learner**.\n\n2. In many real-world scenarios, such as traffic scenarios, channel-wise dependencies can be time-evolving, even varying at different time points. In such cases, it becomes necessary and beneficial to conduct channel-wise modeling for each time point individually to capture the time-varying channel-wise dependencies. This approach is commonly employed in the literature, like in AGCRN and StemGNN.\n\n    Modeling channel-wise dependencies at a segment-wise or series-wise granularity is a promising idea, reminiscent of the successful implementation of series-wise time-wise dependencies in Autoformer. However, this approach overlooks the temporal dynamics within the segment and should integrate time information for comprehensive modeling. Due to time constraints, we were unable to conduct experiments to validate this idea. Nevertheless, we appreciate your constructive suggestion and will take it into account for our future research endeavors.\n\n**W3. ...if the number of variables or the input length is large, the memory usage can become very high.**\n\nFirst, performing time series embedding is widely adopted in recent MTS forecasting models, e.g., StemGNN and AGCRN. Second, we deliberately choose a smaller hidden dimension, specifically $d=128$, in comparison to Transform-based models that typically employ higher dimensions, such as 1024. As a result, the memory usage of $\\mathbf{H}_t$ is perfectly acceptable, which is evidenced by the fact that our FreTS capably works on long-term forecasting on all datasets and achieves the SOTA performance. \n\nIn addition, to avoid **extensive parameters** in the dimension extension block, we employ $\\mathbf{H}_t=\\mathbf{X}_t \\times \\phi_d$ where $\\phi_d \\in \\mathbb{R}^{1 \\times d}$ (see **lines 124-125**). Note that the parameter volume of the embedding matrix of $\\phi_d$ is independent of $N$ and $L$. This approach effectively avoids extensive parameters, as demonstrated in Figure 4(a), where the parameter count of FreTS remains unaffected by changes in $N$.\n\n**Q1. Short-term ... comparison ... suggested to include N-BEATS[1] and N-HITS[2] as baselines...**\n\nThank you for pointing out the two models. In Related Work, we have discussed these models, but we did not compare them with FreTS, considering they are univariate forecasting models. Taking your suggestion, we conducted a comparison between FreTS and the two baselines under the input length and prediction length of 12. The corresponding results are presented below:\n\n| | Solar| |Wiki | |Traffic| |ECG| |Electricity| |COVID-19| |METR-LA| |\n|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|:---|\n| | MAE| RMSE|MAE | RMSE|MAE| RMSE|MAE| RMSE|MAE| RMSE|MAE|RMSE |MAE| RMSE|\n|N-BEATS|0.137|0.182|0.047|0.087|0.016|0.031|0.056|0.088|0.057|0.085|0.162|0.192|0.087|0.166|\n|N-HITS|0.129 |0.174 |0.046|0.083|0.018|0.035|0.055|0.086|0.060|0.089|0.155|0.183|0.090|0.173\n|FreTS|0.120|0.162|0.041|0.074|0.011|0.023|0.053|0.078|0.050|0.076|0.123|0.167|0.080|0.166|\n\nThe results show that FreTS surpasses N-BEATS and N-HITS across all datasets, thereby demonstrating that frequency-domain MLP is more effective than time-domain MLP. We will include these results in the final version.\n\n**Q2. ...the number of these parameters will increase with the prediction length**\n\nThank you for reading our paper carefully. Indeed, the number of parameters in FreTS increases with the prediction length $\\tau$. The raw number of parameters in FreTS for drawing Figure 4(b) is 3.23, 3.26, 3.30, 3.34. However, due to the minimal variation among these data points, the curve exhibits only slight changes.\n\nWe will redraw the figure and put values on the figure to show the change more clearly.\n\n**Q3. From which learner ... which layer are the weights used for obtaining the visualizations?**\n\nNote that both the frequency channel learner and temporal learner adopt one layer of FreMLP (see **line 556 in Appendix B.3**), the reason for one layer is explained in **our response to the question W1 by Reviewer aNLj**. To ensure a fair comparison with DLinear, we follow the visualization settings used in DLinear, where we utilized the weights of frequency temporal learner for visualizations, the same as DLinear does. Further details about the visualization settings are in **Appendix B.4**, and we have uploaded the source codes of the visualization method in the supplementary materials for reference.\n\nWe'll clarify the above in the final version.\n"}, {"review_id": "vqbyQCOLiv", "reviewer": "Reviewer_bjea", "summary": "This paper presents FreTS, a framework that addresses both channel-wise and time-wise dependency learning in the frequency domain for time series prediction. FreTS introduces a specially designed frequency-domain MLP structure that processes the real and imaginary parts of the frequency components interactively. The experimental results on 13 benchmarks demonstrate the superior performance of FreTS compared to other state-of-the-art methods.", "questions": "- Are the “Domain Conversion” and “Domain Inversion” in Fig. 2 differentiable? How is the network trained end-to-end?\n- What are the meanings of the axes in Fig. 1? Whether the results are from a time series or a dataset?\n", "limitations": "The authors are encouraged to discuss the computational complexity after introducing frequency transform and inverse transform into the framework, and its performance on irregular time series. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The two-stage framework and the frequency-domain MLP module introduced in FreTS exhibit some novelty. \n- The experimental results across 13 benchmarks show the potential of the proposed method.", "weaknesses": "- The authors failed to motivate their method both theoretically and in relation to prior work.  1) The authors did not clearly define and explain the terms \"point-wise mappings\" and \"information bottleneck\", not elaborate on how existing methods suffer from these issues, and not clarify how the proposed method addresses and alleviates these problems.  2) While the authors claim that their method has a \"global view\", it did not highlight how it differs from previous MLP-based and Transformer-based methods in capturing long-range dependencies. 3) The claim of the advantage of “energy compaction” is not supported by experimental evidence.\n\n&nbsp;\n- The related work section lacks logical organization and fails to provide a comprehensive summary of previous methods. \n\n&nbsp;\n- The method is not well presented. 1) For the Frequency Temporal Learner, the validity behind performing DFT along the channel dimension needs further discussion. It is more likely to be based on engineering considerations than a theoretical grounding. 2) The rationale behind the design of the proposed frequency-domain MLP is unclear regarding why it calculates the new real and imaginary components as Eq. (7).  \n\n&nbsp;\n- Additional experiments are needed to highlight the benefits of the proposed approach, e.g. the learned global periodic patterns, and the robustness towards noise. Including these experiments will strengthen the empirical evidence for the effectiveness of the proposed method.\n\n&nbsp;\n- Certain phrases or sentences lacks clarity, and need improvements. E.g.,  “stacked MLP layers”, “learning in the frequency spectrum helps capture a greater number of periodic patterns”, .etc.\n\n&nbsp;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your review. Hope our response can address the misunderstandings or concerns\n\nw1.\n1. The concepts of \"point-wise\" and \"information bottleneck\" are widely recognized in the literature. In fact, we provided an explanation of these terms within the context of time series forecasting in **lines 39-42**. The point-wise nature of MLPs limits MLP-based methods' ability to capture long-range global dependencies. Further, these methods suffer from information bottleneck due to the volatile and redundant local momenta of time series, making them hardly capture accurate time-wise dependencies. This work aims to address these issues by leveraging the advantages of frequency-domain MLPs, i.e., global view and energy compaction (see **Lines 43-56**).\n2. FreTS is based on frequency-domain MLPs, obviously differing from MLP- and Transformer-based methods in architecture for capturing long-range dependencies. Intrinsically, Transformer-based methods rely on pair-wise attention mechanism, MLP-based methods rely on point-wise mapping; while ours rely on the global view of frequency techniques and frequency-domain multiplication.\n3. On the contrary, \"energy compaction\" is **discovered** from the experiments. Through experiments (see experimental settings in **Appendix B.4**), we observe that learning in the frequency domain can identify more concentrated diagonal dependencies and patterns than in the time domain (see **Figures 1, 5, 7, and 8**).\n\nW2.   \n+ FreTS is a frequency-based MLP model for time series forecasting. To provide a comprehensive summary, it is essential to introduce time-domain common models, frequency-based models, and MLP-based models.\n+ Correspondingly, the related work is organized from three perspectives: firstly on the time-domain models from classic models to SOTA deep learning models; the second paragraph discusses how existing frequency-based models integrate frequency techniques with neural networks; and the third paragraph describes the representative and SOTA MLP-based models.\n\nThis way logically organizes the models relevant to our work with a comprehensive discussion of the previous related literature.\n\nW3.\n1. Note that Frequency Temporal Learner performs DFT on the time dimension (**see line 165**) while Frequency Channel Learner is on the channel dimension (**see line 152**). We guess you may ask about the theoretical grounding for Frequency Channel Learner, please refer to the general response about the frequency channel learner.\n2. In the frequency domain, the values representing the frequency spectrum are complex numbers. Mathematically, the multiplications in the frequency domain adhere to **the basic rule of complex multiplication**. Please refer to **Appendix C** where we show a calculation example.\n\nW4.   \n1. Note that we have already conducted extensive experiments to verify our claims regarding the benefits of the proposed approach. \n+ To verify the global view of learning in the frequency domain, we performed visualization experiments on the Traffic and Electricity datasets, comparing the weights learned in the time domain with those learned on the frequency spectrum.\nThe results can be shown in **Figures 9, 10, 11, and 12 in Appendix G.2**, verifying that FreTS has a strong capability to capture the global periodic patterns. \n+ To verify energy compaction in the frequency domain, we visualized the weights in the frequency temporal learner on Traffic and Electricity. The results can be reported in **Figures 5, 7, and 8**, exhibiting energy aggregation characteristics with clear diagonal patterns and dependencies.\n\n2. To address your concern regarding the robustness of FreTS towards noise, we add $0.1 \\times \\mathcal{N}(0,1)$ Gaussian random noise into training data on the ETTh1 dataset. The results shown in the following table support our claim of robustness.\n\n| |without noise | |with noise | |\n|:--|:--|:--|:--|:--|\n| Metric| MAE| RMSE| MAE| RMSE|\n|96|0.061|0.087|0.061|0.087|\n|192|0.065|0.091|0.065|0.091|\n|336|0.070|0.096|0.070|0.097|\n|720|0.082|0.108|0.083|0.109|\n\nW5.  \nIn **Line 94**: \"stacked MLP layers\" refers to N-BEATS proposes a deep architecture based on backward and forward residual links and a very deep stack of fully-connected layers (**cf. line 2-3 in Abstract of N-BEATS paper**).\nIn **Line 129-130**: we state \"learning in the frequency spectrum helps capture a greater number of periodic patterns\" to explain the observation from Figure 1(a), meaning the patterns learned in the frequency domain exhibit more obvious global periodic patterns than in the time domain. We have explained this in **the caption of Figure 1** and in **lines 50-51**. We will carefully polish the phrases.\n\nQ1.   \nCertainly, both of them are differentiable, please refer to **Equations (1) and (2)**. Our FreTS is an end-to-end forecasting model.\n\nQ2.   \nFig. 1 visualizes the learned weights $W \\in \\mathbb{R}^{d\\times d}$ where $d$ is the hidden dimension. The axes are the dimension indices of the weight matrix.\nThe weights for visualizing Fig. 1 are learned on the Traffic dataset (see **lines 575 and 584 in Appendix**). A detailed description of the visualization settings is **in Appendix B.4**.\n\nLimitation:\n\n1. In **Section 4.3**, we have conducted experiments to evaluate the efficiency of FreTS, covering short-term forecasting scenarios and long-term forecasting scenarios, in comparison with SOTA models. Contrary to your concern, incorporating Fourier transform can improve efficiency (see reasons in our responses to Reviewer WVuT). Also, this advantage has been widely explored and acknowledged in the literature for time series forecasting, such as Autoformer and FEDformer.\n\n2. In this paper, similar to all baseline methods, we only focus on time series forecasting without focusing on irregular time series. Frequency transformation may also help address irregular time series issues, and we leave this as our future work. \n\nWe'll clarify the above in the final version.\n"}]

Author rebuttal:
Dear Reviewers, ACs and the SAC:

We thank you all for the review and valuable comments. We'll clarify them in the final version to address all relevant questions and constructive suggestions. 
 
To address the common concerns regarding our frequency channel learner (Reviewer u93d, Reviewer BMVc, Reviewer bjea) and our model effectiveness on non-periodic data (Reviewer aNLj, Reviewer BMVc), we provide explanations as follows. We also carefully consider each comment of Reviewer bjea and realize that there might be some misunderstandings towards our method. We've tried our best to clarify these misunderstandings in the specific rebuttal.

## General Response about frequency channel learner and non-periodic data
**Frequency Channel Learner**

Theoretically, the frequency channel learner is equivalent to global convolutions on the variables for each timestamp. In addition, we have performed visualization experiments to verify the channel-wise learning capability of our frequency channel learner.

+ Theoretically: According to **Theorem 2**, we know that the operations of frequency-domain MLPs can be viewed as global convolutions in the time domain, i.e., Eq. (9) $\mathcal{H}\mathcal{W}+\mathcal{B}=\mathcal{F}(\mathbf{H}\ast W+B)$. As a result, the frequency channel learner containing frequency-domain MLPs can be regarded as global convolutions (global CNNs) over the variable dimension.
+ Empirically, we have investigated the channel-wise learning capability of our frequency channel learner on the METR-LA dataset. 

    Specifically, we randomly select 30 detectors and visualize their corresponding adjacency matrix learned by the frequency channel learner via a heatmap. We attach the corresponding heatmap figure in the attached PDF (**see Figure 1 in the attached PDF**). By examining the learned adjacency matrix in conjunction with the actual road map, we can observe that the detectors are very close w.r.t. the physical distance, corresponding to the high values of their correlations with each other in the heatmap. The visualization results demonstrate that the frequency channel learner learns the channel-wise dependencies effectively. 

**FreTS on non-periodic data**

Our proposed FreTS is suitable for time-series data that may not be seasonal and may contain some monotonic tendency. 

Note that the frequency-domain MLPs compared with time-domain MLPs have two advantages: **global view and energy compaction**. 
+ First, the frequency-domain MLPs are equivalent to efficiently performing global convolutions over timestamps (see **Theorem 2**). Accordingly, the frequency temporal learner is able to capture various global temporal information far beyond the periodic information (including periodic patterns, global monotonic tendency, and temporal correlations). 
+ In addition, **Theorem 1** implies that, if most of the energy of a time series is concentrated in a small number of frequency components, learning in the frequency spectrum can facilitate preserving clearer/more effective patterns. This characteristic is beneficial for all types of MTS data, regardless of whether the data is periodic or not.

As a result, FreTS is not only suitable for MTS data with periodic patterns, but also for MTS data without periodic patterns. 

+ Empirically, our extensive experimental results on different kinds of time series datasets from various applications demonstrate the state-of-the-art performance of our proposed FreTS. The benchmark datasets include Wiki, ECG, and Exchange that do not contain significant seasonal patterns.
+ Moreover, we have provided extensive visualization experiments in **Figures 5, 7, and 8** to investigate the learned weights in the frequency temporal learner. From the results, we observe that the weight coefficients of the real or imaginary part exhibit energy aggregation characteristics (energy compaction). The results on various time series data verify that frequency-domain MLPs can facilitate learning informative features and patterns, which is not limited to time series data with periodic patterns.
+ Furthermore, we have synthesized one multivariate time series dataset (linear: $X=aT+\sigma$ where $a$ is the trend of the variable and $\sigma$ is random noise) and conducted experiments on the dataset. The results under different prediction lengths are visualized in the attached PDF (**see Figures 2, 3, 4, 5, and 6 in the attached PDF**).
From these figures, we can find that FreTS can capture a monotonic tendency.

We hope our response has addressed all concerns. We would greatly appreciate any further constructive comments or discussions. 


Meta-review: Dear SAC,

This paper has received the scores of 5, 6, 6, 6, 7, where all reviewers unanimously voted for accepting the paper.

Given the originality and the novelty of the proposed approach of utilizing MLPs in the frequency domain, as well as the relatively high review scores from the reviewers, I recommend to accept the paper as a poster.

Best,

AC
Decision: Accept (poster)