Irregularity Reflection Neural Network for Time Series Forecasting
Authors: YoungJun Choo
Adrian Matias Chung Baek
Namhun Kim
Han-Gyun Woo
Venue: Submitted to ICLR 2023 (2023)
Keywords: DNN
CNN
time series
fourier series
irregularity representation learning

Abstract: Time series forecasting is a long-standing challenge in a variety of industries, and deep learning stands as the mainstream paradigm for handling this forecasting problem. With recent success, representations of time series components (e.g., trend and seasonality) are also considered in the learning process of the models. However, the residual remains under explored due to difficulty in formulating its inherent complexity. In this study, we propose a novel Irregularity Reflection Neural Network (IRN) that reflect the residual for the time series forecasting. First, we redefine the residual as the irregularity and express it as a sum of individual, short regular waves considering the Fourier series in a micro perspective. Second, we design a module, based on the convolutional architectures to mimic the variables of the derived irregularity representation, named Irregularity Representation Block (IRB). IRN comprises IRB on top of a forecasting model to learn the irregularity representation of time series. Extensive experiments on multiple real-world datasets demonstrate that IRN outperforms the state-of-the-art benchmarks in time series forecasting tasks. 

Under review as a conference paper at ICLR 2023
IRREGULARITY REFLECTION NEURAL NETWORK FOR TIME SERIES FORECASTING
Anonymous authors Paper under double-blind review
ABSTRACT
Time series forecasting is long-standing challenge in variety of industries, and deep learning stands as the mainstream paradigm for handling this forecasting problem. With recent success, representations of time series components (e.g.. trend and seasonality) are also considered in the learning process the models. However, the residual remains under explored due difficulty in formulating its inherent complexity. In this study, we propose a novel Irregularity Reflection Neu- ral Network (IRN) that reflect the residual for the time series forecasting. First, we redefine the residual as the irregularity and express it as sum of individual, short regular waves considering the Fourier series in micro perspective. Sec- ond, we design module, based on the convolutional architectures to mimic the variables of the derived irregularity representation, named Irregularity Represen- tation Block (IRB). IRN comprises IRB top of a forecasting model to learn the irregularity representation of time series. Extensive experiments on multiple real- world datasets demonstrate that IRN outperforms the state-of-ther fif-thhertt benchmarks in time series forecasting tasks.
INTRODUCTION
Owing to the ubiquitous computing systems, time series is available wide range of domains in- cluding traffic (Chen et al. 2001), power plant (Gensler al. 2016), stock market indices (Song et al. 2021 and so on (Liu et al., 2015 Duan al. 2021 Spontaneously, interests in time se- ries forecasting have grown, and result, an in- tensive research for a more accurate prediction.
Bend
Scasonatity
In recent literature, many deep learning models have been favored for forecasting problems (Lim & Zohren 2021). Recurrent Neural Network (RNN) and its extensions such as Long Short- Term Memory (LSTM) (Hochreiter & Schmid- huber 1997) and Gated Recurrent Unit (GRU) (Chung al. 2014) are popular choices for ana- lyzing long sequences. Nevertheless, these mod- els tend be restricted in handling multivariate time series. As a powerful alternative, Convolu- tion Neural Networks (CNNs) has been introduced to capture overall characteristics of time series through parallel calculations and filter operations. Building on the success in forecasting task, CNN- based models have been proposed according to the type time series data. Temporal Convolutional Network (TCN) was applied to audio datasets (Oord et al. 2016). whereas Graph Convolutional Network (GCN) was utilized in the time series with graph characteristics (c.g.. human skeleton- based action recognition (hhag et al. 2020) and traffic dataset (Bai et al.) 2020)). The attention models have also been applied to emphasize the specific sequence data that are primarily referenced when making the predictions (Liu et al. 2021b).
Figure 1: The Traffic data and its time series components (i.e., trend, seasonality, and irreg- ularity).

---

Under review as a conference paper at ICLR 2023
Despite the great efforts made, forecasting performance has room for further improvement afore- mentioned models learn feature representations directly from complex real-world time series, often overlooking essential information. Recently, incorporating representations of time series compo- nents (e.g.. trend, seasonality) used in conventional econometric approaches have shown to lead to better performances of the learning models. For instance, N-BEAT'S (Oreshkin et al. 2019). Aut- oformer (Wu et al. (021). and CoST (Woo et al., 2022) reflected the trend and seasonality of the time series and achieved improvements. However, as shown in Figure I, time series also include the irregularity that is not accounted by the trend and seasonality, and is yet under explored (Woo et al., 2022).
To address this challenge, we show how to deal with the irregularity of the time series data to improve the forecasting performance of the deep learning models. To this end, we represent the irregularity into an encodable expression on basis of Fourier series viewed from micro perspective. The derived representation is encoded using convolutional architectures, and named as Irregularity Representation Block (IRB). Then, IRB embedded
base model builds the Irregularity Reflection Neural Network (IRN). We demonstrate that IRN outperforms existing state-of-the-art forecasting models on eleven popular real-world datasets.
2
RELATED WORK
2.1 DEEP LEARNING FOR TIME SERIES FORECASTING
Sequential deep learning models such RNN, LSTM, and GRU have long been used for time series forecasting (Elman [990 Hochreiter & Schmidhuber 1997|
Chung et al. (0144) Although effective
in capturing the temporal dependencies of time series, RNN-based models neglect the correlations
in-between time series. To tackle this issue, Liu et al.
(2020)
propose a dual-stage two-phase (DSTP)
to extract the spatial and temporal features. Shi et al.
(2015 present convLSTM replacing the states
of LSTM block with convolutional states. Another limitation of the sequential models are that the discrepancy between ground truth and prediction is accumulated over time as predictions are referred
to predict further into the future (Liu et al. 2021a).
More recent works have demonstrated that CNNs can be applied in multivariate time series problems as well. Ravi et al. (2016) introduce the ID convolution for human activity recognition, whereas Zhao et al. (2017) suggest the use of 2D convolution. CNN models are parallelizable, and hence show following ad vantages: the consideration of the correlation between variates and the prevention of error accumulation (Liu et al., 2019). A downside is the limited receptive field when predicting long sequences due to the increasing number of the parameters (Zhao al. 2017). [Wang et al. (219) tackle this challenge by decomposing the long sequences according to long, short and close-
CNN-based models have received increasing attention to enhance the forecasting performance. For example, the dilated casual convolutional layer used increase the receptive field by down- sampling and improve long sequences prediction (Sen et al.) 2019 Oord et al.] 2016). Another approach is Graph Convolutional Network (GCN), that analyzes the relation between nodes with specific position and edge relation, especially in traffic data (Fang et al. 2021 Song et al., (220)) and human body skeleton data (Yoon al., 2022 Chen al.) 2021). Attention- based models have also been adopted (Liu et al.) 2019) and further developed into Transformer (Zhou et al., 2021 Liu jet al. (2021)). However, these approaches do not take into account the characteristics of time series such trend, seasonality and irregularity.
2.2 REFLECTING THE REPRESENTATIVE COMPONENTS OF TIME SERIES
Considerable studies on time series analysis have relied on the decomposition of time series into non-random components. For instance, DeJong et al. 1992) conducted analysis on the trends of the macroeconomic time series as well as Lee & Shen (2009) emphasized the importance of obtaining significant trend relationship in linear time complexity. Jonsson & Eklundh (2002) extracted and analyzed the seasonality of the time series data and Taylor & Letham (2018) considered both trend and seasonality. When extracting these non-random components, non non-stationary time series be- comes stationary, meaning -ine-independent. As conventional statistical methods such as ARIMA (Autoregressive Integrated Moving Average) (Williams & Hoel| 2003) and GP (Gaussian Process)
2

---

Under review as a conference paper at ICLR 2023
a
(b)
Bleck
Averr
Sigmaed
Mis
M
M
Multiplication
Residual Connection
ALl
Maser 2 Di. 10Mparree
Masipst
Maver Doa Map M0
Bowmmode!
JDM 10 (parsa))))))
X eitipul
Figure 2: An overview of IRN framework. In IRN, (a) IRB extracts the irregularity feature from the input sequences and (b) Irregularity Reflection module conducts the time series forecasting.
(aaanad Gatelsene) + Altona?
Irrecular (cvee(()
Altana) 2 Catelaaaa
Regular (g/5(()
Reguler wwwe 1 (2)(()
Candiinal
Regular (ar(())
Can be expeessed as regular ware
Regular wave C25
Figure 3: (a) The irregular wave (t) consisting of multiple regular waves and (b) the irregular wave from micro perspective vb (faws).
Van Der Voort et al., 1996) perform better on stationary data (Cheng 2018). differentiation for stationarity has been conducted (Atique et al. 2019). As such, extraction of the representative time series components for forecasting problems has been major research topic (Brockwell & Davis 2009 Cleveland et al. 1990).
Recently, direct learning from input sequences of the deep forecasting models is regarded to be enough, thereupon researchers focus on how to incorporate the components of time series in the learning process. For instance. Oreshkin et al. 2019) proposed hierarchical doubly residual topol- ogy as the interpretable architecture to extract time series representations: trend, and seasonality, Wu al. (202T) proposed transformer-basee model which decomposes and reflects the trend and sonality by using auto-correlation mechanism. WWo et al. (2022) introduced disentangled Seasonal- Trend Representation Learning by using the independent mechanisms. They deviced disentanglers for the trend and seasonal features, mainly composed of a discrete Fourier transform to map the intermediate features frequency domain. These studies successfully reflect representations of trend and seasonality which are the time dependent value, and improve forecasting performances. However, the irregularity, which cannot be explained by the trend seasonality and the time independent value, is not sufficiently addressed. In this paper, we build and reflect the irregularity representation to complement previous researches forecasting tasks.
METHODOLOGY
In this section, we discuss how to reinterpret the irregularity of the time series in term of Fourier series, extract and reflect the irregularity representation using convolutional architectures. Our pro- posed model IRN is shown in Figure 2.
3.1 THRORETICAL APPROACH
3

---

Under review as a conference paper at ICLR 2023
A time series is generally in the form of an ir- regularity. Hence, its representation is essential for time series forecasting, Among many exist- ing approaches to represent irregularity, Fourier series is perhaps the most widely used. Fourier series approximates irregularity by the linear superposition of multiple regular waves with varying height, period, and direction as de- picted Figure 3 (a) (Bloomfield) 2004). The irregularity v (t) can be expressed as:
x imputt
Input layer
Rea
Convv' layer
MaxPoolling
MiniPooling
AverPooling
Pinzz
Pintn
Pay
Conv layer
layer
Conv byyer
Comv layer
Chaze
R Rimex
Rea
v (t) = Σ (trn(t)
(1)
Tanh
xe trreguttr
Trann
n==
Conv layer
where (t))) is n-th regular wave, C, is the coefficient of Tn (t), t is the time. The con- cept of infinity Equation I challenging for the learning model to handle. Therefore, we reinterpret t (1) into an encodable equation by viewing it at the micro level. When the irreg- ularity (t) in the time domain is observed at the moment fa-d. it can be interpreted as regular wave 16 (fawb) with vertical shift, which is the average value of the regular waves. Under this concept, Equation rewritten as:
RGge
Conv layer
Reat
x Boutppt
Figure 4: Architecture of the Irregularity Repre- sentation Block.
v (ta-s) = Coro(tawb) + Σ Carn(tanb)
n=1
where Coro(ta-d) is the representative regular wave characteristic. Cora(ta-a) is the regular wave with mean of 0 without vertical shift. The representative regular wave ro ro(ta-b) oscillates between the constant maximum and minimum values in period of time and can be defined as Amplitude x sin(w x la-s), where the angular velocity w is constant due to the periodicity of the wave, w X tamb is denoted as the angle 0 of ro(la~t). and sin(w x laws) is the phase of ro (law))) Amplitude is calculated with the peaks of the wave. Accordingly, the representative regular wave ro can be rewritten
ro(to-s)= max(tanb) - -min(tand) sin(O(ta>s))
(3)
where sin(O(ta=b)) is the phase of ro(ta-b) at tand. Therefore, Curo(tawb) in Equation 2 is re- defined by referring to Equation The remaining infinity term Ens Carn(tawe) in Equation corresponds to the vertical shift and can be expressed as the average value A(ta-s) of & (fa-a) as depicted in Figure (b). The representative regular wave Coro(lant) and the average value Em Cnrn(luu convert Equation 2 into:
ab (famb) A(tans) + maz(te-s) min(tand) x sin(O(ta>tt)
(4)
2
When the regular waves are sequentially connected, we obtain the irregularity v (t) consisting of regular waves that change with time la-d. We redefine the Equation 4 as follows:
w (t) ~ A(t)+ max(t) - -mi(tt)) x sin(O(t))
(5)
2
where man(t)-min(t) the amplitude of the regular wave Coro(la<s) at lawh, sin(O(()) is the 2 phase of Caro(lans) at la-b, and A(t) is the average which is the sum of remained regular waves Em Cnrx(ta-s) at taws. According to Equation 5, the irregularity (t) can be represented by the combinations of the minimum, maximum, average, and phase values.
3.2 IRREGULARITY REPRESENTATION BLOOK
Based on Equation 5, the irregularity is encoded to incorporate into deep learning models. In this pa- per, convolutional architectures adopted since convolutional layers allow the parallel prediction
4

---

Under review as a conference paper at ICLR 2023
as well analysis of relations existing in multivariate time series through filter operations. We input the multivariate time series x="put E R 7xdd wheere is a look-back window of fixed length, and d is the number of variates. Our model stacks multiple convolution layers with the RELU activation function, a dilation filter and same padding. The RELU activation increases the model complexity through space folding (Montufar al. 2014), and the dilation operation helps expand the receptive fields (Oord et al., 2016). The convolution layer extracts the feature that has the same size of the pmpput through the same padding. Accordingly, Equation is transformed into
maz(zinput) min min(x (pput)
x irregutor
A(x imput A
sin sin(0(z
(6)
2
where @ is the pointwise summation, 69 is the pointwise multiplication, and x: irregaior is the ir- regularity. Through this transformation, the zirreguter is converted from time dependent data dependent and the main operations (i.e., A(, mar(), min(), and sin(0()) are expressed by using the convolution layers and pooling layers which extract the average, the maximum, the minimum, and the phase value from x nnp under microscopic perspective condition. The main operations are encoded like Figure follows:
Maver ( (ninput) = RCav (Pav Pau(RR) (Rc, (x.input))) )) =e A(x ((xpppput)
Mamp Mamp(zinpur (pppt))) Remon (Pmin Rey (a (apue)) })} Rominn ( Ponin (Rei (ar (x""p"()))
max( iupurt Appnt
(8)
2
Mphase (winpat) Teanh TTaan(CCphhae (CCphase RRc, (z sin(0(x
where Pmoz= Pminn and Pay are the max, min, and average pooling operations, respectively. C' the convolution layer without activation, R is C with the RELU activation and Trank is the hyperbolic tangent(tanll (tanh) activation. Through this process, the average, amplitude, and phase values in Equation 6 are converted to trainable values. To extract the representation of the average value from xn we stack the 2D convolution filter and the 2D average pooling as in Equation 7. To decompose construct the structure same Equation 8
the representation of the amplitude from iniput with the 2D max and min pooling. To obtain the representation of the adaptive phase value using impput under microscopic aspect condition, referring to the Equation 9, use the tanh activation (aver(ittpatt (zinppat) Mamp (z"mput), ), and M Mphase(zsimpuly (ppp)))) extract the average, amplitude, and phase values from input, and we redefine Equation follows: r irregalor = Mavver (M (nnt + Mamp (2 (xupptt) @ M Mphase ( (put)
after convolution layer. Consequently, these operations(i.e. Maver
(10)
To consider the x irregular value, we apply the residual stacking principle which enables complex interpretation by combining features in a hierarchical form for each step (Oreshkin et al. 2019).
Therefore, we design the IRB architecture as follows:
x R.utput
(11)
RC.u (Ro., (girregular) + Rc, (winput) + Mamp(w @ Mphose (xinput))
The output of IRB RBBueput is the representation of the irregularity which considers the aver- age, amplitude, phase, and input components. Furthermore, these components are trainable values because they consist of the convolution layers.
3.3 IRREGULARITY REFLECTION NEURAL NETWORK
IRN consists of IRB and irregularity reflection module as in Figure 2. For the forecasting of this study, a recent model that reflects trend and seasonality, known as SCInet (Liu et al. (2021a) is used the base model. The x IRB.uppu is passed to the base model through the irregularity reflection module.
A Rouigut Ssig (2: (BButpptt) @ x input @ x imput
(12)
where x. Roueput is the output of IRN and Ssig is the sigmoid activation. The pointwise multiplica- tion is applied to emphasize the irregularity of the input by using the RRBoutpst with Sang- If we use the x R.uutett as the input value of the time series model, some information (e.g.. trend, sea- sonality) can be omitted. To alleviate this problem. we preserve the original information by residual connection, which also prevents the gradient vanishing (He al. 2016).
5

---

Under review as a conference paper at ICLR 2023
Table 1: Summary of datasets and evaluation metrics used for time series forecasting.
Type
Dataset
Variates
Timesteps
Granularity
Start time
Metries
Train/Val/Tes
ETTh1
17420
hour
MSE MAE
ETT Zhou et al.
2021
ETTh2
7
7/1/2016
12/4/4
ETTml
69680
15 min
PEMS03
358
26209
5/1/2012
MAE MAPE RMSE
PFMS04
307
16992
7/1/2017
PEMS Chen et al. 2001
5 min
6/2/2
PEMS07
883
28224
5/1/2017
PEMS08
170
17856
3/1/2012
Solar.
137
52560
10 min
2016
Traffic.
862
17544
2015
RSE
hour
6/2/2
Electricity,
321
26304
2012
CORR
Exchange-rate (Lai et al. 2018
7588
1 day 1990
Table 2: Multivariate forecasting performance of IRN and baseline models on the ETT datasets. Best results are highlighted in bold.
Model
Metris
ETTb1
ETTh2
ETTml
336
720
168
3,36
LogTans
MAE
MSE
0.604
0.686
0.766
0.757
0.846
1.002
0.952
1.362
1.397
1.291
0.828
0.75
1.806
1,034
1.681
4.07
3.875
1.763
1.552
3.913
0.412
0.419
0.583
0.507
0.792
0.768
1.462
1.32
1,669
1.461
Reformer
MAE
MSE
0.991
0.754
0.906
1,138
2.117
1.28
2.415
1.52
1.513
L.735
1.846
4.66
4.028
1.688
2.015
0.607
0.777
0.945
1.433
1.094
1.232
TCC
MSE
0.766
0.825
0.982
1.099
1.267
1.154
1.579
3,456
3.184
3,538
0.502
0.645
0.675
0.758
0884
MAE
0.786
1,459
1.42
0.583
TST
MSE
0.735
0.8
0.973
1,029
1.02
0.994
1,159
2.609
2.824
2.684
0.471
0.614
0.645
0.749
0.857
MAF
0.633
0.671
0.768
0.797
0.798
0.779
0.85
1.268
1.337
1.334
0.491
0.56
0.581
0.645
0,709
MSE
0.728
2.452
0.707
0.781
MAE
0.6
0.629
0.714
0,779
0.835
0,522
0.684
2.113
1.304
1.399
0.459
0.55
0.593
0.644
0.7
Tripler
MAE
MSE
0.729
0.942
0.746
0.975
0.825
1.135
0.859
1.187
0.916
1.283
0,911
1.285
0.966
1.455
2.175
1.155
2.007
1.101
1.139
2.157
0.689
0.592
0.624
0.752
0.623
0.744
0.508
0.662
0,917
0.72
MoCo
MSE
0.623
0.669
0.981
1.138
0,444
0.613
1.791
2.241
2.425
0.458
0.594
0.621
0.7
MAE
0.555
0.586
0.674
0.755
0.831
0.495
0.595
1 1,034
1.186
1.292
0.444
0.528
0.553
0.606
0.674
TNC
MAE
MSE
0,708
0.592
0.749
0.884
0.768
1.02
1.157
0.83
0.612
0.84
2.359
1.213
2.762
2.753
0.522
0.695
0.731
0.595
0.818
0,932
Informce
MSE
0.577
0.685
0.931
1.128
1.215
0.72
1,457
3,489
2.723
3.467
0.323
0.494
0.678
1.056
1.192
MAE
0,549
0.625
0,752
(1873
0.896
0.665
1 1.001
1.515
1.34
1.473
0.369
0.503
0.614
0.786
0.926
TS2Vee
MAE
MSE
0.531
0.59
0.585
0.639
0.762
0.728
0.799
1.063
0.489
0.605
1 1,074
1,815
2.194
1.197
1.37
0.444
0.521
0.592
0.635
0.554
0.597
0.653
SCInet
MAE
MSE
0.379
0.341
0.395
0.368
0,457
0.451
0,502
0.497
0.583
0,56
0.288
0.188
0.358
0.279
0.505
0.504
0.618
0.56
0.761
1.074
0.126
0.229
0.169
0.274
0.291
0.191
0.415
D.365
0.604
0,713
Pyrafomner
MSE
0,808
0.945
1.022
0.48
0.754
0.857
MAE
0.683
0.766
0.806
0.156
0.659
0,707
MSE
0.386
0.437
0.643
0.812
0.97
0.447
0.699
1.549
1.749
1.971
0.246
0.331
0.378
0.472
0.62
MAE
0.379
0.464
0.582
0.679
0.771
0.502
0.637
0.982
1.042
0.386
0.419
0.4%6
Autoformer
MSE
0.384
0.392
0.49
0.505
0.498
0.261
0.312
0.457
0.471
0.474
0.383
0.454
0.181
0.634
0.606
MAE
0.425
0.419
0.481
0.484
0.5
0.341
0.373
0.455
0.475
0.484
0.403
0.453
0.463
0.528
0.542
IRN
MAP
MSE
0.361
0.368
0.432
0.467
0.474
0.501
0.27
0.314
0.437
0.453
0.498
0.51
0.745
0.223
0.143
0.249
0.184
0.28
0.398
0.522
4 EXPERIMENTS
We conduct experiments on real-world time series datasets and compare the performance with the latest baselines. We analyze the circumstances in which proposed IRB improves the forecasting performance. We refer base model (Liu et al.] 2021a) for the experiment settings. Due to page limits, the implementation details including the loss function, datasets, and metrics are reported in the Appendix.
4.1 DATASET
Experiments conducted on following time series datasets: Electricity Transformer Temperature (Zhou et al, 202I). PEMS (Chen et al. (200I), Solar, Traffic, Electricity. Exchange-rate (Lai et al. 2018). The datasets, experiment settings, and metrics are summarized in Table 1.

---

Under review as a conference paper at ICLR 2023
Table 3: Univariate forecasting performance of IRN and baseline models on the ETT datasets. Best
results are highlighted in bold.
ETThi
ETTh2
FTTml
Model
Merrics
24
48
168
336
720
24
48
168
336
720
24
48
96
288
672
N-BEAT
MSE
0.042
0.065
0.106
0.127
0.269
0.078
0.244
0.27
0.031
0.056
0.157
0.207
MAE
0.156
0.2
0.255
0.284
0.422
0.21
0.271
0.393
0.418
0.432
0.117
0.168
0.234
0.311
0.37
Informer
MSF
0.098
0.158
0.183
0.222
0.269
0.093
0.155
0.232
0,266
0.277
0.03
0.069
0.194
0.401
0,512
MAE
0.247
0.346
0.24
0.389
0.417
0.137
0.203
0.554
TCC
MSE
0.053
0.074
0.133
0.161
0.176
0.111
0.148
0.225
0.232
0.242
0.026
0.045
0.072
0.158
0.239
MAE
0.175
0.209
0.284
0.32
0.343
0.255
0,298
0.374
0.385
0.397
0.122
0.165
0.211
0.318
0.398
TST
MSE
0.127
0.491
0.526
0.717
0.134
0,171
0.261
0.269
0.048
0.064
0.102
0.172
MAE
0.284
0.362
0.596
0.618
0.76
0.281
0.321
0.404
0.413
0.42
0.151
0.183
0.231
0.316
0.366
MSE
0.076
0.104
0.162
0.183
0.212
0.109
0.152
0.251
0,238
0.234
0.018
0.035
0.0:9
0.118
0.177
MAF
0.217
0.259
0.326
0.351
0.387
0.251
0,301
0.392
0,388
0.389
0.035
0.142
0,188
0:271
0,332
Triplet
MSE
0.13
0.145
0.173
0.167
0.195
0.16
0.181
0.214
0.232
0.251
0.071
0.084
0.097
0.13
0.16
MAE
0.289
0.336
0.333
0.368
0.316
0.372
0.18
0.276
MoCo
MSE
0.94
0.063
0.122
0.144
0.183
0.095
0.13
0.204
0.206
0.206
0.015
0.027
0,041
0.083
0.122
MAE
0.151
0.191
0,268
0.297
0.347
0.234
0.279
0.36
0,764
0.369
0
0.122
0.153
0.219
0.268
TNC
MSE
0.057
0.171
0.192
0.235
0.097
0.197
0.207
20207
0.19
0.036
0.054
0.098
0.136
MAE
0.184
0.239
0,329
0.357
0.408
0.238
0.281
0.354
0.366
0.37
0.103
0.142
0.178
0.244
0.29
TS2Vec
MSE
0.039
0.062
0.142
0.16
0.179
0.091
0.124
0.198
0,205
0 208
0,016
0.028
0,045
D.095
0,142
MAE
0.151
0.291
0.364
0.093
0,162
0.235
SCInet
MSE
0.031
0.051
0,081
0.094
0.176
0.07
0.102
0.157
0.177
0.253
0.019
0.045
0.072
0.117
0.18
MAF:
0.132
0.173
0,222
0.242
0.343
0.194
0.242
0:311
0.34
0.403
010888
0.143
0.198
0.266
0,328
CoST
MAE
MSE
0.152
0.04
0.186
0.097
0.236
0.112
0.258
0.148
0.306
0.079
0.207
0.259
0.189
0.339
0.206
0.36
0,371
0.214
0.088
0.015
0.117
0.147
0.077
0.209
0.257
IRN
MSE
0,03
0.045
0.078
0.091
0.168
0.067
0.199
0,154
0,172
0.235
0.018
0.043
0.07
0.116
0.151
MAE
0.131
0.163
0.218
0.241
0.329
0.189
0.232
0.31
0.337
0.392
0.087
0,141
0.196
0.264
0.301
Table 4: Forecasting performance of IRN and baseline models on PEMS datasets. Best results are highlighted in bold.
Model
PEMS03
PEMS04
PEMS07
PEMSO8
MAE
RMSE
MAPE (%)
MAE
RMSE
MAPE (%)
MAE
RMSE
MAPE (%)
MAE
RMSE
MAPE (%)
LSTM
21.33
35.11
21.33
25.14
39.59
20.33
29.98
42.84
15.33
22.2
32,06
15.32
TCN
18.87
32.24
36.87
30.53
13.88
21.42
34,03
13.09
DCRNN
18.18
30.31
18.91
24.7
38.12
17.12
28.3
38, 58
11.66
17.86
27.83
11.45
STGCN
17.49
30.12
17.15
22.7
35.55
14.59
25.38
38.78
11.08
18.02
27.83
11.4
ASTGCN()
17.69
29.66
19.4
22.93
35,22
16.56
28.05
42.57
13.92
18.61
28.16
13.08
STSGEN
17.48
29.21
16.78
21.19
33.65
13.9
24.26
39.03
10.21
19.13
31,05
12.68
STEGNN
16.77
26.28
16.3
20.48
32.51
16.77
23.46
36.6
9.21
17.13
26.8
10,96
AGCRN
15.98
28.25
15,23
19.83
32.3
12.97
22.37
36.55
9.12
16.94
26.25
10.6
SCInet
24.31
14.29
18.95
30.89
11,86
21.19
34.03
8.83
15.72
24.76
9.8
DSTAGNN
15.57
27.21
14.68
19.3
31.46
12.7
21.42
34.51
9.01
15.9
25.24
9.97
IRN
14.98
23.99
14.18
19.03
30.88
11.71
21.11
33.99
8.84
15.71
24.64
4.2 BASELINES
For each dataset, we compare ods (i.c.. LogTrans (Li et
IRN with the latest baselines: (1) For ETT, Transformer-based meth-
2019). Informer (Zhou al., 2021) Autoformer (Wu et al. [021]
Reformer (Kitaev et al. 22200) TST (eervass et al.) 2021). and Pyraformer (Liu et al. (2221b) and
feature representation learning based methods (i.e., TCC (Eldele et al.
(001). N-BEAT' (Oreshiin
et al. 2019)) CPC (Oord et al. 2018), Triplet (Franceschi et al. 2019). MoCo (He et al.) (020). TNC
(Tonekaboni et al., 2021), )), TS2Vec (Yue et 2022). SCInet et al. 2021a) and CoST (Woo et al.) 2022) ): (2) For PEMS, LSTM (Hochreiter & Schmidhuber] 1997), CNN-based methods (i.e., TCN and DCRNN (Li et al. (017)). SCInet, Graph-
methods (i.e., STGCN (Yu et al. 2017). AST- GCNr (Guo et al. 2019). STSGCN (Song et al. 2020), STFGNN (Li & Zhu 2021), AGCRN (Bai ctt al. 2020), and DSTAGNN (Lan et al. (002)): (3) For Solar Energy, Traffic, Electricity, and Ex- change Rate, AR, VAR-MLP (Zhang 2003), GP (Frigola 2015). GRU, L.TNNt (Lai al. 2018
TPA-LSTM (Shih et al. 2019). SCInet, and MTGNN (Wu et al. 2020).
4.3 EXPERIMENTAL RESULTS
We summarize the performances of IRN and baseline models in Table 2 to 5. IRN demonstrates state-of-the-at performances in 36 cases and near-best in 14 cases. Autoformer performs better for long-term forecasting in ETTh2 datasets as it shows strengths in reflecting trends and seasonality, which are more apparent in longer sequences (Wu et al. 2021). In similar vein, features are more
7

---

Under review as a conference paper at ICLR 2023
Table 5: Forecasting performance comparison of IRN and baseline models on the Solar-Energy, Traffic, Electricity, and Exchange-rate datasets. Best results are highlighted in bold.
Methods
Metnis
Trafi:
Electicity
Exchange
6
12
24
3
12
24
12
24
12
24
AR
CORR
RSE
0.3455
0:973
0:0263
0.379
0.5107
0.5911
0.5334
0.8699
0. 1.991
0.7752
0.7568
0.6218
0.7544
0.6252
0.7591
0.63
0. 0.8845
000995
0.1035
0.8633
0.3691
0.105
0.8595
0.1055
0.0228
0.9734
0.0279
0.9656
0.0353
0.9536
0.0445
0.9357
VARMLP
0OKK
RSB
0.922
0.1922
0.9655
0.26'9
0.4244
0:958
0.884
0.149
0 03245
0.558
0.659
0. 0.3895
0.6023
0.792
0.696
0.7911
0.80 8.7008
0.1395
0.8389
0.162
0.8192
0.1557
0.127
0.86/9
0.0055
03009
08825
0.0394
0.0407
0.828
0.0675
0.06 73
RSR
0.2399
0.398
0.5
0.993
08682
0.772
0.6005
5995
015
0.1907
0.162
0.123
00229
0.022
0.019
0.088
CORR
0.0751
0.0488
0.5518
0.597
07691
0.767
0.799
0.967
0.834
0.8818
0.844
0.87%
RNN.GRU
CORR
0.1932
0.9823
05675
0.163
0915
0.452
03882
0.358
0.522
0.562
08365
5633
0.5
0.1102
0.397
0.3185
0,3472
0.0636
LSTNat
CORR
0.9643
0.90
0.467
0.463
0.869
0.8614
0.588
0.9283
0.935
SCinet
CORR
0,9853
0.1775
0.9739
0.2301
0.2997
09112
0.416
0.892
0. 0.359
0.4414
0.4495
0,8325
0.4433
0.3192
0.0748
0.0845
0
0.0976
0.9274
0.01
TPA-LSTM
CORR
RSE
0.1881
6,985
0.9742
0.3547
0.3234
0.047
04459
09961
0.4487
0.812
0.8717
0.465
0.4541
0,8794
0.9629
04766
0.0823
0,9430
0.0916
0.9837
0.0964
0.925
0.913
0.1006
I.134
0.979
1L7719
0.341
0.0341
0J556
0.0444
0
MIGGNN
CORK
RSE
0.1778
0.9852
0.3348
0.9726
0.310
0,909
09001
0.27
0.4162
0,996
0.4754
0 0.856
0.4461
0.8794
0.435
0.581
0.0745
0
0.0878
0
0
0
0
0,923
0.0194
0.778
0.0359
0.908
0.0349
0.951
0.0456
0.9072
IRN
CORR
RSE
0.1720
0.9853
0.2292
0
0.2971
0.9556
0.460
0.9122
0.4171
0 0.990
0.439
0.8%61
0.449
0.8774
0.4449
0.8788
0.0739
0.9493
0.8844
0.939
0,9313
0
0
0
0.0179
0.9005
0.0246
0.9678
RJ337
019622
0,0441
0.9288
(4)
segjasore
(b)
Etnn scaMenoos 900 543
BV
TRLE
abal MAt MAA)
3.0
0.0
-22
-0.2
-0.4
-14
-2.
140
325
Ert Sequowoess 903 $4)
(d)
emni Sequenoes: 510 558
0.5
TRUR
0.6
Caseo
-02
-92
-0.4
20 67 08 380
320 140
20
47 82 333
323
142
Figure 5: Forecasting results of IRN and the base model from (a) sequence 499 to 547, (b) sequence 500 to 548, (c) sequence 501 to 549, (d) sequence 510 to 558 in Ettml data. The ground truths are shown in a solid line. The dotted and dashed lines represent the predicted values of the base model and IRN, respectively. The predicting region is shaded in grey. The bar graph shows the absolute MAE difference between the base model and IRN
evident in univariate time series, which explains the higher performances of MoCo (He et al. 2020) and CoST (Woo et al. 2022), which are feature representation learning models, on cases of ETT univariate datasets. MIGNN (Wu et al.] 2020). a model specialized for analyzing edge relations, yields the best performance on Traffic and Electricity datasets. This is because both datasets contain complex edge between nodes. At last, compared to the attention- based model, IRN shows lesser performances Exchange-rate dataset due the strong random-walk property of the time series (Wright 2008). Overall, our IRN successfully reflects irregularity representation and complements base model to achieve the higher forecasting performances.
4.4 ABLATION STUDY
We perform the ablation study to demonstrate the benefit obtained by IRB. We plot the ground truths and corresponding predictions of IRN and the base model at 499th, 500th, 501th, and 510th sequences of ETTh] data as shown Figure 5. In Figure 5 (a), the original time series has a peak in the predicting region shaded in grey. Up to sequence 499, IRN and the base model make similar predictions having large errors. When a sequence is added as in the Figure 5 (b), the discrepancy between the ground truth and the predicted values of IRN decreases. With additional sequence in the Figure 5 (c), IRN quickly reflects the change and makes a better forecast than the base model. It is observed that the base model is less sensitive the change of the input sequences, giving similar
8

---

Under review as a conference paper at ICLR 2023
(a)
(b)
6.0
FIV
020
mhu (bo
0.30
#D0E HAC MAR!
62%
0.12
70
60
150
372
.00
Figure 6: Forecasting results of (a) sequence 71 to 95 and (b) sequence 217 to 241 in traffic data using IRN and base model.
predictions from sequence 499 sequence 501. Only when 11 sequences have passed, the base model considers the actual changes as in Figure (d). We verify that IRN can reflect the irregular features (instantaneous changes).
Next, we observe the 24 horizon forecasts in Traffic dataset for further analysis. In Figure 6 (a), cycles 2 to consist values lower than 0.2, whereas cycle includes irregular values greater than 0.3. IRN has larger errors than the base model as IRN instantaneously reflects the irregularity. In contrast, IRN performs better than the base model when the irregularity persists as shown in Figure 6 (b). The reflection of the irregularity does not always end in a better forecast, but IRB consistently improves the forecasting performance of the base model, which confirms the effectiveness the our model.
4.5 OBSERVATION THE VARIATION OF THE IRREGULARITY
We further investigate how the perfor- mance of IRN changes with the irregular- ity of the time series. Ettm and Traffic datasets are decomposed into seasonality, trend, and irregularity as depicted in Fig- ure 1, The variation of the irregularity calculated and classified into two cases ac- cording to the degree of variation. Case 1 and Case 2 represent 500 data points with low variation of irregularity and high variation of irregularity, respectively. The difference of average Mean Square Error
Table 6: The difference of average MSE between IRN and base model according to the variation of irregular- ity on ETTml and Traffic datasets. Case and Case refer to 500 data points with low variation in irregular- ity and with high variation in irregularity, respectively.
MSE difference between IRN and base model
Dataset
Horizon
Case
Case 2
Etmm Traffic
48
-0.00097
0.001506
24
0.00709
0.00941
(MSE) between IRN and base model obtained for each case and listed in Table 6. This results indicate that higher performance improvement attained in case 2 than case 1 for both datasets, implying the higher the irregularity variation, the higher performance improvement be achieved.
CONCLUSION
In this paper, we propose Irregularity Reflection Neural Network (IRN), a deep learning based model for time series forecasting that reflects the irregularity in time series. We introduce novel expres- sion of irregularity based on Fourier series under microscopic perspective condition and employ it to design the Irregularity Representation Block (IRB) that captures, preserves, and learns the irreg- ularity representation of time series data. By embedding the IRB on the base model, IRN is further proposed. Experiments on a variety real-world datasets show that IRN can consistently outper- form existing state-of-the-a baselines. The ablation study confirm that the proposed methodology can reflect the irregularity. Accordingly, we argue that the irregularity representation is essential for improving performance of machine learning models.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sharif Atique, Subrina Noureen, Vishwajit Roy, Vinitha Subburaj, Stephen Bayne, and Joshua Mac- fic. Forecasting of total daily solar energy generation using arima: A case study. In 2019 IEEE 9th annual computing and communication workshop and conference (CCWC), pp. 0114-0119. IEEE, 2019.
Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting. Advances in Neural Information Processing Systems, 33:17804- 17815, 2020.
Peter Bloomficld. Fourier analysis of time series: an introduction. John Wiley & Sons, 2004.
Peter J Brockwell and Richard A Davis. Time series: theory and methods. Springer Science & Business Media, 2009.
Chao Chen, Karl Petty, Alexander Skabardonis, Pravin Varaiya, and Zhanfeng Jia. Freeway perfor- mance measurement system: mining loop detector data. Transportation Research Record, 1748 (1):96-102, 2001.
Yuxin Chen, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, and Weiming Hu. Channel-wise topology refinement graph convolution for skeleton- action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13359-13368, 2021.
Changqing Cheng. Multi-scale gaussian process experts for dynamic evolution prediction of com- plex systems. Expert Systems with Applications, 99:25-31, 2018.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Robert B Cleveland, William S Cleveland, Jean McRac, and Irma Terpenning. Stl: seasonal- trend decomposition. J. Off. Stat, 6(1 3:3-33, 1990.
David N DeJong, John Nankervis, N Eugene Savin, and Charles H Whiteman. Integration versus trend stationary in time series. Econometrica: Journal of the Econometric Society, pp. 423-433, 1992.
Haodong Duan, Yue Zhao, Kai Chen, Dian Shao, Dahua Lin, and Bo Dai. Revisiting Ekeleton-based action recognition. arXiv preprint 2rXhv:2104.13586 2021.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2106. 14112, 2021.
Jeffrey L Elman. Finding structure in time. Cognitive science, 14(2):19-211, 1990.
Zheng Fang, Qingqing Long, Guojie Song, and Kunqing Xie. Spatial-temporal graph ode networks for traffic flow forecasting. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pp. 364-373, 2021.
Jean- Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems, 32, 2019.
Roger Frigola. Bayesian time series learning with Gaussian processes. PhD thesis, University of Cambridge, 2015.
André Gensler, Janosch Henze, Bernhard Sick, and Nils Raabe. Deep learning for solar power fore- casting-an an approach using autoencoder and Istm neural networks. In 2016 IEEE international conference on systems, man, and cybernetics (SMC), pp. 002858-002866 IEEF, 2016.
Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial- temporal graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 922-929, 2019.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Rob J Hyndman and Anne B Koehler. Another look measures of forecast accuracy. International journal of forecasting, 22(4)x679-688, 2006.
Per Jonsson and Lars Eklundh. Seasonality extraction by function fitting time-series of satellite sensor data. IEEE transactions on Geoscience and Remote Sensing, 40(8):1824-1832 2002.
Nikita Kitaev, tukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04455 2020.
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41 International ACM SIGIR Conference on Research Development Information Retrieval, pp. 95-104, 2018.
Shiyong Lan, Yitong Ma, Weikang Huang, Wenwu Wang, Hongyu Yang, and Pyang Li. Dstagnn: Dynamic spatial-temporal aware graph neural network for traffic flow forecasting. In Interna- tional Conference on Machine Learning, pp. 1 1196--19917. PMLR, 2022.
Teng-You Yok Lee and Han-Wei Shen. Visualization and exploration of temporal trend relationships in multivariate time-varying data. IEEE Transactions on Visualization and Computer Graphics, 15 (6):1359- 1366, 2009.
Mengzhang Li and Zhanxing Zhu. Spatial-temporal fusion graph neural networks for traffic flow forecasting. Proceedings of the AAAI conference on artificial intelligence, volume 35, pp- 4189 4196, 2021.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances neural information processing systems, 32, 2019.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural net- work: Data-driven traffic forecasting. arXiv preprint arXiv:1707.0192: 2017.
Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: survey. Philosophical Transactions of the Royal Society A, 379(2119):222002099 2021.
Bo Liu, Jianqiang Li, Cheng Chen, Wei Tan, Qiang Chen, and MengChu Zhou. Efficient motif discovery for large-scale time series in healthcare. IEEE Transactions on Industrial Informatics, 11(3):583-590, 2015.
Chengxi Liu, Kenli Li. Jing Liu, and Cen Chen. Lhenn: novel efficient multivariate time series prediction framework utilizing convolutional neural networks. In 2019 IEEE 21 International Conference on High Performance Computing and Communications: IEEE 17th International Conference Smart City: IEEE 5th International Conference Data Science and Systems (HPCC/SmarCity/DSS)) pp. 2324-2332 IEEE, 2019.
Minhao Liu, Ailing Zeng, Zhijian Xu, Qiuxia Lai, and Qiang Xu. Time series is special sequence: Forecasting with sample convolution and interaction. arXiv preprint vrXiv:2106.09300 202 la.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: complcxyyy pyramidal attention for long-range time series modeling and fore- casting. In International Conference Learning Representations, 202

---

Under review as a conference paper at ICLR 2023
Yeqi Liu, Chuanyang Gong, Ling Yang, and Yingyi Chen. Dstp-rmn: A dual-stage two-phase ceantion-based recurrent neural network for long-term and multivariate time series prediction. Expert Systems with Applications, 143:1 13082, 2020.
Spyros Makridakis, Allan Andersen, Robert Carbone, Robert Fildes, Michele Hibon, Rudolf Lewandowski, Joseph Newton, Emanuel Parzen, and Robert Winkler. The accuracy of extrap- olation (time series) methods: Results of a forecasting competition. Journal of forecasting, 1()):
111-153, 1982.
Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in neural information processing systems, 27, 2014.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXfv:1609.03399 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03744. 2018.
Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural ba- sis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2019.
Daniele Ravi, Charence Wong, Benny Lo, and Guang-Zhong Yang. A deep learning approach to on-node sensor data analytics for mobile wearable devices, IEEE journal of biomedical and health informatics, 21(1):56-644 2016.
Rajat Sen, Hsiang-Fu Yu, and Inderjit S Dhillon. Think globally, act locally: A deep neural network approach digeeninnnso time series forecasting. Advances neural information processing systems, 32, 2019.
Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Ad- vances in neural information processing systems, 28, 2015.
Shun-Yao Shih, Fan-Keng Sun, and Hung-yi Lee. Temporal pattern attention for multivariate time series forecasting. Machine Learning, 108()):12111141, 2019.
Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 914-921, 2020.
Donghwan Song, Adrian Matias Chung Back, and Namhun Kim. Forecasting stock market indices using padding-based fourier transform denoising and time series deep learning models. IEEE Access, 9:83786-83799 2021.
Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician, 72(1):37-45, 2018.
Sana Tonekaboni, Danny Fytan, and Anna Goldenberg. Unsupervised representation learning for time series with temporal neighborhood coding. arXiv preprint 1eX.vv2206600777 2021.
Mascha Van Der Voort, Mark Dougherty, and Susan Watson. Combining kohonen maps with arima time series models to forecast traffic flow. Transportation Research Purt C: Emerging Technolo- gies, 4(5):307-318, 1996.
Kang Wang, Kenli Li, Liqian Zhou, Yikun Hu, Zhongyao Cheng, Jing Liu, and Cen Chen. Multiple convolutional neural networks for multivariate time series prediction. Neurocomputiig, 360:107- 119, 2019.
Billy M Williams and Lester A Hoel. Modeling and forecasting vehicular traffic flow as seasonal arima process: Theoretical basis and empirical results. Journal of transportation engineering, (296))6644-67 6722,662,266 2003.
12

---

Under review as a conference paper at ICLR 2023
Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi. Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting, arXiv preprint arXiv:2202.01575. 2022.
Jonathan H Wright. Bayesian model averaging and exchange rate forecasts. Journal of Economet- rics, 146(2):329 2008.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34, 2021.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Con- necting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 753-763, 2020.
Yongsang Yoon, Jongmin Yu, and Moongu Jeon. Predictively encoded graph convolutional network for noise-robust Akeleton-based action recognition. Applied Intelligence, 52(3):2317-23311 2022.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep 2017.
learning framework for traffic forecasting. arXiv preprint
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8980-8987, 2022.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Fick- hoff. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, pp- 2114-2124, 2021.
G Peter Zhang. Time series forecasting using a hybrid arima and neural network model. Neurocom- puling, 50: 159-175, 2003.
Pengfei Zhang, Cuiling Lan, Wenjun Zeng. Junliang Xing, Jianru Xue, and Nanning Zheng. semantics-guided neural networks for efficient skeleton-based human action recognition. In pro- ceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 1112-
1121, 2020.
2020.
Bendong Zhao, Huanzhang Lu, Shangfeng Chen, Junliang Liu, and Dongya Wu. Convolutional neural networks for time series classification. Journal of Systems Engineering and Electronics, 28(1):162-169, 2017.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI, 2021.
A LOSS FUNCTION
We cover stacked cases in which losses are accumulated. When the dataset has enough training data, we apply K layers (Liu al. (021a). To train the K stacked IRN for the k-th intermediate prediction, we compute the LI loss between the k-th prediction and the true value as follows:
h
L
V
where h the horizon size, le is the number of stacks, 1² is i-th horizon prediction of k-th stack, and y is the true value. We apply Equation I to calculate the LI loss of each stacked layer output. The total loss of the stacked IRN is expressed as:
L
O=
13

---

Under review as a conference paper at ICLR 2023
B IMPLEMENTATION DETAILS
Our model and framework are implemented with Pytorch. We train IRN with Adam optimizer by using NVIDIA 2080Ti 8 GPUs for enough batch size. Other parameters such learning rate, level, stack, single, and multi are changed according to the dataset charateristics and referring base model (Liu et al. 2021a).
C DATASETS AND METRICS
C. ELECTRICITY TRANSFORMER TEMPERATURE
ETT contains two-year electric power data gathered from two counties in China (hourly subsets ETTh1, ETTh2 and 15 minutes subsets ETTml). Each data point contains an oil temperature value and six power load components. The train, validation and test sets consist of 12, 4, and 4 months data, respectively. We implement zero-mean normalization for data pre-processing. Mean Absolute Errors (MAE) (Hyndman & Koehler 2006) and Mean Squared Errors (MSE) (Makridakis et al., 1982) are used evaluation metrics.
h
1
MAE
h
1=
MSE
Σ
( x4)
h
where o; is the true value, & the predicted value, and the prediction horizon size.
C.2
PEMS
PeMS consists of four public datasets (i.e., PEMSO3, PEMS04, PEMS07 and PEMS08), which are separately collected from Caltrans Performance Measurement System (PeMS) of four sections in California. The data is collected every five minutes. We predict one hour that consists 12 data points. The zero-mean normalization is applied for the data pre-processing. The evaluation metrics are MAE, Root Mean Squared Errors (RMSE) and Mean Absolute Percentage Errors (MAPE).
1
RMSE
)2
h
MAPE 1 h
(#:
T2
C.3 TRAFFIC, SOLAR ENERGY, ELECTRICITY AND EXCHANGE RATE
Traffic includes the hourly road occupancy rates which ranges from 0 to The sensors gather the road occupancy rates from 2015 to 2016. Solar Energy contains 2016 solar power production which are recorded every 10 minutes from 137 PV plants Alabama State. Electricity collects the hourly electricity consumption (kWh) of 321 clients from 2012 to 2014. Exchange-Rate consists of the daily exchange rates of 8 foreign countries from 1990 to 2016. For four datasets, the size of the lookback window is 168, and horizon sizes are 3,6.12, and 24. The evaluation metrics are Root Relative Squared Error (RSE) and Empirical Correlation Coefficient (CORR) (Lai et al. 2018
RSE Eh-o (2: 22)²
E Ciio (2)))(2:))2
14

---

Under review as a conference paper at ICLR 2023
1 d
Σ oo mean(z;))(eus - mean (ēz)
CORR= - Σ
7 j=0 Li=0(01, - () mean($;))
where d is the number of variates.
15


Reviews:
[{"review_id": "qMWH3Cpvn8D", "reviewer": "Reviewer_ppXw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary: While time-series components like trends or seasonality have been extensively studied and been modeled in existing literature,\n \tmodeling residual or irregularity has been an under explored research area due to its inherent complexity. \n\tIn this work, the authors propose a deep learning based model for time-series forecasting (TSF) that reflects this irregularity\n \tby introducing a novel expression for residual and employing it in TSF.", "summary_of_the_review": "Short review:\n1. The paper’s presentation is decent and comprehensible\n2. This work focuses on an under explored research area and shows great result when applied in real-world dataset\n3. The paper doesn’t show the best technical depth \n4. No link to the code available", "strength_and_weaknesses": "Strength:\n1. Residual modeling of time-series data is quite under explored which gives the research direction of this literature great value. \n2. Authors performed extensive experiments and ablation studies on number of real-world dataset to prove the validity of their approach\n\nWeakness:\n1. The authors didn’t provide any link to their code for reproducibility purpose, so it won’t be easy for peers to validate the experimental results\n2. The technical depth of the work is a tad limited", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Z5wetyEm_p", "reviewer": "Reviewer_Lsd6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new deep learning architecture for multi-horizon forecasting, which aims to improve temporal representation learning using a bespoke Irregularity Representation Block.", "summary_of_the_review": "While I think the results show initial promise -- clarity with regards to the model's terminology, motivations and the problem it is tackling would need to be improved before it can be accepted.", "strength_and_weaknesses": "Strengths\n---\n\nTime series forecasting is a crucial problem across many application domains – with many datasets exhibiting level-trend-seasonality relationships in general. The development of models which can capture these relationships better would be very impactful for said applications.\n\n\nWeaknesses\n---\n\nHowever, I do have several key concerns with the paper:\n\n1. The paper does adopt some seemingly non-standard terms, which can be confusing with similar terms in general time series forecasting/representation learning literature.\n\n     a.\tIrregularity – the paper frequently describes residuals as “the irregularity”, without defining the term concretely or citing its source. Easily confused with modelling irregularly sampled time series – e.g. ICML 2022’s Modelling Irregular Time Series with Continuous Recurrent Units.\n\n     b.\tReflection – appears to be used synonymously with “model”, and could be confused with concepts in vector transformations (https://en.wikipedia.org/wiki/Reflection_(mathematics)).\n\n2. I am not sure I fully understand the motivations behind the architecture\n\n     a.\tEven if we were to understand “irregularity” as “residual”, the underlying model itself appears to be feeding in the inputs as-is – without any residualisation with respect to level-trend-seasonal components as implied by the summary.\n\n     b.\tThe analogy of the IRN as a non-linear Fourier transform appears to breakdown when actual non-time inputs are used (i.e. x_input) – i.e. feeding a exogenous input into a sine wave would not necessarily make it a periodic basis function unless it is monotonic.\n\n     c.\tThe description of the Fourier transform is also slightly unclear in equation 3\n\n          i. Assuming $t_{a\\~b}$ is a single point in time -- as implied by “When the irregularity … is observed at moment $t_{a\\~b}$” – $min(t_{a\\~b})$ becomes unmeaningful as it would simply equal $t_{a~b}$. If it represents a sequence, then sin(…) becomes non-meaningful.\n\n     d. The parallels when we move into input space in equation 6 also require further explanation:\n\n         i. Does dim(x^{input}) have to match dim(x^{irregular})? What if we only have a single input?\n\n         ii. “Through this transformation, x^{irregular} is converted from time dependent to data dependent.”, but x^{input} does not appear to be a time index of any sort? Otherwise does dim d=1?\n\n     e. How is the IRB used to generate multi-step forecasts in the table?\n\n     f. How is hyperparameter optimisation performed for the model and benchmarks – details of which are crucial for reproducibility.\n\n3. Unclear problem formulation – while the paper seemingly implies that the problem being tackled is for improving multi-horizon forecasts for datasets with strong level/trend/seasonal relationships, the precise problem is never fully defined. This makes it difficult to determine the appropriateness of the proposed benchmarks – many of which come from long-term time series forecasting (e.g. LogTran/Reformer/Pyraformer/Autoformer). Even with the simpler models – e.g. LSTM/GRU/VAR-MLP etc – how were they applied for multi-horizon forecasts?\n\n     i. Assuming the paper is tackling models for enhanced level/trend/seasonality modelling – many existing papers do exist in the area that should be compared against (e.g ESRNN, Deep factors for forecasting etc) alongside simple econometric ETS models.\n\n     ii. Assuming the paper is tackling models which used frequency-domain-based representations to improve time series forecasting, it should be compared against models such at the FEDformer for consistency.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PotYogaR7c", "reviewer": "Reviewer_aUx9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a deep learning model for time series forecasting that reflects the irregularity in time series. Irregularity is based on Fourier series and and the authors employ it to design the Irregularity Representation Block  that captures, preserves, and learns the irregularity representation of time series data. Experiments on a variety of realworld datasets show the efficiency of the method.\n\n", "summary_of_the_review": "The paper is interesting overall with mathematical models helping to understand the main concepts. Several experiments are provided to show the efficiency of the method, but we do not have access to the code.", "strength_and_weaknesses": "Mathematical models are used to explain the methodology and extensive experiments on a variety of real word datasets are provided. \n\nThe code of the work is not available making difficult to evaluate if the result can be reproduced easily\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CEHznpxY-uM", "reviewer": "Reviewer_ZKbd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of time series forecasting.\nThe authors propose a model that combines convolutions and\nmin, max and average poolings, aiming to mimick a first order\napproximation of a Fourier transformation. In experiments\nthey show that their method outperforms existing baselines.\n", "summary_of_the_review": "I see three issues with the paper:\n\nw1. misses a closely related, even more simple work.\n\n  The DLinear model by Zeng et al. 2022 proposes an even simpler model with\n  comparable results to yours. What advantages does your model have?\n\nw2. how the proposed model mimicks the Fourier transform is not clear.\n\n  In eq. 9, you define a layer conv-relu-conv-tanh and claim that it approximates\n  the sine basis function. But there is nothing in your layer that would be geared\n  specifically to model a sine function, you could claim that it approximates any\n  function. To me it looks like your derivation breaks here.\n\nw3. the concept of irregularity and the derivation of the approximation \n  is unclear.\n\n  You claim that there is a third component in time series analysis besides\n  seasonality and trend, \"irregularity\". To me this looks wrong and seems to\n  misunderstand what a trend usually models in time series: often it is just\n  a linear trend, and then you are perfectly right, there is room for further\n  basis functions. But generally, trends can be any non-periodic basis functions,\n  e.g. exponentials or logarithms etc. Between periodic (\"seasonality\") and\n  non-periodic (\"trend\") basis functions I do not see a third type of basis\n  functions.\n\n  The basis function you model in the end (eq. 5) actually **is** a simple\n  seasonality, a single sine function.\n\n  The derivation of eq. 5 is problematic: the higher order terms do not\n  \"correspond to the vertical shift\", because they also are periodic. If you\n  want to make an approximation argument here, that **locally** the\n  higher order terms can be approximated by a constant, you need to be\n  more specific, esp. you need to say explicitly that n-th regular waves\n  are the waves with **slower** angular velocity (as your fig. 3a suggests, but\n  you never say that). However, in Fourier analysis by default basis functions\n  are order by **increasing** angular velocity for good reasons, because\n  only if you go to arbitrary high velocities you have arbitrary good\n  approximations.\n\n  So both, your concept of irregularity and the derivation of your layer\n  function look problematic to me.\n\n\nreferences:\n  Ailing Zeng, Muxi Chen, Lei Zhang, Qiang Xu:\n  Are Transformers Effective for Time Series Forecasting? CoRR abs/2205.13504 (2022)\n", "strength_and_weaknesses": "s1. very simple method.\ns2. promising experimental results.\n\nw1. misses a closely related, even more simple work.\nw2. how the proposed model mimicks the Fourier transform is not clear.\nw3. the concept of irregularity and the derivation of the approximation \n  is unclear.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject