DEPTS: Deep Expansion Learning for Periodic Time Series Forecasting
Authors: Wei Fan
Shun Zheng
Xiaohan Yi
Wei Cao
Yanjie Fu
Jiang Bian
Tie-Yan Liu
Venue: ICLR 2022 Spotlight (2022)

Abstract: Periodic time series (PTS) forecasting plays a crucial role in a variety of industries to foster critical tasks, such as early warning, pre-planning, resource scheduling, etc. However, the complicated dependencies of the PTS signal on its inherent periodicity as well as the sophisticated composition of various periods hinder the performance of PTS forecasting. In this paper, we introduce a deep expansion learning framework, DEPTS, for PTS forecasting. DEPTS starts with a decoupled formulation by introducing the periodic state as a hidden variable, which stimulates us to make two dedicated modules to tackle the aforementioned two challenges. First, we develop an expansion module on top of residual learning to perform a layer-by-layer expansion of those complicated dependencies. Second, we introduce a periodicity module with a parameterized periodic function that holds sufficient capacity to capture diversified periods. Moreover, our two customized modules also have certain interpretable capabilities, such as attributing the forecasts to either local momenta or global periodicity and characterizing certain core periodic properties, e.g., amplitudes and frequencies. Extensive experiments on both synthetic data and real-world data demonstrate the effectiveness of DEPTS on handling PTS. In most cases, DEPTS achieves significant improvements over the best baseline. Specifically, the error reduction can even reach up to 20% for a few cases. All codes for this paper are publicly available.

Published as a conference paper at ICLR 2022
DEPTS: DEEP EXPANSION LEARNING FOR PERIODIC TIME SERIES FORECASTING
Wei Fan ; Shunn Zheng', Xiaohan YI?, Wei Cao², Yanjie Fu' Jiang Bian", , Tan Liu? University of Central Florida "Microsoft Research wei weiffn@knights ucf edu, yanjie. fuêucf. edu, {shun. zheng, xiaohan. yi, wei. jiang. bian, tyliu/Jmdccoooott.com
ABSTRACT
Periodic time series (PTS) forecasting plays crucial role in a variety of industries to foster critical tasks, such as early warning, pre-planning, resource scheduling, etc. However, the complicated dependencies of the PTS signal on its inherent periodicity as well as the sophisticated composition of various periods hinder the performance of PTS forecasting. In this paper, introduce deep expansion learning framework, DEPTS, for PTS forecasting. DEPTS starts with a decou- pled formulation by introducing the periodic state hidden variable, which stimulates us to make two dedicated modules to tackle the aforementioned two challenges. First. we develop an expansion module on top of residual learn- ing to perform a layer-by-layer expansion of those complicated dependencies. Second, we introduce a periodicity module with a parameterized periodic fune- tion that holds sufficient capacity to capture diversified periods. Moreover, our two customized modules also have certain interpretable capabilities, such as at- tributing the forecasts to either local momenta global periodicity and charac- terizing certain core periodic properties, e.g., amplitudes and frequencies. Ex- tensive experiments both synthetic data and real- worldw data demonstrate the effectiveness of DEPTS on handling PTS. In most cases, DEPTS achieves sig- nificant improvements
the best baseline. Specifically, the error reduction can even reach up to 20% for a few cases. All codes are publicly available at https://githur:
INTRODUCTION
Time series (TS) with apparent periodic (seasonal) oscillations, referred to as periodic time series (PTS) this paper, is pervasive in wide range critical industries, such as seasonal electricity spot prices in power industry (Koopman et al. 2007), periodic traffic flows in transportation (Lippi)
al. 2013), periodic carbon dioxide exchanges and water flows sustainability domain (Seymour, 2001 Testaye et al. 2006). Apparently, PTS forecasting plays crucial role in these industries since it can foster their business development by facilitating variety of capabilities, including early warning, pre-plamning, and resource scheduling (Kahn) 2003 Jain 2017).
Given the pervasiveness and importance of PTS, two obstacles, however, largely hinder the perfor- mance of existing forecasting models. First, future TS signals yield complicated dependencies on both adjacent historical observations and inherent periodicity. Nevertheless, many existing stud- ies did not consider this distinctive periodic property (Salinas al.) 2020) Toubeau et al.) 2018 Wang et al., 2019; Oreshkin et al., 2020). The performance of these methods has been greatly restrained due its ignorance of periodicity modeling. Some other efforts, though explicitly intro- ducing periodicity modeling, only followed some arbitrary yet simple assumptions, such as additive or multiplicative seasonality, capture certain plain periodic effects (Holi 1957| 2004 Vecchia 1985b Taylor & Letham, 2018). These methods failed to model complicated periodic dependencies beyond much simplified assumptions. The second challenge lies that the inherent periodicity of typical real-world TS is usually composed of various periods with different amplitudes and fre- quencies. For example, Figure[]xxentn exxempliies the sophisticated composition of diversified periods via
Work is done during the internship Microsoft Research.

---

Published as a conference paper at ICLR 2022
20300
10000
2013:01-01
201 2013-11-65...
2015-33-41
2016-51-01
2012 01:01
2010-53-05
2019-32-01
2028-01-01
01
2013/07-01
2016-01-01
2015-02-01 2010-01-01
2019-01-08
Figure I: We visualize the electricity load TS in region of California to show diversified periods. In the upper part, we depict the whole TS with the length eight years, and in the bottom part, we plot three segments with the lengths of half year, one month, and one week, respectively.
real-world eight-years hourly TS of electricity load in a region of California. However, existing methods (Taylor& & Letham 2018 Smyl| 2020) required the pre-specification of periodic frequencies before estimating other parameters from data, which attempted to evade this obstacle by transferring the burden of periodicity coefficient initialization to practitioners.
To better tackle the aforementioned two challenges, we develop deep expansion learning frame- work, DEPTS, for PTS forecasting. The core idea of DEPTS is to build a deep neural network that conducts the progressive expansions of the complicated dependencies of PTS signals on periodicity to facilitate forecasting. We start from novel decoupled formulation for PTS forecasting by intro- make more
ducing the periodic state as hidden variable. This new formulation stimulates customized and dedicated designs to handle the two specific challenges mentioned above.
For the first challenge, we develop an expansion module on top of residual learning (He et al. 2016 Oreshkin et al., (2020) to conduct layer-by-layer expansions between observed TS signals and hidden periodic states. With such a design, we can build a deep architecture with both high capacities and efficient parameter optimization to model those complicated dependencies of TS signals on periodicity. For the second challenge, we build a periodicity module to estimate the periodic states from observational data. We represent the hidden periodic state with respect to time as a parameterized periodic function with sufficient expressiveness. In this work, for simplicity, we instantiate this function series of cosine functions. To release the burden of manually setting periodic coefficients for different data, we develop a data-driven parameter initialization strategy on top of Diserete Cosine Transform (Ahmed et al. 1974). After that, we combine the periodicity module with the expansion module to perform end-to-end learning.
To the best of knowledge, DEPTS is very early attempt to build a customized deep learn- ing (DL) architecture for PTS that explicitly takes account of the periodic property. Moreover, with two delicately designed modules, DEPTS also owns certain interpretable capabilities. First, the ex- pansions of forecasts can distinguish the contributions from either adjacent TS signals or inherent periodicity, which intuitively illustrate how the future TS signals may vary based on local momenta and global periodicity. Second, coefficients of the periodicity module have their own practical mean- ings, such as amplitudes and frequencies, which provide certain interpretable effects inherently.
We conduct experiments on both synthetic data and real-world data, which all demonstrate the su- periority of DEPTS on handling PTS. On average, DEPTS reduces the error of the best baseline by about 10%. In few cases, the error reduction can even reach up to 20%. Besides, we also include extensive ablation tests to verify our critical designs and visualize specific model components to interpret model behaviors.
2 RELATED WORK
TS forecasting is a longstanding research topic that has been extensively studied for decades. After a comprehensive review of the literature, we find three types of paradigms in developing TS models. At an early stage, researchers developed simple yet effective statistical modeling approaches, in- cluding exponentially weighted moving averages (Holt 1957 2004 Winters 1960), auto-regressive moving averages (ARMA) (Whittle] 1951; 1963), the unified state-space modeling approach as well as other various extensions (Hyndman & Khandakar, 2008). However, these statistical approaches only considered the linear dependencies of future TS signals on past observations. To handle high- order dependencies, researchers attempted to adopt a hybrid design that combines statistical mod-
eling with more advanced high-capacity models MMntteo--anns al. 2020:
|myyl 2020). At the
2

---

Published as a conference paper at ICLR 2022
same time, with the great successes of DL in computer vision (He et al. 2016) and natural language processing (Vaswami et al. (017), various DL models have also been developed for TS forecast-
ing (Rangapuram et al., 2018 Toubeau al. 2018 Salinas et al, 2020; Zia & Razzaq 2020 Cao et al. 2020). Among them, the most representative one is N-BEATS (Oreshkin et al.) (020). which pure DI. architecture that has achieved state-of-the-arr performance across wide range of benchmarks. The connections between DEPTS and N-BEATS have been discussed in Section 4.2
As for PTS forecasting, many traditional statistical approaches explicitly considered the periodic property, such as periodic ARMA (PARMA) (Vecchia 1985a by and its variants (Tsstaee et al.
2006 Anderson et al., 2007, Dudek et al., 2016). However, as discussed Sections [ and 3 these methods only followed some arbitrary yet simple assumptions, such as additive or multiplicative seasonality, and thus cannot well handle complicated periodic dependencies in many real-world scenarios. Besides, other recent studies either followed the similar assumptions for periodicity or required the pre-specification of periodic coefficients (Taylor & Letham 2018 Smyl| 2020). To the
best of our knowledge, we are the first work that develops a customized DL architecture to model complicated periodic dependencies and to capture diversified periodic compositions simultaneously.
PROBLEM FORMULATIONS
We consider the point forecasting problem of regularly sampled uni-variate TS. Let Te denote the time series value at time-step t, and the classical auto-regressive formulation is to project the historical observations Tt-La 26-1 into its subsequent future values Te .
Ura+ Fo(x; + Eci+H
where H is the length of the forecast horizon, I, is the length of the lookback window, Fe RR - RR is a mapping function parameterized by θ, and E = [et..... H-+ denotes a vector of independent and identically distributed Gaussian noises. Essentially, the fundamental assumption behind this formulation is the Markov property Tt.+H 1 Z0.-- - which assumes that the future values H2:++H are independent of all farther historical values Tu:t- L given the adjacent short- term observations 20 Lat-- Note that most existing DL models (Salinas et al. 2020) Toubeau et al.) 2018 Wang al., 2019 Oreshkin al. 2020) directly follow this formulation
solve TS. Even traditional statistical TS models (Holt) 1957] 2004 Winters] 1960) are indeed consistent with that omitting those long-tail exponentially decayed dependencies introduced by moving averages.
To precisely formulate PTS, on the other hand, this assumption needs to be slightly modified such that the dependency of Fe++H on L;-L:t is further conditioned on the inherent periodicity, which can be anchored by associated time-steps. Accordingly, we alter the equation ( into
IEit = (ZI + Cc:c+Hf
(2)
where other than Lat F R R R takes extra argument 1, which denotes the forecasting time-step. Existing methods for PTS adopt a few different instantiations of F For example, [Holi (1957] (2044) developed several exponentially weighted moving average processes with additive or multiplicative seasonality. Vecchia (1985a) b) adopted the multiplicative seasonality by treating the coefficients of the auto-regressive moving average process as time dependent. [Smyl (2020) also adopted the multiplicative seasonality and built a hybrid method by coupling that with recurrent neural networks (Hochreiter & Schmidhuber 1997), while Taylor & Letham| (2018) chose the additive seasonality by adding the periodic forecast with other parts as the final forecast.
DEPTS
In this section, we elaborate our framework, DEPTS. First, we start with a decoupled formu- lation of (2) in Section 4.1 Then, we illustrate the proposed neural architecture for this formulation
in Sections 4.2 and 4.3 Last, we discuss the interpretable capabilities in Section 4.4
4.1 THE DECOUPLED FORMULATION
To explicitly tackle the two-sided challenges of PTS forecasting, i.c., complicated periodic depen- dencies and diversified periodic compositions, introduce decoupled formulation that refines
3

---

Published as a conference paper at ICLR 2022
AVHLE
Layer
Perledi-Mack (iui
Layer
Time
DEPTS
Local Blcck
Periodicity Module
Ayy
Layer
Expansion
Roce
Figure 2: In the right part, we visualize the overall data flows for our framework, DEPTS. In the mid- dle part, we plot the integral structure of three layer-by-layer expansion branches in the expansion module fa. In the left part, we depict the detailed residual connections within single layer.
by introducing hidden variable 24 to represent the periodic state at time-step
Bet+ = To Lit, 2t L+++H) + 6::+ 2 gt(():
where we treat 2 E TR. scalar value to be consistent with the uni-variate TS J. E R!, we use fe : R² x IRL + _R" to model complicated dependencies of the future signals Drs++ on the local observations Lt-L: and the corresponding periodic states Lt-L:tH within the lookback and forecast horizons, and Js :R' + R is to produce a periodic state 2, for a specific time-step t. deepiess the overall data flows of this formulation, in which the expansion module fa and the periodicity module Iø are responsible for handling the two aforementioned PTS- specific challenges, respectively.
The right part of Figure depicts
4.2 THE EXPANSION MODULE
To effectively model complicated periodic dependencies, the main challenge lies in the trade-off between model capacity and generalization. To avoid the over-fitting issue, many existing PTS ap- proaches relied on the assumptions of additive or multiplicative seasonality (Holi 1957 Vecchia (8856)Anddesse et al. 2007 Taylor & Letham 2018 Smy] 2020), which however can hardly press periodicity beyond such simplified assumptions. Lately, residual learning has shown its great potentials in building expressive and generalizable DL architectures for variety of crucial applica- tions, such as computer vision (He et al. Specifically. N-BEATS (Oreshkin et al. 2020) conducted pioneer demonstration of introducing
2016 and language understanding (Vaswani et al. 2017).
residual learning to TS forecasting. Inspired by these successful examples and with full considera- tion of PTS-specific challenges, develop novel expansion module to characterize the complicated dependencies of ota++ on æ L.t and z L.t+H.
top of residual learning
The proposed architecture for fe. as shown in the middle part Figure 2 consists of N layers in total. As further claboration in the left part of Figure 2 each layer f, share an identical residual struc- ture consisting of three residual branches, which correspond to the recurrence relations of (t) 2 and H after Ê-layers expansions, and (f) Lt-Ltt denote the residual terms of æt-Lat +it++ denotes the cumulative forecasts after layers.
(2) x L-t L.c. and (f) t. respectively. Here x and (f)
In layer f, three residual branches are specified by two parameterized blocks, local block félce) and a periodic block fh (0)) where (f( and (e) are their respective parameters.
First, we present the updating equation for (f-1) H1 which aims produce the forecasts from periodic states and exclude the periodic effects that have been used. To be more concrete, for takes
in z LiL+ and emits the ε-th expansion term of periodic states, denoted as v.(0) V L. E L+ (2)
(E-1)
o L has two components, a backeast component v P/-L:t and forecast one t We leverage (2)
((-1) and adopt (()) v as the portion of forecasts purely
e L.t to exclude the periodic effects from æ;
from the l-th periodic block. Besides, when moving to the next layer, exclude ()()) L from
4

---

Published as a conference paper at ICLR 2022
((-1) (0) L La+ 2t-L L t L encourage the subsequent periodic blocks to (2)
(t-1) (6)
focus on the unresolved residue x L
Then, since t is related to the periodic components that have been used to produce part of forecasts, we construct the input to (e(()) as (x,-La v(-L:))- Here the purpose is to en- courage (0 to focus the unresolved patterns within Lt-Lit" (e, emits ui-Lt and (2) Wi++H which correspond the local backcast and forecast expansion terms of the f-th layer, respec- tively. After that, we update Fi-Lot L:t by further subtracting 41 from (x) v ( = x((--)
(f-1)
(6)
as 10 viLLa Lat Here the insight is also to exclude all analyzed patterns of
(6)
(f-1)
(8)
(8)
T L- =
this layer to let the following layers focus on unresolved information. Besides, we update a If (6) by adding both 11++ and v Uit+H as F(atH #b++ Ua+H + 1 The motivation of (4) such expansion is to decompose the forecasts from the f-th layer into two parts, u If and vas+ which correspond to the part from local observations excluding redundant periodic information and the other part purely from periodic states, respectively.
(E)
Note that before the first layer, we have (2(0) = Li-L: L--LI+ = - and E (())
+i++ O.
Besides, we collect cumulative forecasts # of the N-th layer as the overall forecasts &
Therefore, after stacking N layers of 2 - -.+++ +--+:: L,t and a:(0) + we have the following triply residual expansions that encapsulate the left and middle parts of Figure
(1)
(0)
(6) La+H H
(N)
L:-- 21-L:tH
2
Li-Lit = 2-
(0)
u u Lit + + + +
(4)
(N)
(8) (144 DeL+H).
(2)
where 2, L and 2 deemed be the residues irrelevant forecasting.
(N)
(N)
Connections and differences to N-BEATS. Our design of fo shares the similar insight with N- BEATS (Oreshkin et al. 2020), which stimulating deep neural network to learn expansions of raw TS signals progressively, whereas N-BEATS only considered the generic TS by modeling the dependencies of Fi:++ on Ltt- contrast, our design is to capture the complicated depen- dencies of T1:+H on æt-L:t and bt-L:+ for PTS. Moreover, to achieve periodicity modeling, N-BEATS produces coefficients solely based on the input signals within lookback window for group of predefined seasonal basis vectors with fixed frequencies and phases. However, work can capture diversified periods in practice and model the inherent global periodicity.
Inner architectures local and periodic blocks. The local block Seicee aims produce a part of forecasts based on the local observations excluding redundant periodic information as (æt- Lat D (0) (. Thus, we reuse the generie block developed by Oreshkin et al. (2020), which consists of series of fully connected layers. As for the periodic block "o which handles the relatively stable periodic states, we can adopt simple two-layer perception. Due to the space limit, we include more details of inner block architectures in Appendix A
4.3 THE PERIODICITY MODULE
To represent the sophisticated periodicity composed of various periodic patterns, we estimate via a parameterized periodic function (t) that holds sufficient capacities to incorporate diversified periods. In this work, for simplicity, we instantiate this function as a series of cosine functions as
(+(() Ap EK Aş cos(2 (2 Fgt Pk), where K is hyper-parameter denoting the total number
5

---

Published as a conference paper at ICLR 2022
of periods, Ag is scalar parameter for the base scale, Ag. Fg. and Ps are the scalar parameters for the amplitude, the frequency, and the phase of the k-th cosine function, respectively, and ø represents the set of all parameters. Coupling Ie with fo illustrated in Section 4 we can effectively model the periodicity-awanl auto-regressive forecasting process in the equation ( However, it is extremely challenging to directly conduct the joint optimization of & and & from random initialization. The reason is that in such a highly non-convex condition, the coefficients in ø are easily trapped into numerous local optima, which do not necessarily characterize our desired periodicity.
Parameter Initialization. To overcome the optimization obstacle mentioned above, we formalize a two-stage optimization problem based on raw PTS signals to find good initialization for & First, we construct a surrogate function, g (t) = Au + Ex=1
k=1 M₁ A2 Fgt + Pk), to enable the selection of a subset of periods via M = ( MM k E {1.... K}), where each Mg. E {0, 1 is a mask variable to enable or disable certain periods. Note that (t) is equivalent to gø (t) when every
K
M₁ is equal to one. Then, construct the following two-stage optimization problem:
M arg min CD..r (9 (t)), 6" = arg min DD (go(t)),
[|M]]
where L. and D.... denote the discrepancy losses on training and validation, respectively; the inner stage is to obtain o² that minimizes the discrepancy between 2; and 2; the training data Drraini the outer stage is a binary integer programming on the validation data Deal to find M that can select certain periods with good generalization, and the hyper-parameter J controls the maximal number of periods being selected. With the help of such two-stage optimization, we are able to estimate generalizable periodic coefficients from observational data good starting point for & to be jointly optimized with 0. Nevertheless, it is still costly to perform exact optimization of equations 5 in practice. Thus, we develop fast approximation algorithm to obtain acceptable solution with affordable costs. Our approximation algorithm contains the following two steps: 1) conducting Discrete Cosine Transform (Ahmed al., 1974) of PTS signals on Dtrain to select top- K cosine bases with the largest amplitude as an approximated solution of ø"; 2) iterating over the selected K cosine bases from the largest amplitude to the smallest one and greedily select J periods that generalize well on the validation set. Due to the space limit, we include more details of this approximation algorithm in Appendix B After obtaining approximated solutions 3² and M we fix M = M to exclude those unstable periodic coefficients and initialize & with ø to avoid being trapped into bad local optima. Then, we follow the formulation to perform the joint learning of & and 0 in an end-to-end manner.
4.4 INTERPRETABILITY
Owing to the specific designs of fe and ga. our architecture is born with a degree of interpretabil- ity. First, for fe, shown in equations (4). we decompose 0c:c+ into two types of components,
(2) u E+++ H and v U- Note that ttt+ is conditioned on L and independent of æt-Lat- Thus,
(E)
E
N v 114H represents the portion of forecasts purely from periodic states. Mcanwhile, (f)
depends on both Pt- Lat and I1 and transformed by feeding the subtraction of V Lct (f-1) 2 Let into the f-th local block. Thus, we can regard Ee u Lat as the forecasts from the local historical observations excluding the periodic effects, referred to as the local momenta in this paper. this way, we can differentiate the contribution to the final forecasts into both the global pe- riodicity and the local momenta. Second, Is. the periodicity estimation module in our architecture, also has interpretable effects. Specifically, the coefficients in (f) have practical meanings, such as amplitudes, frequencies, and phases. We can interpret these coefficients the inherent properties of the series and connect them to practical scenarios. Furthermore, by grouping various periods together, ds provides us with the essential periodicity of TS filtering out various local momenta.
(2)
from x
EXPERIMENTS
Our empirical studies aim to answer three questions. 1) Why is it important model the complicated dependencies of PTS signals on its inherent periodicity? 2) How much benefit can DEPTS gain for PTS forecasting compared with existing state-of-thh-art models? 3) What kind of interpretability can

---

Published as a conference paper at ICLR 2022
Linear
Quadratic
Cubic
0.085
0.042
0.12
0.080
0.040
0.075
N-BEATS
DEPTS
N: BEATS
DEPTS
N-BEATS
DEPTS
Figure 3: Performance comparisons of N-BEATS and DEPTS (ours) on synthetic data, in which we simulate different periodic dependencies, such as linear, quadratic, and cubic.
DEPTS offer based on our two customized modules, fo and gs? To answer the first two questions, conduct extensive experiments both synthetic data and real-world data, which illustrated in Section 5.1 552 respectively. Then, Section answers the third question by comparing and interpreting model behaviors for specific cases.
Baselines. We adopt the state-of-the-art DL architecture, N-BEATS (Oreshkin et al. 2020), as our primary baseline since it has been shown to outperform wide range of DL models. including MatFact (Yu et al. (2)66). Deep State (Rangapuram et al., 2018). Deep Factors (Wang et al. 2019 and DeepAR (Salinas et al. 2020), and many competitive hybrid methods (Montero-Manne Manso et al., 2020 Smyl 2020). Besides, we also include PARMA as reference to be aware of the positions of conventional statistical models, For this baseline, we leverage the AutoARIMA implementation provided by Löning et al. (2019) to search for the best configurations automatically.
compare different models, we utilize the following two metrics, normal- ized deviation, abbreviated nd, and normalized root-mean-soee square error, denoted as nrmse, which
Evaluation Metries.
are conventionally adopted by Yu et al.] (2016); Rangapuram et al. (2018); Salinas et al. (2020);
Oreshkin et al. (2020) on PTS-related benchmarks.
[ E Eeneen |at 1--$j ofl
nrmse FT En,oen =:)²
nd
(6)
[ Eo.nen
Euneen |=:|
where is the index of TS in a dataset, is the time index, and Ω denotes the whole evaluation space.
5.1 EVALUATION ON SYNTHETIC DATA
To intuitively illustrate the importance of periodicity modeling, we generate synthetic data with various periodic dependencies and multiple types of periods. Specifically, we generate a simulated TS signal z; by composing an auto-regressive signal 4, corresponding to the local momentum, and compounded periodic signal pt, denoting the global periodicity, via a function f as ff. Pt), which characterizes the dependency of s4 on 4 and pt. First, we produce 4 via an auto-regressive process, 4 = -EL= El aqlei + c,, in which a; is a coefficient for the i-lag dependency, and the error term ei ~ ~N o¹) follows a zero-mean Gaussian distribution with standard deviation o'_ Then, we produce pt by sampling from another Gaussian distribution N(z,, d²), in which 2, is characterized by a periodic function (instantiated as I in Section 4.3). and oº is standard deviation to adjust the degree of dispersion for periodic samples. Next, we take three types of fl pt). (le + Pt), (4e + p,)² and (4 + p;) 3 to characterize the linear, quadratic, and cubic dependencies of on l and Pt, respectively. Last, after data generation, all models only have access to the final mixed signal I, for training and evaluation.
Due to the space limit, include the main results in Figure and leave finer grained parameter specifications and more experimental details to Appendix C For each setup (linear, quadratic, cubic) Figure 3 we have searched for the best lookback length (I.) for N-BEATS and the best number of periods (J) for DEPTS on the validation set and re-run the model training with five different random seeds to produce robust results on the test set. We can observe that for all cases, even with an exhaustive search of proper lookback lengths for N-BEATS, there exists considerable performance gap between and DEPTS, which verifies the utility of explicit periodicity modeling. Moreover, as the periodic dependency becomes of DEPTS over N- N-BEATS keeps increasing (from 7% to 11%), which further demonstrates the importance of modeling high-order periodic effects.
complex (from linear to cubic). the average error reduction

---

Published as a conference paper at ICLR 2022
Table 1:
Performance comparisons (nd) on ELECTRICITY, TRAFFIC, and M4 (HOURLY). For the first two, we follow two different test splits defined in previous studies.
ELECTRICITY
TRAFFIC
M4 (HOURLY)
Model 2014-09-01
2014-12-25
0008-06-15
2009-03-24
MatFact
0.16
0.255
0.20
0.187
n/a
DeepAR
0.07
n/a
0.17
n/a
0.09
Deep State
0.083
n/a
0.167
n/a
0.044
N-BEATS
0.064
0.171
0.114
0.112
0,023
DEPTS
0.060
0.139
0.111
0.107
0.021
Table 2: Performance comparisons (nd and nrmse) on CAISO and NP, where we define four test splits to cover all four seasons of the last year for each benchmark.
2020-01-01
2020-04-01
2020-07-01
2020-10-01 nd
Dataset
Model
nd
nrsse
nd
nrmse
nd
nrmse
nrmse
PARMA
0.089
0.169
0.107
0.214
0.116
0.215
0.079
0.148
CAISO
N-BEATS
0.029
0.058
0.031
0.073
0.030
0.064
0.026
0.057
DEPTS
0.024
0.049
0.028
0.063
0.029
0.058
0.020
0.042
PARMA
0.220
0.350
0.201
0.321
0.216
0.352
0.199
0.305
NP
N-BEATS
0.207
0.434
0.154
0.237
0.195
0.315
0,211
0.332
DEPTS
0.196
0.377
0.145
0.224
0.169
0.269
0.179
0.281
5.2 EVALUATION ON WORLD DATA
Other than simulation experiments, further demonstrate the effectiveness of DEPTS real- world data. We adopt three existing PTS-related datasets, ELECTRICITY TRAFFIC and M4 (HOURLY which contain various long-term (quarterly, yearly), mid-term (monthly, weekly), and short-term (daily, hourly) periodic effects corresponding to regular economic and social activities. These datasets serve as common benchmarks for many recent studies Yu et al. 2016 Rangapuram et al. 2018 Salinas et al. 2020 Oreshkin et al. 2020). For ELECTRICITY and TRAFFIC, we follow two different test splits defined by (Sllinas et (2020) and Yu et (206)), and the evaluation hori- zon covers the first week starting from the split date. As for M4 (HOURLY),
adopt the official test set. Besides, we note that the time horizons covered by these three benchmarks are still too short, which results in very limited data being left for periodicity learning if we alter the time split too early. This drawback of lacking enough long PTS limits the power of periodicity modeling and thus may hinder the research development in this field. To further verify the importance of peri- odicity modeling in real-world scenarios, we construct two new benchmarks with sufficiently long PTS from public data sources. The first one, denoted CAISO, contains eight-years hourly actual electricity load series in different zones of California f The second one, referred to as NP, includes eight-years hourly energy production volume series multiple European countrie Accordingly, we define four test splits that correspond to all four seasons of the last year for robust evaluation.
For all benchmarks, we search for the best hyper- parameters of DEPTS on the validation set. Similar to N-BEATS (Oreshkin et al. (2020), we also produce ensemble forecasts of multiple models trained with different lookback lengths and random initialization seeds. Tables and 2 show the overall performance comparisons. On average, the error reductions (nd) of DEPTS over N-BEATS on ELECTRICITY, TRAFFIC, M4 (HOURLY), CAISO, and NP are 12.5%. 3.5%, 8.7%, 13.3%, and 9.9%, respectively, Interestingly, we observe some prominent improvements few specific cases,
8

---

Published as a conference paper at ICLR 2022
PARMA
DCPTSP
SEPTLL
933-
200
5.75
0,375
5.000
0.000
0.250
0.005
2.02
0.000
Figure 4: We compare the forecasts of different models (in the left side) and visualize the intermedi- ate states within DEPTS (in the middle and right parts), where DEPTS-P denotes the forecasts from the global periodicity, DEPTS-L denotes the forecasts from the local momenta, and DEPTS the summation of these two parts, as illustrated in Section 4.2
such as 18.7% in ELECTRICITY (2014-09-01). 23.1% in CAISO (2020-10-01) and 15.2% in NP (2020-10-01). At the same time, we also observe some tiny improvements, such as 2.6% in TRAFFIC (2008-06-15 15) and 3.3% in CAISO (2020-07-01). These observations imply that the predictive abilities and the complexities of periodic effects may vary over time, which corresponds to the changes in performance gaps between DEPTS and N-BEATS. Nevertheless, most of the time, DEPTS still brings stable and significant performance gains for PTS forecasting, which clearly demonstrate the importance of periodicity modeling practice.
Due to the space limit, we leave more details about datasets and hyper-parameters used in real-world experiments to Appendix D] Moreover, to achieve effective periodicity modeling, we have made several critical designs, such the triply residual expansions in Section 4.2 and the composition of diversified periods in Section 4.3 We also conduct extensive ablation tests to verify these critical designs, which are included in Appendix H
5.3 INTERPRETABILITY
In Figure we illustrate the interpretable effects of DEPTS via two cases, the upper one from ELECTRICITY and the bottom one from TRAFFIC. First, from subplots in the left part, we observe that DEPTS obtains much more accurate forecasts than N-BEATS and PARMA. Then, in the middle and right parts, we can visualize the inner states of DEPTS to interpret how it makes such forecasts. As Section states, DEPTS
differentiate the contributions to the final forecasts t into
the local momenta Σ H and the global periodicity Σ/1 v 2 H- Interestingly, we can see that DEPTS has Icarned two different decomposition strategies: 1) for the upper case, most of the contributions to the final forecasts come from the global periodicity part, which implies that this case follows strong periodic patterns; 2) for the bottom case, the periodicity part just characterizes major oscillation frequency, while the model relies more on the local momenta to refine the final forecasts. Besides, the right part of Figure 4 depicts the hidden periodic state 2, estimated by our periodicity module (t). We can see that (t) indeed captures some inherent periodicity. Moreover, the actual PTS signals also present diverse variations at different time, which further demonstrate the importance of leveraging So to model the dependencies of # on both 2;- L.t and L.t We include more case studies and interpretability analysis in Appendix
6 CONCLUSION
In this paper, we develop novel DL. framework, DEPTS, for PTS forecasting. Our core contri- butions are to model complicated periodic dependencies and to capture sophisticated compositions of diversified periods simultaneously. Extensive experiments on both synthetic data and real-world data demonstrate the effectiveness of DEPTS on handling PTS. Moreover, periodicity modeling is actually old and crucial topic for traditional TS modeling but is rarely studied in the context of DL. Thus we hope that the new DL framework together the two new benchmarks with evident periodicity and sufficiently long observations facilitate more future research on PTS.

---

Published as a conference paper at ICLR 2022
REFERENCES
Nasir Ahmed, T- Natarajan, and Kamisetty R Rao. Discrete cosine transform. JEEE Transactions on Computers, 100(1):90-99) 1974.
Paul L Anderson, Yonas Gebeyehu Testaye, and Mark M Meerschaert, Fourier-parma models and their application to river flows. Journal of Hydrologic Engineering, 12(5):462-472, 2007.
Defu Cao, Yujing Wang. Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bix- iong Xu, Jing Bai, Jie Tong, et al. Spectral temporal graph neural network for multivariate time- series forecasting. Advances neural information processing systems, 33:17766-17778, 2020.
Anna E Dudek, Harry Hurd, and Wioletta Wójtowicz. Periodic autoregressive moving average methods based on fourier representation of periodic coefficients. Wiley Interdisciplinary Reviews: Computational Statisties, 8(3):130-149, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pailern recognition, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Charles C Holt. Forecasting trends and seasonal by exponentially weighted moving averages. ONR Memorandum, 52(2), 1957.
Charles C Holt. Forecasting seasonals and trends by exponentially weighted moving averages. International Journal of Forecasting, 20(1):5-10, 2004.
Rob Hyndman and Yeasmin Khandakar. Automatic time series forecasting: the forecast package for r. Journal of statistical software, 27(1):1-22 2008
Chaman L Jain. Answers to your forecasting questions. The Journal of Business Forecasting, 36(1): 3, 2017.
Kenneth B Kahn. How to measure the impact of forecast error on an enterprise? The Journal of Business Forecasting, 22(1):21, 2003.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.660 6980, 2014.
Siem Jan Koopman, Marius Ooms, and M Angeles Carnero. Periodic seasonal reg-arfima-gan garch models for daily electricity spot prices. Journal of the American Statistical Association, 102 (477):16-27, 2007.
Marco Lippi, Matteo Bertini, and Paolo Frasconi. Short-term traffic flow forecasting: An experimen- tal comparison of time-series analysis and supervised learning. IEEE Transactions Intelligent Transportation Systems, 14(2):871-882 2013.
Markus Löning, Anthony Bagnall, Sajaysurya Ganesh, Viktor Kazakov, Jason Lines, and Franz Király, sktime: A unified interface for machine learning with time series. arXiv preprint SSt1::190900872... 2019.
Pablo Montero-Manso George Athanasopoulos, Rob Hyndman, and Thiyanga S Talagala. Fforma: Feature-based forecast model averaging. International Journal of Forecasting. 36(1):86-92, 2020.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning, 2010.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural ba- sis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2020.
10

---

Published as a conference paper at ICLR 2022
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting, Advances neural information processing systems, 31:7785-7794, 2018.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic fore- casting with autoregressive recurrent networks. International Journal of Forecasting, 36(3):1181 1191, 2020.
Lynne Seymour. overview of periodic time series with examples. IFAC Proceedings Volumes, 34(12):61-66, 2001.
Slawek Smyl. A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecasting, 36(1):75-8)) 77585, 2020.
Sean J Taylor and Benjamin Letham. Forecasting at scale. The American Statistician, 72(1):37-45, 2018.
Yonas Gebeyehu Tesfaye, Mark M Meerschaert, and Paul Anderson. Identification of periodic autoregressive moving average models and their application to the modeling of river flows. Water resources research, 42(1), 2006.
lean-François Toubeau, Jérémie Bottieau, François Vallée, and Zacharie De Grève. Deep learning- based multivariate probabilistic forecasting for short-tems scheduling in power markets. 1EEE Transactions on Power Systems, 34(2): 1203-122,,, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
AV Vecchia. Maximum likelihood estimation for periodic autoregressive moving average models. Technometrics, 2754):375-384, 1985a.
AV Vecchia. Periodic autoregressive-moviin average (parma) modeling with applications to water resources. JAWRA Journal of the American Water Resources Association, 21(5):721-730 1985b.
Yuyang Wang, Alex Smola, Danielle Maddix, Jan Gasthaus, Dean Foster, and Tim Januschowski. Deep factors for forecasting. In International Conference on Machine Learning, pp. 6607-6617. PMLR, 2019.
Peter Whittle. Hypothesis testing in time series analysis. Almqvist Wiksells boktr., 1951.
Peter Whittle. Prediction and regulation by linear least-square methods. English Universities Press, 1963.
Peter R Winters. Forecasting sales by exponentially weighted moving averages. Management sci- ence, 6(3):324 342, 1960.
Hsiang-Fu Yu, Nikhil Rao, and Inderjit S Dhillon. Temporal regularized matrix factorization for high-dimensional time series prediction. Advances in neural information processing systems, 29: 847-855, 2016.
Tehseen Zia and Saad Razzaq. Residual recurrent highway networks for learning deep sequence prediction models. Journal of Grid Computing, 1811)169-176, 2020.
11

---

Published as a conference paper at ICLR 2022
A BLOCK ARCHITECTURES
As illustrated in Section 4.2 the local block f ()) produces forecasts based on local observed PTS signals excluding redundant periodic effects. This goal aligns with that of N-BEATS to extract in- formative representations from generic TS signals. Therefore, we reuse the generic block design of N-BEATS to instantiate fey Here we include brief description of the local block for complete- ness. Please refer to Section 3.1 in (Oreshkin al. 2020 for more details.
(f--1)
(f-1) 1)
T-Lit
FC Layers
(4 layers)
FC Layer (1 layer)
FC
FC
(?)
(2)
cD
Linear
Linear
h (cp))
(t)
h (c.
Periodic Block
Local Block
(O)
( u 1-lst
5 24 EE+ H
Ltt
tx+H
Figure 5: Detailed architectures of the local block and the periodic block in DEPTS.
Local Block.
The left part of Figure 5 shows the detailed architecture within a local block, where we use o Lat
(f-1) excluding -1-1 l,t v to denote the portion of the local observations Φ;-
the periodic effects U for the l-th layer. After taking in 2 we pass it through four fully-
connected layers and then obtain the backcast coefficients c/ and the forecast coefficients of via two linear projections:
(f),1 (0
(6)2 = FC FCr (6).1
(0.3 = FCe,3 (9). (-)
u L.t
t-Ltt u
i-Lt
(0)4
(0)3
(6) (I (0),
of LINEAR (u,-L.)
(9.4
where FC is a standard fully-comnected layer with ReLU activation (Nair & Hinton 2010). and LINEAR denotes a linear projection function. Then, we pass these coefficients to the basis layers, (.)) and he (-), to obtain the backcast term u and the forecast term W respectively. The generic choice for h²()) can simply be another linear projection function, which is also adopted by us since it produces more competitive and stable performance on PTS-related benchmarks than other interpretable basis layers, shown by (Oreshkin al. 2020 in Appendix C.4.
(E)
Periodic Block. The periodic block fi aims extract predictive information from associated periodic states, which are relatively simple and stable compared with rapidly shifting PTS signals. Therefore, we adopt simple architecture while still maintain desired effects. In this work, we use lny-r standard fully-connected layer to encode zt-Li
(f-1) and leverage another two linear
(t) projection functions to obtain the backeast term e 1 La and the forecast term v #1++ as the periodic effects of the f-th layer.
(0.1 Lat=
FC2 (f-1) (C
0.0 U
LINEAR
(0.1 (1)
O0 614H = LINEAR (o (0.1
(1)44H)
12

---

Published as a conference paper at ICLR 2022
where FC and LINEAR share the same meanings mentioned above. Moreover, when training for multiple series simultaneously, we use series-specific scalar parameter 0s (i is the series index) to
(8) take account of differences in the strengths of periodicity by updating v La a; tt
B PARAMETER INITIALIZATION FOR THE PERIODICITY MODULE
As illustrated Section 4.3 we leverage a fast approximation approach to obtain an acceptable solution of the two-stage optimization problem (5) with affordable costs in practice. Algorithm summarizes the overall procedure for this fast approximation.
Algorithm 1: Parameter initialization for the periodicity module.
Input:
Deraen = Zo.T, vall= = TT,:T. K, and
Conduct DCT over Fo:T. Sort the top-K' cosine bases by amplitudes in descending order to obtain 3- U (Át, Fx, PøJK=1-
Initialize M = 0.
for j in [1,.... K] do
|M]]| ||| < J then
Update Mj by arg migmin M, C (0 (f))
else
return 3. and M
end end
Output: 3z and M
First, we divide the whole PTS signals 20. into the training part Dirain Bg.T, and the validation part Deal = TT,TT, where Tu is the split time-step. Then, the inner optimization stage is to identify the optimal parameter set ø² that can best fit the training data:
K
6² arg min (t), (t = Ao + Σ A cos(2 + Pk).
where the pyper--arameter K controls the capacity of go(t) and the discrepancy training loss Derwin can be instantiated the mean square error -i--1 (1)(() xz1... Directly optimizing 7 via gradient descent from random initialization is inefficient and time- tone-consuming since it involves numerous gradient updates and is easily trapped into bad local optima. Fortunately, instantiation of (t) as a group of cosine functions shares the similar format with Discrete Cosine Transform conduct DCT over æp.r, and select top-K K cosine bases with the largest amplitudes, which characterize the major periodic oscillations of this series, as the approximated solution Ø+ of
(DCT) (Ahmed et al.) 1974). Accordingly,
Next, enter the outer optimization stage to select certain periods with good generalization:
M arg min LD (2) (1) AØ Mr AG cos (20 PE),
|M]k==
where the ayper-parameter... further constrains the expressiveness of g (t) for good generalization. Conducting exact optimization of this binary integer programming is also costly since it involves an exponentially growing parameter space. Similarly, to capture the major periodic oscillations as much as possible, we develop a greedy strategy that iterates the selected cosine bases from the largest amplitude to the smallest and greedily assigns 0 to Mj depending whether the k-th period further reduces the discrepancy loss on the validation data. Specifically, assuming periods already sorted by their amplitudes descendingly and are indexed by h; construct another
13

---

Published as a conference paper at ICLR 2022
M/ (t) for the j-th greedy step: surrogate function 96-
M3 (L) = MM A3 cos (2 Pÿ) + Ag + Σ Mi 96-
A c2 Pk)
where { {Mgg is determined in previous steps, M, is an integer parameter to be set in the current step. Thus, for the j-th step, we are actually updating Mj by
Mf arg min CD.. (9g. M, (t)).
(10)
M2
Besides, to tolerate the approximation errors introduced by お", which may result in shifted periodic oscillations, we use Dynamic Time Warping to measure the discrepancy of g (t) and xy on Deat- We continue this greedily updating steps until selecting periods in total or completing the traverses of all K selected periods. Finally, obtain approximated solution M of (8).
My
Complexity Analyses. We also provide the complexity analyses of Algorithm [ which runs very fast in practice and takes up negligible time compared with training neural networks. Let us denote the length of training series as L, and the length of validation series as L,. First, the complexity of conducting DCT over training series is ([,)) Then, the complexity of selecting top-K frequencies with the largest amplitudes is O(Llogg( K)), which can be ignored since K << L,- Next, we need to select at most J frequencies greedily based the generalization errors on the validation set. Since we measure the generalization errors via dynamic time warping, the total worst complexity for this selection procedure is ((K In total, the worse complexity of our approximation algorithm for series is (L,logg(zz L.) + KL2). In practice, Lus the length of the validation series, is relatively small, and K, the maximum number of frequencies, can be regarded as a constant. So, the squared complexity term (TT)) is not a big trouble.
C MORE DETAILS ON SYNTHETIC EXPERIMENTS
As Section 5.1 states, we produce TS 4 via an auto-regressive process, le - EtEn ashi= d. in which a; coefficient for the i-lag dependence. and the error term et NO, o') follows a zero-mean Gaussian distribution with standard deviation of. Specifically, we set L as 3 and o' as We leverage uniform samples from 1] to initialize {a and also uniformly sample three values from J0,5 5 for the initial points, 1.3, 12,, and 1-1. Then, we produce pt by sampling from another Gaussian distribution N N(z,, in which 2, is characterized by a periodic function (instantiated (() in Section(4.3) and oº is standard deviation to adjust the degree of dispersion for periodic samples. Specifically, we also set σº as and produce 2, via a composition of three cosine bases, (cos(2x(t + 2)/50). 4cos(2 + 3)/10), (cos(2xt/4) and a base level, 30. These three cosine bases represent long-term, mid-term, short-term periodic effects, respectively, which very similar to the cireumstance in practice. Next, take three types of (4,,,))). (4 Pt), (4e + p²) and (1, + )3, to characterize the linear, quadratic, and cubic dependencies of da on l and pt. respectively. We repeat the above procedure for 5000 time steps and divide them into 4000, 100, and 900 for training, validation, and evaluation, respectively. Figure the first 1000 time steps of these synthetic series. Note that after data generation, all models only have access to the final mixed signal 22 for training and evaluation.
Moreover, as illustrated in Section 5.1 we search for the best loobkack length (L) for N-BEATS and the best number of periods (J) for DEPTS. The lookback length for DEPTS is fixed as 3, which is also determined by hyper ppprr-paraetter tuning on the validation set. Figure shows detailed comparisons of N-BEATS and DEPTS for different configurations of L and J. We can see that N-BEATS always needs a relatively long lookback window, such as 48 or 96 time steps, to capture those periodic patterns effectively. Besides, further increasing the lookback length will introduce more irrelevant noises, which overwhelm effective predictive signals and thus result in more worse performance. In contrast, with effective periodicity modeling, DEPTS can achieve better perfor- mance by using a short lookback window, which is also consistent with the regresssee process that governs the local momenta.
14

---

Published as a conference paper at ICLR 2022
Auto-regressive Signal
0.0
-2.5
200
400
600
800
1000
Periodic Signal pr:
40
20
200
400
1000
Linear Dependence: p.
40
20
200
460
Quadratic Dependence: x, = (e, 102
2000
1000
WN
200
400
600
800
1000
Cubic Dependence: x,
100000
50000
WN
200
400
600
800
1000
Figure 6: Synthetic Data.
D MORE DETAILS ON REAL- EXPERIMENTS
D.1 DATASETS
Table includes main statisties of the five datasets used by experiments. We can that the existing datasets (ELECTRICITY, TRAFFIC, and M4 (HOURLY)) utilized by recent studies usually have large number of series but with relatively short lengths. Therefore, it is hard identify or evaluate yearly or quarterly periods on these benchmarks. In contrast, CAISO and NP contain tens of series with the lengths of several years, which can better illustrate the inherent periodicity of these series and serve as complementary benchmarks for PTS modeling.
Table 3: Dataset statisties.
Dataset
ELECTRICITY
TRAFFIC M4 (HOURLY)
CAISO
NP
# Series
370
963
414
Frequency
hourly
hourly
hourly
hourly
hourly
Start Date
2012-01-01
2008-01-09
n/a
2013-01-01
2013-01-01
End Date
2015-01-01
2009-03-31
n/a
2020-12-31
9020-12-31
Min. Length
4008
10560
700
37272
69984
Max. Length
26304
10560
960
70128
70128
Avg. Length
24556
10560
854
54259
70120
Max. Value
764500
1.0000
352000
49909
27513
Avg. Value
2378.9
0.0528
1351.6
5582.4
4671.4
15

---

Published as a conference paper at ICLR 2022
Linear
0.050
0.045
0.040
N-BEATS
N-BEATS
N-BEATS L-48
N-BEATS L-96
N-BEATS L-192
N-BEATS L-384
N-BEATS L-768
DEPTS
DEPTS
DEPTS J-9
DEPTS
L-12
L-24
Quadratic
0.11
0.10
2 0.09
0.08
N-BEATS
N-BEATS
N-BEATS L-48
N-BEATS L-96
N-BEATS L-192
N-BEATS L-384
N-BEATS L-768
DEPTS /-3
DEPTS J-6
DEPTS J-9
DEPTS J-12
L-12
L-24
Cubic
0.16
0.15
0.14
0.13
0.12
0.11
N-BEATS
N-BEATS
N-BEATS
N-BEATS L-96
N-BEATS
N-BEATS L-384
N-BEATS 768
DEPTS J-3
DEPTS J-6
DEPTS J-9
DEPTS J-12
L-12
2-24
L-48
L-192
Figure 7: Performance comparisons of N-BEATS and DEPTS with different lookback lengths (L) and number of periods (J).
Table 4: slpper-parameters of N BEATS CAISO and NP.
Dataset
CAISO / NP
Split
2020-01-01
2020-04-01
2020-07-01 2020-10-01
Iterations
4000 12000 sMAPE
Loss
Forecast horizon (H)
24
Lookback horizon
2H, 3H, 4H, 5H, GH 7H
Training horizon
720H (most recent points before the split)
Layer number
30
Layer size
512
Batch size
1024
Learning rate
le-3 1e-6
Optimizer
Adam (Kingma & Ba 2014)
D.2 HYPER-PARE HYPER-PARAMETTES
its default hyper-parameter for ELECTRICITY, TRAFFIC, and M4 (HoURLY), and we report its hyper-parameters searched on CAISO and NP in Table 4 Besides, N-BEATS used multiple loss functions, such sMAPE or MASE, for model training, and also follow these setups. Tables JablesSJand@jncclud the hyper-parameters of DEPTS for all five datasets. Note
For N-BEATS,
https://ggitee whw.. com/E1 ementAl /N- BEATS
16

---

Published as a conference paper at ICLR 2022
Table 5: Hyper-parameters of DEPTS ELECTRICITY, TRAFFIC, and M4 (HOURLY).
Dataset
ELECTRICITY
TRAFFIC
M4 (HOURLY)
Split
2014-09-01
2014-12-25
2008-06-15
2009-03-24
Iterations
72000
12000
Loss
sMAPE
MASE 48
Forecast horizon (H)
Lookback horizon
211 3H, 4H, 5H, 611, 7/1
4H, 511 611, 7H
Training horizon
10/1
4
32
K
128
Layer number
30
Layer size
512
Batch size
1024
Learning rate (fe)
lc-3
Learning rate (gx)
5e-7
Optimizer
Adam (Kingma & Ba 2014)
Table 6: Hyper-parameters of DEPTS CAISO and NP.
Dataset
CAISO / NP
Split
2020-01-01
2020-04-01
2020-07-01 2020-10-01
Iterations
4000 / 12000
Loss
sMAPE
Forecast horizon (H)
Lookback horizon
2H,3H, 4H, 5H, 6H, 7H
Training horizon
720H
J
8/ 8
32/8 8
32/33 32
8 32
K
128
Layer number
30
Layer size
512
Batch size
1024
Learning rate (fe)
le-3 le-6 5e-7
Learning (g₂)
Optimizer
Adam Kingmaa & Ba 2014)
that, all these hyper- parameersss are searched on validation set, which is defined as the last week before the test split. Moreover, for typical dataset with multiple series, we build an independent periodicity module ds for each series and perform respective parameter initialization procedures as described in Appendix B Then, for all datasets (splits), we train 30 models (6 lookback lengths x 5 random seeds) for both N-BEATS and DEPTS and then produce ensemble forecasts for fair and robust evaluation.
E ABLATION TESTS
As Figure shows, adopt three ablated variants of DEPTS to demonstrate critical designs in the expansion module (Section 4.2
DEPTS-1: removing the residual connection of (x/_1)) - so that the outputs the local block (5) are only conditioned on the raw PTS signals -t- which correspond to the mixed observations of local momenta and global periodicity.
removing the residual connection of (a((-1) w((+H) that the contribu- tions to the forecasts only come from the local block, which takes in the signals excluding periodic effects progressively.
DEPTS-2:
17

---

Published as a conference paper at ICLR 2022
DEPTS-3: removing the residual connection of (t-1)
L+++ 0 ---La+ so that the inputs
La++
to the periodic block of each layer are the same hidden variables 2 La+
We also construct another four baselines to demonstrate the importance of our customized periodic- ity learning:
NoPeriod: removing the periodic blocks by directly feeding (æ, ze) to N-BEATS.
RandInit:
randomly initializing periodic coefficients (o) and directly applying the end-to-
end learning.
FixPeriod:
fixing the periodic coefficients (Ø) after the initialization stage and only tuning / during end-to-end optimization.
MultiVar:
treating 2t as a covariate of 2y and feeding (az, ze) into an N-BEATS-style model via two channels.
Moreover, as illustrated in Section 4.3 and Appendix B the maximal number of selected periods is critical hyper-parameter balance expressiveness and generalization of go Thus, we conduct experiments with different J to verify its sensitivity on different datasets. Tables 7 and include experimental results of these model variants on ELECTRICITY, TRAFFIC, CAISO, and NP with different J. Since only identify one reliable period via Algorithm Don M4 (HOURLY), we report results separately in Table
Layer
Periodic Block
Periodic Black
12
Local Block
Local Block
feet
M
(a) DEPTS
(b) DEPTS-1
Layer
Layer
Perlodic
Local Block
Local Block
xifl
D
(d) DEPTS-3
The residual structures of DEPTS and its three ablated variants, where the dashed line denotes the removed connection.
Figure 8:
First, as Tables 7 and & show, J is a crucial hyper-parameter that has huge impacts on forecasting performance. The reason is that if / is too small, the periodicity module Jo cannot produce effective representations of the inherent periodicity to boost the predictive ability. While is too large, Je has high risk of over-fitting the irrelevant noises contained by the training data, which also sults in poor predictive performance. Moreover, the interactions between local momenta and global periodicity may vary over time. Therefore, it critical search for proper J for each PTS and
18

---

Published as a conference paper at ICLR 2022
Table 7: Performance comparisons of DEPTS-1, DEPTS-2, DEPTS-3, and DEPTS.
Model
ELECTRICITY
TRAFFIC
CAISO
NP
2014-12-25 nd
2009-03-24
2020-10-01
10200-0-00
nrmse
nd
nrmse
nd
numse
nd
nrmss
4
DEPTS-1
0.15870
0,97571
0.11167
0.40790
0.02429
0.05184
0.19818
0.31224
DEPTS-2
0.15391
0.99258
0.10786
0.39716
0.02190
0.04402
0.19213
0.30317
DEPTS-3
0.14955
0,96602
0.10784
0.39811
0.01951
0.04087
0.19201
0,30469
DEPTS
0.14931
0.96488
0.10745
0.39730
0.02061
0.04373
0.19128
0,30381
8
DEPTS-1 1 DEPTS-2
0.15632
0.98683
0.11108
0.40529
0.02256
0.04634
0.19194
0.30167
0,15070
0,97949
0.10688
0.39421
0.02103
0.04329
0.18412
0.28983
DEPTS-3
0.14908
0,96270
0.10714
0.39687
0.02017
0.04532
0.18618
0,29525
DEPTS
0.14929
0.95627
0.10653
0.39567
0.02008
0.04176
0.18475
0,29214
16
DEPTS-1
0.14954
0.92162
0.11216
0,40335
0.02419
0.05041
0.18740
0.29442
DEPTS-2
0.14719
0.95648
O. 0.10
0.39448
0.02236
0.04672
0.18124
0,28434
DEPTS-3
0.14742
0.94554
0.10678
0.39444
0.01991
0.04384
0.18180
0,28786
DEPTS
0.14653
0,94929
0.10770
0.39554
0.02116
0.04276
0.18095
0,28620
32
D
0,14730
0,90305
0.11425
0.39968
0.02476
0.05193
0.18445
0.28860
DEPTS-2
0.14765
0.95478
0.11061
0.39699
0.02171
0.04526
0.18024
0,28110
DEPTS-3
0.14179
0.90319
0.10801
0.39403
0.01975
0.04270
0.18057
0,28355
DEPTS
0.13915
0.87498
0.11076
0.39453
0.02156
0.04446
0.17885
0.28031
Table 8: Performance comparisons of NoPeriod, RandInit, FixPeriod, MultiVar, and DEPTS.
Model
ELECTRICITY
TRAFFIC
CAISO
NP
2014-12-25
2009-03-24
2020-10-01 nd
2020-10-00 nd
nd
nrmse
nd
nrmse
nrmse
nrmse
4 NoPeriod
0.20615
1,27117
0.11829
0.40465
0.08195
0.14208
0.27128
0.40491
RandInit 0.17677
1.07514
0.11051
0.40383
0.02504
0.05293
0.20869
0.32853
FixPeriod
0.16756
0.99876
0.10816
0.39833
0.02282
0.04576
0.20145
0.31604
MultiVar
0.15743
1.02039
0.10733
0.39635
0.02018
0.04038
0.19998
0.31393
DEPTS 0.14931
0.96488
0.10745
0.39730
0.02061
0.04373
0.19128
0.30381
8
NoPeriod
0.23969
1.47537
0.11940
0.40536
0.08182
0.15585
0.24796
0.37781
RandInit 0.17463
1.05695
0.11065
0.40500
0.02639
0.05680
0.20972
0.33044
FixPeriod 0.16431
0.98414
0.10796
0.39764
0.02228
0.04661
0.20163
0.31666
MultiVar 0.15482
0.99266
0.10667
0.39599
0.02160
0.04855
0.19494
0.30513
DEPTS 0.14929
0.95627
0.10653
0.39567
0.02008
0.04176
0.18475
0.29214
16
NoPeriod
0.26851
1.65571
0.12203
0.40410
0.07275
0.15280
0.23281
0.34943
RandInit 0.18167
1,08529
0.11048
0.40287
0.02496
0.05347
0.20917
0.32936
FixPeriod 0.15792
0.95356
0.10803
0.39699
0.02138
0.04417
0.20293
0.31963
MultiVar
0.15479
0.98805
0.10724
0.39366
0,02289
0.05023
0.19045
0.29784
DEPTS 0.14653
0.94929
0.10770
0.39554
0.02116
0.04276
0.18095
0.28620
32
NoPeriod
0.31358
1,73706
0.12835
0.40741
0.07696
0.15433
0.22503
0.34109
RandInit
0.19399
1.14278
0.11075
0.40082
0.02577
0.05723
0.20916
0.32944
FixPeriod 0.15539
0.92896
0.10862
0.39637
0.02055
0.04115
0.20272
0.31940
MultiVar 0.16405
1.01227
0.10907
0.39596
0.02159
0.04582
0.18844
0,29294
DEPTS 0.13915
0.87498
0.11076
0.39453
0.02156
0.04446
0.17885
0.28031
each split point to pursue better performance. Fortunately, demonstrate that the hyper- parameer tuning of J on the validation set can ensure its good generalization abilities on the subsequent test horizons.
Then, us focus on Table 7 to compare DEPTS with DEPTS-1, DEPTS-2, and DEPTS-3, First, see that DEPTS-1 usually produces the worst performance in most cases, which demon-
19

---

Published as a conference paper at ICLR 2022
Table 9: Overall ablation studies on M4 (HOURLY).
DEPTS-1
DEPTS-2
DEPTS-3
DEPTS
nd nemse
0.03712
0.02161
0.02252
0.02050
0.22785
0.07710
0.08851
0.06872
NoPeriod
RandInit
FixPeriod
MultiVar
nd
0.03848
0.02401
0.02526
0.02937
nrmse
0.16568
0.09192
0.11221
0.14130
strates that excluding periodic effects from raw PTS signals can stably and significantly boost the performance of PTS forecasting. Second, for most cases, DEPTS outperforms DEPTS-2, and the performance gaps can be remarkable, such as 0.139 vs. 0.148 on ELECTRICITY and 0.020 vs. 0.021 on CAISO. These results verify the importance of including the portion of forecasts solely from the periodicity module. Third, DEPTS-3 can produce competitive results compared with DEPTS in many cases. Nevertheless, after selecting the best J for each dataset, DEPTS still slightly out- performs DEPTS-3 in most cases. Besides, from Table | much better than DEPTS-3 M4 (HOURLY). Thus we retain the residual connection reduce the periodic effects leveraged by previous layers.
we also observe that DEPTS performs
Next, let focus Table compare DEPTS with other four baselines, NoPeriod, RandInit, MultiVan and FixPeriod. First, we observe that NoPeriod usually produces the worst preformance.
The reason is that (xz=2e) denotes the raw time-series signal subtracting the periodic effect, so is challenging for the model to forecast the future signals, Tes+H solely based on the periodicity- agnostic inputs, (= Second, RandInit also produces much more worse results than DEPTS, which demonstrate the importance of initializing periodic coefficients (Section 4.3). Third, DEPTS performs much better than FixPeriod in most cases, which demonstrate the effectiveness of fine-tuning periodic coefficients after the initialization stage. Last, we observe that sometimes Mul- tiVar can produce comparable and even slightly better results than DEPTS. However, after selecting the best // each dataset for these two models, we find that DEPTS still outperforms MultiVar consistently and significantly, which also demonstrates the superiority of our expansion learning. Moreover, Table 9 shows, DEPTS outperforms all these baselines by large margin M4 (HOURLY), which containing very short PTS with 854 observations on average. Given limited data, all our critical designs, such properly initializing periodic coefficients, fine-tuning periodic co- efficients, and conducting expansion learning to decouple the dependencies of I; on 2,, play much erucial roles in producing accurate forecasts.
F MORE CASE STUDIES AND INTERPRETABILITY ANALYSES
In the following. we further study the interpretable effects of DEPTS with more cases. Fig- ures 19, [0] I1 and 12 show the additional two cases on ELECTRICITY, TRAFFIC, CAISO, and NP, respectively. Following Figure 4 we compare the forecasts of N-BEATS and DEPTS in the left side, differentiate the forecasts of DEPTS into the local part (DEPTS-L) and the periodic part (DEPTS-P) in the middle side, and plot the hidden state 2, together with the PTS signals in the right side. The general observations are that with the help of explicit periodicity modeling, DEPTS achieves better performance than N-BEATS PTS forecasting, and DEPTS has learned diverse behaviors for dif- ferent cases. Besides, also include their critical periodic coefficients (amplitude Ag. frequency Fk. and phase P&) in Tables 10 periods that consistent with practical domains.
12 and [3] We find that DEPTS can learn many meaningful
20

---

Published as a conference paper at ICLR 2022
Labdl
N-BBATS
DEPTS
DEFTS
DEPTS-P
DEPTS
1250
1400
Electriitt 1e224
1200-
1000
1200
800
500
800
250
100
150
BEATS
DEPTS
DEPTS
DEPTS
DEPTS-L
Label
800
800
800
Electrickt 18235
600
600
600
400
400
400
200
200
200
0.
0:
50
160 150
50
100 150
0
100 150
Figure 9: We show two some inherent periodicity, the real PTS signals usually have various irregular oscillations at different time steps, while DEPTS can produce periodicity simultaneously. For these two cases with evident and stable periodicity, DEPTS relies more the periodic forecasts (DEPTS-P) and thus achieves more competitive and stable results.
ELECTRICITY dataset. It is clear to see that other than following
stable forecasts by analyzing local momenta and global
Table 10: Periodic coefficients of the ELECTRICITY examples shown in Figure 9 We find that DEPTS has learned both short-term and long-term periods, such as three hours (11/Fx x 3), six hours (11/Fe) x 6), 12 hours (11/Fxl 12), one day (1/Fkk 24), and half year (1/FMl R 4380), which are very similar to the patterns of electricity utilization in practice.
ELECTRICITY
id 224
id 235
A
11/Pa
PA
|Ag
[1/Fx
P&
362.601
23.995
0.088
160.144
23.997
0.092
196.829
8320.428
0.422
77.804
8256.523
0.451
87.138
4470.598
0.487
36.918
23.969
0.092
66.418
24.035
0.122
19.517
23.921
0.102
52.736
11.999
0.052
17.714
4.800
0.024
44.248
23.920
0.096
12.810
11.993
0.054
27.172
6.000
0.027
11.964
6068.298
0.684
23.220
6.001
0.030
11.186
3.000
0.015
21

---

Published as a conference paper at ICLR 2022
Lab
N-BEATS
DEPTS
DEPTS
DEPTS
DEPTS
0.20
0.20
0.15
0.100
0.15
0.075
Treette
0.10
0.10
0.050
0.05
0.025
0.05
0.00
0.000
0.00
150
Label
0.08
0.08
0.06
0.06
0.04
Traffic
0.04
0.04
0.02
0.00
50
100
150
0.00
50
100 150
0.00
50
100
150
Figure 10: We show two quite different periodic effects. For the upper case, there are unexpected peaks at different time steps. For the bottom case, there different types of periodic oscillations. Similar cases in Figure 9] DEPTS has estimated roughly consistent periodic states 2; and then combined DEPTS-P and DEPTS-L to produce stable and accurate forecasts.
TRAFFIC dataset. We
that DEPTS is able to characterize
Table 11:
Periodic coefficients of the TRAFFIC examples shown in Figure 10 DEPTS has also learned multiple types of periods.
We find that
TRAFFIC
id 398 [1/Fkl
id 532
|4
PP
[Ag
[1/Fkl
|&&l
0.0231
23,993
0.233
0.0267
23.987
0.209
0.0066
164.055
2.293
0.0122
3192.218
0.527
0.0062
1845.505
2.226
0.0104
4906.324
0.382
0.0054
11.999
0.200
0.0066
24.270
0.434
0.0046
8.003
0.205
0.0048
1265.645
0.330
0.0045
23,920
0.234
0.0039
4.801
0.114
0.0044
28.097
0.282
0.0032
23.637
0.332
0.0038
12.011
0.513
0.0030
28.053
0.248
22

---

Published as a conference paper at ICLR 2022
Labal
BEATS
DEPTS
DEPTS
OEPTSIP
DEPTSLL
40000
40000
40000
id4
30000
35000
35000
20000
30000
25000
25000
20000
20000
0-
50
100
150
50
100
150
50
150
Labal
BEATS
DEPTS
DEPTS
DEPTS-P
DEPTSIL
Lobel
4:
20000
20000
mmw
20000
CAISO
17500
15000
17500
15000
10000
5000
10000
10000
Figure 11: We show two cases on CAISO dataset. These two cases present relatively regular oscil- lations, and thus N-BEATS with enough lookback lengths can also produce pretty good forecasts. Even though, DEPTS can better capture the curves of future PTS signals by modeling the depen- dencies of them on estimated periodicity. We can see that DEPTS first relies on the periodic part (DEPTS-P) to form the basic shape of forecasts and then leverages the forecasts from the local part (DEPTS-L) stretch condense the forecasting curve.
Periodic coefficients of the two CAISO examples shown in Figure 11 Other than daily and yearly periods, which are observed similarly in ELECTRICITY and TRAFFIC cases, we find that DEPTS has identified some weekly periods (11/ Fil s 168) for two cases CAISO.
Table 12:
CAISO
id 4
id
[&
11/Fa
PP
|Ae
1 FA
PP
3411.731
24.004
0.000
1851.303
24.002
0.307
3207.279
8344.825
0.081
1754.576
8629.984
0.606
1712.536
4509.138
0.042
720.007
23.934
0.704
1493.276
23.934
0.000
625.465
4299.993
0.334
1434.023
23,992
0.000
536.312
167.907
0.369
1225.926
9408.412
0.086
452.345
11.999
0.086
963.309
24.062
0.000
409.348
24.018
0.323
854.321
168.236
0.001
326.875
24.069
0.215
23

---

Published as a conference paper at ICLR 2022
Labdl
N-B6AT5
DEPTS
DEFTS
DEPTS-P
DEPTS
1250-
1500
1000
1500
MM
(d]
1000
750
1000
500
500
250
500
100
100
150
N-GEATS
DEPTS
DEPTS
DEPTS-P
DEPTS
Lakel
9000
id10
0000
6000
8000
7000
4000
7000
6000
2000
6000
5000
5000
50
100 150
0
50
100
150
0
50
1.00
150
Figure 12: We show two cases on NP dataset. We can see that these cases are rather difficult, and both N-BEATS and DEPTS struggle to make sufficiently accurate forecasts. Nevertheless, as shown in the right side, DEPTS has relatively stable estimation of the future trending and thus can obtain relatively good performance in forecasting future curves.
We can see that the dominant periods belong to the long-term type, which characterizes the overall variation but omits those local volatile oscillations. Since this dataset contains massive noises in local oscillations, in some splits, N-BEATS even produces forecasts that are inferior to the projections of simple statistical approaches, such as PARMA, as shown in Table 2
Table 13: Periodic coefficients of the two NP examples shown in Figure 12
NP
id 1
id 10
[&
[1/Pi
[P
|Ap
11/P%
[P
252.601
8529.557
0.960
2131.096
8506.317
0.809
140.967
670.924
0.715
1643.707
24.003
0.212
134.343
366.735
0.668
1158.437
366.648
0.797
117.755
24.004
0.378
1132.171
670.602
0.734
107.298
244.195
0.768
942.847
244.106
0.988
97.824
794.904
0.376
909.762
795.685
0.417
69.710
6217.354
0.746
867.654
12.001
0.309
65,251
182.112
0.452
729.163
737.857
0.696
24


Reviews:
[{"review_id": "ypyvqgQs8J", "reviewer": "Reviewer_xtYj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written and easy to follow. The proposed\nmodel is plausible and the experiments show a lift. The paper is\naccompagnied by a detailed appendix with further experiments,\nesp. ablation studies. Methodologically, the paper is a\nsmall extension of N-BEATS, but due to modelling the periodicity\nexplicitly an interesting one.\n\nHowever, two questions about the proposed model are not\nanswered in the paper currently:\n\n1. Does learning the periodic time encoding g_phi and the forecasting\n   model f_theta really have advantages? One could imagine a simple\n   baseline, where the learnt periodic time encoding is fed as covariate\n   channel into N-BEATS. Is this as good as the proposed model?\n   -- Also an ablation study in which the time encoder g_phi is frozen\n   after the first stage, could shed some light on this question.\n\n2. It is not clear on what hyperparameter grid the hyperparameter\n   optimization has been conducted. Esp. in table 4 and 6, for N-BEATS\n   and the proposed model the same backbone architecture seems to\n   be optimal, while the proposed model gets additional complexity\n   through the time encoder. Have N-BEATS architectures with a similar\n   number of parameters as the optimal architecture for the proposed\n   model been inside the hyperparameter grid for N-BEATS?\n\n\nSmall questions:\n- what is the role of the two-stage optimization problem in eq. 5? If\n  I understand it correctly, this problem never is tackled, but the\n  authors use the heuristic fitting outlined in the two bullet points\n  below eq. 5. Why do we need eq. 5 then?\n", "summary_of_the_paper": "The paper addresses the problem of time series forecasting, esp.\nwith periodic dependencies. The authors propose a model that combines\na learnt one-dimensional sum of cosines as periodic signal with\nresidual feedforward neural network (N-BEATS). They propose to\nlearn the model in two stages: first to estimate the cosines\nwith a discrete cosine transform and greedily selecting the\nK ones with largest amplitude, second to refine both the\nperiodic time encoding and the overall forecasting model\nend-to-end. In experiments on synthetic, three real-life datasets\nfrom the literature and two new real-life datasets with longer\ntraining segments, that should allow to identify periodicities\nmore easily, they show that they outperform the underlying\nN-BEATS model mostly consistently. \n", "summary_of_the_review": "A well written paper with a small, but interesting contribution. \nSome aspects of the evaluation still can be made more clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jf5M8SWQjB2", "reviewer": "Reviewer_qo99", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### strong points\n1. a customized deep learning (DL) architecture for periodic time series to explicitly take account of the periodic property\n2. the proposed techniques are reasonable and sound novel.\n3. extensive experiments on a synthetic dataset and several real-world datasets about the effectiveness and interpretability.\n\n### weak points\n1. The writing of the paper should be improved. \n2. Some technical details are missing and some motivations are not very clear.\n3. lack of significant test\n4. lack of complexity analysis", "summary_of_the_paper": "This paper proposed a novel DL framework for periodic time series forecasting. The contributions of the paper include modeling complicated periodic dependency and capturing compositions of diversified periods. The authors conduct extensive experiments on synthetic data and real-world datasets, showing the effectiveness and interpretability of the proposed methods", "summary_of_the_review": "In this paper, the authors proposed a customized deep learning (DL) architecture for periodic time series to explicitly take account of the periodic property. The proposed solution is technically sound and effective based on the extensive evaluation. The periodical module sounds novel and reasonable, and parameter initialization resorts to a two-stage optimization problem, which is also interesting.  However, I have the following comments that the authors should take into account to improve the paper.\n\n1. the writing should be improved. For example, Section 4.2 is not easy to read and the left part of Figure 2 is not very easy to understand. The authors are suggested to overview the framework before delving into the detailed description. The figure should be made more clear to understand. \n2. a few technical parts are missing. For example, how to design local block function f^l and periodic block function f^p. \n3. some motivations are not very clear. For example, why instantiate the period function as a series of cosine functions and how such a formulation can lead to the modeling of periodic patterns. \n4. Though the results of the proposed method are good, but in some cases, the improvements are small. The authors are suggested to include significant testing and show some standard error in the result table.\n5.  Though the proposed method performs well, it is unclear the efficiency of the proposed method compared to baselines. The authors are suggested to add detailed complexity analysis to the proposed methods and some related baselines.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_XBXI7Jt6kj", "reviewer": "Reviewer_3Zt1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper brings us more attention to the periodicity modeling of time series forecasting in deep learning scenarios. After carefully checking the experiments on synthetic data (Figure 3 and Figure 7), I have noticed that even adding simple operations and periods on the auto-regressive signal data can lead to the state-of-the-art model (N-BEATS) degrade. This is quite meaningful and signifies the importance of periodicity on pure DL-based forecasting models. I think such simple yet intuitive observation can bring up good insights.\n\nThe paper has proposed a new decoupled formulation of time series forecasting and formulate the forecasting problem in the model DEPTS by introducing a prior estimated hidden periodic state and a expansion framework based on residue learning. Actually, this formulation is very interesting because most existing works are only to sample series based on the time window for training, which naturally fail to consider given series as whole and thus fail to capture all the information of a series globally. The proposed formulation cleverly makes use of periodicity to relieve this problem by the hidden periodic state, as those low-frequency periods need such modeling in particular. DEPTS framework builds upon the residual learning and expand the learned local residuals and periodic residuals block by block. The periodic blocks and local blocks process the periodicity and the local momenta respectively through expansion. In brief, the whole architecture design is straightforward and it can also be interpretable by outputs of different blocks.\n\nThis paper seems to have solid, extensive and diverse experiments. The extensive experimental results have clearly demonstrated the paper intuition (synthetic experiments), the performance improvement (real-world experiments) and the interpretability. The real-world experiments are on 5 different datasets, in which they follow settings of previous work on 3 datasets and construct 2 new datasets. The proposed model achieves better performances across all the datasets. Especially, I find out that DEPTS has a 10% improvement for M4 competition compared with state-of-the-art N-BEATS, which could very challenging. The sufficient interpretability cases are showed in each dataset. Overall, the experiment part is convincing.\n\nAlso, I do have some concerns. My first concern is about periodic module. As the paper states, the expansion module and the periodicity module jointly work for the forecasting based on many stacked layers. Each layer includes a local block and a periodic block for the residual expansion. The periodicity module is started with a parameter initialization, which is firstly initialized on the observation data. So my question is if the data is with an obvious trend (e.g., upward trend), how does the periodicity work since it can only handle periods with accumulated cosine functions? My another concern is with the training of the periodic module. If the periodic module is randomly initialized, can this DEPTS model still perform well? Would it still can accurately capture the obvious periods? I am very curious about the forecasting results without initialization of the periodicity module. ", "summary_of_the_paper": "This paper focuses on an important property of time series, periodicity, and mainly studies the problem of periodic time series (PTS) forecasting. This work solves two main research challenges of PTS forecasting: (1) to learn the dependencies of observation data on periodicity; (2) to learn the sophisticated compositions of various periods. The authors propose a deep expansion framework on top of residue learning for dependency learning and a parameterized surrogate function for periodicity learning. Extensive experiments on both synthetic data and 5 real-world datasets demonstrate a significant improvement on time series forecasting tasks when considering periodicity especially.", "summary_of_the_review": "The paper solves the periodic time series forecasting problem; the proposed DEPTS model is intuitive, effective and interpretable; the experiment is solid and extensive; the improvement is significant. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ZmcDfyPqKMU", "reviewer": "Reviewer_SZrb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, this paper is well-written and easy to follow. Periodicity is a property rarely explicitly handled by previous deep learning models. The proposed model achieves better performance compared to previous methods such as N-BEATS. Experiments on different settings and parameters are provided. \n\nHowever, I have the following questions:\n(1). Why does the expansion module need periodic blocks? Considering estimated hidden state z_t has already included inherent periodicity (assumed by the paper), why not directly taking x_t minus z_t to the local block for the predictions?\n\n(2). The authors discuss the connections between the model and N-BEATS. What is unclear is the main difference between N-BEATS (interpretable) and DEPTS, since both of them have applied the periodicity. More illustrations should be provided. \n\n(3). Other time series analysis baselines such as ARIMA should be included in the experiments, although they may not beat the proposed method. ", "summary_of_the_paper": "This paper proposes a deep learning model tailor-designed for periodic time series forecasting problem. The inner architecture is composed of an expansion module to tackle complicated dependencies between inherent periods and time series signals and a periodicity module to capture sophisticated period signals. ", "summary_of_the_review": "Overall the paper appears to have good quality, with a new time series prediction model proposed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a novel deep learning model specifically designed for periodic time series forecasting problems. The approach includes lay-by-layer expansion, residual learning, and periodic parametrization. The model outperforms state-of-the-art baselines on several time series forecasting benchmarks.  The reviewers appreciate the extensive experimental results, but also suggested improvement on writing and comparison regarding the parameter efficiency of the model.
Decision: Accept (Spotlight)