DEPTS: Deep Expansion Learning for Periodic Time Series Forecasting

1 INTRODUCTION . Time series ( TS ) with apparent periodic ( seasonal ) oscillations , referred to as periodic time series ( PTS ) in this paper , is pervasive in a wide range of critical industries , such as seasonal electricity spot prices in power industry ( Koopman et al. , 2007 ) , periodic traffic flows in transportation ( Lippi et al. , 2013 ) , periodic carbon dioxide exchanges and water flows in sustainability domain ( Seymour , 2001 ; Tesfaye et al. , 2006 ) . Apparently , PTS forecasting plays a crucial role in these industries since it can foster their business development by facilitating a variety of capabilities , including early warning , pre-planning , and resource scheduling ( Kahn , 2003 ; Jain , 2017 ) . Given the pervasiveness and importance of PTS , two obstacles , however , largely hinder the performance of existing forecasting models . First , future TS signals yield complicated dependencies on both adjacent historical observations and inherent periodicity . Nevertheless , many existing studies did not consider this distinctive periodic property ( Salinas et al. , 2020 ; Toubeau et al. , 2018 ; Wang et al. , 2019 ; Oreshkin et al. , 2020 ) . The performance of these methods has been greatly restrained due to its ignorance of periodicity modeling . Some other efforts , though explicitly introducing periodicity modeling , only followed some arbitrary yet simple assumptions , such as additive or multiplicative seasonality , to capture certain plain periodic effects ( Holt , 1957 ; 2004 ; Vecchia , 1985b ; Taylor & Letham , 2018 ) . These methods failed to model complicated periodic dependencies beyond much simplified assumptions . The second challenge lies in that the inherent periodicity of a typical real-world TS is usually composed of various periods with different amplitudes and frequencies . For example , Figure 1 exemplifies the sophisticated composition of diversified periods via an eightyears hourly TS of electricity load in a region of California . However , existing methods ( Taylor & Letham , 2018 ; Smyl , 2020 ) required the pre-specification of periodic frequencies before estimating other parameters from data , which attempted to evade this obstacle by transferring the burden of periodicity coefficient initialization to practitioners . To better tackle the aforementioned two challenges , we develop a deep expansion learning framework , DEPTS , for PTS forecasting . The core idea of DEPTS is to build a deep neural network that conducts the progressive expansions of the complicated dependencies of PTS signals on periodicity to facilitate forecasting . We start from a novel decoupled formulation for PTS forecasting by introducing the periodic state as a hidden variable . This new formulation stimulates us to make more customized and dedicated designs to handle the two specific challenges mentioned above . For the first challenge , we develop an expansion module on top of residual learning ( He et al. , 2016 ; Oreshkin et al. , 2020 ) to conduct layer-by-layer expansions between observed TS signals and hidden periodic states . With such a design , we can build a deep architecture with both high capacities and efficient parameter optimization to model those complicated dependencies of TS signals on periodicity . For the second challenge , we build a periodicity module to estimate the periodic states from observational data . We represent the hidden periodic state with respect to time as a parameterized periodic function with sufficient expressiveness . In this work , for simplicity , we instantiate this function as a series of cosine functions . To release the burden of manually setting periodic coefficients for different data , we develop a data-driven parameter initialization strategy on top of Discrete Cosine Transform ( Ahmed et al. , 1974 ) . After that , we combine the periodicity module with the expansion module to perform end-to-end learning . To the best of our knowledge , DEPTS is a very early attempt to build a customized deep learning ( DL ) architecture for PTS that explicitly takes account of the periodic property . Moreover , with two delicately designed modules , DEPTS also owns certain interpretable capabilities . First , the expansions of forecasts can distinguish the contributions from either adjacent TS signals or inherent periodicity , which intuitively illustrate how the future TS signals may vary based on local momenta and global periodicity . Second , coefficients of the periodicity module have their own practical meanings , such as amplitudes and frequencies , which provide certain interpretable effects inherently . We conduct experiments on both synthetic data and real-world data , which all demonstrate the superiority of DEPTS on handling PTS . On average , DEPTS reduces the error of the best baseline by about 10 % . In a few cases , the error reduction can even reach up to 20 % . Besides , we also include extensive ablation tests to verify our critical designs and visualize specific model components to interpret model behaviors . 2 RELATED WORK . TS forecasting is a longstanding research topic that has been extensively studied for decades . After a comprehensive review of the literature , we find three types of paradigms in developing TS models . At an early stage , researchers developed simple yet effective statistical modeling approaches , including exponentially weighted moving averages ( Holt , 1957 ; 2004 ; Winters , 1960 ) , auto-regressive moving averages ( ARMA ) ( Whittle , 1951 ; 1963 ) , the unified state-space modeling approach as well as other various extensions ( Hyndman & Khandakar , 2008 ) . However , these statistical approaches only considered the linear dependencies of future TS signals on past observations . To handle highorder dependencies , researchers attempted to adopt a hybrid design that combines statistical modeling with more advanced high-capacity models ( Montero-Manso et al. , 2020 ; Smyl , 2020 ) . At the same time , with the great successes of DL in computer vision ( He et al. , 2016 ) and natural language processing ( Vaswani et al. , 2017 ) , various DL models have also been developed for TS forecasting ( Rangapuram et al. , 2018 ; Toubeau et al. , 2018 ; Salinas et al. , 2020 ; Zia & Razzaq , 2020 ) . Among them , the most representative one is N-BEATS ( Oreshkin et al. , 2020 ) , which is a pure DL architecture that has achieved state-of-the-art performance across a wide range of benchmarks . The connections between DEPTS and N-BEATS have been discussed in Section 4.2 . As for PTS forecasting , many traditional statistical approaches explicitly considered the periodic property , such as periodic ARMA ( PARMA ) ( Vecchia , 1985a ; b ) and its variants ( Tesfaye et al. , 2006 ; Anderson et al. , 2007 ; Dudek et al. , 2016 ) . However , as discussed in Sections 1 and 3 , these methods only followed some arbitrary yet simple assumptions , such as additive or multiplicative seasonality , and thus can not well handle complicated periodic dependencies in many real-world scenarios . Besides , other recent studies either followed the similar assumptions for periodicity or required the pre-specification of periodic coefficients ( Taylor & Letham , 2018 ; Smyl , 2020 ) . To the best of our knowledge , we are the first work that develops a customized DL architecture to model complicated periodic dependencies and to capture diversified periodic compositions simultaneously . 3 PROBLEM FORMULATIONS . We consider the point forecasting problem of regularly sampled uni-variate TS . Let xt denote the time series value at time-step t , and the classical auto-regressive formulation is to project the historical observations xt−L : t = [ xt−L , . . . , xt−1 ] into its subsequent future values xt : t+H = [ xt , . . . , xt+H−1 ] : xt : t+H = FΘ ( xt−L : t ) + t : t+H , ( 1 ) where H is the length of the forecast horizon , L is the length of the lookback window , FΘ : RL → RH is a mapping function parameterized by Θ , and t : t+H = [ t , . . . , t+H−1 ] denotes a vector of independent and identically distributed Gaussian noises . Essentially , the fundamental assumption behind this formulation is the Markov property xt : t+H ⊥ x0 : t−L|xt−L : t , which assumes that the future values xt : t+H are independent of all farther historical values x0 : t−L given the adjacent shortterm observations xt−L : t. Note that most existing DL models ( Salinas et al. , 2020 ; Toubeau et al. , 2018 ; Wang et al. , 2019 ; Oreshkin et al. , 2020 ) directly follow this formulation to solve TS . Even traditional statistical TS models ( Holt , 1957 ; 2004 ; Winters , 1960 ) are indeed consistent with that if omitting those long-tail exponentially decayed dependencies introduced by moving averages . To precisely formulate PTS , on the other hand , this assumption needs to be slightly modified such that the dependency of xt : t+H on xt−L : t is further conditioned on the inherent periodicity , which can be anchored by associated time-steps . Accordingly , we alter the equation ( 1 ) into xt : t+H = F ′ Θ ( xt−L : t , t ) + t : t+H , ( 2 ) where other than xt−L : t , F ′ Θ : RL × R → RH takes an extra argument t , which denotes the forecasting time-step . Existing methods for PTS adopt a few different instantiations of F ′Θ . For example , Holt ( 1957 ; 2004 ) developed several exponentially weighted moving average processes with additive or multiplicative seasonality . Vecchia ( 1985a ; b ) adopted the multiplicative seasonality by treating the coefficients of the auto-regressive moving average process as time dependent . Smyl ( 2020 ) also adopted the multiplicative seasonality and built a hybrid method by coupling that with recurrent neural networks ( Hochreiter & Schmidhuber , 1997 ) , while Taylor & Letham ( 2018 ) chose the additive seasonality by adding the periodic forecast with other parts as the final forecast . 4 DEPTS . In this section , we elaborate on our new framework , DEPTS . First , we start with a decoupled formulation of ( 2 ) in Section 4.1 . Then , we illustrate the proposed neural architecture for this formulation in Sections 4.2 and 4.3 . Last , we discuss the interpretable capabilities in Section 4.4 . 4.1 THE DECOUPLED FORMULATION . To explicitly tackle the two-sided challenges of PTS forecasting , i.e. , complicated periodic dependencies and diversified periodic compositions , we introduce a decoupled formulation ( 3 ) that refines ( 2 ) by introducing a hidden variable zt to represent the periodic state at time-step t : xt : t+H = fθ ( xt−L : t , zt−L : t+H ) + t : t+H , zt = gφ ( t ) , ( 3 ) where we treat zt ∈ R1 as a scalar value to be consistent with the uni-variate TS xt ∈ R1 , we use fθ : RL × RL+H → RH to model complicated dependencies of the future signals xt : t+H on the local observations xt−L : t and the corresponding periodic states zt−L : t+H within the lookback and forecast horizons , and gφ : R1 → R1 is to produce a periodic state zt for a specific time-step t. The right part of Figure 2 depicts the overall data flows of this formulation , in which the expansion module fθ and the periodicity module gφ are responsible for handling the two aforementioned PTSspecific challenges , respectively .

Summary: This paper focuses on an important property of time series, periodicity, and mainly studies the problem of periodic time series (PTS) forecasting. This work solves two main research challenges of PTS forecasting: (1) to learn the dependencies of observation data on periodicity; (2) to learn the sophisticated compositions of various periods. The authors propose a deep expansion framework on top of residue learning for dependency learning and a parameterized surrogate function for periodicity learning. Extensive experiments on both synthetic data and 5 real-world datasets demonstrate a significant improvement on time series forecasting tasks when considering periodicity especially.