DEPTS: Deep Expansion Learning for Periodic Time Series Forecasting

1 INTRODUCTION . Time series ( TS ) with apparent periodic ( seasonal ) oscillations , referred to as periodic time series ( PTS ) in this paper , is pervasive in a wide range of critical industries , such as seasonal electricity spot prices in power industry ( Koopman et al. , 2007 ) , periodic traffic flows in transportation ( Lippi et al. , 2013 ) , periodic carbon dioxide exchanges and water flows in sustainability domain ( Seymour , 2001 ; Tesfaye et al. , 2006 ) . Apparently , PTS forecasting plays a crucial role in these industries since it can foster their business development by facilitating a variety of capabilities , including early warning , pre-planning , and resource scheduling ( Kahn , 2003 ; Jain , 2017 ) . Given the pervasiveness and importance of PTS , two obstacles , however , largely hinder the performance of existing forecasting models . First , future TS signals yield complicated dependencies on both adjacent historical observations and inherent periodicity . Nevertheless , many existing studies did not consider this distinctive periodic property ( Salinas et al. , 2020 ; Toubeau et al. , 2018 ; Wang et al. , 2019 ; Oreshkin et al. , 2020 ) . The performance of these methods has been greatly restrained due to its ignorance of periodicity modeling . Some other efforts , though explicitly introducing periodicity modeling , only followed some arbitrary yet simple assumptions , such as additive or multiplicative seasonality , to capture certain plain periodic effects ( Holt , 1957 ; 2004 ; Vecchia , 1985b ; Taylor & Letham , 2018 ) . These methods failed to model complicated periodic dependencies beyond much simplified assumptions . The second challenge lies in that the inherent periodicity of a typical real-world TS is usually composed of various periods with different amplitudes and frequencies . For example , Figure 1 exemplifies the sophisticated composition of diversified periods via an eightyears hourly TS of electricity load in a region of California . However , existing methods ( Taylor & Letham , 2018 ; Smyl , 2020 ) required the pre-specification of periodic frequencies before estimating other parameters from data , which attempted to evade this obstacle by transferring the burden of periodicity coefficient initialization to practitioners . To better tackle the aforementioned two challenges , we develop a deep expansion learning framework , DEPTS , for PTS forecasting . The core idea of DEPTS is to build a deep neural network that conducts the progressive expansions of the complicated dependencies of PTS signals on periodicity to facilitate forecasting . We start from a novel decoupled formulation for PTS forecasting by introducing the periodic state as a hidden variable . This new formulation stimulates us to make more customized and dedicated designs to handle the two specific challenges mentioned above . For the first challenge , we develop an expansion module on top of residual learning ( He et al. , 2016 ; Oreshkin et al. , 2020 ) to conduct layer-by-layer expansions between observed TS signals and hidden periodic states . With such a design , we can build a deep architecture with both high capacities and efficient parameter optimization to model those complicated dependencies of TS signals on periodicity . For the second challenge , we build a periodicity module to estimate the periodic states from observational data . We represent the hidden periodic state with respect to time as a parameterized periodic function with sufficient expressiveness . In this work , for simplicity , we instantiate this function as a series of cosine functions . To release the burden of manually setting periodic coefficients for different data , we develop a data-driven parameter initialization strategy on top of Discrete Cosine Transform ( Ahmed et al. , 1974 ) . After that , we combine the periodicity module with the expansion module to perform end-to-end learning . To the best of our knowledge , DEPTS is a very early attempt to build a customized deep learning ( DL ) architecture for PTS that explicitly takes account of the periodic property . Moreover , with two delicately designed modules , DEPTS also owns certain interpretable capabilities . First , the expansions of forecasts can distinguish the contributions from either adjacent TS signals or inherent periodicity , which intuitively illustrate how the future TS signals may vary based on local momenta and global periodicity . Second , coefficients of the periodicity module have their own practical meanings , such as amplitudes and frequencies , which provide certain interpretable effects inherently . We conduct experiments on both synthetic data and real-world data , which all demonstrate the superiority of DEPTS on handling PTS . On average , DEPTS reduces the error of the best baseline by about 10 % . In a few cases , the error reduction can even reach up to 20 % . Besides , we also include extensive ablation tests to verify our critical designs and visualize specific model components to interpret model behaviors . 2 RELATED WORK . TS forecasting is a longstanding research topic that has been extensively studied for decades . After a comprehensive review of the literature , we find three types of paradigms in developing TS models . At an early stage , researchers developed simple yet effective statistical modeling approaches , including exponentially weighted moving averages ( Holt , 1957 ; 2004 ; Winters , 1960 ) , auto-regressive moving averages ( ARMA ) ( Whittle , 1951 ; 1963 ) , the unified state-space modeling approach as well as other various extensions ( Hyndman & Khandakar , 2008 ) . However , these statistical approaches only considered the linear dependencies of future TS signals on past observations . To handle highorder dependencies , researchers attempted to adopt a hybrid design that combines statistical modeling with more advanced high-capacity models ( Montero-Manso et al. , 2020 ; Smyl , 2020 ) . At the same time , with the great successes of DL in computer vision ( He et al. , 2016 ) and natural language processing ( Vaswani et al. , 2017 ) , various DL models have also been developed for TS forecasting ( Rangapuram et al. , 2018 ; Toubeau et al. , 2018 ; Salinas et al. , 2020 ; Zia & Razzaq , 2020 ) . Among them , the most representative one is N-BEATS ( Oreshkin et al. , 2020 ) , which is a pure DL architecture that has achieved state-of-the-art performance across a wide range of benchmarks . The connections between DEPTS and N-BEATS have been discussed in Section 4.2 . As for PTS forecasting , many traditional statistical approaches explicitly considered the periodic property , such as periodic ARMA ( PARMA ) ( Vecchia , 1985a ; b ) and its variants ( Tesfaye et al. , 2006 ; Anderson et al. , 2007 ; Dudek et al. , 2016 ) . However , as discussed in Sections 1 and 3 , these methods only followed some arbitrary yet simple assumptions , such as additive or multiplicative seasonality , and thus can not well handle complicated periodic dependencies in many real-world scenarios . Besides , other recent studies either followed the similar assumptions for periodicity or required the pre-specification of periodic coefficients ( Taylor & Letham , 2018 ; Smyl , 2020 ) . To the best of our knowledge , we are the first work that develops a customized DL architecture to model complicated periodic dependencies and to capture diversified periodic compositions simultaneously . 3 PROBLEM FORMULATIONS . We consider the point forecasting problem of regularly sampled uni-variate TS . Let xt denote the time series value at time-step t , and the classical auto-regressive formulation is to project the historical observations xt−L : t = [ xt−L , . . . , xt−1 ] into its subsequent future values xt : t+H = [ xt , . . . , xt+H−1 ] : xt : t+H = FΘ ( xt−L : t ) + t : t+H , ( 1 ) where H is the length of the forecast horizon , L is the length of the lookback window , FΘ : RL → RH is a mapping function parameterized by Θ , and t : t+H = [ t , . . . , t+H−1 ] denotes a vector of independent and identically distributed Gaussian noises . Essentially , the fundamental assumption behind this formulation is the Markov property xt : t+H ⊥ x0 : t−L|xt−L : t , which assumes that the future values xt : t+H are independent of all farther historical values x0 : t−L given the adjacent shortterm observations xt−L : t. Note that most existing DL models ( Salinas et al. , 2020 ; Toubeau et al. , 2018 ; Wang et al. , 2019 ; Oreshkin et al. , 2020 ) directly follow this formulation to solve TS . Even traditional statistical TS models ( Holt , 1957 ; 2004 ; Winters , 1960 ) are indeed consistent with that if omitting those long-tail exponentially decayed dependencies introduced by moving averages . To precisely formulate PTS , on the other hand , this assumption needs to be slightly modified such that the dependency of xt : t+H on xt−L : t is further conditioned on the inherent periodicity , which can be anchored by associated time-steps . Accordingly , we alter the equation ( 1 ) into xt : t+H = F ′ Θ ( xt−L : t , t ) + t : t+H , ( 2 ) where other than xt−L : t , F ′ Θ : RL × R → RH takes an extra argument t , which denotes the forecasting time-step . Existing methods for PTS adopt a few different instantiations of F ′Θ . For example , Holt ( 1957 ; 2004 ) developed several exponentially weighted moving average processes with additive or multiplicative seasonality . Vecchia ( 1985a ; b ) adopted the multiplicative seasonality by treating the coefficients of the auto-regressive moving average process as time dependent . Smyl ( 2020 ) also adopted the multiplicative seasonality and built a hybrid method by coupling that with recurrent neural networks ( Hochreiter & Schmidhuber , 1997 ) , while Taylor & Letham ( 2018 ) chose the additive seasonality by adding the periodic forecast with other parts as the final forecast . 4 DEPTS . In this section , we elaborate on our new framework , DEPTS . First , we start with a decoupled formulation of ( 2 ) in Section 4.1 . Then , we illustrate the proposed neural architecture for this formulation in Sections 4.2 and 4.3 . Last , we discuss the interpretable capabilities in Section 4.4 . 4.1 THE DECOUPLED FORMULATION . To explicitly tackle the two-sided challenges of PTS forecasting , i.e. , complicated periodic dependencies and diversified periodic compositions , we introduce a decoupled formulation ( 3 ) that refines ( 2 ) by introducing a hidden variable zt to represent the periodic state at time-step t : xt : t+H = fθ ( xt−L : t , zt−L : t+H ) + t : t+H , zt = gφ ( t ) , ( 3 ) where we treat zt ∈ R1 as a scalar value to be consistent with the uni-variate TS xt ∈ R1 , we use fθ : RL × RL+H → RH to model complicated dependencies of the future signals xt : t+H on the local observations xt−L : t and the corresponding periodic states zt−L : t+H within the lookback and forecast horizons , and gφ : R1 → R1 is to produce a periodic state zt for a specific time-step t. The right part of Figure 2 depicts the overall data flows of this formulation , in which the expansion module fθ and the periodicity module gφ are responsible for handling the two aforementioned PTSspecific challenges , respectively .

Summary: The paper addresses the problem of time series forecasting, esp. with periodic dependencies. The authors propose a model that combines a learnt one-dimensional sum of cosines as periodic signal with residual feedforward neural network (N-BEATS). They propose to learn the model in two stages: first to estimate the cosines with a discrete cosine transform and greedily selecting the K ones with largest amplitude, second to refine both the periodic time encoding and the overall forecasting model end-to-end. In experiments on synthetic, three real-life datasets from the literature and two new real-life datasets with longer training segments, that should allow to identify periodicities more easily, they show that they outperform the underlying N-BEATS model mostly consistently.