Topological Attention for Time Series Forecasting

The problem of ( point ) forecasting univariate time series is considered . Most approaches , ranging from traditional statistical methods to recent learning-based techniques with neural networks , directly operate on raw time series observations . As an extension , we study whether local topological properties , as captured via persistent homology , can serve as a reliable signal that provides complementary information for learning to forecast . To this end , we propose topological attention , which allows attending to local topological features within a time horizon of historical data . Our approach easily integrates into existing end-to-end trainable forecasting models , such as N-BEATS , and , in combination with the latter , exhibits state-of-the-art performance on the large-scale M4 benchmark dataset of 100,000 diverse time series from different domains . Ablation experiments , as well as a comparison to a broad range of forecasting methods in a setting where only a single time series is available for training , corroborate the beneficial nature of including local topological information through an attention mechanism . 1 Introduction . Time series are ubiquitous in science and industry , from medical signals ( e.g. , EEG ) , motion data ( e.g. , speed , steps , etc . ) or economic operating figures to ride/demand volumes of transportation network companies ( e.g. , Uber , Lyft , etc. ) . Despite many advances in predicting future observations from historical data via traditional statistical [ 5 ] , or recent machine learning approaches [ 29 , 42 , 34 , 32 , 24 , 44 ] , reliable and accurate forecasting remains challenging . This is not least due to widely different and often heavily domain dependent structural properties of time related sequential data . In this work , we focus on the problem of ( point ) forecasting univariate time series , i.e. , given a length-T vector of historical data , the task is to predict future observations for a given time horizon H . While neural network models excel in situations where a large corpus of time series is available for training , the case of only a single ( possibly long ) time series is equally important . The arguably most prominent benchmarks for the former type of forecasting problem are the “ ( M ) akridakis ” competitions , such as M3 [ 26 ] or M4 [ 28 ] . While combinations ( and hybrids ) of statistical and machine learning approaches have largely dominated these competitions [ 28 , see Table 4 ] , Oreshkin et al . [ 29 ] have recently demonstrated that a pure learning-based model ( N-BEATS ) attains state-ofthe-art performance . Interestingly , the latter approach is simply built from a collection of common neural network primitives which are not specific to sequential data . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , the majority of learning-based approaches directly operate on the raw input signal , implicitly assuming that viable representations for forecasting can be learned via common neural network primitives , composed either in a feedforward or recurrent manner . This raises the question of whether there exist structural properties of the signal , which are not easily extractable via neural network components , but offer complementary information . One prime example1 are topological features , typically obtained via persistent homology [ 7 , 13 ] . In fact , various approaches [ 31 , 16 , 11 , 21 , 15 ] have successfully used topological features for time series analysis , however , mostly in classification settings , for the identification of certain phenomena in dynamical systems , or for purely exploratory analysis ( see Section 2 ) . Contribution . We propose an approach to incorporate local topological information into neural forecasting models . Contrary to previous works , we do not compute a global topological summary of historical observations , but features of short , overlapping time windows to which the forecasting model can attend to . The latter is achieved via self-attention and thereby integrates well into recent techniques , such as N-BEATS [ 29 ] . Notably , in our setting , computation of topological features ( via persistent homology ) comes with moderate computational cost , which allows application in large-scale forecasting problems . Problem statement . In practice , neural forecasting models typically utilize the last T observations of a time series in order to yield ( point ) forecasts for a given time horizon H . Under this perspective , the problem boils down to learning a function ( parametrized as a neural network ) ϕ : RT → RH x 7→ ϕ ( x ) = y , ( 1 ) from a given collection of inputs ( i.e. , length-T vectors ) and targets ( i.e. , length-H vectors ) . Specifically , we consider two settings , where either ( 1 ) a large collection of time series is available , as in the M4 competition , or ( 2 ) we only have access to a single time series . In the latter setting , a model has to learn from patterns within a time series , while the former setting allows to exploit common patterns across multiple time series . 2 Related work . Persistent homology and time series . Most approaches to topological time series analysis are conceptually similar , building on top of work by de Silva et al . [ 10 ] and Perea & Harer [ 31 , 30 ] . Herein , time series observations are transformed into a point cloud via a time-delay coordinate embedding [ 38 ] from which Vietoris-Rips ( VR ) persistent homology is computed . The resulting topological summaries , i.e. , persistence barcodes , are then used for downstream processing . Within this regime , Gidea et al . [ 15 ] analyze the dynamics of cryptocurrencies using persistence landscapes [ 6 ] , Khasawneh et al . [ 21 ] study chatter classification in synthetic time series from turning processes 1Although learning-based approaches [ 36 ] exist to approximate topological summaries ( without guarantees ) . and Dłotko et al . [ 11 ] identify periodicity patterns in time series . In [ 22 ] , Kim et al . actually compute one-step forecasts for Bitcoin prices and classify price patterns , essentially feeding barcode statistics as supplementary features to a MLP/CNN-based regression model . Surprisingly , very few works deviate from this pipeline , with the notable exception of [ 16 ] , where VR persistent homology is not computed from a time-delay coordinate embedding , but rather from assembling observations ( within sliding windows of size n ) from a d-variate time series into a d-dimensional point cloud , followed by VR persistent homology computation . Although these works clearly demonstrate that capturing the “ shape ” of data via persistent homology provides valuable information for time series related problems , they ( 1 ) rely on handcrafted features ( i.e. , predefined barcode summary statistics , or a fixed barcode-vectorization strategy ) , ( 2 ) consider topological summaries as the single source of information and ( 3 ) only partially touch upon forecasting problems ( with the exception of [ 22 ] ) . Furthermore , in this existing line of work , sweeping a sliding window over the time series is , first and foremost , a way to construct a point cloud which represents the entire time series . Instead , in our approach , each window yields its own topological summary in the form of a persistence barcode , reminiscent to representing a sentence as a sequence of word embeddings in NLP tasks . When combined with learnable representations of persistence barcodes [ 19 , 8 ] , this perspective paves the way for leveraging recent techniques for handling learning problems with sequential data , such as attention [ 41 ] , and allows to seamlessly integrate topological features into existing neural forecasting techniques . Neural network approaches to time series forecasting . Recently , various successful neural network approaches to ( mostly probabilistic ) time series forecasting have emerged , ranging from auto-regressive neural networks as in DeepAR [ 34 ] , to ( deep ) extensions of traditional state space models , such as DeepFactors [ 42 ] or DeepState [ 32 ] . While these models are inherently tailored to the sequential nature of the forecasting problem , Li et al . [ 24 ] instead rely on the concept of ( log-sparse ) self-attention [ 41 ] , fed by the outputs of causal convolutions , and Oreshkin et al . [ 29 ] even abandon sequential neural network primitives altogether . The latter approach , solely based on operations predominantly found in feed-forward architectures , achieves state-of-the-art performance for ( point ) forecasts across several benchmarks , including the large-scale M4 competition . Yet , a common factor in all aforementioned works is that raw time series observations are directly input to the model , assuming that relevant structural characteristics of the signal can be learned . While we choose an approach similar to [ 24 ] , in the sense that we rely on self-attention , our work differs in that representations fed to the attention mechanism are not obtained through convolutions , but rather through a topological analysis step which , by its construction , captures the “ shape ” of local time series segments . 3 Topological attention . The key idea of topological attention is to analyze local segments within an input time series , x , through the lens of persistent homology . As mentioned in Section 2 , the prevalent strategy in prior work is to first construct a point cloud from x via a time-delay coordinate embedding and to subsequently compute VR persistent homology . Historically , this is motivated by studying structural properties of an underlying dynamical system , with a solid theoretical foundation , e.g. , in the context of identifying periodicity patterns [ 31 , 30 ] . In this regime , x is encoded as a point cloud in Rn by considering observations within a sliding window of size n as a point in Rn . While the time-delay embedding strategy is adequate in settings where one aims to obtain one global topological summary , it is inadequate when local structural properties of time series segments are of interest . Further , unless large ( computationally impractical ) historical time horizons are considered , one would obtain relatively sparse point clouds that , most likely , carry little information . 3.1 Time series as local topological summaries . Instead of a time-delay coordinate embedding , we follow an alternative strategy : a time series signal , x , is still decomposed into a sequence of ( overlapping ) windows , but not to yield a point cloud element , but rather to be analyzed in isolation . In the following , we only discuss the necessities specific to our approach , and refer the reader to [ 14 , 7 , 4 ] for a thorough treatment of persistent homology . To topologically analyze a length-T time series , over the time steps { 1 , . . . , T } = [ T ] , in a computationally tractable manner , lets consider a 1-dimensional simplicial complex of the form K = { { 1 } , . . . { T } , { 1 , 2 } , . . . , { T − 1 , T } } , where { i } denote 0-simplices ( i.e. , vertices ) and 1-simplices { i , j } ( i.e. , edges ) are in K if and only if i and j are two consecutive time indices . Topologically , K carries the connectivity properties of a time series of length T , which is equivalent to a straight line . This is the same for all time series of length T and thus offers no discriminative information . Persistent homology , however , lets us combine the purely topological representation of the time series with its actual values . For a specific x , let a1 ≤ · · · ≤ aT denote its increasingly sorted values and consider K0x = ∅ , Kjx = { σ ∈ K : ∀i ∈ σ : xi ≤ aj } for j ∈ [ T ] . In fact , ∅ = K0x ⊆ K1x ⊆ · · · ⊆ KTx = K forms an increasing sequence of subsets of K , i.e. , a filtration . Importantly , while K is the same for all time series of length T , the filtration , ( Kjx ) Tj=0 , is determined by the values of x . Persistent homology then tracks the evolution of topological features throughout this sequence and summarizes this information in the form of persistence barcodes . In our specific case , as K is topologically equivalent to a straight line , we only get 0-dimensional features , i.e. , connected components . Hence , we obtain one ( 0 degree ) barcode Bx . This barcode is a multiset of ( birth , death ) tuples , representing the birth ( b ) and death ( d ) of topological features . Informally , we may think of building K , piece-by-piece , according to the sorting of the xi , starting with the lowest value , and tracking how connected components appear / merge , illustrated in Fig . 2 . Remark 3.1 . The information captured throughout this process has two noteworthy properties . First , it is stable in the sense that small changes in the observation values may not cause arbitrary changes in the respective barcodes , see [ 9 ] . Second , one may equally order the negative observations , i.e. , −x , and thus obtain B−x . In that manner , the signal is analyzed from below and above . Finally , to extract local topological information , we do not compute one single barcode for x , but one for each sliding window of size n , see Fig . 1 . Given a decomposition of x into W subsequent windows , we obtain W barcodes , B1 , . . . , BW , which constitute the entry point for any downstream operation . Informally , those barcodes encode the evolution of local topological features over time .

Summary: The paper presents an approach to combine topological features to time-series forecasting. Essentially, the method computes 1D topological barcodes at each time-steps, pass them to a transformer and then provides those features as additional input to NBEATS. Experiments are conducted on two datasest when performing local forecast (e.g. forecasting the time-series independently) and on M4 when using global forecast (learning the model jointly across time-series). An improvement of 0.4%/1.2% is reported on M4 for OWA/sMAPE.