Topological Attention for Time Series Forecasting

The problem of ( point ) forecasting univariate time series is considered . Most approaches , ranging from traditional statistical methods to recent learning-based techniques with neural networks , directly operate on raw time series observations . As an extension , we study whether local topological properties , as captured via persistent homology , can serve as a reliable signal that provides complementary information for learning to forecast . To this end , we propose topological attention , which allows attending to local topological features within a time horizon of historical data . Our approach easily integrates into existing end-to-end trainable forecasting models , such as N-BEATS , and , in combination with the latter , exhibits state-of-the-art performance on the large-scale M4 benchmark dataset of 100,000 diverse time series from different domains . Ablation experiments , as well as a comparison to a broad range of forecasting methods in a setting where only a single time series is available for training , corroborate the beneficial nature of including local topological information through an attention mechanism . 1 Introduction . Time series are ubiquitous in science and industry , from medical signals ( e.g. , EEG ) , motion data ( e.g. , speed , steps , etc . ) or economic operating figures to ride/demand volumes of transportation network companies ( e.g. , Uber , Lyft , etc. ) . Despite many advances in predicting future observations from historical data via traditional statistical [ 5 ] , or recent machine learning approaches [ 29 , 42 , 34 , 32 , 24 , 44 ] , reliable and accurate forecasting remains challenging . This is not least due to widely different and often heavily domain dependent structural properties of time related sequential data . In this work , we focus on the problem of ( point ) forecasting univariate time series , i.e. , given a length-T vector of historical data , the task is to predict future observations for a given time horizon H . While neural network models excel in situations where a large corpus of time series is available for training , the case of only a single ( possibly long ) time series is equally important . The arguably most prominent benchmarks for the former type of forecasting problem are the “ ( M ) akridakis ” competitions , such as M3 [ 26 ] or M4 [ 28 ] . While combinations ( and hybrids ) of statistical and machine learning approaches have largely dominated these competitions [ 28 , see Table 4 ] , Oreshkin et al . [ 29 ] have recently demonstrated that a pure learning-based model ( N-BEATS ) attains state-ofthe-art performance . Interestingly , the latter approach is simply built from a collection of common neural network primitives which are not specific to sequential data . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , the majority of learning-based approaches directly operate on the raw input signal , implicitly assuming that viable representations for forecasting can be learned via common neural network primitives , composed either in a feedforward or recurrent manner . This raises the question of whether there exist structural properties of the signal , which are not easily extractable via neural network components , but offer complementary information . One prime example1 are topological features , typically obtained via persistent homology [ 7 , 13 ] . In fact , various approaches [ 31 , 16 , 11 , 21 , 15 ] have successfully used topological features for time series analysis , however , mostly in classification settings , for the identification of certain phenomena in dynamical systems , or for purely exploratory analysis ( see Section 2 ) . Contribution . We propose an approach to incorporate local topological information into neural forecasting models . Contrary to previous works , we do not compute a global topological summary of historical observations , but features of short , overlapping time windows to which the forecasting model can attend to . The latter is achieved via self-attention and thereby integrates well into recent techniques , such as N-BEATS [ 29 ] . Notably , in our setting , computation of topological features ( via persistent homology ) comes with moderate computational cost , which allows application in large-scale forecasting problems . Problem statement . In practice , neural forecasting models typically utilize the last T observations of a time series in order to yield ( point ) forecasts for a given time horizon H . Under this perspective , the problem boils down to learning a function ( parametrized as a neural network ) ϕ : RT → RH x 7→ ϕ ( x ) = y , ( 1 ) from a given collection of inputs ( i.e. , length-T vectors ) and targets ( i.e. , length-H vectors ) . Specifically , we consider two settings , where either ( 1 ) a large collection of time series is available , as in the M4 competition , or ( 2 ) we only have access to a single time series . In the latter setting , a model has to learn from patterns within a time series , while the former setting allows to exploit common patterns across multiple time series . 2 Related work . Persistent homology and time series . Most approaches to topological time series analysis are conceptually similar , building on top of work by de Silva et al . [ 10 ] and Perea & Harer [ 31 , 30 ] . Herein , time series observations are transformed into a point cloud via a time-delay coordinate embedding [ 38 ] from which Vietoris-Rips ( VR ) persistent homology is computed . The resulting topological summaries , i.e. , persistence barcodes , are then used for downstream processing . Within this regime , Gidea et al . [ 15 ] analyze the dynamics of cryptocurrencies using persistence landscapes [ 6 ] , Khasawneh et al . [ 21 ] study chatter classification in synthetic time series from turning processes 1Although learning-based approaches [ 36 ] exist to approximate topological summaries ( without guarantees ) . and Dłotko et al . [ 11 ] identify periodicity patterns in time series . In [ 22 ] , Kim et al . actually compute one-step forecasts for Bitcoin prices and classify price patterns , essentially feeding barcode statistics as supplementary features to a MLP/CNN-based regression model . Surprisingly , very few works deviate from this pipeline , with the notable exception of [ 16 ] , where VR persistent homology is not computed from a time-delay coordinate embedding , but rather from assembling observations ( within sliding windows of size n ) from a d-variate time series into a d-dimensional point cloud , followed by VR persistent homology computation . Although these works clearly demonstrate that capturing the “ shape ” of data via persistent homology provides valuable information for time series related problems , they ( 1 ) rely on handcrafted features ( i.e. , predefined barcode summary statistics , or a fixed barcode-vectorization strategy ) , ( 2 ) consider topological summaries as the single source of information and ( 3 ) only partially touch upon forecasting problems ( with the exception of [ 22 ] ) . Furthermore , in this existing line of work , sweeping a sliding window over the time series is , first and foremost , a way to construct a point cloud which represents the entire time series . Instead , in our approach , each window yields its own topological summary in the form of a persistence barcode , reminiscent to representing a sentence as a sequence of word embeddings in NLP tasks . When combined with learnable representations of persistence barcodes [ 19 , 8 ] , this perspective paves the way for leveraging recent techniques for handling learning problems with sequential data , such as attention [ 41 ] , and allows to seamlessly integrate topological features into existing neural forecasting techniques . Neural network approaches to time series forecasting . Recently , various successful neural network approaches to ( mostly probabilistic ) time series forecasting have emerged , ranging from auto-regressive neural networks as in DeepAR [ 34 ] , to ( deep ) extensions of traditional state space models , such as DeepFactors [ 42 ] or DeepState [ 32 ] . While these models are inherently tailored to the sequential nature of the forecasting problem , Li et al . [ 24 ] instead rely on the concept of ( log-sparse ) self-attention [ 41 ] , fed by the outputs of causal convolutions , and Oreshkin et al . [ 29 ] even abandon sequential neural network primitives altogether . The latter approach , solely based on operations predominantly found in feed-forward architectures , achieves state-of-the-art performance for ( point ) forecasts across several benchmarks , including the large-scale M4 competition . Yet , a common factor in all aforementioned works is that raw time series observations are directly input to the model , assuming that relevant structural characteristics of the signal can be learned . While we choose an approach similar to [ 24 ] , in the sense that we rely on self-attention , our work differs in that representations fed to the attention mechanism are not obtained through convolutions , but rather through a topological analysis step which , by its construction , captures the “ shape ” of local time series segments . 3 Topological attention . The key idea of topological attention is to analyze local segments within an input time series , x , through the lens of persistent homology . As mentioned in Section 2 , the prevalent strategy in prior work is to first construct a point cloud from x via a time-delay coordinate embedding and to subsequently compute VR persistent homology . Historically , this is motivated by studying structural properties of an underlying dynamical system , with a solid theoretical foundation , e.g. , in the context of identifying periodicity patterns [ 31 , 30 ] . In this regime , x is encoded as a point cloud in Rn by considering observations within a sliding window of size n as a point in Rn . While the time-delay embedding strategy is adequate in settings where one aims to obtain one global topological summary , it is inadequate when local structural properties of time series segments are of interest . Further , unless large ( computationally impractical ) historical time horizons are considered , one would obtain relatively sparse point clouds that , most likely , carry little information . 3.1 Time series as local topological summaries . Instead of a time-delay coordinate embedding , we follow an alternative strategy : a time series signal , x , is still decomposed into a sequence of ( overlapping ) windows , but not to yield a point cloud element , but rather to be analyzed in isolation . In the following , we only discuss the necessities specific to our approach , and refer the reader to [ 14 , 7 , 4 ] for a thorough treatment of persistent homology . To topologically analyze a length-T time series , over the time steps { 1 , . . . , T } = [ T ] , in a computationally tractable manner , lets consider a 1-dimensional simplicial complex of the form K = { { 1 } , . . . { T } , { 1 , 2 } , . . . , { T − 1 , T } } , where { i } denote 0-simplices ( i.e. , vertices ) and 1-simplices { i , j } ( i.e. , edges ) are in K if and only if i and j are two consecutive time indices . Topologically , K carries the connectivity properties of a time series of length T , which is equivalent to a straight line . This is the same for all time series of length T and thus offers no discriminative information . Persistent homology , however , lets us combine the purely topological representation of the time series with its actual values . For a specific x , let a1 ≤ · · · ≤ aT denote its increasingly sorted values and consider K0x = ∅ , Kjx = { σ ∈ K : ∀i ∈ σ : xi ≤ aj } for j ∈ [ T ] . In fact , ∅ = K0x ⊆ K1x ⊆ · · · ⊆ KTx = K forms an increasing sequence of subsets of K , i.e. , a filtration . Importantly , while K is the same for all time series of length T , the filtration , ( Kjx ) Tj=0 , is determined by the values of x . Persistent homology then tracks the evolution of topological features throughout this sequence and summarizes this information in the form of persistence barcodes . In our specific case , as K is topologically equivalent to a straight line , we only get 0-dimensional features , i.e. , connected components . Hence , we obtain one ( 0 degree ) barcode Bx . This barcode is a multiset of ( birth , death ) tuples , representing the birth ( b ) and death ( d ) of topological features . Informally , we may think of building K , piece-by-piece , according to the sorting of the xi , starting with the lowest value , and tracking how connected components appear / merge , illustrated in Fig . 2 . Remark 3.1 . The information captured throughout this process has two noteworthy properties . First , it is stable in the sense that small changes in the observation values may not cause arbitrary changes in the respective barcodes , see [ 9 ] . Second , one may equally order the negative observations , i.e. , −x , and thus obtain B−x . In that manner , the signal is analyzed from below and above . Finally , to extract local topological information , we do not compute one single barcode for x , but one for each sliding window of size n , see Fig . 1 . Given a decomposition of x into W subsequent windows , we obtain W barcodes , B1 , . . . , BW , which constitute the entry point for any downstream operation . Informally , those barcodes encode the evolution of local topological features over time .

Summary: The paper proposes topological attention, a type of self-attention mechanism that captures the topological information of the signals. In contrast to earlier works, this work focuses on local instead of global topological summaries. The main difference is attributed to the extraction of such summaries for each sliding window and not over the global historic data. The paper discusses how such sets of local summaries can be vectorized and incorporated into neural network architectures. The experimental results on an existing forecasting approach show the benefit of topological attention vs. the regular (global) attention mechanism.