Topological Attention for Time Series Forecasting
Year: 2021

Abstract:
The problem of (point) forecasting univariate time series is considered. Most approaches, ranging from traditional statistical methods to recent learning-based techniques with neural networks, directly operate on raw time series observations. As an extension, we study whether local topological properties, as captured via persistent homology, can serve as a reliable signal that provides complementary information for learning to forecast. To this end, we propose topological attention, which allows attending to local topological features within a time horizon of historical data. Our approach easily integrates into existing end-to-end trainable forecasting models, such as N-BEATS, and, in combination with the latter, exhibits state-of-the-art performance on the large-scale M4 benchmark dataset of 100,000 diverse time series from different domains. Ablation experiments, as well as a comparison to a broad range of forecasting methods in a setting where only a single time series is available for training, corroborate the beneficial nature of including local topological information through an attention mechanism.

text: The problem of (point) forecasting univariate time series is considered. Most approaches, ranging from traditional statistical methods to recent learning-based techniques with neural networks, directly operate on raw time series observations. As an extension, we study whether local topological properties, as captured via persistent homology, can serve as a reliable signal that provides complementary information for learning to forecast. To this end, we propose topological attention, which allows attending to local topological features within a time horizon of historical data. Our approach easily integrates into existing end-to-end trainable forecasting models, such as N-BEATS, and, in combination with the latter, exhibits state-of-the-art performance on the large-scale M4 benchmark dataset of 100,000 diverse time series from different domains. Ablation experiments, as well as a comparison to a broad range of forecasting methods in a setting where only a single time series is available for training, corroborate the beneficial nature of including local topological information through an attention mechanism.

heading: 1 Introduction
text: Time series are ubiquitous in science and industry, from medical signals (e.g., EEG), motion data (e.g., speed, steps, etc.) or economic operating figures to ride/demand volumes of transportation network companies (e.g., Uber, Lyft, etc.). Despite many advances in predicting future observations from historical data via traditional statistical [5], or recent machine learning approaches [29, 42, 34, 32, 24, 44], reliable and accurate forecasting remains challenging. This is not least due to widely different and often heavily domain dependent structural properties of time related sequential data.
In this work, we focus on the problem of (point) forecasting univariate time series, i.e., given a length-T vector of historical data, the task is to predict future observations for a given time horizon H . While neural network models excel in situations where a large corpus of time series is available for training, the case of only a single (possibly long) time series is equally important. The arguably most prominent benchmarks for the former type of forecasting problem are the “(M)akridakis”competitions, such as M3 [26] or M4 [28]. While combinations (and hybrids) of statistical and machine learning approaches have largely dominated these competitions [28, see Table 4], Oreshkin et al. [29] have recently demonstrated that a pure learning-based model (N-BEATS) attains state-ofthe-art performance. Interestingly, the latter approach is simply built from a collection of common neural network primitives which are not specific to sequential data.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
However, the majority of learning-based approaches directly operate on the raw input signal, implicitly assuming that viable representations for forecasting can be learned via common neural network primitives, composed either in a feedforward or recurrent manner. This raises the question of whether there exist structural properties of the signal, which are not easily extractable via neural network components, but offer complementary information. One prime example1 are topological features, typically obtained via persistent homology [7, 13]. In fact, various approaches [31, 16, 11, 21, 15] have successfully used topological features for time series analysis, however, mostly in classification settings, for the identification of certain phenomena in dynamical systems, or for purely exploratory analysis (see Section 2).
Contribution. We propose an approach to incorporate local topological information into neural forecasting models. Contrary to previous works, we do not compute a global topological summary of historical observations, but features of short, overlapping time windows to which the forecasting model can attend to. The latter is achieved via self-attention and thereby integrates well into recent techniques, such as N-BEATS [29]. Notably, in our setting, computation of topological features (via persistent homology) comes with moderate computational cost, which allows application in large-scale forecasting problems.
Problem statement. In practice, neural forecasting models typically utilize the last T observations of a time series in order to yield (point) forecasts for a given time horizon H . Under this perspective, the problem boils down to learning a function (parametrized as a neural network)
ϕ : RT → RH x 7→ ϕ(x) = y , (1)
from a given collection of inputs (i.e., length-T vectors) and targets (i.e., length-H vectors). Specifically, we consider two settings, where either (1) a large collection of time series is available, as in the M4 competition, or (2) we only have access to a single time series. In the latter setting, a model has to learn from patterns within a time series, while the former setting allows to exploit common patterns across multiple time series.

heading: 2 Related work
text: Persistent homology and time series. Most approaches to topological time series analysis are conceptually similar, building on top of work by de Silva et al. [10] and Perea & Harer [31, 30]. Herein, time series observations are transformed into a point cloud via a time-delay coordinate embedding [38] from which Vietoris-Rips (VR) persistent homology is computed. The resulting topological summaries, i.e., persistence barcodes, are then used for downstream processing. Within this regime, Gidea et al. [15] analyze the dynamics of cryptocurrencies using persistence landscapes [6], Khasawneh et al. [21] study chatter classification in synthetic time series from turning processes
1Although learning-based approaches [36] exist to approximate topological summaries (without guarantees).
and Dłotko et al. [11] identify periodicity patterns in time series. In [22], Kim et al. actually compute one-step forecasts for Bitcoin prices and classify price patterns, essentially feeding barcode statistics as supplementary features to a MLP/CNN-based regression model. Surprisingly, very few works deviate from this pipeline, with the notable exception of [16], where VR persistent homology is not computed from a time-delay coordinate embedding, but rather from assembling observations (within sliding windows of size n) from a d-variate time series into a d-dimensional point cloud, followed by VR persistent homology computation.
Although these works clearly demonstrate that capturing the “shape” of data via persistent homology provides valuable information for time series related problems, they (1) rely on handcrafted features (i.e., predefined barcode summary statistics, or a fixed barcode-vectorization strategy), (2) consider topological summaries as the single source of information and (3) only partially touch upon forecasting problems (with the exception of [22]). Furthermore, in this existing line of work, sweeping a sliding window over the time series is, first and foremost, a way to construct a point cloud which represents the entire time series. Instead, in our approach, each window yields its own topological summary in the form of a persistence barcode, reminiscent to representing a sentence as a sequence of word embeddings in NLP tasks. When combined with learnable representations of persistence barcodes [19, 8], this perspective paves the way for leveraging recent techniques for handling learning problems with sequential data, such as attention [41], and allows to seamlessly integrate topological features into existing neural forecasting techniques.
Neural network approaches to time series forecasting. Recently, various successful neural network approaches to (mostly probabilistic) time series forecasting have emerged, ranging from auto-regressive neural networks as in DeepAR [34], to (deep) extensions of traditional state space models, such as DeepFactors [42] or DeepState [32]. While these models are inherently tailored to the sequential nature of the forecasting problem, Li et al. [24] instead rely on the concept of (log-sparse) self-attention [41], fed by the outputs of causal convolutions, and Oreshkin et al. [29] even abandon sequential neural network primitives altogether. The latter approach, solely based on operations predominantly found in feed-forward architectures, achieves state-of-the-art performance for (point) forecasts across several benchmarks, including the large-scale M4 competition.
Yet, a common factor in all aforementioned works is that raw time series observations are directly input to the model, assuming that relevant structural characteristics of the signal can be learned. While we choose an approach similar to [24], in the sense that we rely on self-attention, our work differs in that representations fed to the attention mechanism are not obtained through convolutions, but rather through a topological analysis step which, by its construction, captures the “shape” of local time series segments.

heading: 3 Topological attention
text: The key idea of topological attention is to analyze local segments within an input time series, x, through the lens of persistent homology. As mentioned in Section 2, the prevalent strategy in prior work is to first construct a point cloud from x via a time-delay coordinate embedding and to subsequently compute VR persistent homology. Historically, this is motivated by studying structural properties of an underlying dynamical system, with a solid theoretical foundation, e.g., in the context of identifying periodicity patterns [31, 30]. In this regime, x is encoded as a point cloud in Rn by considering observations within a sliding window of size n as a point in Rn. While the time-delay embedding strategy is adequate in settings where one aims to obtain one global topological summary, it is inadequate when local structural properties of time series segments are of interest. Further, unless large (computationally impractical) historical time horizons are considered, one would obtain relatively sparse point clouds that, most likely, carry little information.

heading: 3.1 Time series as local topological summaries
text: Instead of a time-delay coordinate embedding, we follow an alternative strategy: a time series signal, x, is still decomposed into a sequence of (overlapping) windows, but not to yield a point cloud element, but rather to be analyzed in isolation. In the following, we only discuss the necessities specific to our approach, and refer the reader to [14, 7, 4] for a thorough treatment of persistent homology.
To topologically analyze a length-T time series, over the time steps {1, . . . , T} = [T ], in a computationally tractable manner, lets consider a 1-dimensional simplicial complex of the form
K = { {1}, . . . {T}, {1, 2}, . . . , {T − 1, T} } ,
where {i} denote 0-simplices (i.e., vertices) and 1-simplices {i, j} (i.e., edges) are in K if and only if i and j are two consecutive time indices. Topologically, K carries the connectivity properties of a time series of length T , which is equivalent to a straight line. This is the same for all time series of length T and thus offers no discriminative information.
Persistent homology, however, lets us combine the purely topological representation of the time series with its actual values. For a specific x, let a1 ≤ · · · ≤ aT denote its increasingly sorted values and consider
K0x = ∅, Kjx = {σ ∈ K : ∀i ∈ σ : xi ≤ aj} for j ∈ [T ] .
In fact, ∅ = K0x ⊆ K1x ⊆ · · · ⊆ KTx = K forms an increasing sequence of subsets of K, i.e., a filtration. Importantly, while K is the same for all time series of length T , the filtration, (Kjx)Tj=0, is determined by the values of x. Persistent homology then tracks the evolution of topological features throughout this sequence and summarizes this information in the form of persistence barcodes.
In our specific case, as K is topologically equivalent to a straight line, we only get 0-dimensional features, i.e., connected components. Hence, we obtain one (0 degree) barcode Bx. This barcode is a multiset of (birth, death) tuples, representing the birth (b) and death (d) of topological features. Informally, we may think of building K, piece-by-piece, according to the sorting of the xi, starting with the lowest value, and tracking how connected components appear / merge, illustrated in Fig. 2. Remark 3.1. The information captured throughout this process has two noteworthy properties. First, it is stable in the sense that small changes in the observation values may not cause arbitrary changes in the respective barcodes, see [9]. Second, one may equally order the negative observations, i.e., −x, and thus obtain B−x. In that manner, the signal is analyzed from below and above.
Finally, to extract local topological information, we do not compute one single barcode for x, but one for each sliding window of size n, see Fig. 1. Given a decomposition of x into W subsequent windows, we obtain W barcodes, B1, . . . ,BW , which constitute the entry point for any downstream operation. Informally, those barcodes encode the evolution of local topological features over time.

heading: 3.2 Barcode vectorization
text: Although persistence barcodes concisely encode topological features, the space of persistence barcodes, denoted as B, carries no linear structure [40] and the nature of barcodes as multisets renders them difficult to use in learning settings. Myriad approaches have been proposed to alleviate this issue, ranging from fixed mappings into a vector space (e.g., [6, 1]), to kernel techniques (e.g., [33, 23]) and, more recently, to learnable vectorization schemes (e.g., [18, 8]). Here, we follow the latter approach, as it integrates well into the regime of neural networks. In particular, the core element in learnable vectorization schemes is a differentiable map of the form
Vθ : B → R, B 7→ ∑
(b,d)∈B
sθ(b, d) , (2)
where sθ : R2 → R denotes a so called barcode coordinate function [18], designed to preserve the stability property in Remark 3.1. Upon assembling a collection of e ∈ N such coordinate functions and subsuming parameters into Θ, one obtains an e-dimensional vectorization of B ∈ B via
VΘ : B → Re, B 7→ a = ( Vθ1(B), . . . ,Vθe(B) )⊤ . (3)
Taking into account the representation of x as W persistence barcodes, we summarize the vectorization step as
TopVec : BW → RW×e, (B1, . . . ,BW ) 7→ (a1, . . . ,aW )⊤ = ( VΘ(B1), . . . ,VΘ(BW ) )⊤ . (4)
This is distinctly different to [31, 30, 21, 22] (see Section 2), where one barcode is obtained and this barcode is represented in a fixed manner, e.g., via persistence landscapes [6] or via barcode statistics.

heading: 3.3 Attention mechanism
text: In order to allow a forecasting model to attend to local topological patterns, as encoded via the aj , we propose to use the encoder part of Vaswani et al.’s [41] transformer architecture, implementing a repeated application of a (multi-head) self-attention mechanism. Allowing to attend to local time series segments is conceptually similar to Li et al. [24], but differs in the way local structural properties are captured: not via causal convolutions, but rather through the lens of persistent homology. In this setting, the scaled dot-product attention, at the heart of a transformer encoder layer, computes
O = softmax ( (AWq)(AWk)⊤√
dk
) AWv , with A Eq. (4) = (a1, . . . ,aW ) ⊤ , (5)
and Wq ∈ Re×dq ,Wk ∈ Re×dk , Wv ∈ Re×dv denoting learnable (key, value, query) projection matrices. Recall that A ∈ RW×e holds all e-dimensional vectorizations of the W persistence barcodes. In its actual incarnation, one transformer encoder layer2, denoted as AttnEnc : RW×e → RW×e, computes and concatenates M parallel instances of Eq. (5), (i.e., the attention heads), and internally adjusts dv such that d = Mdv . Composing E such AttnEnc maps, one obtains
TransformerEncoder : RW×e → RW×e
A 7→ AttnEnc1 ◦ · · · ◦ AttnEncE(A) . (6)
Finally, we use a two-layer MLP : RWe → RT (with ReLU activations) to map the vectorized output of the transformer encoder to a T -dimensional representation. Topological attention thus implements
TopAttn : BW → RT
(B1, . . . ,BW ) 7→ MLP(vec(TransformerEncoder ◦ TopVec(B1, . . . ,BW ))) , (7)
where vec(·) denotes row-major vectorization operation. Remark 3.2. Notably, as the domain of TopAttn is BW , error backpropagation stops at the persistent homology computation. However, we remark that, given recent works on differentiating through the persistent homology computation [17, 25], one could even combine topological attention with, e.g., [24], in the sense that the outputs of causal convolutions could serve as the filter function for persistent homology. Error backpropagation would then consequently allow to learn this filter function.

heading: 3.4 Forecasting model
text: While the representation yielded by topological attention (see Eq. (7)) can be integrated into different neural forecasting approaches, it specifically integrates well into the N-BEATS model of Oreshkin et al. [29]. We briefly describe the generic incarnation of N-BEATS next, but remark that topological attention can be similarly integrated into the basis-expansion variant without modifications.
2omitting the additional normalization and projection layers for brevity
x − S2 S1 −
−
SL
+
v
−
−
+
N-BEATS
y
y1
y2
yL
x
Ŝ2
Ŝ1
y
xL−1
−
ŜL
(xL−1,v)
N-BEATS (+TopAttn) Essentially, the generic N-BEATS model is assembled from a stack of L double-residual blocks. For 1 ≤ l ≤ L, each block consists of a non-linear map (implemented as a MLP)
Sl : RT → Rh , (8)
with h denoting the internal dimensionality, and two subsequent maps that yield the two-fold output of the l-th block as
xl = xl−1 −Ul ( Sl(xl−1) ) , and yl = Vl ( Sl(xl−1) ) ,
with Ul ∈ RT×h, Vl ∈ RH×h and x0 = x. While the xl yield the connection to the following computation block, the yl are used to compute the final model prediction y ∈ RH via (component-wise) summation, i.e., y = y1 + · · ·+ yL. Importantly, in this computational chain, the xl can be leveraged as an interface to integrate additional information. In our case, we enrich the input signal to each block by the output of the topological attention mechanism through concatenation (hence, Ŝl : R2T → Rh, see figure to the top-right), i.e.,
xlTopAttn = (x l,v) where v = TopAttn(B1, . . . ,BW ) . (9)
This means that the time series signal x is (1) input (in its raw form) to N-BEATS and (2) its topological attention representation, v, is supplied to each block as a complementary signal. In a similar manner (i.e., through concatenation), v can be included in much simpler models as well (see Section 4.2).
Computational complexity. Aside from the computational overhead incurred by the (multi-head) attention module, we need to compute 0-dimensional persistent homology for each sliding window. This can be done efficiently, using union find data structures, with complexity O ( mα−1(m) ) , where m = |K| = 2n−1 with n the sliding window size and α−1(·) denoting the inverse of the Ackermann function. As the latter grows very slowly, computational complexity is roughly linear for this part; see suppl. material for a detailed runtime study.

heading: 4 Experiments
text: We assess the quality of point forecasts in two different settings and perform ablation studies to isolate the impact of topological attention in the proposed regime.
Throughout all experiments, we compute persistent homology from x and −x (see Remark 3.1) using Ripser [3]. Barcode vectorization, see Eq. (3), is based on rational hat coordinate functions [18] with the position parameters (i.e., the locations of each coordinate function in R2) initialized by k-means++ clustering over all barcodes in the training data (with k set to the number of coordinate functions). This yields a representation a ∈ R2e per sliding window. Full architecture details can be found in the suppl. material.
Ablation setup. When assessing each component of topological attention in isolation, we refer to +Top as omitting the TransformerEncoder part in Eq. (7), and to +Attn as directly feeding the time series observations to the transformer encoder, i.e., omitting TopVec in Eq. (7).

heading: 4.1 Evaluation metrics
text: To evaluate the quality of point forecasts, two commonly used metrics are the symmetric mean absolute percentage error (sMAPE) and the mean absolute scaled error (MASE). Letting ŷ = (x̂T+1, . . . , x̂T+H)
⊤ denote the length-H forecast, y = (xT+1, . . . , xT+H)⊤ the true observations and x = (x1, . . . , xT )⊤ the length-T history of input observations, both scores are defined as [28]
sMAPE(y, ŷ) = 200
H H∑ i=1 |xT+i − x̂T+i| |xT+i|+ |x̂T+i| , MASE(x,y, ŷ) = 1 H
∑H i=1 |xT+i − x̂T+i|
1 T−m ∑T i=m+1 |xi − xi−m| (10)
with m depending on the observation frequency. For results on the M4 benchmark (see Section 4.3), we adhere to the competition guidelines and additionally report the overall weighted average (OWA) which denotes the arithmetic mean of sMAPE and MASE (with m pre-specified), both measured relative to a naïve (seasonally adjusted) forecast (also provided by the M4 competition as Naive2).

heading: 4.2 Single time series experiments
text: We first consider the simple, yet frequently occurring, practical setting of one-step forecasts with historical observations available for only a single length-N time series. Upon receiving a time series x ∈ RT (with T ≪ N ), a model should yield a forecast for the time point T + 1 (i.e., H = 1).

heading: 4.2.1 Dataset
text: To experiment with several single (but long) time series of different characteristics, we use 10 time series from the publicly available electricity [12] demand dataset3 and four (third-party) time series of car part demands, denoted as car-parts. Based on the categorization scheme of [37], the time series are chosen such that not only smooth time series (regular demand occurrence and low demand quantity variation) are represented, but also lumpy ones (irregular demand occurrence and high demand quantity variation). For electricity, the respective percentages are 70% vs. 30%, and, for car-parts, 75% vs. 25%. All observations are non-negative. In case of electricity, which contain measurements in 15min intervals, we aggregate (by summation) within 7h windows, yielding a total of 3,762 observations. For car-parts, demand is measured on daily basis across a time span of 7-8 years (weekends and holidays excluded), yielding 4,644 observations on average. For each time series, 20% of held-out consecutive observations are used for testing, 5% for validation.

heading: 4.2.2 Forecasting model
text: We employ a simple incarnation of the forecasting model from Section 3.4. In particular, we replace N-BEATS by a single linear map (with bias), implementing(
(B1, . . . ,BW ),x ) 7→ w⊤(x, TopAttn(B1, . . . ,BW )) + b , (11)
with (x, TopAttn(B1, . . . ,BW )) denoting the concatenation of the topological attention signal and the input time series x, as in Eq. (9). During training, we randomly extract T + 1 consecutive observations from the training portion of the time series. The first T observations are used as input x, the observation at T + 1 is used as target. Forecasts for all testing observations are obtained via a length-T rolling window, moved forward one step at a time.
In terms of hyperparameters for topological attention, we use a single transformer encoder layer with four attention heads and 32 barcode coordinate functions. We minimize the mean-squared-error via ADAM over 1.5k (electricity) and 2k (car-parts) iterations, respectively, with a batch size of 30. The initial learning rate of the linear map in Eq. (11) is set to 9e-2, the initial learning rates for the components of Eq. (7) are listed in Section 4.3.2, scaled up by a factor of 10. Regarding the latter, we empirically found that models trained on single time series typically require larger learning rates, most likely due to the reduced variation in the batches throughout training (as we sample from a single time series). All learning rates are annealed following a cosine learning rate schedule.

heading: 4.2.3 Results & Ablation study
text: We compare against several techniques from the literature that are readily available to a practitioner. This includes autoARIMA [20], Prophet [39], a vanilla LSTM model, as well as several approaches implemented within the GluonTS [2] library. With respect to the latter, we list results for a singlehidden-layer MLP, DeepAR [34] and MQ-CNN/MQ-RNN [43]. By Naive, we denote a baseline, yielding xT as forecast for xT+1. Importantly, each model is fit separately to each time series in the dataset.
For a fair comparison, we further account for the fact that forecasting models typically differ in their sensitivity to the length of the input observations, x. To this end, we cross-validate T (for all methods) using the sMAPE on the validation set. Cross-validation points are determined by the topological attention parameters W and n, i.e., the number and lengths of the sliding windows. For n ranging from 10 to 200 and W ranging from 5 to 45, we obtain a wide range of input lengths, from
3https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014
14 to 244. Instead of listing absolute performance figures, we focus on the average rank4 within the cohort of methods, as well as the average percentual difference to the best-ranking approach per time series.
Table 1 lists the overall statistics for electricity and car-parts. We observe that, while the overall ranking per dataset differs quite significantly, Lin.+TopAttn consistently ranks well. Second, the average percentual difference to the best-ranking approach per time series is low, meaning that while Lin.+TopAttn might not yield the most accurate (wrt. the sMAPE) forecasts on a specific time series, it still produces forecasts of comparable quality.
Table 2 provides the same performance statistics for an ablation study of the topological attention components. Specifically, we combine the linear model of Eq. (11) with each component of topological attention in isolation.
Some observations are worth pointing out: First, the linear model (Lin.) alone already performs surprisingly well. This can possibly be explained by the fact that the task only requires one-step forecasts, for which the historical length-T observations (directly preceding the forecast point) are already quite informative. Second, directly including topological features (i.e., +Top) has a confounding effect. We hypothesize that simply vectorizing local topological information from all sliding windows, without any focus, obfuscates relevant information, rather than providing a reliable learning signal. This also highlights the importance of attention in this context, which, even when directly fed with observations from each sliding window (i.e., +Attn), exhibits favorable performance (particularly on car-parts). However, in the latter strategy, the input dimensionality for the transformer encoder scales with the sliding window size n. Contrary to that, in case of topological attention, the
input dimensionality is always fixed to the of number of coordinate functions, irrespective of the sliding window size n.

heading: 4.3 Large-scale experiments on the M4 benchmark
text: Different to Section 4.2, we now consider having multiple time series of different lengths and characteristics available for training. Further, instead of one-step forecasts, the sought-for model needs to output (multi-step) point forecasts for time horizons H > 1.
4the sMAPE determines the rank of a method per time series; these ranks are then averaged over all time series

heading: 4.3.1 Dataset
text: Experiments are based on the publicly available M4 dataset5, consisting of 100,000 time series from six domains, aggregated into six subgroups that are defined by the frequency of observations (i.e., yearly, quarterly, monthly, etc.). Forecasting horizons range from H = 6 (yearly) to H = 48 (hourly). The test set is fixed and contains, for all time series per subgroup, exactly H observations to be predicted (starting at the last available training time point); see suppl. material for dataset statistics.

heading: 4.3.2 Forecasting model
text: We employ the forecasting model6 of Section 3.4 and closely follow the architecture and training configuration of [29, Table 18]. Our implementation only differs in the hidden dimensionality of N-BEATS blocks (128 instead of 512) and in the ensembling step. In particular, for each forecast horizon (i.e., for each subgroup), [29] train multiple models, varying T from T = 2H to T = 7H , using ten random initializations and three separate loss functions (sMAPE, MASE, MAPE). One final forecast per time series is obtained by median-aggregation of each model’s predictions. In our setup, we solely rely on the sMAPE as loss function, vary T only from T = 2H to T = 5H , but still use ten random initializations. Even with this (smaller) ensemble size (40 models per subgroup, instead of 180), N-BEATS alone already outperforms the winner of M4 (see Table 3). As we are primarily interested in the effect of integrating topological attention, sacrificing absolute performance for a smaller ensemble size is incidental.
In terms of hyperparameters for topological attention, the length, n, of sliding windows is set to n = ⌊0.7 · T ⌋, where T varies per subgroup as specified above. The model uses 20 transformer encoder layers with four attention heads and 64 structure elements for barcode vectorization. For optimization, we use ADAM with initial learning rates of 1e-3 (for N-BEATS and the MLP part of Eq. (7)), 8e-3 (TopVec) and 5e-3 (TransformerEncoder). All learning rates are annealed according to a cosine learning rate schedule over 5,000 iterations with a batch size of 1,024.

heading: 4.3.3 Results & Ablation study
text: Table 3 lists the sMAPE and OWA for the winner of the M4 competition [35], as well as the Naive2 baseline (with respect to which the OWA is computed) and the M4 benchmark approach, obtained as the arithmetic mean over simple, Holt, and damped exponential smoothing.
Table 4: Ablation study
Method sMAPE OWA
N-BEATS 11.488 0.827
+Top 11.505 0.920 +Attn 11.492 0.826 +TopAttn 11.466 0.824
In terms of the OWA, we see an overall 0.4% improvement over N-BEATS and a 1.2% improvement over the M4 winner [35]. In particular, topological attention performs well on the large yearly / monthly subgroups of 23k and 48k time series, respectively. While OWA scores are admittedly quite close, the differences are non-negligible, considering the large corpus of 100k time series. In fact, several methods in the official M4 ranking differ by an even smaller amount with respect to the OWA measure.
5available at https://github.com/Mcompetitions/M4-methods 6based on the N-BEATS reference implementation https://github.com/ElementAI/N-BEATS
Similar to the ablation results of Section 4.2, the ablation study in Table 4 (conducted for T = 2H only) reveals the beneficial effect of topological attention, in particular, the beneficial nature of allowing to attend to local topological features. Contrary to the ablation in Table 2, we see that in this large-scale setting, neither topological features (+Top), nor attention (+Attn) alone yield any improvements over the already strong N-BEATS model. In fact, when integrated separately into N-BEATS, both components even deteriorate performance in terms of the sMAPE.
Number of parameters. Regarding a comparison of N-BEATS and N-BEATS+TopAttn in terms of the number of parameters, we note that the particular N-BEATS incarnation in our experiments has a total of ≈1.7M parameters; our approach adds about 700k additional parameters. To assess whether the reported improvements are simply due to a larger overall model, we ran additional experiments increasing the size of N-BEATS (by doubling the hidden dimensionality) to ≈6.4M parameters. As in Table 4, this study was performed using a historical time horizon of T = 2H and all results are averaged over 10 random initializations of each model. In terms of the OWA, the larger N-BEATS model achieves an average score of 0.826 vs. 0.827 for N-BEATS and 0.824 for N-BEATS+TopAttn (the latter two numbers correspond to the results in Table 4). Hence, substantially increasing the number of parameters for N-BEATS indeed improves the overall score (as expected), but does not reach the OWA result of our model.

heading: 5 Conclusion
text: While several prior forecasting works have pointed out the relevance of local structural information within historical observations (e.g., [24]), it is typically left to the model to learn such features from data. Instead, we present a direct approach for capturing the “shape” of local time series segments via persistent homology. Different to the typical application of the latter in signal analysis, we capture the evolution of topological features over time, rather than a global summary, and allow a forecasting model to attend to these local features. The so obtained topological attention mechanism yields a complementary representation that easily integrates into neural forecasting approaches. In combination with N-BEATS [29], for instance, large-scale experiments on the M4 benchmark provide evidence that including topological attention indeed allows to obtain more accurate point forecasts.
Societal impact. Due to the ubiquity of time series data, forecasting in general, certainly touches upon a variety of societally relevant and presumably sensible areas. As our work has potential impact in that sense, we perform large-scale experiments over a wide variety of time series from different domains, thereby obtaining a broad picture of the overall forecasting performance.
Source code is publicly available at https://github.com/plus-rkwitt/TAN.
Acknowledgments & Disclosure of funding
This research was supported in part by the Austrian Science Fund (FWF): project FWF P31799-N38 and the Land Salzburg (WISS 2025) under project numbers 20102- F1901166-KZP and 20204- WISS/225/197-2019. The first author also gratefully acknowledges financial support from Porsche Holding Austria and Land Salzburg within the WISS 2025 project “KFZ” (P1900123). Finally, we like to thank all anonymous reviewers for their constructive feedback during the review process.

Review summaries:
The paper presents an approach to combine topological features to time-series forecasting. Essentially, the method computes 1D topological barcodes at each time-steps, pass them to a transformer and then provides those features as additional input to NBEATS. Experiments are conducted on two datasest when performing local forecast (e.g. forecasting the time-series independently) and on M4 when using global forecast (learning the model jointly across time-series). An improvement of 0.4%/1.2% is reported on M4 for OWA/sMAPE. 

The paper proposes topological attention, a type of self-attention mechanism that captures the topological information of the signals. In contrast to earlier works, this work focuses on local instead of global topological summaries. The main difference is attributed to the extraction of such summaries for each sliding window and not over the global historic data. The paper discusses how such sets of local summaries can be vectorized and incorporated into neural network architectures. The experimental results on an existing forecasting approach show the benefit of topological attention vs. the regular (global) attention mechanism.

This paper proposes a new approach for one-dimensional time-series forecasting that is based on augmenting the data with topological information. Specifically, the authors design an attention module that extract the local connections in their windowed persistent homology barcodes, and concatenates this information to the original time-series. The authors evaluate their method on two scenarios: a single series and multiple time-series elements. Their techniques shows modest improvement over existing approaches.