TAMP-S2GCNets: Coupling Time-Aware Multipersistence Knowledge Representation with Spatio-Supra Graph Convolutional Networks for Time-Series Forecasting

Graph Neural Networks ( GNNs ) are proven to be a powerful machinery for learning complex dependencies in multivariate spatio-temporal processes . However , most existing GNNs have inherently static architectures , and as a result , do not explicitly account for time dependencies of the encoded knowledge and are limited in their ability to simultaneously infer latent time-conditioned relations among entities . We postulate that such hidden time-conditioned properties may be captured by the tools of multipersistence , i.e , a emerging machinery in topological data analysis which allows us to quantify dynamics of the data shape along multiple geometric dimensions . We make the first step toward integrating the two rising research directions , that is , time-aware deep learning and multipersistence , and propose a new model , Time-Aware Multipersistence Spatio-Supra Graph Convolutional Network ( TAMP-S2GCNets ) . We summarize inherent time-conditioned topological properties of the data as time-aware multipersistence Euler-Poincaré surface and prove its stability . We then construct a supragraph convolution module which simultaneously accounts for the extracted intra- and inter-dependencies in the data . Our extensive experiments on highway traffic flow , Ethereum token prices , and COVID-19 hospitalizations demonstrate that TAMP-S2GCNets outperforms the state-of-the-art tools in multivariate time series forecasting tasks . 1 INTRODUCTION . Multivariate time series forecasting plays an integral role in virtually every aspect of societal functioning , from biosurveillance to financial analytics to intelligent transportation solutions . In the last few years , Graph Convolutional Networks ( GCNs ) have emerged as a powerful alternative to more conventional time series predictive models . Despite their proven success , GCNs tend to be limited in their ability to simultaneously infer latent temporal relations among entities ( such as correlations both within a time series and in-between time series or the joint spatio-temporal dependencies ) . More generally , most existing GCNs architectures are inherently static and as such , do not explicitly account for time-conditioned properties of the encoded knowledge about the complex dynamic phenomena . At the same time , recent studies on integrating shape properties of the complex data into deep learning ( DL ) models indicate that topological representations , obtained using the tools of single parameter persistence , can bring an invaluable insight into the system organization and enhance the resulting graph learning mechanisms ( Hofer et al. , 2019 ; Carrière et al. , 2020 ; Carlsson & Gabrielsson , 2020 ; Horn et al. , 2021 ) . ( By shape here we broadly understand data properties which are invariant under continuous deformations , e.g. , stretching , bending , and twisting ) . However , in many applications , particularly , involving spatio-temporal processes , the data exhibit richer structures which can not be well encoded with a single parameter persistence . We postulate that many critical hidden time-conditioned interrelations which are inaccessible with other methods can be captured by the emerging machinery of multipersistence . Multipersistence , or multiparameter persistence ( MP ) generalizes the notion of single parameter persistence to a case when dynamics of the inherent data shape is discerned along multiple geometric dimensions ( Carlsson & Zomorodian , 2009 ) . Despite its premise , applications of MP in any discipline remain nascent at best ( Riess & Hansen , 2020 ; Kerber , 2020 ) . We make the first step on a path of bridging the two emerging directions , namely , time-aware DL with time-conditioned MP representations of complex dynamic phenomena . By time-conditioned MP representations , we mean the most salient topological properties of the data that manifest themselves over time . To summarize such time-conditioned topological properties , we first introduce a dynamic Euler-Poincaré surface as a new MP summary . We then propose a directed multilayer supra graph abstraction to represent a sequence of time-varying objects and develop a supragraph convolution module which allows us to simultaneously learn co-evolving intra- and inter-dependencies ( i.e. , spatial and temporal correlations ) in the complex high-dimensional data . The key novelty of this paper are summarized as follows : • This is the first work to bridge the concepts of MP with the time-aware learning paradigm . Applications of MP in any field of study are currently nascent . • We introduce a new time-aware multipersistence invariant , a dynamic Euler-Poincaré surface . We prove its stability and show its substantial computational gains and high utility for encoding the time-conditioned knowledge . • We propose a mathematical abstraction of directed multilayer supra graph for time-conditioned knowledge representation and construct a new Time-Aware Multipersistence Spatio-Supra Graph Convolutional Network ( TAMP-S2GCNets ) which simultaneously learns latent temporal interand intra-relations among entities in the complex high-dimensional data . • We perform expansive forecasting experiments , in application to highway traffic flow , Ethereum token prices , and COVID-19 hospitalizations . Our findings demonstrate superior predictive performance , versatility and computational efficiency of TAMP-S2GCNets , compared to the stateof-the-art methods in multivariate time series forecasting . 2 RELATED WORKS . Multipersistence Despite that MP demonstrates very promising results in terms of improving accuracy , tractability and robustness , applications of MP in ML are virtually non-existent ( Wright & Zheng , 2020 ; Riess & Hansen , 2020 ; Kerber , 2020 ) . Some notable efforts in the direction to develop MP summaries which are suitable for integration with ML models include Multiparameter Persistence Kernel of Corbet et al . ( 2019 ) , Multiparameter Persistence Landscapes ( MP-L ) of Vipond ( 2020 ) , and Multiparameter Persistence Images ( MP-I ) of Carrière & Blumberg ( 2020 ) which are based on the concept of slicing , that is , restricting the MP module to an affine line ( or single parameter persistence ) ( Cerri et al. , 2013 ; Landi , 2014 ) . Such slicing methods enjoy a number of important stability guarantees but tend to be computationally expensive even in static scenarios , which makes them infeasible for time series forecasting tasks . In turn , the most recent results of Beltramo et al . ( 2021 ) ( i.e. , Euler Characteristic Surfaces ) and Coskunuzer et al . ( 2021 ) ( i.e. , Multiparameter Persistence Grids ) introduce pointwise MP representations , in application to static point clouds and graphs . Such pointwise representations are weaker invariants but are substantially more computationally efficient . Integration of pointwise representation with ML models has not been yet investigated . Here we propose the first time-aware pointwise MP representation , a dynamic Euler-Poincaré surface , derive its theoretical properties and integrate it with GCN in time series forecasting tasks . Spatial-Temporal Graph Models and Forecasting Recent studies ( Li et al. , 2018 ; Yu et al. , 2018 ; Yao et al. , 2018 ) introduce graph convolution methods into spatio-temporal networks for multivariate time series forecasting which , as a result , allows for better modeling of dependencies among entities ( Wu et al. , 2019 ; Bai et al. , 2020 ; Cao et al. , 2020 ) and handling data heterogeneity . Despite the GCN successes , designs of the existing spatial-temporal GCNs largely rely on the pre-defined graph structures . As such , GCNs are restricted in their ability to explicitly integrate time dimension into the knowledge representation and learning mechanisms , thereby , limiting model adaptivity to the dynamic environments and requiring more frequent retraining . Most recently , Chen et al . ( 2021 ) propose a time-aware GCN , Z-GCNETs , for time series forecasting which integrates zigzag persistence images based on a single filtration , as the primary time-conditioned topological representation . In general , the zigzag concept can be combined with MP , but it requires more fundamental advances in the theory of algebraic topology . As such , our time-aware MP learning approach may be viewed as complementary to zigzag persistence , while considering time-changing connections of graph structures in dynamic networks . 3 TIME-AWARE MULTIPERSISTENCE EULER CHARACTERISTIC SURFACES . Spatio-Temporal Graph Construction We define a spatial network at time step t as Gt = ( Vt , Et , At , Xt ) , where Vt is a set of nodes and Et is a set of edges . We let |Vt| = N and |Et| = Mt . The adjacency matrix At ∈ RN×N , and Xt = { xt,1 , xt,2 , . . . , xt , N } > ∈ RN×FN is the node feature matrix with feature dimension FN . To construct the spatial network Gt , we can build the adjacency matrix At based on ( i ) the prior knowledge of graph structure : first-order neighbours , i.e. , At , uv = 1 if the node u and node v have a connection in the dynamic graph at the time step t ; and ( ii ) the Radial Basis Function ( RBF ) : degrees of similarity between instances ( i.e. , nodes ) in Xt , i.e. , At , uv = 1exp ( −||xt , u−xt , v||2/γ ) ≤ , where γ denotes the length scale parameter and denotes the threshold parameter filters noisy edges . Let T be the total number of time steps . Given a sequence of observations on a multivariate variable , X = { X1 , X2 , . . . , XT } ∈ RN×FN×T with T timestamps and FN node attributes , we construct spatio-temporal networks G = { G1 , G2 , . . . , GT } via either prior knowledge of network structure or applying RBF to the node feature matrix . Single Parameter Persistence Persistent homology ( PH ) based on one parameter discerns shape of the complex data along a single geometric dimension . The goal is to select some suitable parameter of interest and then to study a graph Gt not as a single object , but as a sequence of nested subgraphs , or a graph filtration G1t ⊆ G2t . . . ⊆ Gmt = Gt , induced by this evolving scale parameter . Armed with such filtration , we can then assess which structural patterns ( e.g. , loops and cavities ) appear/disappear and record their lifespans . To make the counting process more efficient and systematic , we build a simplicial complex Cit from each subgraph Git , resulting in a filtration C1t ⊆ C2t . . . ⊆ Cmt ( e.g. , clique complexes ) . For example , to construct such filtration , a common method is to consider a filtering function f : Vt 7→ R and the corresponding increasing set of thresholds { αi } m1 such that Cit = { ∆ ∈ Ct : maxv∈∆ f ( v ) ≤ αi } . The resulting construction is called a sublevel set filtration of f , and f can be selected , for instance , as degree , centrality , or eccentricity function ( Hofer et al. , 2020 ; Cai & Wang , 2020 ) . Similarly , f can be defined on the set of edges Et . More details on a single filtration PH can be found in Appendix B. Time-Aware Multiparameter Persistence ( TAMP ) Data in many applications , particularly , involving spatio-temporal modeling , might be naturally indexed by multiple parameters , e.g. , real time traffic flow and optimal route in urban transportation analytics . Alternatively , the primary focus might be on discerning shape properties of the complex data along multiple dimensions . For instance , to better predict cryptocurrency prices and manage cryptomarket investment performance , we may need to evaluate structural patterns in cryptoasset dynamics not along one dimension but simultaneously along the volume of transactions and transaction graph betweenness , as a measure of joint perception of the cryptomarket volatility among the key investors . Such multidimensional analysis of topological and geometric properties can be addressed using generalization of PH based on a single filtration to a multifiltration case . That is , the MP idea is to simultaneously assess shape characteristics of Gt based on a multivariate filtering function F : Vt 7→ Rd . As a result , e.g. , for d = 2 and a set of nondecreasing thresholds { αi } m1 and { βj } n1 , instead of a single filtration of complexes , we get a bifiltration of complexes { Cαi , βjt | 1 ≤ i ≤ m , 1 ≤ j ≤ n } such that if βk < βl , then C αi , βk t ↪→ C αi , βl t and if αi < αj , then C αi , βk t ↪→ C αj , βk t . Finally , this bifiltration induces a bigraded MP module { Hk ( Cαi , βkt ) } , where Hk is the kth homology group . Inspired by Beltramo et al . ( 2021 ) ; Coskunuzer et al . ( 2021 ) , we propose a new time-aware MP summary , namely , a Dynamic Euler-Poincaré Surface . Definition 3.1 ( Dynamic Euler-Poincaré Surface ) . Let { Gt } Tt=1 be a series of time-varying graphs . Let F = ( f , g ) be a multivariate filtering function F : Vt 7→ R2 with thresholds I = { ( αi , βj ) | 1 ≤ i ≤ m , 1 ≤ j ≤ n } . Let Cαi , βjt be the clique complex of the induced subgraph G αi , βj t = F−1 ( ( −∞ , αi ] × ( −∞ , βj ] ) , t = 1 , 2 , . . . , T and χ be the Euler–Poincaré characteristic . Then , a sequence of time-evolving m × n-matrices { Et } Tt=1 such that Etij = χ ( C αi , βj t ) for 1 ≤ i ≤ m , 1 ≤ j ≤ n is called Dynamic Euler-Poincaré Surface ( DEPS ) . ( Figure 2 in Appendix C shows a toy example how DEPS is computed . ) Theoretical Guarantees of DEPS Consider two graphs G+ and G− , where time index t is suppressed for brevity . Let F : V± 7→ R2 be a multivariate filtering function with thresholds I = { ( αi , βj ) | 1 ≤ i ≤ m , 1 ≤ j ≤ n } . Let C± be the clique complexes of G± , and let Ĉ± = { C+ij } be the bifiltration induced by ( C± , F , I ) as before . Let E± be the corresponding Euler-Poincaré Surfaces ( i.e. , m×n matrices ) . Then , set ‖E+−E−‖1,1 = ∑m i=1 ∑n j=1 |E + ij −E − ij | as the distance between E+ and E− , where ‖ · ‖1,1 is the vectorized L1 matrix norm . We now introduce an L1-based MP metric instead of using L∞-based metrics like , e.g. , matching or interleaving , due to the nature of our summaries E± ( see Remark C.1 in Appendix C ) . Let Dfk ( C± ) and Dgk ( C± ) be the kth single parameter persistence diagrams ( PDs ) of C± for filtrations induced by functions f , g : V± 7→ R , respectively ( see Appendix B ) . Let C±i∗ and C ± ∗j be clique complexes corresponding to G±i∗ = f−1 ( ( −∞ , αi ] ) and G ± ∗j = g −1 ( ( −∞ , βj ] ) . Define the ith column distance for the kth PDs as Dki∗ ( Ĉ + , Ĉ− ) =W1 ( Dgk ( C + i∗ ) , D g k ( C − i∗ ) ) , whereW1 is the Wasserstein-1 distance . Similarly , the jth row distance for kth PDs is Dk∗j ( Ĉ + , Ĉ− ) =W1 ( Dfk ( C + ∗j ) , D f k ( C − ∗j ) ) . Definition 3.2 ( Weak L1-metric for Multipersistence ) . The weak L1-metric between Ĉ± is D ( Ĉ+ , Ĉ− ) = max { Dc ( Ĉ + , Ĉ− ) , Dr ( Ĉ + , Ĉ− ) } , s.t . Dc ( Ĉ+ , Ĉ− ) = ∑M k=0 ∑m i=1 D k i∗ ( Ĉ + , Ĉ− ) and Dr ( Ĉ+ , Ĉ− ) = ∑M k=0 ∑n j=1 D k ∗j ( Ĉ + , Ĉ− ) . Now , we can state our stability result for Euler-Poincaré Surfaces . Theorem 3.1 . Let G± , F , Ĉ± , E± be as defined above . Then , the Euler-Poincaré Surfaces are stable with respect to the weak L1-metric , i.e. , ‖E+ − E−‖1,1 ≤ C ·D ( Ĉ+ , Ĉ− ) for some C > 0 . The proof of the theorem is given in Appendix C. This stability result implies that the distances between multiparameter PDs control the distance between the resulting Euler-Poincaré Surfaces . By combining with the stability result for PDs ( Cohen-Steiner et al. , 2007 ) , one can conclude that the small changes in the MP filtering function F : Vt 7→ Rd or the small changes in the input data can result only in a small change in DEPS surfaces . For further discussion on implications of the stability result , please refer to Remark C.2 .

Summary: This paper proposes the use of multi-parameter persistence (an emergent research topic in topological data analysis) to capture latent time-conditioned relations among nodes in a GNN. To do so, the authors introduce a dynamic Euler-Poincaré surface as a new multi-parameter persistence summary, and prove its stability and empirically show its computational efficiency. A "supra" graph convolution module is then proposed in order to allow for simultaneously learning co-evolving spatial and temporal correlations in the complex multivariate time-series data. The superior predictive performance of the proposed approach is shown empirically based on 3 different multi-variate time-series datasets from 3 different domains: highway traffic flow, Ethereum token prices, and COVID-19 hospitalizations.