Multivariate Time Series Forecasting with Latent Graph Inference

This paper introduces a new architecture for multivariate time series forecasting that simultaneously infers and leverages relations among time series . We cast our method as a modular extension to univariate architectures where relations among individual time series are dynamically inferred in the latent space obtained after encoding the whole input signal . Our approach is flexible enough to scale gracefully according to the needs of the forecasting task under consideration . In its most straight-forward and general version , we infer a potentially fully connected graph to model the interactions between time series , which allows us to obtain competitive forecast accuracy compared with the state-of-the-art in graph neural networks for forecasting . In addition , whereas previous latent graph inference methods scale O ( N2 ) w.r.t . the number of nodes N ( representing the time series ) , we show how to configure our approach to cater for the scale of modern time series panels . By assuming the inferred graph to be bipartite where one partition consists of the original N nodes and we introduce K nodes ( taking inspiration from low-rank-decompositions ) we reduce the time complexity of our procedure to O ( NK ) . This allows us to leverage the dependency structure with a small trade-off in forecasting accuracy . We demonstrate the effectiveness of our method for a variety of datasets where it performs better or very competitively to previous methods under both the fully connected and bipartite assumptions . 1 INTRODUCTION . Time Series Forecasting ( TSF ) has been widely studied due to its practical significance in a wide variety of applications such as climate modelling ( Mudelsee , 2019 ) , supply chain management in retail ( Larson , 2001 ; Böse et al. , 2017 ) , market analysis in finance ( Andersen et al. , 2005 ) , traffic control ( Li et al. , 2017 ) and medicine ( Kaushik et al. , 2020 ) . Petropoulos et al . ( 2020 ) provide a non-systematic overview of further applications . In TSF , given a sequence of data points indexed over time , we aim to estimate its future values based on previously observed data . Data is often multivariate , meaning that multiple variables vary over time , each variable may not only depend on its own historical values , but also on other variables ’ past . Efficiently modelling the dependencies between these variables is still an open problem . Multivariate Time Series ( MTS ) methods aim to leverage the dependencies between variables in order to improve the forecasting accuracy . Some classical MTS forecasting algorithms such as Vector Autoregression ( VAR ) ( Lütkepohl , 2005 ) or Gaussian Processes ( Roberts et al. , 2013 ) only consider linear dependencies among variables . A natural way to model non-linear dependencies in deep learning is via Graph Neural Networks ( GNNs ) ( Bruna et al. , 2013 ; Defferrard et al. , 2016 ; Kipf and Welling , 2016 ) . In fact , GNNs have been successfully applied for the Multivariate Time Series forecasting ( Li et al. , 2017 ; Yu et al. , 2017 ; Chen et al. , 2020a ) , leveraging the relations among different series . But these methods require a pre-defined adjacency matrix which may only be available in some specific datasets , for example , traffic datasets , where it can be constructed from the spatial structure of a city . More recently , a family of methods that do not require a pre-defined adjacency matrix have been proposed ( Wu et al. , 2019 ; Franceschi et al. , 2019 ; Wu et al. , 2020 ; Shang et al. , 2021 ) . In this case , a latent graph representation is inferred while forecasting , allowing to operate on a larger variety of datasets . Our work belongs to this category . However , inferring all pairwise relations may come at a higher computational cost , as the number of relations scales quadratically O ( N2 ) w.r.t . the number of nodes . This limits the scalability to large datasets . Additionally , the above mentioned latent graph inference methods perform message passing update at every time step iteration which can also be expensive . To overcome these limitations , we propose a new latent graph inference algorithm for Multivariate Time Series forecasting that is more efficient than previous algorithms while achieving better or competitive performance . We cast the latent graph inference as a modular and easy-to-implement extension to current univariate models . The graph is dynamically inferred per inputted data stream allowing a more flexible representation than a static graph for the whole dataset . Additionally , we optionally reduce the complexity from O ( N2 ) ( Fully Connected Assumption ) to O ( NK ) ( Bipartite Assumption ) where K ⌧ N for a small trade off in performance . 2 BACKGROUND . 2.1 TIME SERIES FORECASTING . In time series forecasting we want to estimate a future time series xt+1 : T given its past xt0 : t where t0 t T indexes over time , and ( optionally ) some context information c. For the multivariate case we assume the time series is composed of N variates at a time such that xt0 : T = { x1 , t0 : T , . . . , xN , t0 : T } 2 RN⇥T t0+1 . In this section we distinguish two main categories of time series forecasting methods , Global Univariate and Multivariate . Global Univariate methods : In this case we only use the past of each univariate to predict its future . However , the model weights ✓u are shared across all univariate time series . More formally : x̂i , t+1 : T = fu ( xi , t0 : t , ci ; ✓u ) ( 1 ) where x̂ denotes the estimated values , i 2 { 1 , . . . , N } indexes over multivariates and fu ( · ) is the estimator function with learnable parameters ✓u shared across time series . Conditioning on the past of each univariate may limit the performance of the forecasting algorithm compared to multivariate ones . Despite that , it simplifies the design of f✓ and already provides reasonable results . A popular example of univariate models would be ( Salinas et al. , 2020 ) . Multivariate methods : Multivariate methods condition on all past data ( all N variates ) and directly predict the multivariate target . More formally : x̂t+1 : T = fm ( xt0 : t , c ) . ( 2 ) Different variables may be correlated and/or depend on the same con-founders . For example , in retail forecasting , PPE masks and antibacterial soaps jointly increased in demand during the early days of the COVID-19 pandemic . In traffic forecasting , an increase of the outcome traffic flow in a given neighborhood may result in an increase of the income traffic flow on another one . Modelling these dependencies may improve the forecasting accuracy , but it may come at a cost of higher complexity and hence more expensive algorithms , specially when trying to model all pairwise interactions between variates . 2.2 GRAPH NEURAL NETWORKS . Graph Neural Networks ( GNNs ) ( Bruna et al. , 2013 ; Defferrard et al. , 2016 ; Kipf and Welling , 2016 ) operate directly on graph structured data . They have gained a lot of attention in the last years due to their success in a large variety of domains which benefit from modelling interactions between different nodes/entities . In the context of multivariate time series , GNNs can be used to model the interactions between time series . In this work we consider the type of GNN introduced by ( Gilmer et al. , 2017 ) . Given a graph G = ( V , E ) with nodes vi 2 V and edges eij 2 E , we define a graph convolutional layer as : mij = e ( h l i , h l j ) mi = X j2N ( i ) ↵ijmij h l+1 i = h ( h l i , mi ) ( 3 ) Where e and h are the edge and node functions , usually approximated as Multi Layer Perceptrons ( MLPs ) , hli 2 Rnf is the nf-dimensional embedding of a node vi at layer l and mij is the edge embedding that propagates information from node vj to vi . A GNN is constructed by stacking multiple of these Graph Convolutional Layers hl+1 = GCL [ hl , E ] . Additionally , in Equation 3 we include ↵ij 2 ( 0 , 1 ) which is a scalar value that performs the edge inference or attention over the neighbors similarly to Veličković et al . ( 2017 ) . As done in ( Satorras et al. , 2021 ) , we choose this value to be computed as the output of a function ↵ij = ↵ ( mij ) where ↵ is composed of just a linear layer followed by a sigmoid activation function . 3 RELATED WORK . Time series forecasting has been extensively studied in the past due to its practical significance with a number of recent overview articles available ( et al. , 2020 ; Benidis et al. , 2020 ; Lim and Zohren , 2021 ) . Traditionally , most classical methods are univariate in nature ( see e.g. , Hyndman and Athanasopoulos ( 2017 ) for an overview ) . While some of these have multi-variate extensions ( e.g. , ARMA and VARMA models ) , they are limited by the amount of related time series information they can incorporate . Dynamic factor models ( Geweke , 1977 ; Wang et al. , 2019a ) are fore-runners of a family of models that has recently received more attention , the so-called global models ( Januschowski et al. , 2019 ; Montero-Manso and Hyndman , 2022 ) . These global models estimate their parameters over an entire panel of time series , so thereby taking advantage of cross time series learning , but still produce a univariate forecast . Many such global models have been proposed building on the main neural network architectures like RNNs ( Salinas et al. , 2020 ; Liberty et al. , 2020 ; Bandara et al. , 2019 ) , CNNs ( Wen et al. , 2017 ; Chen et al. , 2020b ) , Transformers ( Li et al. , 2019 ; Lim et al. , 2021 ; Eisenach et al. , 2020 ) and also combining classical probabilistic models with deep learning ( Rangapuram et al. , 2018 ; Kurle et al. , 2020 ; de Bézenac et al. , 2020 ) . However , these global models do not explicitly model the relationship between the time series in the panel . Most recently , global multi-variate forecasting models have received attention , in particular models that attempt to capture the relationship of the time series via a multi-variate likelihood ( Rasul et al. , 2020 ; 2021 ; de Bézenac et al. , 2020 ; Salinas et al. , 2019 ) . Here , we attempt to capture the multivariate nature of many modern forecasting problems primarily by using a multi-variate time series as input . For this , a natural way to model and exploit the relationship between time series is via Graph Neural Networks ( Bruna et al. , 2013 ; Defferrard et al. , 2016 ; Kipf and Welling , 2016 ) which have been successfully applied to a wide variety of deep learning domains where exploiting relations between entities/nodes can benefit the prediction task . Even in those cases where edges are not explicitly provided in the dataset , attention or a latent graph can be inferred from the node embeddings such that GNNs can still leverage the structure of the data . Some examples of latent graph inference or attention are ( Wang et al. , 2019b ) in point clouds , ( Franceschi et al. , 2019 ) in semi-supervised graph classification , ( Ying et al. , 2018 ) in hierarchical graph representation learning , ( Kipf et al. , 2018 ) in modelling dynamical systems , ( Kazi et al. , 2020 ) in zero-shot learning and 3D point cloud segmentation , ( Garcia and Bruna , 2017 ; Kossen et al. , 2021 ) in image classification , ( Cranmer et al. , 2020 ) in inferring symbolic representations and ( Fuchs et al. , 2020 ; Satorras et al. , 2021 ) in molecular property prediction . In Multivariate Time Series ( MTS ) forecasting we can leverage dependencies between time series by exchanging information among them . Lai et al . ( 2018 ) ; Shih et al . ( 2019 ) are some of the first deep learning approaches designed to exploit those pair-wise dependencies . More recent methods , ( Li et al. , 2017 ; Yu et al. , 2017 ; Seo et al. , 2018 ; Zhao et al. , 2019 ) are built in the intersection of Graph Neural Networks and time series forecasting but they require a pre-defined adjacency matrix . Lately , new methods that infer a latent graph from the node embeddings have been introduced in MTS forecasting ( Wu et al. , 2020 ; Shang et al. , 2021 ; Cao et al. , 2021 ) , and in MTS anomaly detection ( Zhang et al. , 2020 ; Deng and Hooi , 2021 ) . These methods can be applied to any dataset even when there is not an explicitly defined adjacency matrix . But this comes with a limitation , inferring the edges can be expensive since those scale quadratically O ( N2 ) w.r.t the number of nodes/variables N or O ( N3 ) in ( Cao et al. , 2021 ) . Our approach is related to ( Wu et al. , 2020 ; Shang et al. , 2021 ) , but in contrast a ) Our latent graph is dynamically inferred for each inputted data stream instead of a static graph for the whole dataset which allows a more flexible graph representation . b ) It is modular since it can be added as an extension to standard univariate methods after encoding the input signal xt0 : t , this also makes the graph operation cheaper since the message exchange is only done at time step t. c ) Optionally reduces the number of edges from O ( N2 ) to O ( NK ) where K ⌧ N with the bipartite assumption .

Summary: The paper proposes using graph neural net (GNN) operations to combine per-series embeddings, to enable multivariate forecasting.  Specifically, N individual series are separately encoded for a given time window to get representations per series.  These representations are then updated with a GNN - either assuming a fully connected graph (with edge weights computed as part of the model), or using a bipartite graph (using a smaller set of K << N auxilliary nodes).  I.e., after multiple layers of the GNN, the representations are updated with information from the other series' representations.  Finally, the final representations are passed through per-series decoders.  The latent bipartite graph formulation enables more efficient operation of the GNN component as instead of computing O(N^2) messages only O(NK) need to be computed in a given pass.  The authors compare the proposed approach with other Graph based forecasting approaches (including ablated versions of the proposed approach) on 6 datasets - 2 where there is a given graph structure (so past work requiring the graph structure to be known can be used) and 4 where there is not.  They demonstrate competitive performance of the proposed approach (including the bipartite graph approach) and significant speed up of forward passes using the bipartite formulation especially for larger N.  They also examine the inferred adjacency matrices for different methods on some synthetic data.  Additionally, they show hyper parameter sensitivity results for varying K (number of latent nodes in the bipartite graph) for one dataset.