Multivariate Time Series Forecasting with Latent Graph Inference

This paper introduces a new architecture for multivariate time series forecasting that simultaneously infers and leverages relations among time series . We cast our method as a modular extension to univariate architectures where relations among individual time series are dynamically inferred in the latent space obtained after encoding the whole input signal . Our approach is flexible enough to scale gracefully according to the needs of the forecasting task under consideration . In its most straight-forward and general version , we infer a potentially fully connected graph to model the interactions between time series , which allows us to obtain competitive forecast accuracy compared with the state-of-the-art in graph neural networks for forecasting . In addition , whereas previous latent graph inference methods scale O ( N2 ) w.r.t . the number of nodes N ( representing the time series ) , we show how to configure our approach to cater for the scale of modern time series panels . By assuming the inferred graph to be bipartite where one partition consists of the original N nodes and we introduce K nodes ( taking inspiration from low-rank-decompositions ) we reduce the time complexity of our procedure to O ( NK ) . This allows us to leverage the dependency structure with a small trade-off in forecasting accuracy . We demonstrate the effectiveness of our method for a variety of datasets where it performs better or very competitively to previous methods under both the fully connected and bipartite assumptions . 1 INTRODUCTION . Time Series Forecasting ( TSF ) has been widely studied due to its practical significance in a wide variety of applications such as climate modelling ( Mudelsee , 2019 ) , supply chain management in retail ( Larson , 2001 ; Böse et al. , 2017 ) , market analysis in finance ( Andersen et al. , 2005 ) , traffic control ( Li et al. , 2017 ) and medicine ( Kaushik et al. , 2020 ) . Petropoulos et al . ( 2020 ) provide a non-systematic overview of further applications . In TSF , given a sequence of data points indexed over time , we aim to estimate its future values based on previously observed data . Data is often multivariate , meaning that multiple variables vary over time , each variable may not only depend on its own historical values , but also on other variables ’ past . Efficiently modelling the dependencies between these variables is still an open problem . Multivariate Time Series ( MTS ) methods aim to leverage the dependencies between variables in order to improve the forecasting accuracy . Some classical MTS forecasting algorithms such as Vector Autoregression ( VAR ) ( Lütkepohl , 2005 ) or Gaussian Processes ( Roberts et al. , 2013 ) only consider linear dependencies among variables . A natural way to model non-linear dependencies in deep learning is via Graph Neural Networks ( GNNs ) ( Bruna et al. , 2013 ; Defferrard et al. , 2016 ; Kipf and Welling , 2016 ) . In fact , GNNs have been successfully applied for the Multivariate Time Series forecasting ( Li et al. , 2017 ; Yu et al. , 2017 ; Chen et al. , 2020a ) , leveraging the relations among different series . But these methods require a pre-defined adjacency matrix which may only be available in some specific datasets , for example , traffic datasets , where it can be constructed from the spatial structure of a city . More recently , a family of methods that do not require a pre-defined adjacency matrix have been proposed ( Wu et al. , 2019 ; Franceschi et al. , 2019 ; Wu et al. , 2020 ; Shang et al. , 2021 ) . In this case , a latent graph representation is inferred while forecasting , allowing to operate on a larger variety of datasets . Our work belongs to this category . However , inferring all pairwise relations may come at a higher computational cost , as the number of relations scales quadratically O ( N2 ) w.r.t . the number of nodes . This limits the scalability to large datasets . Additionally , the above mentioned latent graph inference methods perform message passing update at every time step iteration which can also be expensive . To overcome these limitations , we propose a new latent graph inference algorithm for Multivariate Time Series forecasting that is more efficient than previous algorithms while achieving better or competitive performance . We cast the latent graph inference as a modular and easy-to-implement extension to current univariate models . The graph is dynamically inferred per inputted data stream allowing a more flexible representation than a static graph for the whole dataset . Additionally , we optionally reduce the complexity from O ( N2 ) ( Fully Connected Assumption ) to O ( NK ) ( Bipartite Assumption ) where K ⌧ N for a small trade off in performance . 2 BACKGROUND . 2.1 TIME SERIES FORECASTING . In time series forecasting we want to estimate a future time series xt+1 : T given its past xt0 : t where t0 t T indexes over time , and ( optionally ) some context information c. For the multivariate case we assume the time series is composed of N variates at a time such that xt0 : T = { x1 , t0 : T , . . . , xN , t0 : T } 2 RN⇥T t0+1 . In this section we distinguish two main categories of time series forecasting methods , Global Univariate and Multivariate . Global Univariate methods : In this case we only use the past of each univariate to predict its future . However , the model weights ✓u are shared across all univariate time series . More formally : x̂i , t+1 : T = fu ( xi , t0 : t , ci ; ✓u ) ( 1 ) where x̂ denotes the estimated values , i 2 { 1 , . . . , N } indexes over multivariates and fu ( · ) is the estimator function with learnable parameters ✓u shared across time series . Conditioning on the past of each univariate may limit the performance of the forecasting algorithm compared to multivariate ones . Despite that , it simplifies the design of f✓ and already provides reasonable results . A popular example of univariate models would be ( Salinas et al. , 2020 ) . Multivariate methods : Multivariate methods condition on all past data ( all N variates ) and directly predict the multivariate target . More formally : x̂t+1 : T = fm ( xt0 : t , c ) . ( 2 ) Different variables may be correlated and/or depend on the same con-founders . For example , in retail forecasting , PPE masks and antibacterial soaps jointly increased in demand during the early days of the COVID-19 pandemic . In traffic forecasting , an increase of the outcome traffic flow in a given neighborhood may result in an increase of the income traffic flow on another one . Modelling these dependencies may improve the forecasting accuracy , but it may come at a cost of higher complexity and hence more expensive algorithms , specially when trying to model all pairwise interactions between variates . 2.2 GRAPH NEURAL NETWORKS . Graph Neural Networks ( GNNs ) ( Bruna et al. , 2013 ; Defferrard et al. , 2016 ; Kipf and Welling , 2016 ) operate directly on graph structured data . They have gained a lot of attention in the last years due to their success in a large variety of domains which benefit from modelling interactions between different nodes/entities . In the context of multivariate time series , GNNs can be used to model the interactions between time series . In this work we consider the type of GNN introduced by ( Gilmer et al. , 2017 ) . Given a graph G = ( V , E ) with nodes vi 2 V and edges eij 2 E , we define a graph convolutional layer as : mij = e ( h l i , h l j ) mi = X j2N ( i ) ↵ijmij h l+1 i = h ( h l i , mi ) ( 3 ) Where e and h are the edge and node functions , usually approximated as Multi Layer Perceptrons ( MLPs ) , hli 2 Rnf is the nf-dimensional embedding of a node vi at layer l and mij is the edge embedding that propagates information from node vj to vi . A GNN is constructed by stacking multiple of these Graph Convolutional Layers hl+1 = GCL [ hl , E ] . Additionally , in Equation 3 we include ↵ij 2 ( 0 , 1 ) which is a scalar value that performs the edge inference or attention over the neighbors similarly to Veličković et al . ( 2017 ) . As done in ( Satorras et al. , 2021 ) , we choose this value to be computed as the output of a function ↵ij = ↵ ( mij ) where ↵ is composed of just a linear layer followed by a sigmoid activation function . 3 RELATED WORK . Time series forecasting has been extensively studied in the past due to its practical significance with a number of recent overview articles available ( et al. , 2020 ; Benidis et al. , 2020 ; Lim and Zohren , 2021 ) . Traditionally , most classical methods are univariate in nature ( see e.g. , Hyndman and Athanasopoulos ( 2017 ) for an overview ) . While some of these have multi-variate extensions ( e.g. , ARMA and VARMA models ) , they are limited by the amount of related time series information they can incorporate . Dynamic factor models ( Geweke , 1977 ; Wang et al. , 2019a ) are fore-runners of a family of models that has recently received more attention , the so-called global models ( Januschowski et al. , 2019 ; Montero-Manso and Hyndman , 2022 ) . These global models estimate their parameters over an entire panel of time series , so thereby taking advantage of cross time series learning , but still produce a univariate forecast . Many such global models have been proposed building on the main neural network architectures like RNNs ( Salinas et al. , 2020 ; Liberty et al. , 2020 ; Bandara et al. , 2019 ) , CNNs ( Wen et al. , 2017 ; Chen et al. , 2020b ) , Transformers ( Li et al. , 2019 ; Lim et al. , 2021 ; Eisenach et al. , 2020 ) and also combining classical probabilistic models with deep learning ( Rangapuram et al. , 2018 ; Kurle et al. , 2020 ; de Bézenac et al. , 2020 ) . However , these global models do not explicitly model the relationship between the time series in the panel . Most recently , global multi-variate forecasting models have received attention , in particular models that attempt to capture the relationship of the time series via a multi-variate likelihood ( Rasul et al. , 2020 ; 2021 ; de Bézenac et al. , 2020 ; Salinas et al. , 2019 ) . Here , we attempt to capture the multivariate nature of many modern forecasting problems primarily by using a multi-variate time series as input . For this , a natural way to model and exploit the relationship between time series is via Graph Neural Networks ( Bruna et al. , 2013 ; Defferrard et al. , 2016 ; Kipf and Welling , 2016 ) which have been successfully applied to a wide variety of deep learning domains where exploiting relations between entities/nodes can benefit the prediction task . Even in those cases where edges are not explicitly provided in the dataset , attention or a latent graph can be inferred from the node embeddings such that GNNs can still leverage the structure of the data . Some examples of latent graph inference or attention are ( Wang et al. , 2019b ) in point clouds , ( Franceschi et al. , 2019 ) in semi-supervised graph classification , ( Ying et al. , 2018 ) in hierarchical graph representation learning , ( Kipf et al. , 2018 ) in modelling dynamical systems , ( Kazi et al. , 2020 ) in zero-shot learning and 3D point cloud segmentation , ( Garcia and Bruna , 2017 ; Kossen et al. , 2021 ) in image classification , ( Cranmer et al. , 2020 ) in inferring symbolic representations and ( Fuchs et al. , 2020 ; Satorras et al. , 2021 ) in molecular property prediction . In Multivariate Time Series ( MTS ) forecasting we can leverage dependencies between time series by exchanging information among them . Lai et al . ( 2018 ) ; Shih et al . ( 2019 ) are some of the first deep learning approaches designed to exploit those pair-wise dependencies . More recent methods , ( Li et al. , 2017 ; Yu et al. , 2017 ; Seo et al. , 2018 ; Zhao et al. , 2019 ) are built in the intersection of Graph Neural Networks and time series forecasting but they require a pre-defined adjacency matrix . Lately , new methods that infer a latent graph from the node embeddings have been introduced in MTS forecasting ( Wu et al. , 2020 ; Shang et al. , 2021 ; Cao et al. , 2021 ) , and in MTS anomaly detection ( Zhang et al. , 2020 ; Deng and Hooi , 2021 ) . These methods can be applied to any dataset even when there is not an explicitly defined adjacency matrix . But this comes with a limitation , inferring the edges can be expensive since those scale quadratically O ( N2 ) w.r.t the number of nodes/variables N or O ( N3 ) in ( Cao et al. , 2021 ) . Our approach is related to ( Wu et al. , 2020 ; Shang et al. , 2021 ) , but in contrast a ) Our latent graph is dynamically inferred for each inputted data stream instead of a static graph for the whole dataset which allows a more flexible graph representation . b ) It is modular since it can be added as an extension to standard univariate methods after encoding the input signal xt0 : t , this also makes the graph operation cheaper since the message exchange is only done at time step t. c ) Optionally reduces the number of edges from O ( N2 ) to O ( NK ) where K ⌧ N with the bipartite assumption .

Summary: This paper presents a method to combine information of multivariate time series by extending univariate architectures. The technique uses a graph representation to represent interactions by assuming a bipartite structure, which allows the technique to scale the representation to reduce the complexity from $O(N^2)$ to $O(N K)$ using K additional nodes. The architecture represents each time series as one of N nodes and associate K embeddings (nodes) and thus there are connections between the N nodes and the K nodes but not within each group of nodes. The architecture encodes the confounders, co-linearities, and other information among the time series to improve forecasting. The experiments show the performance (and time efficiency) of the technique against several baselines on METR-LA and PEMS-BAY datasets. The experiments also show the performance on single-step forecasting on four publicly available datasets. Finally, synthetic datasets were used to evaluate the adjacency matrices in control scenarios.