Discrete Graph Structure Learning for Forecasting Multiple Time Series

1 INTRODUCTION . Time series data are widely studied in science and engineering that involve temporal measurements . Time series forecasting is concerned with the prediction of future values based on observed ones in the past . It has played important roles in climate studies , market analysis , traffic control , and energy grid management ( Makridakis et al. , 1997 ) and has inspired the development of various predictive models that capture the temporal dynamics of the underlying system . These models range from early autoregressive approaches ( Hamilton , 1994 ; Asteriou & Hall , 2011 ) to the recent deep learning methods ( Seo et al. , 2016 ; Li et al. , 2018 ; Yu et al. , 2018 ; Zhao et al. , 2019 ) . Analysis of univariate time series ( a single longitudinal variable ) has been extended to multivariate time series and multiple ( univariate or multivariate ) time series . Multivariate forecasting models find strong predictive power in stressing the interdependency ( and even causal relationship ) among the variables . The vector autoregressive model ( Hamilton , 1994 ) is an example of multivariate analysis , wherein the coefficient magnitudes offer hints into the Granger causality ( Granger , 1969 ) of one variable to another . For multiple time series , pairwise similarities or connections among them have also been explored to improve the forecasting accuracy ( Yu et al. , 2018 ) . An example is the traffic network where each node denotes a time series recording captured by a particular sensor . The spatial connections of the roads offer insights into how traffic dynamics propagates along the network . Several graph neural ∗This work was done while C. Shang was an intern at MIT-IBM Watson AI Lab , IBM Research . †To whom correspondence should be addressed . network ( GNN ) approaches ( Seo et al. , 2016 ; Li et al. , 2018 ; Yu et al. , 2018 ; Zhao et al. , 2019 ) have been proposed recently to leverage the graph structure for forecasting all time series simultaneously . The graph structure however is not always available or it may be incomplete . There could be several reasons , including the difficulty in obtaining such information or a deliberate shielding for the protection of sensitive information . For example , a data set comprising sensory readings of the nation-wide energy grid is granted access to specific users without disclosure of the grid structure . Such practical situations incentivize the automatic learning of the hidden graph structure jointly with the forecasting model . Because GNN approaches show promise in forecasting multiple interrelated time series , in this paper we are concerned with structure learning methods applicable to the downstream use of GNNs . A prominent example is the recent work of Franceschi et al . ( 2019 ) ( named LDS ) , which is a meta-learning approach that treats the graph as a hyperparameter in a bilevel optimization framework ( Franceschi et al. , 2017 ) . Specifically , let Xtrain and Xval denote the training and the validation sets of time series respectively , A ∈ { 0 , 1 } n×n denote the graph adjacency matrix of the n time series , w denote the parameters used in the GNN , and L and F denote the the loss functions used during training and validation respectively ( which may not be identical ) . LDS formulates the problem as learning the probability matrix θ ∈ [ 0 , 1 ] n×n , which parameterizes the element-wise Bernoulli distribution from which the adjacency matrix A is sampled : min θ EA∼Ber ( θ ) [ F ( A , w ( θ ) , Xval ) ] , s.t . w ( θ ) = argmin w EA∼Ber ( θ ) [ L ( A , w , Xtrain ) ] . ( 1 ) Formulation ( 1 ) gives a bilevel optimization problem . The constraint ( which by itself is an optimization problem ) defines the GNN weights as a function of the given graph , so that the objective is to optimize over such a graph only . Note that for differentiability , one does not directly operate on the discrete graph adjacency matrix A , but on the continuous probabilities θ instead . LDS has two drawbacks . First , its computation is expensive . The derivative of w with respect to θ is computed by applying the chain rule on a recursive-dynamics surrogate of the inner optimization argmin . Applying the chain rule on this surrogate is equivalent to differentiating an RNN , which is either memory intensive if done in the reverse mode or time consuming if done in the forward mode , when unrolling a deep dynamics . Second , it is challenging to scale . The matrix θ has Θ ( n2 ) entries to optimize and thus the method is hard to scale to increasingly more time series . In light of the challenges of LDS , we instead advocate a unilevel optimization : min w EA∼Ber ( θ ( w ) ) [ F ( A , w , Xtrain ) ] . ( 2 ) Formulation ( 2 ) trains the GNN model as usual , except that the probabilities θ ( which parameterizes the distribution from which A is sampled ) , is by itself parameterized . We absorb these parameters , together with the GNN parameters , into the notation w. We still use a validation set Xval for usual hyperparameter tuning , but these hyperparameters are not θ as treated by ( 1 ) . In fact , formulation ( 1 ) may need a second validation set to tune other hyperparameters . The major distinction of our approach from LDS is the parameterization θ ( w ) , as opposed to an inner optimization w ( θ ) . In our approach , a modeler owns the freedom to design the parameterization and better control the number of parameters as n2 increases . To this end , time series representation learning and link prediction techniques offer ample inspiration for modeling . In contrast , LDS is more agnostic as no modeling is needed . The effort , instead , lies in the nontrivial treatment of the inner optimization ( in particular , its differentiation ) . As such , our approach is advantageous in two regards . First , its computation is less expensive , because the gradient computation of a unilevel optimization is straightforward and efficient and implementations are mature . Second , it better scales , because the number of parameters does not grow quadratically with the number of time series . We coin our approach GTS ( short for “ graph for time series ” ) , signaling the usefulness of graph structure learning for enhancing time series forecasting . It is important to note that the end purpose of the graph is to improve forecasting quality , rather than identifying causal relationship of the series or recovering the ground-truth graph , if any . While causal discovery of multiple scalar variables is an established field , identifying causality among multiple multivariate time series requires a nontrivial extension that spans beyond the current study . On the other hand , the graph , either learned or preexisting , serves as additional information that helps the model better capture global signals and apply on each series . There does not exist a golden measure for the quality of the learned graph except forecasting accuracy . For example , the traffic network does not necessarily offer the best pairwise relationship a GNN can exploit for forecasting traffic series . Nevertheless , to robustify GTS we incorporate regularization that penalizes significant departure from one ’ s prior belief . If a certain “ ground-truth ” graph is believed , the learned graph will be a healthy variation of it for a more accurate forecast . 2 RELATED WORK . Time series forecasting has been studied for decades by statisticians . It is out of the scope of this paper to comprehensively survey the literature , but we will focus more on late developments under the deep learning context . Early textbook methods include ( vector ) autoregressive models ( Hamilton , 1994 ) , autoregressive integrated moving average ( ARIMA ) ( Asteriou & Hall , 2011 ) , hidden Markov models ( HMM ) ( Baum & Petrie , 1966 ) , and Kalman filters ( Zarchan & Musoff , 2000 ) . Generally speaking , these are linear models that use a window of the past information to predict the next time step , although nonlinear versions with parameterization are subsequently developed . A notable nonlinear extension was the RNN ( Williams et al. , 1986 ) , which later evolved into LSTM ( Hochreiter & Schmidhuber , 1997 ) , BiLSTM ( Schuster & Paliwal , 1997 ) , and GRU ( Cho et al. , 2014 ) , which addressed several limitations of the vanilla RNN , such as the vanishing gradient problem . These architectures are hard to parallelize because of the recurrent nature of the forward and backward computation . More recently , Transformer ( Vaswani et al. , 2017 ) and BERT ( Devlin et al. , 2019 ) were developed to address parallelization , by introducing attention mechanisms that simultaneously digested past ( and future ) information . Although these models are more heavily used for sequence data under the context of natural language processing , they are readily applicable for time series as well ( Shih et al. , 2019 ; Li et al. , 2019 ) . Graph neural networks ( Zhang et al. , 2018 ; Zhou et al. , 2018 ; Wu et al. , 2019 ) emerged quickly in deep learning to handle graph-structured data . Typically , graph nodes are represented by feature vectors , but for the case of time series , a number of specialized architectures were recently developed ; see , e.g. , GCRN ( Seo et al. , 2016 ) , DCRNN ( Li et al. , 2018 ) , STGCN ( Yu et al. , 2018 ) , and TGCN ( Zhao et al. , 2019 ) . These architectures essentially combine the temporal recurrent processing with graph convolution to augment the representation learning of the individual time series . Graph structure learning ( not necessarily for time series ) appears in various contexts and thus methods span a broad spectrum . One field of study is probabilistic graphical models and casual inference , whereby the directed acyclic structure is enforced . Gradient-based approaches in this context include NOTEARS ( Zheng et al. , 2018 ) , DAG-GNN ( Yu et al. , 2019 ) , and GraN-DAG ( Lachapelle et al. , 2020 ) . On the other hand , a general graph may still be useful without resorting to causality . LDS ( Franceschi et al. , 2019 ) is a meta-learning approach that demonstrates to improve the performance on node classification tasks . MTGNN ( Wu et al. , 2020 ) parameterizes the graph as a degree-k graph , which is learned end-to-end with a GNN for forecasting time series . We , on the other hand , allow a more general structural prior for the graph . NRI ( Kipf et al. , 2018 ) adopts a latent-variable approach and learns a latent graph for forecasting system dynamics . Our approach is closely related to NRI and we will compare with it in the following section after introducing the technical details . 3 METHOD . In this section , we present the proposed GTS method , elaborate the model parameterization , and describe the training technique . We also highlight the distinctions from NRI ( Kipf et al. , 2018 ) . Let us first settle the notations . Denote by X the training data , which is a three dimensional tensor , with the three dimensions being feature , time , and the n series . Superscript refers to the series and subscript refers to time ; that is , Xi denotes the i-th series for all features and time and Xt denotes the t-th time step for all features and series . There are in total S time steps for training . The model will use a window of T steps to forecast the next τ steps . For each valid t , denote by feature extractor link predictor recurrent graph convolution recurrent graph convolution … entire data X sampled graph A windowed data forecast sampling learned structure 𝜃 t + 1 t + 2 t + T t + T + 1 t + T + 𝜏 recurrent graph convolution recurrent graph convolution … Figure 1 : GTS architecture . X̂t+T+1 : t+T+τ = f ( A , w , Xt+1 : t+T ) the model , which forecasts X̂t+T+1 : t+T+τ from observations Xt+1 : t+T , through exploiting the graph structureA and being parameterized byw . Using ` to denote the loss function between the prediction and the ground truth , a typical training objective reads∑ t ` ( f ( A , w , Xt+1 : t+T ) , Xt+T+1 : t+T+τ ) . ( 3 ) Three remaining details are the parameterization of A , the model f , and the loss ` .

Summary: This paper proposes an approach for time series forecasting that learns the graph structure among multiple (multivariate) time series simultaneously with the parameters of a Graph Neural Network (GNN). The problem is formulated as learning a probabilistic graphical model by optimizing the expectation over the graph distribution, which is parameterized by a neural network and encapsulated in a single differentiable objective. Empirical evidence suggests that the proposed GTS obtains superior forecasting performance to both deep and non-deep learning based, as well as graph and non-graph based, competitor forecasting models. In addition, GTS appears to be more computationally efficient compared to LDS, a recently proposed meta-learning graph-based approach.