Discrete Graph Structure Learning for Forecasting Multiple Time Series

1 INTRODUCTION . Time series data are widely studied in science and engineering that involve temporal measurements . Time series forecasting is concerned with the prediction of future values based on observed ones in the past . It has played important roles in climate studies , market analysis , traffic control , and energy grid management ( Makridakis et al. , 1997 ) and has inspired the development of various predictive models that capture the temporal dynamics of the underlying system . These models range from early autoregressive approaches ( Hamilton , 1994 ; Asteriou & Hall , 2011 ) to the recent deep learning methods ( Seo et al. , 2016 ; Li et al. , 2018 ; Yu et al. , 2018 ; Zhao et al. , 2019 ) . Analysis of univariate time series ( a single longitudinal variable ) has been extended to multivariate time series and multiple ( univariate or multivariate ) time series . Multivariate forecasting models find strong predictive power in stressing the interdependency ( and even causal relationship ) among the variables . The vector autoregressive model ( Hamilton , 1994 ) is an example of multivariate analysis , wherein the coefficient magnitudes offer hints into the Granger causality ( Granger , 1969 ) of one variable to another . For multiple time series , pairwise similarities or connections among them have also been explored to improve the forecasting accuracy ( Yu et al. , 2018 ) . An example is the traffic network where each node denotes a time series recording captured by a particular sensor . The spatial connections of the roads offer insights into how traffic dynamics propagates along the network . Several graph neural ∗This work was done while C. Shang was an intern at MIT-IBM Watson AI Lab , IBM Research . †To whom correspondence should be addressed . network ( GNN ) approaches ( Seo et al. , 2016 ; Li et al. , 2018 ; Yu et al. , 2018 ; Zhao et al. , 2019 ) have been proposed recently to leverage the graph structure for forecasting all time series simultaneously . The graph structure however is not always available or it may be incomplete . There could be several reasons , including the difficulty in obtaining such information or a deliberate shielding for the protection of sensitive information . For example , a data set comprising sensory readings of the nation-wide energy grid is granted access to specific users without disclosure of the grid structure . Such practical situations incentivize the automatic learning of the hidden graph structure jointly with the forecasting model . Because GNN approaches show promise in forecasting multiple interrelated time series , in this paper we are concerned with structure learning methods applicable to the downstream use of GNNs . A prominent example is the recent work of Franceschi et al . ( 2019 ) ( named LDS ) , which is a meta-learning approach that treats the graph as a hyperparameter in a bilevel optimization framework ( Franceschi et al. , 2017 ) . Specifically , let Xtrain and Xval denote the training and the validation sets of time series respectively , A ∈ { 0 , 1 } n×n denote the graph adjacency matrix of the n time series , w denote the parameters used in the GNN , and L and F denote the the loss functions used during training and validation respectively ( which may not be identical ) . LDS formulates the problem as learning the probability matrix θ ∈ [ 0 , 1 ] n×n , which parameterizes the element-wise Bernoulli distribution from which the adjacency matrix A is sampled : min θ EA∼Ber ( θ ) [ F ( A , w ( θ ) , Xval ) ] , s.t . w ( θ ) = argmin w EA∼Ber ( θ ) [ L ( A , w , Xtrain ) ] . ( 1 ) Formulation ( 1 ) gives a bilevel optimization problem . The constraint ( which by itself is an optimization problem ) defines the GNN weights as a function of the given graph , so that the objective is to optimize over such a graph only . Note that for differentiability , one does not directly operate on the discrete graph adjacency matrix A , but on the continuous probabilities θ instead . LDS has two drawbacks . First , its computation is expensive . The derivative of w with respect to θ is computed by applying the chain rule on a recursive-dynamics surrogate of the inner optimization argmin . Applying the chain rule on this surrogate is equivalent to differentiating an RNN , which is either memory intensive if done in the reverse mode or time consuming if done in the forward mode , when unrolling a deep dynamics . Second , it is challenging to scale . The matrix θ has Θ ( n2 ) entries to optimize and thus the method is hard to scale to increasingly more time series . In light of the challenges of LDS , we instead advocate a unilevel optimization : min w EA∼Ber ( θ ( w ) ) [ F ( A , w , Xtrain ) ] . ( 2 ) Formulation ( 2 ) trains the GNN model as usual , except that the probabilities θ ( which parameterizes the distribution from which A is sampled ) , is by itself parameterized . We absorb these parameters , together with the GNN parameters , into the notation w. We still use a validation set Xval for usual hyperparameter tuning , but these hyperparameters are not θ as treated by ( 1 ) . In fact , formulation ( 1 ) may need a second validation set to tune other hyperparameters . The major distinction of our approach from LDS is the parameterization θ ( w ) , as opposed to an inner optimization w ( θ ) . In our approach , a modeler owns the freedom to design the parameterization and better control the number of parameters as n2 increases . To this end , time series representation learning and link prediction techniques offer ample inspiration for modeling . In contrast , LDS is more agnostic as no modeling is needed . The effort , instead , lies in the nontrivial treatment of the inner optimization ( in particular , its differentiation ) . As such , our approach is advantageous in two regards . First , its computation is less expensive , because the gradient computation of a unilevel optimization is straightforward and efficient and implementations are mature . Second , it better scales , because the number of parameters does not grow quadratically with the number of time series . We coin our approach GTS ( short for “ graph for time series ” ) , signaling the usefulness of graph structure learning for enhancing time series forecasting . It is important to note that the end purpose of the graph is to improve forecasting quality , rather than identifying causal relationship of the series or recovering the ground-truth graph , if any . While causal discovery of multiple scalar variables is an established field , identifying causality among multiple multivariate time series requires a nontrivial extension that spans beyond the current study . On the other hand , the graph , either learned or preexisting , serves as additional information that helps the model better capture global signals and apply on each series . There does not exist a golden measure for the quality of the learned graph except forecasting accuracy . For example , the traffic network does not necessarily offer the best pairwise relationship a GNN can exploit for forecasting traffic series . Nevertheless , to robustify GTS we incorporate regularization that penalizes significant departure from one ’ s prior belief . If a certain “ ground-truth ” graph is believed , the learned graph will be a healthy variation of it for a more accurate forecast . 2 RELATED WORK . Time series forecasting has been studied for decades by statisticians . It is out of the scope of this paper to comprehensively survey the literature , but we will focus more on late developments under the deep learning context . Early textbook methods include ( vector ) autoregressive models ( Hamilton , 1994 ) , autoregressive integrated moving average ( ARIMA ) ( Asteriou & Hall , 2011 ) , hidden Markov models ( HMM ) ( Baum & Petrie , 1966 ) , and Kalman filters ( Zarchan & Musoff , 2000 ) . Generally speaking , these are linear models that use a window of the past information to predict the next time step , although nonlinear versions with parameterization are subsequently developed . A notable nonlinear extension was the RNN ( Williams et al. , 1986 ) , which later evolved into LSTM ( Hochreiter & Schmidhuber , 1997 ) , BiLSTM ( Schuster & Paliwal , 1997 ) , and GRU ( Cho et al. , 2014 ) , which addressed several limitations of the vanilla RNN , such as the vanishing gradient problem . These architectures are hard to parallelize because of the recurrent nature of the forward and backward computation . More recently , Transformer ( Vaswani et al. , 2017 ) and BERT ( Devlin et al. , 2019 ) were developed to address parallelization , by introducing attention mechanisms that simultaneously digested past ( and future ) information . Although these models are more heavily used for sequence data under the context of natural language processing , they are readily applicable for time series as well ( Shih et al. , 2019 ; Li et al. , 2019 ) . Graph neural networks ( Zhang et al. , 2018 ; Zhou et al. , 2018 ; Wu et al. , 2019 ) emerged quickly in deep learning to handle graph-structured data . Typically , graph nodes are represented by feature vectors , but for the case of time series , a number of specialized architectures were recently developed ; see , e.g. , GCRN ( Seo et al. , 2016 ) , DCRNN ( Li et al. , 2018 ) , STGCN ( Yu et al. , 2018 ) , and TGCN ( Zhao et al. , 2019 ) . These architectures essentially combine the temporal recurrent processing with graph convolution to augment the representation learning of the individual time series . Graph structure learning ( not necessarily for time series ) appears in various contexts and thus methods span a broad spectrum . One field of study is probabilistic graphical models and casual inference , whereby the directed acyclic structure is enforced . Gradient-based approaches in this context include NOTEARS ( Zheng et al. , 2018 ) , DAG-GNN ( Yu et al. , 2019 ) , and GraN-DAG ( Lachapelle et al. , 2020 ) . On the other hand , a general graph may still be useful without resorting to causality . LDS ( Franceschi et al. , 2019 ) is a meta-learning approach that demonstrates to improve the performance on node classification tasks . MTGNN ( Wu et al. , 2020 ) parameterizes the graph as a degree-k graph , which is learned end-to-end with a GNN for forecasting time series . We , on the other hand , allow a more general structural prior for the graph . NRI ( Kipf et al. , 2018 ) adopts a latent-variable approach and learns a latent graph for forecasting system dynamics . Our approach is closely related to NRI and we will compare with it in the following section after introducing the technical details . 3 METHOD . In this section , we present the proposed GTS method , elaborate the model parameterization , and describe the training technique . We also highlight the distinctions from NRI ( Kipf et al. , 2018 ) . Let us first settle the notations . Denote by X the training data , which is a three dimensional tensor , with the three dimensions being feature , time , and the n series . Superscript refers to the series and subscript refers to time ; that is , Xi denotes the i-th series for all features and time and Xt denotes the t-th time step for all features and series . There are in total S time steps for training . The model will use a window of T steps to forecast the next τ steps . For each valid t , denote by feature extractor link predictor recurrent graph convolution recurrent graph convolution … entire data X sampled graph A windowed data forecast sampling learned structure 𝜃 t + 1 t + 2 t + T t + T + 1 t + T + 𝜏 recurrent graph convolution recurrent graph convolution … Figure 1 : GTS architecture . X̂t+T+1 : t+T+τ = f ( A , w , Xt+1 : t+T ) the model , which forecasts X̂t+T+1 : t+T+τ from observations Xt+1 : t+T , through exploiting the graph structureA and being parameterized byw . Using ` to denote the loss function between the prediction and the ground truth , a typical training objective reads∑ t ` ( f ( A , w , Xt+1 : t+T ) , Xt+T+1 : t+T+τ ) . ( 3 ) Three remaining details are the parameterization of A , the model f , and the loss ` .

Summary: The paper considers learning both graph structures and NNs for time series data, similar to the idea of LDS (Franceschi et al., 2019). Observing the computation and scalability issues with LDS, authors propose a unilevel optimization form wrt. the mean performance over the graph distribution. This is done via NNs, with input being the observed sequence, to output a real matrix whose elements are then treated as weights for the Gumbel trick. NN structures, training procedure, etc. mostly follow existing works.