Filling the G_ap_s: Multivariate Time Series Imputation by Graph Neural Networks

1 INTRODUCTION . Imputation of missing values is a prominent problem in multivariate time-series analysis ( TSA ) from both theoretical and practical perspectives ( Little & Rubin , 2019 ) . In fact , in a world of complex interconnected systems such as those characterizing sensor networks or the Internet of Things , faulty sensors and network failures are widespread phenomena that cause disruptions in the data acquisition process . Luckily , failures of these types are often sparse and localized at the single sensor level , i.e. , they do not compromise the entire sensor network at once . In other terms , it is often the case that , at a certain time step , missing data appear only at some of the channels of the resulting multivariate time series . In this context , spatio-temporal imputation methods ( Yi et al. , 2016 ; Yoon et al. , 2018b ) aim at reconstructing the missing parts of the signals by possibly exploiting both temporal and spatial dependencies . In particular , effective spatio-temporal approaches would reconstruct missing values by taking into account past and future values , and the concurrent measurements of spatially close neighboring sensors too . Here , spatial similarity does not necessarily mean physical ( e.g. , geographic ) proximity , but rather indicates that considered sensors are related w.r.t . a generic ( quantifiable ) functional dependency ( e.g. , Pearson correlation or Granger causality – Granger , 1969 ) and/or that are close in a certain latent space . Relational information , then , can be interpreted as a set of constraints – linking the different time series – that allows replacing the malfunctioning sensors with virtual ones . Among different imputation methods , approaches based on deep learning ( LeCun et al. , 2015 ; Schmidhuber , 2015 ; Goodfellow et al. , 2016 ) have become increasingly popular ( Yoon et al. , 2018a ; Cao et al. , 2018 ; Liu et al. , 2019 ) . However , these methods often completely disregard available relational information or rely on rather simplistic modifications of standard neural architectures tailored ∗Equal contribution . Correspondence to andrea.cini @ usi.ch . for sequence processing ( Hochreiter & Schmidhuber , 1997 ; Chung et al. , 2014 ; Bai et al. , 2018 ; Vaswani et al. , 2017 ) . We argue that stronger , structural , inductive biases are needed to advance the state of the art in time series imputation and allow to build effective inference engines in the context of large and complex sensor networks as those found in real-world applications . In this work , we model input multivariate time series as sequences of graphs where edges represent relationships among different channels . We propose graph neural networks ( GNNs ) ( Scarselli et al. , 2008 ; Bronstein et al. , 2017 ; Battaglia et al. , 2018 ) as the building block of a novel , bidirectional , recurrent neural network for multivariate time series imputation ( MTSI ) . Our method , named Graph Recurrent Imputation Network ( GRIN ) , has at its core a recurrent neural network cell where gates are implemented by message-passing neural networks ( MPNNs ; Gilmer et al. , 2017 ) . Two of these networks process the input multivariate time series in both forward and backward time directions at each node , while hidden states are processed by a message-passing imputation layer which is constrained to learn how to perform imputation by looking at neighboring nodes . In fact , by considering each edge as a soft functional dependency that constraints the value observed at corresponding nodes , we argue that operating in the context of graphs introduces a positive inductive bias for MTSI . Our contributions are manifold : 1 ) we introduce a methodological framework to exploit graph neural networks in the context of MTSI , 2 ) we propose a novel , practical and effective implementation of a GNN-based architecture for MTSI , and 3 ) we achieve state-of-the-art results on several and varied MTSI benchmarks . Our method does not rely on any assumption on the distribution of the missing values ( e.g. , presence and duration of transient dynamics and/or length of missing sequences ) other than stationarity of the underlying process . The rest of the paper is organized as follows . In Section 2 we discuss the related works . Then , in Section 3 , we formally introduce the problem settings and the task of MTSI . We present our approach to MTSI in Section 4 , by describing the novel framework to implement imputation architectures based on GNNs . We proceed with an empirical evaluation of the presented method against state-of-the-art baselines in Section 5 and , finally , we draw our conclusions in Section 6 . 2 RELATED WORKS . Time series imputation There exists a large literature addressing missing value imputation in time series . Besides the simple and standard interpolation methods based on polynomial curve fitting , popular approaches aim at filling up missing values by taking advantage of standard forecasting methods and similarities among time series . For example , several approaches rely on k-nearest neighbors ( Troyanskaya et al. , 2001 ; Beretta & Santaniello , 2016 ) , the expectation-maximization algorithm ( Ghahramani & Jordan , 1994 ; Nelwamondo et al. , 2007 ) or linear predictors and statespace models ( Durbin & Koopman , 2012 ; Kihoro et al. , 2013 ) . Low-rank approximation methods , such as matrix factorization ( Cichocki & Phan , 2009 ) , are also popular alternatives which can also account for spatial ( Cai et al. , 2010 ; Rao et al. , 2015 ) and temporal ( Yu et al. , 2016 ; Mei et al. , 2017 ) information . Among linear methods , STMVL ( Yi et al. , 2016 ) combines temporal and spatial interpolation to fill missing values in geographically tagged time series . More recently , several deep learning approaches have been proposed for MTSI . Among the others , deep autoregressive methods based on recurrent neural networks ( RNNs ) found widespread success ( Lipton et al. , 2016 ; Che et al. , 2018 ; Luo et al. , 2018 ; Yoon et al. , 2018b ; Cao et al. , 2018 ) . GRU-D ( Che et al. , 2018 ) learns how to process sequences with missing data by controlling the decay of the hidden states of a gated RNN . Cao et al . ( 2018 ) propose BRITS , a bidirectional GRU-D-like RNN for multivariate time series imputation that takes into account correlation among different channels to perform spatial imputation . Other successful strategies in the literature have been proposed that exploit the adversarial training framework to generate realistic reconstructed sequences ( Yoon et al. , 2018a ; Fedus et al. , 2018 ; Luo et al. , 2018 ; 2019 ) . Notably , GAIN ( Yoon et al. , 2018a ) uses GANs ( Goodfellow et al. , 2014 ) to learn models that perform imputation in the i.i.d . settings . Luo et al . ( 2018 ; 2019 ) aim , instead , at learning models that generate realistic synthetic sequences and exploit them to fill missing values . Miao et al . ( 2021 ) use an approach similar to GAIN , but condition the generator on the predicted label for the target incomplete time series . Concurrently to our work , Kuppannagari et al . ( 2021 ) developed a graph-based spatio-temporal denoising autoencoder for spatio-temporal data coming from smart grids with known topology . Liu et al . ( 2019 ) , instead , uses adversarial learning to train a multiscale model that imputes highly sparse time series in a hierarchical fashion . However , we argue that none of the above-cited methods can take full advantage of relational information and nonlinear spatio-temporal dependencies . Most importantly , the above methods do not fully exploit the flexibility and expressiveness enabled by operating in the context of graph processing . Graph neural networks for TSA Graph neural networks have been exploited in TSA mostly in spatio-temporal forecasting methods . The idea behind most of the methods present in the literature is to modify standard neural network architectures for sequential data by relying on operators that work in the graph domain . For example , Seo et al . ( 2018 ) propose a GRU cell where gates are implemented by spectral GNNs ( Defferrard et al. , 2016 ) ; Li et al . ( 2018 ) propose an analogous architecture replacing spectral GNNs with a diffusion-convolutional network ( Atwood & Towsley , 2016 ) . Note that these models are different w.r.t . approaches that use recurrent networks to propagate information graph-wise ( Scarselli et al. , 2008 ; Li et al. , 2016 ) . Yu et al . ( 2017 ) and Wu et al . ( 2019 ; 2020b ) propose , instead , spatio-temporal convolutional neural networks that alternate convolutions on temporal and spatial dimensions . Similar approaches have also been studied in the context of attention-based models ( Vaswani et al. , 2017 ) with spatio-temporal Transformer-like architectures ( Zhang et al. , 2018 ; Cai et al. , 2020 ) . Another particularly interesting line of research is related to the problem of learning the graph structure underlying an input multivariate time series ( Kipf et al. , 2018 ; Wu et al. , 2020b ; Shang et al. , 2020 ) . While previously mentioned approaches focus on multivariate time series prediction , other methods aim at predicting changes in graph topology ( Zambon et al. , 2019 ; Paassen et al. , 2020 ) . Conversely , methods such as Temporal Graph Networks ( Rossi et al. , 2020 ) are tailored to learn node embeddings in dynamical graphs . Finally , recent works have proposed GNNs for imputing missing features in the context of i.i.d . data . Among the others , Spinelli et al . ( 2020 ) propose an adversarial framework to train GNNs on the data reconstruction task , while You et al . ( 2020 ) propose a bipartite graph representation for feature imputation . Lately , GNNs have also been exploited for spatial interpolation ( Appleby et al. , 2020 ; Wu et al. , 2020a ) – sometimes referred to as kriging ( Stein , 1999 ) . To the best of our knowledge , no previous GNN-based method targeted missing value imputation for generic multivariate time series . 3 PRELIMINARIES . Sequences of graphs We consider sequences of weighted directed graphs , where we observe a graph Gt withNt nodes at each time step t. A graph is a couple Gt = 〈Xt , Wt〉 , where Xt ∈ RNt×d is the node-attribute matrix whose i-th row contains the d-dimensional node-attribute vector xit ∈ Rd associated with the i-th node ; entry wi , jt of the adjacency matrix Wt ∈ RNt×Nt denotes the scalar weight of the edge ( if any ) connecting the i-th and j-th node . Fig . 1 exemplifies this modelling framework . We assume nodes to be identified , i.e. , to have a unique ID that enables time-wise consistent processing . This problem setting can be easily extended to more general classes of graphs with attributed edges and global attributes . In this work , we mainly focus on problems where the topology of the graph is fixed and does not change over time , i.e. , at each time step Wt = W and Nt = N . Any generic multivariate time series fits the above framework by letting each channel of the sequence ( i.e. , each sensor ) correspond to a node and using the available relation information to build an adjacency matrix . If no relational information is available , one could use the identity matrix , but this would defeat the purpose of the formulation . A more proper choice of Wt can be made using any standard similarity score ( e.g. , Pearson correlation ) or a ( thresholded ) kernel . A more advanced approach instead could aim at learning an adjacency directly from data by using , for instance , spatial attention scores or resorting to graph learning techniques , e.g. , Kipf et al . ( 2018 ) . From now on , we assume that input multivariate time series have homogeneous channels , i.e. , sensors are of the same type . Note that this assumption does not imply a loss in generality : it is always possible to standardize node features by adding sensor type attributes and additional dimensions to accommodate the different types of sensor readings . Alternatively , one might directly model the problem by exploiting heterogeneous graphs ( Schlichtkrull et al. , 2018 ) . Multivariate time series imputation To model the presence of missing values , we consider , at each step , a binary mask Mt ∈ { 0 , 1 } Nt×d where each row mit indicates which of the corresponding node attributes of xit are available in Xt . It follows that , m i , j t = 0 implies x i , j t to be missing ; conversely , if mi , jt = 1 , then x i , j t stores the actual sensor reading . We denote by X̃t the unknown ground truth node-attribute matrix , i.e. , the complete node-attribute matrix without any missing data . We assume stationarity of missing data distribution and , in experiments , we mostly focus on the missing at random ( MAR ) scenario ( Rubin , 1976 ) . We neither make assumptions on the number of concurrent sensor failures , nor on the length of missing data blocks , i.e. , multiple failures extended over time are accounted for . Clearly , one should expect imputation performance to scale with the number of concurrent faults and the time length of missing data bursts . The objective of MTSI is to impute missing values in a sequence of input data . More formally , given a graph sequence G [ t , t+T ] of length T , we can define the missing data reconstruction error as L ( X̂ [ t , t+T ] , X̃ [ t , t+T ] , M [ t , t+T ] ) = ∑t+T h=t ∑Nt i=1 〈mih , ` ( x̂ih , x̃ i h ) 〉 〈mih , mih〉 , ( 1 ) where x̂ih is the reconstructed x̃ i h ; M [ t , t+T ] and m i h are respectively the logical binary complement of M [ t , t+T ] and mih , ` ( · , · ) is an element-wise error function ( e.g. , absolute or squared error ) and 〈 · , · 〉 indicates the standard dot product . Note that , in practice , it is impossible to have access to X̃ [ t , t+T ] and , as a consequence , it is necessary to define a surrogate optimization objective by , for example , using a forecasting loss or generating synthetic missing values . In the context of trainable , parametric , imputation methods , we consider two different operational settings . In the first one , named in-sample imputation , the model is trained to reconstruct missing values in a given fixed input sequence X [ t , t+T ] , i.e. , the model is trained on all the available data except those that are missing and those that have been removed from the sequence to emulate additional failures for evaluation . Differently , in the second one ( referred to as out-of-sample imputation ) , the model is trained and evaluated on disjoint sequences . Note that in both cases the model does not have access to the ground-truth data used for the final evaluation . The first operational setting simulates the case where a practitioner fits the model directly on the sequence to fill up its gaps . The second , instead , simulates the case where one wishes to use a model fitted on a set of historical data to impute missing values in an unseen target sequence .

Summary: A novel recurrant graph neural network, for spatio-temporal encoding, coupled with a message passing neural network, for spatial decoding, is proposed for imputing missing values in a multivariate time series; The architecture uses temporal and spatial properties of the data. It seems that the approach of gated recurrent neural networks (and related architectures using gated recurrent units, GRUs) is extended to include the graph topology through neighbourhood message passing. Experiments show impressive results on three benchmark datasets, and comparison with simple and state-of-the-art baselines. A short ablation study excludes the spatial decoder and omits the bi-directional architecture to empricially show that they have a significant impact on the imputation performance. Another short experiment tests the ability of this architecture to replace entire missing sensors, using an air pollution dataset with multiple sensors, results are convincing.