Graph-Guided Network for Irregularly Sampled Multivariate Time Series

1 INTRODUCTION . Multivariate time series are prevalent in a variety of domains including healthcare , space science , cybersecurity , biology , and finance ( Ravuri et al. , 2021 ; Sousa et al. , 2020 ; Sezer et al. , 2020 ; Fawaz et al. , 2019 ; Abanda et al. , 2019 ; Tang et al. , 2018 ) . Practical issues often exist in collecting sensor measurements that lead to various types of irregularities caused by missing observations , such as cost saving , sensor failures , external forces in physical scenarios , medical interventions , to name a few ( Choi et al. , 2020 ) . While temporal machine learning models usually assume fully observable and fixed-size input data , irregularly sampled time series raise considerable challenges . For example , the observations of multiple sensors are not well-aligned ; the time intervals among adjacent observations are different across sensors ; and different samples have different numbers of observations for different subsets of sensors recorded at different time points . Prior methods for dealing with irregularly sampled time series involve filling in missing values , using interpolation , kernel methods , and probabilistic approaches ( Schafer & Graham , 2002 ) . The missingness of observations carry informative power ( Little & Rubin , 2014 ) and thus imputation of missing data is not always beneficial ( Agniel et al. , 2018 ) . While modern techniques involve recurrent neural network architectures ( e.g. , RNN , LSTM , GRU ) ( Hochreiter & Schmidhuber , 1997 ; Cho et al. , 2014 ) and transformers ( Vaswani et al. , 2017 ) , they are restricted to regular sampling or assume aligned measurements across modalities . For misaligned measurements , existing methods tend to rely on a two-stage process of imputation to get a regularly-sampled version of a dataset , and then performing a downstream task such as classification . This decoupled approach might not fully exploit informative missingness patterns that could be essential for the downstream task thus achieving suboptimal performance ( Wells et al. , 2013 ; Li & Marlin , 2016 ) . Thus , several approaches circumvent imputation and directly model irregularly sampled time series data ( Che et al. , 2018 ; Horn et al. , 2020 ; Shukla & Marlin , 2021 ; 2018 ) . However , few explicitly consider relational structure to address the characteristics of irregularly sampled multivariate time series and none leverages graph neural networks . Present work . To address the characteristics of irregularly sampled time series data , we propose to model temporal dynamics of sensor dependencies and how those relationships evolve over time . Previous studies ( Wu et al. , 2021 ; Li et al. , 2020a ; Zhang et al. , 2019 ) have noted that the inter-sensor correlations bring rich information in modeling time series . Our intuitive assumption is that the observed sensors can indicate how the unobserved sensors currently behave , which can further improve the representation learning of irregular multivariate time series . We develop RAINDROP1 , a graph-guided network that leverages relational structure to embed and classify irregularly sampled multivariate time series . RAINDROP takes samples as input , each sample containing multiple sensors and each sensor consisting of irregularly recorded observations ( e.g. , in clinical data , an individual patient ’ s state of health , recorded at irregular time intervals with different subsets of sensors observed at different times ) . RAINDROP model is inspired by the idea of raindrops falling into a pool at sequential but non-uniform time intervals and thereby creating ripple effects that propagate across the pool . Mathematically , in RAINDROP , observations ( i.e. , raindrops ) hit a sensor graph ( i.e. , pool ) asynchronously and at irregular time intervals , and each observation is processed by passing messages to neighboring sensors ( i.e. , causes a ripple effect in the pool ) taking into account the learned sensor dependencies ( Figure 1 ) . As such , RAINDROP can handle misaligned observations , varying time gaps , arbitrary numbers of observations , and produce embeddings via a novel hierarchical attention . RAINDROP is the first to explicitly model sensor dependencies in learning representations of irregularly sampled time series . We represent dependencies with a separate sensor graph for every sample , wherein nodes indicate sensors and edges denote relationships between them . In addition to capturing distinct sensor dependencies within each sample , RAINDROP i ) takes advantage of similarities between different samples via sharing of parameters in the calculation of attention weights , and ii ) considers importance of successive observations via temporal attention . RAINDROP adaptively estimates missing observations based on recorded information and the learned graph structure . We compare RAINDROP to five state-of-the-art methods across three datasets and four experimental settings , including a setup where a subset of sensors in the test set have malfunctioned ( i.e. , have no readouts at all ) . Experiments show that RAINDROP outperforms baselines on all datasets with an average AUROC improvement of 3.5 % in absolute points on classification tasks . Further , RAINDROP achieves a considerable margin ( 9.3 % absolute points in accuracy on activity recognition ) when a subset of sensors malfunction . 2 RELATED WORK . Learning with irregularly sampled multivariate time series . Irregular time series indicate that the time intervals between adjacent observations are varying ( Zerveas et al. , 2021 ; Tipirneni & Reddy , 2021 ; Chen et al. , 2020 ) . In the multivariate case , irregularity means that observations are misaligned across different sensors . Further , because of a multitude of sampling frequencies and varying time intervals , the number of observations can vary considerably across samples ( Fang & Wang , 2020 ; Kidger et al. , 2020 ) . Predominant downstream tasks for time series are classification ( i.e. , predicting a label for a given sample , e.g. , Tan et al . ( 2020 ) ; Ma et al . ( 2020 ) ) and forecasting ( i.e. , anticipating future observations based on historical observations , e.g. , Wu et al . ( 2020a ) ) . The above mentioned data characteristics create considerable challenges for machine learning models that expect wellaligned and fixed-size input ( Shukla & Marlin , 2020 ) . An intuitive way to deal with irregular time series is to impute missing values and process them as a regular time series dataset ( Mikalsen et al. , 2021 ; Li & Marlin , 2020 ; Shan & Oliva , 2021 ) . However , imputation methods can distort underlying 1Code and datasets are available at https : //anonymous.4open.science/r/Raindrop . distributions and introduce unwanted bias in the dataset . To this end , recent methods directly learn from irregularly sampled time series . For example , Che et al . ( 2018 ) propose a decay mechanism based on gated recurrent units ( GRU-D ) , incorporating binary masking indicators and time intervals to capture long-term temporal dependencies . SeFT ( Horn et al. , 2020 ) re-formulates irregularly sampled time series into a set of observations and learn representations through set functions that are insensitive to alignment . Chen et al . ( 2018 ) model continuous-time hidden dynamics by latent ordinary differential equations ( Latent-ODE ) , combined with an RNN for temporal representations . mTAND ( Shukla & Marlin , 2021 ) leverages multi-time attention mechanism to learn temporal similarity from non-uniformly collected measurements and produces continuous-time embeddings . IP-Net ( Shukla & Marlin , 2018 ) and DGM2 ( Wu et al. , 2021 ) adopt similar imputation paradigms to interpolate the irregular time series against a set of reference points using a kernel-based method . The learned inter-sensor relations are static without considering sample-specific and time-specific characteristics . In contrast with these methods , RAINDROP leverages dynamic graphs to address the characteristics of irregular time series and improve the quality of learned representations . Learning with graphs and neural message passing . There has been a surge of interest in applying neural networks to graphs , leading to the development of graph embeddings ( Zhou et al. , 2020 ; Li et al. , 2021 ) , graph neural networks ( Wu et al. , 2020b ) , and message passing neural networks ( Gilmer et al. , 2017 ) . Our approach is mainly related to methods that perform message passing along edges to update node representations through neural transformations ( Riba et al. , 2018 ; Nikolentzos et al. , 2020 ; Galkin et al. , 2020 ; Fey et al. , 2020 ; Lin et al. , 2018 ; Zhang et al. , 2020 ) . However , in contrast to message passing used to make predictions on graphs , we are interested in leveraging it to meet the challenges of irregularly sampled time series . In particular , we consider message passing where nodes are sensors , describing a particular sample ( e.g. , patient , Figure 1 ) , and we design a message-passing network with learnable adjacency matrices . The key difference with the predominant use of message passing is that RAINDROP uses it to estimate edges ( dependencies ) between sensors rather than applying it on a fixed , apriori-given graph . To the best of our knowledge , no prior work utilized sensor dependencies for learning representations of irregularly sampled multivariate time series . Finally , while prior work used message passing for regular time series ( Wang et al. , 2020 ; Wu et al. , 2020c ; Kalinicheva et al. , 2020 ) , no modeling of irregularly sampled time series has been attempted . 3 RAINDROP Let D = { ( Si , yi ) | i = 1 , . . . , N } denote an irregular time series dataset with N labeled samples ( Figure 2 ) . Every sample Si is an irregular multivariate time series with a corresponding label yi ∈ { 1 , . . . , C } , indicating which of the C classes Si is associated with . Each sample contains M non-uniformly measured sensors that are denoted as u , v , etc . RAINDROP can also work on samples with a subset of sensors ( see Sec . 4.1 ) . Each sensor is given by a sequence of observations ordered by time . For sensor u in sample Si , we denote a single observation as a tuple ( t , xti , u ) , meaning that sensor u was recorded with value xti , u ∈ R at timestamp t ∈ R+ . We omit sample index i and sensor index u in timestamp t. Sensor observations are irregularly recorded , meaning that time intervals between successive observations can vary across sensors . For sensor u in sample Si , we use Ti , u to denote the set of timestamps that u , or at least one of u ’ s L-hop neighbors ( L is the number of layers in RAINDROP ’ s message passing ) is recorded . We use || and T to denote concatenation and transpose , respectively . We omit layer index l ∈ { 1 , . . . , L } for simplicity when text is clear . Problem ( Representation learning for irregularly sampled multivariate time series ) . A datasetD of irregularly sampled multivariate time series is given , where each sample Si has multiple sensors and each sensor has a variable number of observations . RAINDROP learns a function f : Si → zi that maps Si to a fixed-length representation zi suitable for downstream task of interest , such as classification . Using learned zi , RAINDROP can predict label ŷi ∈ { 1 , . . . , C } for Si . RAINDROP learns informative embeddings for irregularly samples time series . The learned embeddings capture temporal patterns of irregular observations and explicitly consider varying dependencies between sensors . While we focus on time-series classification in this work , the proposed method can be easily extended to broader applications such as regression , clustering and generation tasks .

Summary: This manuscript describes a method that models multiple time series data (sensors) across different individuals (samples). A graph is used to model the dependency between the sensors, such that the data of one sensor could be used to infer another, which is the key point of this ms. How one sensor affects another is modelled as a message passing problem on the graph. At each time point, an observation embedding is generated, either from a sensor or its neighbours in the graph. Observation embeddings of a sensor is then aggregated into a fixed-length sensor embedding, using temporal attention. These sensor embeddings are concatenated into a sample embedding for an individual for downstream analysis such as classification.