Graph-Guided Network for Irregularly Sampled Multivariate Time Series

1 INTRODUCTION . Multivariate time series are prevalent in a variety of domains including healthcare , space science , cybersecurity , biology , and finance ( Ravuri et al. , 2021 ; Sousa et al. , 2020 ; Sezer et al. , 2020 ; Fawaz et al. , 2019 ; Abanda et al. , 2019 ; Tang et al. , 2018 ) . Practical issues often exist in collecting sensor measurements that lead to various types of irregularities caused by missing observations , such as cost saving , sensor failures , external forces in physical scenarios , medical interventions , to name a few ( Choi et al. , 2020 ) . While temporal machine learning models usually assume fully observable and fixed-size input data , irregularly sampled time series raise considerable challenges . For example , the observations of multiple sensors are not well-aligned ; the time intervals among adjacent observations are different across sensors ; and different samples have different numbers of observations for different subsets of sensors recorded at different time points . Prior methods for dealing with irregularly sampled time series involve filling in missing values , using interpolation , kernel methods , and probabilistic approaches ( Schafer & Graham , 2002 ) . The missingness of observations carry informative power ( Little & Rubin , 2014 ) and thus imputation of missing data is not always beneficial ( Agniel et al. , 2018 ) . While modern techniques involve recurrent neural network architectures ( e.g. , RNN , LSTM , GRU ) ( Hochreiter & Schmidhuber , 1997 ; Cho et al. , 2014 ) and transformers ( Vaswani et al. , 2017 ) , they are restricted to regular sampling or assume aligned measurements across modalities . For misaligned measurements , existing methods tend to rely on a two-stage process of imputation to get a regularly-sampled version of a dataset , and then performing a downstream task such as classification . This decoupled approach might not fully exploit informative missingness patterns that could be essential for the downstream task thus achieving suboptimal performance ( Wells et al. , 2013 ; Li & Marlin , 2016 ) . Thus , several approaches circumvent imputation and directly model irregularly sampled time series data ( Che et al. , 2018 ; Horn et al. , 2020 ; Shukla & Marlin , 2021 ; 2018 ) . However , few explicitly consider relational structure to address the characteristics of irregularly sampled multivariate time series and none leverages graph neural networks . Present work . To address the characteristics of irregularly sampled time series data , we propose to model temporal dynamics of sensor dependencies and how those relationships evolve over time . Previous studies ( Wu et al. , 2021 ; Li et al. , 2020a ; Zhang et al. , 2019 ) have noted that the inter-sensor correlations bring rich information in modeling time series . Our intuitive assumption is that the observed sensors can indicate how the unobserved sensors currently behave , which can further improve the representation learning of irregular multivariate time series . We develop RAINDROP1 , a graph-guided network that leverages relational structure to embed and classify irregularly sampled multivariate time series . RAINDROP takes samples as input , each sample containing multiple sensors and each sensor consisting of irregularly recorded observations ( e.g. , in clinical data , an individual patient ’ s state of health , recorded at irregular time intervals with different subsets of sensors observed at different times ) . RAINDROP model is inspired by the idea of raindrops falling into a pool at sequential but non-uniform time intervals and thereby creating ripple effects that propagate across the pool . Mathematically , in RAINDROP , observations ( i.e. , raindrops ) hit a sensor graph ( i.e. , pool ) asynchronously and at irregular time intervals , and each observation is processed by passing messages to neighboring sensors ( i.e. , causes a ripple effect in the pool ) taking into account the learned sensor dependencies ( Figure 1 ) . As such , RAINDROP can handle misaligned observations , varying time gaps , arbitrary numbers of observations , and produce embeddings via a novel hierarchical attention . RAINDROP is the first to explicitly model sensor dependencies in learning representations of irregularly sampled time series . We represent dependencies with a separate sensor graph for every sample , wherein nodes indicate sensors and edges denote relationships between them . In addition to capturing distinct sensor dependencies within each sample , RAINDROP i ) takes advantage of similarities between different samples via sharing of parameters in the calculation of attention weights , and ii ) considers importance of successive observations via temporal attention . RAINDROP adaptively estimates missing observations based on recorded information and the learned graph structure . We compare RAINDROP to five state-of-the-art methods across three datasets and four experimental settings , including a setup where a subset of sensors in the test set have malfunctioned ( i.e. , have no readouts at all ) . Experiments show that RAINDROP outperforms baselines on all datasets with an average AUROC improvement of 3.5 % in absolute points on classification tasks . Further , RAINDROP achieves a considerable margin ( 9.3 % absolute points in accuracy on activity recognition ) when a subset of sensors malfunction . 2 RELATED WORK . Learning with irregularly sampled multivariate time series . Irregular time series indicate that the time intervals between adjacent observations are varying ( Zerveas et al. , 2021 ; Tipirneni & Reddy , 2021 ; Chen et al. , 2020 ) . In the multivariate case , irregularity means that observations are misaligned across different sensors . Further , because of a multitude of sampling frequencies and varying time intervals , the number of observations can vary considerably across samples ( Fang & Wang , 2020 ; Kidger et al. , 2020 ) . Predominant downstream tasks for time series are classification ( i.e. , predicting a label for a given sample , e.g. , Tan et al . ( 2020 ) ; Ma et al . ( 2020 ) ) and forecasting ( i.e. , anticipating future observations based on historical observations , e.g. , Wu et al . ( 2020a ) ) . The above mentioned data characteristics create considerable challenges for machine learning models that expect wellaligned and fixed-size input ( Shukla & Marlin , 2020 ) . An intuitive way to deal with irregular time series is to impute missing values and process them as a regular time series dataset ( Mikalsen et al. , 2021 ; Li & Marlin , 2020 ; Shan & Oliva , 2021 ) . However , imputation methods can distort underlying 1Code and datasets are available at https : //anonymous.4open.science/r/Raindrop . distributions and introduce unwanted bias in the dataset . To this end , recent methods directly learn from irregularly sampled time series . For example , Che et al . ( 2018 ) propose a decay mechanism based on gated recurrent units ( GRU-D ) , incorporating binary masking indicators and time intervals to capture long-term temporal dependencies . SeFT ( Horn et al. , 2020 ) re-formulates irregularly sampled time series into a set of observations and learn representations through set functions that are insensitive to alignment . Chen et al . ( 2018 ) model continuous-time hidden dynamics by latent ordinary differential equations ( Latent-ODE ) , combined with an RNN for temporal representations . mTAND ( Shukla & Marlin , 2021 ) leverages multi-time attention mechanism to learn temporal similarity from non-uniformly collected measurements and produces continuous-time embeddings . IP-Net ( Shukla & Marlin , 2018 ) and DGM2 ( Wu et al. , 2021 ) adopt similar imputation paradigms to interpolate the irregular time series against a set of reference points using a kernel-based method . The learned inter-sensor relations are static without considering sample-specific and time-specific characteristics . In contrast with these methods , RAINDROP leverages dynamic graphs to address the characteristics of irregular time series and improve the quality of learned representations . Learning with graphs and neural message passing . There has been a surge of interest in applying neural networks to graphs , leading to the development of graph embeddings ( Zhou et al. , 2020 ; Li et al. , 2021 ) , graph neural networks ( Wu et al. , 2020b ) , and message passing neural networks ( Gilmer et al. , 2017 ) . Our approach is mainly related to methods that perform message passing along edges to update node representations through neural transformations ( Riba et al. , 2018 ; Nikolentzos et al. , 2020 ; Galkin et al. , 2020 ; Fey et al. , 2020 ; Lin et al. , 2018 ; Zhang et al. , 2020 ) . However , in contrast to message passing used to make predictions on graphs , we are interested in leveraging it to meet the challenges of irregularly sampled time series . In particular , we consider message passing where nodes are sensors , describing a particular sample ( e.g. , patient , Figure 1 ) , and we design a message-passing network with learnable adjacency matrices . The key difference with the predominant use of message passing is that RAINDROP uses it to estimate edges ( dependencies ) between sensors rather than applying it on a fixed , apriori-given graph . To the best of our knowledge , no prior work utilized sensor dependencies for learning representations of irregularly sampled multivariate time series . Finally , while prior work used message passing for regular time series ( Wang et al. , 2020 ; Wu et al. , 2020c ; Kalinicheva et al. , 2020 ) , no modeling of irregularly sampled time series has been attempted . 3 RAINDROP Let D = { ( Si , yi ) | i = 1 , . . . , N } denote an irregular time series dataset with N labeled samples ( Figure 2 ) . Every sample Si is an irregular multivariate time series with a corresponding label yi ∈ { 1 , . . . , C } , indicating which of the C classes Si is associated with . Each sample contains M non-uniformly measured sensors that are denoted as u , v , etc . RAINDROP can also work on samples with a subset of sensors ( see Sec . 4.1 ) . Each sensor is given by a sequence of observations ordered by time . For sensor u in sample Si , we denote a single observation as a tuple ( t , xti , u ) , meaning that sensor u was recorded with value xti , u ∈ R at timestamp t ∈ R+ . We omit sample index i and sensor index u in timestamp t. Sensor observations are irregularly recorded , meaning that time intervals between successive observations can vary across sensors . For sensor u in sample Si , we use Ti , u to denote the set of timestamps that u , or at least one of u ’ s L-hop neighbors ( L is the number of layers in RAINDROP ’ s message passing ) is recorded . We use || and T to denote concatenation and transpose , respectively . We omit layer index l ∈ { 1 , . . . , L } for simplicity when text is clear . Problem ( Representation learning for irregularly sampled multivariate time series ) . A datasetD of irregularly sampled multivariate time series is given , where each sample Si has multiple sensors and each sensor has a variable number of observations . RAINDROP learns a function f : Si → zi that maps Si to a fixed-length representation zi suitable for downstream task of interest , such as classification . Using learned zi , RAINDROP can predict label ŷi ∈ { 1 , . . . , C } for Si . RAINDROP learns informative embeddings for irregularly samples time series . The learned embeddings capture temporal patterns of irregular observations and explicitly consider varying dependencies between sensors . While we focus on time-series classification in this work , the proposed method can be easily extended to broader applications such as regression , clustering and generation tasks .

Summary: The core idea of the paper is transforming multivariate time series data into a vector z and using z for classification. The transformation has two steps. Firstly, each time record of each sensor is mapped into a  space of 20 dimensions. The first 4 dimensions are used for feature extraction, and the last 16 dimensions are used for positional encoding. Secondly, self-attention is used to transform the embedding into a vector z and use it for classification.