Graph-Guided Network for Irregularly Sampled Multivariate Time Series
Authors: Xiang Zhang
Marko Zeman
Theodoros Tsiligkaridis
Marinka Zitnik
Venue: ICLR 2022 Poster (2022)
Keywords: time seres
irregular time series
graph neural networks
attention mechanism
time series classification
multivariate time series
representation learning
embeddings

Abstract: In many domains, including healthcare, biology, and climate science, time series are irregularly sampled with varying time intervals between successive readouts and different subsets of variables (sensors) observed at different time points. Here, we introduce RAINDROP, a graph neural network that embeds irregularly sampled and multivariate time series while also learning the dynamics of sensors purely from observational data. RAINDROP represents every sample as a separate sensor graph and models time-varying dependencies between sensors with a novel message passing operator. It estimates the latent sensor graph structure and leverages the structure together with nearby observations to predict misaligned readouts. This model can be interpreted as a graph neural network that sends messages over graphs that are optimized for capturing time-varying dependencies among sensors. We use RAINDROP to classify time series and interpret temporal dynamics on three healthcare and human activity datasets. RAINDROP outperforms state-of-the-art methods by up to 11.4% (absolute F1-score points), including techniques that deal with irregular sampling using fixed discretization and set functions. RAINDROP shows superiority in diverse setups, including challenging leave-sensor-out settings. 

Published as a conference paper at ICLR 2022
GRAPH-GUIDED NETWORK FOR IRREGULARLY SAMPLED MULTIVARIATE TIME SERIES
Xiang Zhang
Marko Zeman
Harvard University xi wwangggaaggggaaaeeee hww.aad..aadddeeaaaoo
University of Ljubljana
marko. zeman@ uni 1 j.s.s
Theodoros Tsiligkaridis
Marinka Zitnik
MIT Lincoln Laboratory ttsiliel1 mim.mii......................... edu
Harvard University
marinka hms hww.aaaraaa..oo.
ABSTRACT
In many domains, including healthcare, biology, and climate science, time series are irregularly sampled with varying time intervals between successive readouts and different subsets of variables (sensors) observed different time points. Here, we introduce RAINDROP, graph neural network that embeds irregularly sampled and multivariate time series while also learning the dynamics of sensors purely from observational data. RAINDROP represents every sample as separate sensor graph and models time- varying dependencies between sensors with novel mes- sage passing operator. It estimates the latent sensor graph structure and leverages the structure together with nearby observations to predict misaligned readouts. This model can be interpreted as graph neural network that sends messages over graphs that are optimized for capturing time-varying dependencies among sensors. We use RAINDROP classify time series and interpret temporal dynamics on three health- care and human activity datasets. RAINDROP outperforms state-of-the methods by up to 11.4% (absolute F1-score points), including techniques that deal with irregular sampling using fixed discretization and set functions. RAINDROP shows superiority in diverse setups, including challenging leave-sensor-out settings.
INTRODUCTION
Multivariate time series are prevalent in variety of domains, including healthcare, space science, cyber security, biology, and finance (Ravuri et al., 2021; Sousa et al., 2020; Sezer et al., 2020; Fawaz et al., 2019). Practical issues often exist in collecting sensor measurements that lead to various types of irregularities caused by missing observations, such as saving costs, sensor failures, external few (Choi al., 2020). While temporal
forces in physical systems, medical interventions, machine learning models typically assume fully observed and fixed-size inputs, irregularly sampled time series raise considerable challenges (Shukla & Marlin, 2021; Hu al., 2021). For example, observations of different sensors might not be aligned, time intervals among adjacent observations are different across sensors, and different samples have different numbers of observations for different subsets of sensors recorded at different time points (Horn et al., 2020; Wang et al., 2011).
Prior methods for dealing with irregularly sampled time series involve filling in missing values using interpolation, kernel methods, and probabilistic approaches (Schafer & Graham, 2002). However, the absence of observations can be informative on its own (Little Rubin, 2014) and thus imputing missing observations is not necessarily beneficial (Agniel et al., 2018) 8). While modern techniques involve recurrent neural network architectures (e.g. RNN, LSTM, GRU) (Cho et al., 2014) and transformers (Vaswani et al., 2017), they are restricted to regular sampling or assume aligned measurements across modalities. For misaligned measurements, existing methods tend to rely on a two-stage approach that first imputes missing values to produce regulaly-samplee dataset and then optimizes a model choice for downstream performance. This decoupled approach does not fully exploit informative missingness patterns deal with irregular sampling, thus producing suboptimal

---

Published as a conference paper at ICLR 2022
performance (Wells et al., 2013; Li & Marlin, 2016). Thus, recent methods circumvent the imputation stage and directly model irregularly sampled time series (Che et al., 2018; Horn et al., 2020).
Previous studies (Wu et al., 2021; Li et al., 2020a; Zhang et al., 2019) have noted that inter-sensor correlations bring rich information in modeling time series. However, only few studies consider relational structure of irregularly sampled time series, and those which do have limited ability in capturing inter-sensor connections (Wu et al., 2021; Shukla & Marlin, 20. In contrast, we integrate recent advances in graph neural networks take advantage of relational structure among sensors. We learn latent graphs from multivariate time series and model time-varying inter-sensor dependencies through neural message passing, establishing graph neural networks as a way model sample-varynn and time-var varying structure in complex time series.
Present work. To address the characteristics of irregu- larly sampled time series, we propose to model temporal dynamics of sensor dependencies and how those relation- ships evolve over time. Our intuitive assumption is that the observed sensors can indicate how the unobserved sensors currently behave, which can further improve the represen- tation learning of irregular multivariate time series. We develop RAINDROP', graph neural network that lever- ages relational structure to embed and classify irregularly sampled multivariate time series. RAINDROP takes ples as input, each sample containing multiple sensors and each sensor consisting of irregularly recorded observa- tions (e.g., in clinical data, an individual patient's 's state of health is recorded at irregular time intervals with different subsets of sensors observed at different times). RAINDROP model is inspired by how raindrops hit a surface varying times and create ripple effects that propagate through the surface. Mathematically, in RAINDROP, observations (i.e., raindrops) hit sensor graph (i.e., surface) asynchronously and at irregular time intervals. Every observation is processed by passing messages to neighboring sensors (i.e., creating ripples), taking into account the learned sensor dependencies (Figure 1). As such, RAINDROP can handle misaligned observations, varying time gaps, arbitrary numbers of observations, and produce multi-scale embeddings via novel hierarchical attention.
Blood pressure
Blood test
Heart rala
Tamper aturr
Observaiooe tuaneroopai rydlyydd et cfffooent semeore. and have imogu or
t
Chowraation on Aia
Survicl
Death
The RAINDROP approach. For sample St, sensor is recorded at time i as value x triggering propagation and trans- formation neural messages along edges of S, 's sensor dependency graph.
Figure 1:
We represent dependencies with a separate sensor graph for every sample wherein nodes indicate sensors and edges denote relationships between them. Sensor graphs latent in the sense that graph connectivity is learned by RAINDROP purely from observational time series. In addition to capturing sensor dependencies within each sample, RAINDROP i) takes advantage of similarities between different samples by sharing parameters when calculating attention weights, and ii) considers importance of sequential sensor observations via temporal attention.
RAINDROP adaptively estimates observations based on both neighboring readouts in the temporal domain and similar sensors as determined by the connectivity of optimized sensor graphs. We compare RAINDROP to five state-of-the-ar methods on two healthcare datasets and an activity recognition dataset across three experimental settings, including a setup where a subset of sensors in the test set is malfunctioning (i.c., have no readouts at all). Experiments show that RAINDROP outperforms baselines on all datasets with an average AUROC improvement of 3.5% in absolute points on various classification tasks. Further, RAINDROP improves prior work by 9.3% margin (absolute points in accuracy) when varying subsets of sensors malfunction.
2 RELATED WORK
Our work here builds on time-series representation learning and notions of graph neural networks and attempts to resolve them by developing single, unified approach for analysis of complex time series.
Learning with irregularly sampled multivariate time series. Irregular time series are character- ized by varying time intervals between adjacent observations (Zerveas et al., 2021; Tipirenni &
Code and datasets are available http:///ittaahcommmmimmmmaaaaaar
2

---

Published as a conference paper at ICLR 2022
Chen al., 2020). In multivariate case, irregularity means that observations can be misaligned across different sensors, which can further complicate the analysis. Further, because of multitude of sampling frequencies and varying time intervals, the number of observations can also vary considerably across samples (Fang & Wang, 2020; Kidger et al., 2020). Predominant downstream tasks for time series are classification (i.e., predicting label for given sample, e-g., Tan et al. (2020); Ma et al. (2020)) and forecasting (i.e., anticipating future observations based on historical observations, e.g. Wu et al. (2020a)). The above mentioned characteristics create considerable challenges for models that expect well-aligned and fixed-size size inputs (Shukla & Marlin, 2020). An intuitive way to deal with irregular time series to impute missing values and process them as regular time series (Mikalsen et al., 2021; Li & Marlin, 2020; Shan & Oliva, 2021). However, imputation methods can distort the underlying distribution and lead to unwanted distribution shifts. To this end, recent methods directly learn from irregularly sampled time series (Chen et al., 201 8). For example, Che et al. (2018) develop a decay mechanism based on gated recurrent units (GRU-D) and binary masking to capture long-range temporal dependencies. SeFT (Horn et al., 2020) takes set-based approach and transforms irregularly sampled time series datasets into sets of observations modeled by set functions insensitive to misalignment. mTAND (Shukla & Marlin, 2021) leverages multi-time attention mechanism to learn temporal similarity from non-uniformly collected measure- ments and produce continuous-time embeddings. IP-Ner (Shukla & Marlin, 2018) and DGM (Wu et al., 2021) adopt imputation to interpolate irregular time series against set of reference points using a kernel- basee approach. The learned inter-sensor relations are static ignoring and time-speeite characteristics. In contrast with the above methods, RAINDROP leverages dynamic graphs to address the characteristics of irregular time series and produce high-quality representations.
Reddy, 2021;
Learning with graphs and neural message passing. There has been surge of interest in applying neural networks to graphs, leading to the development of graph embeddings (Zhou al., 2020; Li et al., 2021), graph neural networks (Wu et al., 2020b), and message passing neural networks (Gilmer et al., 2017). To address the challenges of irregular time series, RAINDROP specifies a message passing strategy to exchange neural message along edges of sensor graphs and deal with misaligned sensor readouts (Riba et al., 2018; Nikolentzos et al., 2020; Galkin et al., 2020; Fey et al., 2020; Lin et al., 2018; Zhang et al., 2020). In particular, RAINDROP considers message passing on latent sensor graphs, each graph deseribing a different sample (e.g., patient, Figure 1), and it specifies a passage-passing network with learnable adjacency matrices. The key difference with the predominant use of message passing is that RAINDROP uses it to estimate edges (dependencies) between sensors rather than applying it on fixed, apriori-given graph. To the best of our knowledge, prior work did not utilize sensor dependencies for irregularly sampled time series. While prior work used message passing for regular time series (Wang et al., 2020; Wu et al., 2020c; Kalinicheva et al., 2020; Zha et al., 2022), its utility for irregularly sampled time series has not yet been studied.
3 RAINDROP
Let D {(S,, y1) -i=1.... N) denote an irregular time series dataset with N labeled samples (Figure 2). Every sample S; is an irregular multivariate time series with a corresponding label Vi E [fl...., C), indicating which of the C' classes S; is associated with. Each sample contains M non-uniformly measured sensors that are denoted as u, etc. RAINDROP can also work on samples with only a subset of active sensors (see Sec. 4.). Each sensor is given by a sequence of observations ordered by time. For sensor in sample S, we denote a single observation as a tuple (4, ",, meaning that sensor was recorded with value 24 R timestamp & R We omit sample index i and sensor index u in timestamp f. Sensor observations are irregularly recorded, meaning that time intervals between successive observations can vary across sensors. For sensor in sample S; we use Tr, to denote the set of timestamps that u, or at least one of u's L-hop neighbors (L is the number of layers in RAINDROP's message passing) is recorded. We use || and T to denote concatenation and transpose, respectively. We omit layer index E L} for simplicity when clear from the text.
Problem (Representation learning for irregularly sampled multivariate time series). A dataset D of irregularly sampled multivariate time series is given. where each sample S, has multiple sensors and each sensor has variable number of observations. RAINDROP learns a function f St > 2; that maps S, to a fixed-lengrt representation z, suitable for downstream task of interest, such as classification. Using learned 2,, RAINDROP can predict label iJ {1,..., C} for S..
3

---

Published as a conference paper at ICLR 2022
RAINDROP learns informative embeddings for inregularly samples time series. The learned embed- dings capture temporal patterns of irregular observations and explicitly consider varying dependencies between sensors. While focus time-series classification in this work, the proposed method can be easily extended to broader applications such as regression, clustering and generation tasks.
3.1 OVERVIEW OF RAINDROP
RAINDROP aims to learn a fixed-dimensional embedding 2; for a given sample S and predict the associated label ÿt- To this end, it generates sample embeddings using a hierarchical architecture composed of three levels to model observations (sensor readouts), sensors, and whole samples (Figure 2). Without loss of generality, we describe RAINDROP'S procedure as if observations arrive one at time (one sensor is observed at time t and other sensors do not have observations). there are multiple observations at the same time, RAINDROP effortlessly process them in parallel.
(Section 3.5) Sample
S
(Secionn 3.4)
RAINDROP first constructs graph for every sample where nodes rep- resent sensors and edges indicate relations between sensors (Sec. 3.2). We use Gj to denote the sensor graph for sample S; and éi,wr to rep- resent the weight of directed edge from sensor u sensor Gū- Sensor graphs are automatically optimized considering sample- and time-wise wise specificity.
Observation (Section
Figure 2: Hicrarchical structure of irregular multivariate time se- ries dataset. RAINDROP embeds individual observations consid- ering inter-sensor dependencies (Sec. 3.3), aggregates them into a sensor embedding using tempo- ral attention (Sec. 3,4), and finally integrates sensor embeddings into sample emhedding (Sec. 3.5).
The key idea of RAINDROP is to borrow information from u's neigh- bors based on estimated relationships between and other sensors. This is achieved via message passing carried out on S, 's dependency graph and initiated at node u in the graph. When an observation (1, (,at is recorded for sample S; at time I, RAINDROP first em- beds the observation at active sensor u (i.e., sensor whose value recorded) and then propagates messages (i.e., the observation embeddings) from и to neighboring sensors along edges in sensor dependency graph G;- As result, recording the value of u affect u's embedding as well as embeddings of other sensors that related to u (Sec. 3.3). Finally, RAINDROP generates sensor embeddings by aggregating all observation embeddings for each sensor (across all timestamps) using temporal attention weights (Sec. 3.4). At last, RAINDROP embeds sample S; based on sensor embeddings (Sec. 3.5) and feeds the sample embedding into downstream predictor.
3.2 CONSTRUCTING SENSOR DEPENDENCY GRAPHI
We build a directed weighted graph G; = {V, for every sample S, and refer to as the sensor dependency graph for S, Nodes V represent sensors and edges &, describe dependencies between sensors in sample S; that RAINDROP infers. As we show in experiments, RAINDROP can directly used with samples that only contain subset of sensors in V. We denote edge from triplet (u, Ci,uv; v)), where 6j,av E [0, 1) represents the strength of relationship between sensors u and in sample S, Edge (u, Ci, v) describes the relationship between and v: when receives an observation, it will send a neural message to following edge éi If Gi O, there is no exchange of neural information between and v, indicating that the two sensors assume that the importance of u to u is different than the importance of v to and so we treat sensor dependency graphs directed, i.e., Ci, # Ci,vu. All graphs are initialized fully-connected graphs (i.e,, fi, 1 for any u, and S;) and edge weights ei,wr updated following Eq. 3 during model training. available, it is easy integrate additional domain knowledge into graph initialization.
unrelated. We
3.3 GENERATING EMBEDDINGS OF INDIVIDUAL. OBSERVATIONS
Let u indicate active sensor at time E Te.,s i.e., sensor whose value of is observed at f, and let u be connected to e through edge (u, Ci,uv; v.. We next deseribe how to produce observation embeddings hi.. E &d and ht E dd for sensors and v, respectively (Figure 3a). We omit layer index and note that the proposed strategy applies to any number of layers.
4

---

Published as a conference paper at ICLR 2022
Sample S; records the value sipy sensor us at time
b
Observation-level processing In sample S h
Sensor-level processing In sample S
P
Lnter SERGAY
t
h
UDD
ClAm Edge weight
h
Kiy
B
ORDD
t2
Attention weights PLy
Update edge weight Lise in
Sample St
21,y Sensor embedding
A9.4 $,00
L-11
tr
Stacked observation embeddings
Learned embaddinge
Observation (mput)
Sensors
O product
Message
0003 Weight vestor
1039 Time represeritiion
Figure 3: (a) RAINDROP generates observation embedding h based on observed value xi.u at t, passes message to neighbor sensors such U, and generates hip through inter-sensor dependencics. The of denotes time-specific attention weight, calculated based on time representation p and weight vector r.. Edge weight embedding. Apply the message passing in (a) to timestamps and produce corresponding observation embeddings. number of observation embeddings into a fixed-length sensor embedding &.. while paying distinctive attentions to different observations. We independently apply the processing procedure to all sensors. (c) RAINDROP updates edge weight e based on the edge weight e ((-1) from previous layer and the learned inter-sensor attention weights in all time steps. We explicitly show layer index I as multiple layers are involved.
ei is shared by all timestamps. (b) An illustration of generating
aggregate arbitrary
Embedding an observation of active sensor. Let u denote an active sensor whose value has just been observed as u!... For sufficient expressive power (Veličković et al., 2018), we map observation of to a high-dimensional space using nonlinear transformation: h a (x R.... We use sensor- specific transformations because values recorded different sensors can follow different distributions, which is achieved by trainable weight vectors R., depending what sensor is activated (Li et al., 2020b). Alternatives, such as multilayer perceptron, can be considered to transform oं.u into h As h represents information brought on by observing x! u we regard h as the embedding of w's observation l. Sensor-specific weight vectors R., shared across samples.
Passing messages along sensor dependency graphs. For sensors that are not active at timestamp but are neighbors of the active sensor in the sensor dependency graph G., RAINDROP uses relationships between u and those sensors to estimate observation embeddings for them. We proceed by describing how RAINDROP generates observation embedding h for sensor assuming is neighbor of u in Ge. Given ht,, and edge (u, Ci,uv; v), we first calculate inter-sensor attention weight a E [0, 1], representing how important is to v via the following equation:
a = ar (h" D D(r.D(rrll) D(r-lln))²)
where To E Rdd is trainable weight vector that is specific to the sensor receiving the message (i.e., h. Vector r, allows the model to learn distinct attention weights for different edges going out from the same sensor u. Further, p! E de is the time representation obtained by converting I-dimensional timestamp & into trigonometric functions (Horn et al., 2020). See Appendix A. for details. RAINDROP uses p to calculate attention weights that are sensitive to time. Finally, is trainable weight matrix mapping h from d, dimensions to (d, dc) dimensions, Taken this together, we estimate the embedding h for w's neighbor e as follows:
dalei-iimensional vector p by passing t through series of
h o ohh W, W. af
where w.,, ww E 4 are trainable weight vectors shared across all samples. The w,, is specific to active sensor and w. is specific neighhoring sensor In the above equation, eg denotes edge
5

---

Published as a conference paper at ICLR 2022
weight shared across all timestamps. The above message passing describes the processing of single observation at a single timestamp. In case multiple sensors are active at time and connected with v, normalize ai (with softmax function) across active sensors and aggregate messages at v.
Overall, RAINDROP produces observation embedding h for sensor e through its relational con- nection with u, even though there is no direct measurement of at time l. These message passing operations are performed adaptively and dynamically estimate missing observations in the embed- ding space based on recorded information and learned graph structure.
Updating sensor dependency graphs. We describe the update of edge weights and prune of graph structures in the situation that stacks multiple R RAINDROP layers (Figure 3). Here we explicitly show layer index / because multiple layers are involved in the computation. As no prior knowledge is assumed, we initialize the graph as all sensors connected with each other. However, the fully connected edges may bridge sensors that should be independent, which will introduce spurious correlations and prevent the model from paying attention to the truly important connections. Addressing this issue, RAINDROP automatically updates edge weights and prunes out less important edges. Based on the (t), aggregated temporal influence driven by the inter-sensor attention weights ", we update edge weights e,uu in each layer / {1,..., by:
(!)
(1-1)
(1)
Σ
( 0iaan
C,a0
[Tr,ul
TET
where T., denotes the set of all timestamps where there is message passes from u to v. In particular, (0) set Ci 1 in the initialization of graph structures. We use L = 2 in all our experiments. In (1) every layer, order the estimated values au for all edges in sample S, and prune bottom K% edges with smallest edge weights (Yang et al., 2021). Pruned edges are not re-added in later layers.
3.4 GENERATING SENSOR EMBEDDINGS
Next we describe how to aggregate observation embeddings into sensor embeddings 2,,u, taking sensor as an example (Figure 3b). Previous step (Sec. 3.3) generates observation embeddings for every timestamp when either u or v neighbor is observed. The observation embeddings different timestamps have unequal importance to the the sensor embedding (Zerveas et al., 2021). We use the temporal attention weight (scalar) " to represent the importance of observation embedding at
We use Tr,, = (4, .2,...; LT} to denote all the timestamps when a readout is observed in v (we can directly generate h or in v's neighbor (we can generate h through message passing). The [ is the corresponding element of vector Br,, which include the temporal attention weights at all timestamps E T....
We use temporal self-attention calculate Bi,.. which is different from the standard self-attention (Hu et al., 2020; Yun et al., 2019). The standard dot-product self-attention generates an atlention malrix with dimension of T T' (where T [T,.vl can vary across samples) that has an attention weight for each pair of observation embeddings. In our case, we only need a single attention vector where each element denotes the temporal attention weight of an observation embedding when generating the sensor embedding. Thus, we modify the typical self-attention model to fit our case: using a trainable to map the self-attention matrix TxT) ) to T-dimensional vector Bi, T'11) through matrix product (Appendix A.2).
8 E IR Tx
The following steps describe how to generate sensor embeddings. We first concatenate observation embedding h with time representation pt to include information of timestamp. Then, we stack the concatenated embeddings [h$, I..,llp'' [p:] for all Tr into matrix H.,. The H, contains all
information of observations and timestamps for sensor v. We calculate 8 through:
Qu.cK.n Vile
B,0 = softmax
(4)
where Qi, and K are two intermediate matrices that are derived from the stacked observation embeddings. In practice, Qú, H Wq and Ki, H., Wx linearly mapped from H.,

---

Published as a conference paper at ICLR 2022
parameterized by We and WK, respectively (Vaswani et al., 2017). The vd is scaling factor where dy is the dimension after linear mapping. Based on the learned temporal attention weights B L. we calculate sensor embedding 2i, through:
21,0
Σ TET.
(Pr (... W))
(5)
where weight matrix W is linear projector shared by sensors and samples. is worth to mention that all attention weights (such as af au and (,,,) can be multi-head. In this work, we describe the model the context of single head for brevity.
Using attentional aggregation, RAINDROP can learn a fixed-length sensor embedding for arbitrary number of observations. Meanwhile, RAINDROP capable of focusing on the most informative observation embeddings. We process all observation embeddings which allows parallel computation for faster training and also mitigates the performance drop caused by modeling long dependencies sequentially. In the case of sensors with very large number of observations, we can reduce the length of time series by subsampling or splitting a long series into multiple short series.
whole instead of sequentially,
3.5 GENERATING SAMPLE EMBEDDINGS
Finally, for sample St. we aggregate sensor embeddings ži,0 (Eq. 5) across all sensors to obtain an embedding z; E dR through readout functiin g as follows: z; g( v== .,,, MM as concatenation). When a sample contains large number of sensors, RAINDROP can seamlessly use a set-based readout function such as averaging aggregation (Appendix A.3). Given an input sample S, RAINDROP'S s strategy outlined in Sec. 3.2-3.5 produces a sample embedding z; that can be further optimized for downstream tasks.
......,,.
(such
3.6 IMPLEMENTATION AND PRACTICAL CONSIDERATIONS
Loss function. RAINDROP'S loss function is formulated as: C LeE + XC,, where &, TJ² En,vE Ei,jev ||ei (/.ullll/( - 1)?, where LCE is cross entropy and C, is a regu- larizer to encourage the model to learn similar sensor dependency graphs for similar samples. The C, measures averaged Euclidean distance between edge weights across all samples pairs, in all sensor pairs (including self-connections). The À is user-defined coefficient. Practically, as N can be large, we calculate L, only for samples in a batch.
Downstream tasks. If a sample has auxiliary attributes (e.g., a patient's demographics) that do not change over time, we can project the attribute vector to a d, diddeennsiooal vector a; with a fully- connected layer and concatenate it with the sample embedding, getting [zi|aa]] At last, we feed [z,] [a (or only 24 if a; is not available) into a neural classifier yp &d. - {1...., C). In our experiments, ip is 2-layer fully-connected network with C' neurons the output layer returning prediction ÿ = ( [[z] (a:) for sample S,.
Sensor dependencies. While modeling sensor dependencies, involve observation embedding (h Eq. 1) of each sample in the calculation of attention weights. Similarly, to model time-wise specificity in graph structures, we consider time information (pf. Eq. 1) when measuring of a RAINDROP can capture similar graph structures across samples from three aspects (Appendix A.4): (1) the initial graphs are the same in all samples; (2) the parameters in message passing (R,,; wa, w., Eq. 2), inter-sensor attention weights calculation (D, Eq. 1), and temporal attention weights calculation (s, Eq. 4; W, Eq. 5) shared by all samples; (3) encourage the model to learn similar graph structures by adding a penalty to disparity structures (C,).
Scalability. RAINDROP is efficient because embeddings can be learned in parallel. In particular, pro- cessing of observation embeddings is independent across timestamps. Similarly, sensor embeddings can be processed independently across different sensors (Figure 3). While the complexity of temporal self-attention calculation grows quadratically with the number of observations, it can practically implemented using hiphly-optimized matrix multiplication.
7

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTS
Datasets. Below we briefly overview healthcare and human activity datasets. (1) P19 (Reyna et al., 2020) includes 38,803 patients that are monitored by 34 sensors. Each patient is associated with binary label representing the occurrence of sepsis. (2) P12 (Goldberger et al., 2000) records temporal measurements of 36 sensors of 11,988 patients in the first 48-hour stay in ICU. The samples labeled based on hospitalization length. (3) PAM (Reiss & Stricker, 2012) contains 5,333 segments from 8 activities of daily living that are measured by 17 sensors. Details are in Appendix A.5.
Baselines. We compare RAINDROP with five state-of-the-art baselines: Transformer (Vaswani et al., 2017), Trans-mean, GRU-D (Che et al., 2018), SeFT (Horn et al., 2020), and mTAND (Shukla & Marlin, 2021). The Trans-mean is an imputation method combining transformer architecture with commonly used average interpolation (i.e., missing values are replaced by average observations in each sensor). The mTAND (Shukla & Marlin, 2021) method has been shown to outperform numerous recurrent models including RNN-Impun INN-tepute (Che al., 2018). RNN-Simple, and Piaased-LSTM, (Neil et al., 2016), along with ordinary differential equations (ODE)-based models such as LATENT-ODE and ODE-RNN (Chen al., 2018). For this reason, we compare with mTAND and do not report comparison with those techniques in this paper. Even though, to better show the superiority of RAINDROP, we provide extensive comparison with popular approaches, such as DGM (Wu et al., 2021) and MTGNN (Wu et al., 2020c), that are designed for forecasting tasks. Further details are in Table I and Appendix 1. Details on hyperparameter selection and baselines are in Appendix A.6, and evaluation metries are presented in Appendix A.7.
4.1 RESULTS ACROSS DIVERSE EVALUATION SETTINGS
Classic time series classification. Setup. We randomly split the dataset into training (80%), validation (10%), and test (10%) set. The indices of these splits are fixed across all methods. Results. As shown in Table 1, RAINDROP obtains the best performance across three benchmark datasets, suggesting its strong performance for time series classification. In particular, in binary classification (P19 and P12). RAINDROP outperforms the strongest baselines by 5.3% in AUROC and 4.8% in AUPRC average. In more challenging 8- way classification on the PAM dataset, RAINDROP outperforms existing approaches by 5.7% in accuracy and 5.5% in F1 score. Further exploratory analyses and benchmarking results are shown in Appendix A.9-A.10.
Setting 1:
Table 1: Method henchmarking on irregularly sampled time series classification (Setting 1).
P19
P12
PAM
Methods
AUROC
AUPRC
AUROC
AUPRC
Accuracy
Precision
Recall
F1 score
Transformer
83.2 1,3
47.6 388
65.1 + 5.6
95.7 + 1.6
83.5 + 1.5
84.8
1.5
86.0 + 1.2
85.0 1.3
Trans-mean
84.1 + 1.7
47.4
66.8 66.8
95.9 + 1.1
83.7
+ 2.3
849 - 26
86.4
+ 2.1
85.1
GRU-D
83.9 1.7
2.1
67.2
95.9 + 2.1
83.3
+
1.6
84.6
146=11
85.2
+ 1.6
84.8 + 1.2
SelT
78.7
2.4
31.1
1
66.8 66.8 0.8
96.2 1 0.2
67.1
1
2.2
70.0
24
68.2
1
1.5
68.5
1.8
ITAND
80.4 +: 1.3
32.4
65.3 1.7
96.5 + 1.2
74.6
1
4.3
74.3
=.3=400
79.5
+ 2.8
76,8
JP.Net
84.6 1 1.3
38.1
3.7
72.5 1
96.7
1
0.3
74.3
1 3.8
2.1 75.6
77.9
1
2.2
76.6
2.8
DGM2-O
86.7 1:
44.7
11.7
2.
96.9
1 0.4
824
85.2
1
2.1
84.3
MTGNN
8..9
39.9
- 8.9
67.5 3.1
96.4
0.7
83.4 1.9
85.2 1.7
86.1 + 1.9
85.9 2.4
RAINDROP
87.0 +: 2.3
51.8 3. 5.5
72.1 1.3
97.0 1 0.4
88.5 1.5
89.9 1.5
89.9 Lt 0.6
89.8 1.0
Setting 2: Leave-fixed-sensors-ott Setup, RAINDROP can compensate for missing sensor obser- vations by exploiting dependencies between sensors. To this end, we test whether RAINDROP can achieve good performance when subset of sensors are completely missing. This setting practically relevant in situations when, for example, sensors fail or are unavailable. We select a fraction of sensors and hide all their observations in both validation and test sets (training samples are not changed). In particular, we leave out the most informative sensors as defined by information gain analysis (Appendix A. The left-out sensors are fixed across samples and models. Results, We report results taking PAM as example. In Table 2 (left block), we observe that RAINDROP achieves top performance in 18 out of 20 settings when the number of left-out sensors goes from 10% to 50%. With the increased amount of missing data, RAINDROP yield greater performance improvements. RAINDROP outperforms baselines by up to 24.9% in accuracy, 50.3% in precision, 29.3% in recall, and 42.8% in Fl score.
8

---

Published as a conference paper at ICLR 2022
Setup. Setting 3 is similar to Setting 2 except that left-out sensors are randomly selected in each sample instead of being fixed. In each test sample, we select subset of sensors and regard them missing by replacing all of their observations with zeros. Results. We provide results for the PAM dataset in Table (right block). We find that RAINDROP achieves better performance than baselines in 16 out of 20 settings and that Trans-mean and GRU-D) are the strongest competitors. Further, we evaluated RAINDROP in another setting where the model is trained on one group of samples (e. (.g.. females) and tested on another group not seen during training (e.g., males). Experimental setup and results are detailed in Appendix A.13.
Setting 3: Leave-raddom--ensorssses
Table 2: Classification performance on samples with a fixed set of left-out sensors (Setting 2) or random missing sensors (Setting 3) on the PAM dataset. Results for P19 dataset (Settings 2-3) are shown in Appendix A.
Missing sensur
Methods
PAM (Seting lcave fixed sensees out)
PAM (Setting rundom
Acconcy
Procision
Recall
Fl score
Accumay
Precision
Rocall
Fl score
59.8
16.2
Trans-men
60:4 11.2
6188 14,9
60.2 1: 13.8
$8.0 15.2
62.4 3.5
59.6 1 7.2
63.7 8.1
62.7
6/4
10%
GRU-D
63.4
1.7
72.6 2.6
14.3.55 5.1
63.6 0:::::44 0.4
68.4
2 3.7
74.2
70.8 4:2
72.0
3.7
SeFT
$8.9 23
62.5 + 188
59.6 2.6
99.6 2.6
40.0 + 1.9
40.8
41.0 0.7
39. + 1.5
aTAND
2.7
64.4 +
6.1
2.0
54.
2.2
RAINDROP
78.4
1.8
77.9
Transfrmmr
66
71.1 +
62.2 + 8.2
63.2 8.7
62.3 +: 11.5
65. 12.7
614 13.9
618 156
Trans-mran
61.2+30
63.5
4.1
59.4 3,4
53.2
3.5
20%
64
73.3
63.5
64.8
0.4
69.8
65.8
SeFT
0.
42.1 + 4.8
38.1 +:
35.0
2.2
34,2 4 288
34,9 + 5.2
34,6 H: 2.1
333 2.7
mTAND
13.2
26,9 1.7
37.7 1. 3.7
37.3
1 3.4
15.6
1 1.6
49.2 1 2.1
49.0 1 1.6
49.0 1,0
RAINDROP
66.5 4.00
72.0 1.001:3..
67,9 5.
65.1 7.0
71.3 2.5
75.8 1.
72.5
2.0
734 2.1
Transformer
31.6 10.0
26.4 +: 9.7
24.0 + 10.0
19.0
12.8
52.0 #: 11,9
55.2 + 15.3
50.1
# 13.3
48.4 18:2
Trans mem
42.5 1:6.1:8.6 8.6
45.3 1: 9.6
37.0 4: 7.9
33.9
8.2
65.1
1 1.9
63.8 1 L.2
67.9 1 1.8
649 1.7
30%
GRU-D
4.
51,7 6.2
42.1 1
47.2
3.9
58.0
2.0
63.2 1
S8.2
1 3.1
59.3 3.5
S:FT
34.5
1.5
32.0
ITAND
31.2
30.6
5.5
43.4
36.3
RAINDROP
$2.4
60.9 38
$1.3 7.1
48. 1.8
60.3 1.
68.1 1. 3.1
60.3 3.6
61.9 3.9
23.0
14.5
6.66
23.0
Trans-menn
25.
185
99 1.1
48.7 2-27
$5.8 2.6
$4.3
2.9
40%
GRU-D
1:.4.1:25
64.5 1: 688
42.6 1.
44.3
7.9
47.7
1.4
63.4 4:44100
$4.5
0.5
47.5 0.0
SEFT
26.3..09
29,9 1. 4,5
27.3 1 1.6
22.3 1 1.9
26.8
2.6
2..1
28.0 1.2
23.3 3.0
mITAND
19.4
15.1 1 4.4
20.2 1. 3.8
17001:3 3,4
23.7 2: 1.0
33
26.4
29.3 1 1.9
3.7
44.7
65.4
2.5
Transformer
1.
2.7 0.2
12.5 0.04
9.4 0.3
43.2 2.5
$2.0.122.55
36,9 1.9.....
41.9 1
mean
1.
28
12.5
4.6
1.4
59.1
43.1.12.2
16.51.3.
50%
GRU-D
27.
56
L.2
S:FT
1.7
159
25.3
182-224
1.4
27.5
aTAND
3.1
12.6 4:
17.0 1.6
13.9 4.0
20.9
+ 3.1
35.1 6.1
23.0 3.2
21.7 39
RAINDROP
46.6 2: 2.6
44.5 4: 2.6
42.4 3.9
38.0 38 4.0
47.2
2 4.4
59.4 3,9
44.8 2 5.3
47.6 2 5.2
4.2 ABLATION STUDY AND VISUALIZATION OF OPTIMIZED SENSOR GRAPHS
Ablation study. Considering the PAM dataset and a typical setup (Setting 1), we conduct an ablation study to evaluate how much various RAINDROP'S components contribute towards its final performance. We examine the following components: inter-sensor dependencies (further decomposed into weights including Ei,au, Tv, p and a ,,,,,, temporal attention, and sensor-level concatenation. We show in Appendix A.14 (Table 7) that all model components are necessary and that regularization C, contributes positively to RAINDROP'S performance.
Visualizing sensor dependency graphs. We investigate whether samples with the same labels get more similar sensor dependency graphs than samples with different labels. To this end, visualize inter-sensor dependencies (P19; Setting 1) and explore them. Figure 4 shows distinguishable patterns between graphs of negative and positive samples, indicating that RAINDROP can extract relationships that are specific to downstream sample labels. Further differential analysis provides insights that can inform early detection of sepsis from P19 clinical data. Details are in Appendix A.15.
CONCLUSION
We introduce RAINDROP, graph-guided network for irregularly sampled time series. RAINDROP learns distinct sensor dependency graph for every sample capturing time-varying dependencies be- tween sensors. The ability leverage graph structure gives RAINDROP unique capability to naturally handle misaligned observations, non-uniform time intervals between successive observations, and sensors with varying numbers of recorded observations. Our findings have implications for using message passing as way to leverage relational information in multivariate time series.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This material is based upon work supported by the Under Secretary of Defense for Research and Engineering under Air Force Contract No. FA8702-15-D-0001. M.Z. is supported, in part, by NSF under nos. IIS-2030459 and IIS- Harvard Data Science Initiative, Amazon Research Award, Bayer Early Excellence in Science Award, AstraZeneca Research, and Roche Alliance with Distinguished Scientists Award. Any opinions, findings, conclusions or recommendations expressed in this material are those the authors and do not necessarily reflect the views of the funders. The authors declare that there are no conflict of interests.
REPRODUCIBILITY STATEMENT
We ensure the reproducibility of our work by clearly presenting the model and providing publicly accessible code and data. For all datasets used in this work, we share downloadable links to the raw sources and processed and ready-to-run datasets with the research community through this link: https://githu..com/mit com/mims- con/mmins-aarvvad//eeiiii havvard/Raindrop. We specify all training details (e.g.. preprocessing, data splits, hyperparameters, sensor selection) in the main text and Appendix. Python implementation of RAINDROP and all baseline methods is available at the aforementioned link. Detailed description of data, scripts, and configurations along with examples of usage are also provided.
ETHICS STATEMENT
The ability of RAINDROP learn robust information about sensors' representations and dependencies creates new opportunities for applications, where time series are predominant, e.g., in healthcare, biology, and finance. In all these fields, especially in healthcare applications, be used with caution. Although our model can gain valuable insights from time series, users must consider the limitations of machine-guided predictions. As with all data-driven solutions, may make biased predictions. In the case of biomedical data, biases can exist within the data itself, which can be, for example, caused by considering demographic attributes, such as age, weight, and gender, that might correlate with protected/regulated attributes. When target classes are highly imbalanced, our model can mitigate the issues by upsampling minority classes every processed batch.
method should
model
All datasets in this paper are publicly available and are not associated with any privacy or security concern. Further, all data are anonymized to guard against breaching patients' protected health information. We followed PhysioNet privacy policy and guidelines privacy.shtml) when experimenting with P12 and P19 datasets.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Denis Agniel, Isaac S Kohane, and Griffin M Weber. Biases electronic health record data due to processes within the healthcare system: retrospective observational study, British Medical Journal, , 2018.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 8(1):1-12, 2018.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differen- tial equations. In NeurIPS, 2018.
Zekai Chen, Jiaze, Xiao Zhang, Hao Sheng, and Xiuzheng Cheng. Multi task time series forecasting with shared attention. In ICDM Workshop, pp. 917-925, 2020.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. 2014.
Edward Choi, Zhen Xu, Yujia Li, Michael Dusenberry, Gerardo Flores, Emily Xue, and Andrew Dai. Learning the graphical structure of electronic health records with graph convolutional transformer. In AAAL, volume 34, pp. 606-613, 2020.
Federico Errica, Davide Bacciu, and Alessio Micheli. Graph mixture density networks. In ICML, pp. 3025-3035, 2021.
Chenguang Fang and Chen Wang. Time series data imputation: A survey on deep learning approaches. arXiv:2011.1 1347, 2020.
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. Deep learning for time series classification: review. Data Mining and Knowledge Discovery, 3344)9917-963, 2019.
Matthias Fey, Jan-Gin Yuen, and Frank Weichert. Hierarchical inter-message passing for learning on molecular graphs. arXiv:2006. 12179, 2020.
Mikhail Galkin, Priyansh Trivedi, Gaurav Maheshwari, Ricardo Usbeck, and Jens Lehmann. Message passing for hyper-relational knowledge graphs. 2XXi:2009910084) 2020.
Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang. Hassan Sajjad, Preslav Nakov, Deming Chen, and Marianne Winslett. Compressing large-scale transformer-basee models: A case study on bert. Transactions of the Association for Computational Linguistics. 9:1061-1080, 2021.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In ICML, pp. 1272-1272. PMLR, 2017.
Ary L. Goldberger, Luis A. Nunes Amaral, L Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung-Kang Peng, and Harry Eugene Stanley. PhysioBank, PhysioToolkii, and PhysioNet: components of new research resource for complex physiologic signals. Circulation, 101 23:E215-20 20, 2000.
Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for time series. pp. 4303-430 2020.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-exciiation networks. In CVPR, pp. 7132-7141, 2018.
Wenjie Hu, Yang Yang, Ziqiang Cheng, Carl Yang, and Xiang Ren. Time-series event prediction with evolutionary state graph. In WSDM, pp. 580.588, 2021.
Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In The Web Conference, pp. 2704-2710, 2020.
Ekaterina Kalinicheva, Dino Ienco, Jérémie Sublime, and Maria Trocan. Unsupervised change detection analysis in satellite image time series using deep learning combined with graph based approaches. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 13:1450 13:1450-1466. 2020.
11

---

Published as a conference paper at ICLR 2022
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. arXiv:2005.08926 2020.
Byung-Hoon Kim, Jong Chul Ye, and Jae-Jin Kim. Learning dynamic graph representation of brain connectome with pratio-temporal attention. arXiv:2105.1343 13495, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv: 112..6980 2014.
Liying Li, Yang Liu, Tongquan Wei, and Xin Li. Exploring inter-sensor correlation for missing data estimation. In IECON, pp. 2108-2114, IEEE, 2020a.
Michelle M Li, Kexin Huang, and Marinka Zitnik. Representation learning for networks in biology and medicine: Advancements, challenges, and opportunities. arXiv:2104.04883 2021.
S. C.-X. Li and B. M. Marlin. A scalable end-to-end gaussian process adapter for irregularly sampled time series classification. In NIPS, pp. 1804-1812, 2016.
Steven Cheng-Xian Li and Benjamin Marlin. Learning from irreullary-ssumlle time series: A missing data perspective. In ICML, pp. 5937-5946. PMLR, 2020.
Xiaoxue Li. Yanmin Shang, Yanan Cao, Yangxi Li, Jianlong Tan, and Yanbing Liu. Type-aware anchor link prediction across heterogeneous networks based on graph attention network. In AAAI volume 34, pp. 147-155, 2020b.
Wu Lin, Nicolas Hubacher, and Mohammad Emtiyaz Khan. Variational message passing with structured inference networks. ICLR, 2018
R. J. Little and D. B. Rubin. Statistical Analysis with Missing Data. John Wiley & Sons, 3 edition, 2014.
Qianli Ma. Sen Li, and Garrison Cottrell. Adversarial joint-learning recurrent neural network for incomplete time series classification. TPAMI, 2020.
Karl Øyvind Mikalsen, Cristina Soguero- Ruiz, Filippo Maria Bianchi, Arthur Revhaug, and Robert Jenssen. Time series cluster kemnels to exploit informative missingness and incomplete label information. Pattern Recognition, 1 105:10789,, 2021.
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased Istm: accelerating recurrent network training for long or event-based sequences. In NIPS, pp. 3889-3897, 2016.
Giannis Nikolentzos, Antoine Tixier, and Michalis Vazirgiannis. Message passing attention networks for document understanding. In AAAL, volume 34, pp. 8544-8551, 2020.
S Ravuri, K Lenc, M Willson, D Kangin, R Lam, Mirowski, M Athanassiadou, S Kashem, S Madge, R Prudden, al. Skillful precipitation nowcasting using deep generative models of radar, arxiv. Nature, 597:672-677, 2021.
Attila Reiss and Didier Stricker. Introducing a new benchmarked dataset for activity monitoring, In ISWC, pp. 108 109, 2012.
Matthew Reyna, Christopher Josef, Russell Jeter, Supreeth Shashikumar, Brandon Westover, Shamim Nemati, Gari D Clifford, and Ashish Sharma. Early prediction of sepsis from clinical data: The physionet/computing in cardiology challenge 2019. Critical Care Medicine, 4802):210-217, 2020.
Pau Riba, Andreas Fischer, Josep Lladós, and Alicia Foréés. Learning graph distances with message passing neural networks. In ICPR, pp. 2239-2244. IEEE, 2018
J. L. Schafer and W. Graham. Missing data: Our view of the state the art. Psychological Methods, 7(2), 2002.
Omer Berat Sezer, Mehmet Ugur Gudelek, and Ahmet Murat Ozbayoglu. Financial time series fore- casting with deep learning: A systematic literature review: 2005-2019, Applied Soft Computing, 90:106181, 2020.
12

---

Published as a conference paper at ICLR 2022
Siyuan Shan and Junier B Oliva. Nrtsi: Non-recurrent time series imputation for sregularry-sampple data. arXiv:2102.03340, 2021.
Paul Shannon, Andrew Markiel, Owen Ozier, Nitin S Baliga, Jonathan T Wang, Daniel Ramage, Nada Amin, Benno Schwikowski, and Trey Ideker. Cytoscape: a software environment for integrated models of biomolecular interaction networks. Genome Research, 13(1 (3(11):2498-2504, 2003.
Satya Narayan Shukla and Benjamin Marlin. Interpolation-preciitton networks for irregularly sampled time series. In ICLR, 2018.
Satya Narayan Shukla and Benjamin Marlin. Multi-time attention networks for irregularly sampled time series. In ICLR, 2021.
Satya Narayan Shukla and Benjamin M Marlin. A survey on principles, models and methods for learning from irregularly sampled time series. 2020.
Rafael T Sousa, Lucas A Pereira, and Anderson S Soares. Improving irregularly sampled time series learning with dense descriptors of time. arXiv:2003.09291, 2020.
Qingxiong Tan, Mang Yc, Baoyao Yang. Siqi Liu, Andy Jinhua Ma, Terry Cheuk-Fune Yip. Grace Lai-Hung Wong, and PongChi Yuen. DATA-GRU: Dual-attention time-aware gated recurrent unit for irregular multivariate time series. In AAAI, volume 34, pp. 930-937, 2020.
Sindhu Tipireni and Chandan K Reddy. Self-supervised transformer for multivariate clinical time-series with missing values. arXiv:2107.12993 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, pp. 5998-6008, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018
Xiaoyang Wang, Yao Ma, Yiqi Wang, Wei Jin, Xin Wang, Jiliang Tang, Caiyan Jia, and Jian Yu. Traffic flow prediction via spatial temporal graph neural network. In The Web Conference 2020, pp. 1082-1092, 2020.
Zhen Wang. Yang Zhang. Ai Jiang, Ji Zhang, Zhao Li, Jun Gao, Ke Li, and Chenhao Lu. Dama-net: novel predictive model for irregularly asynchronously and sparsely sampled multivariate time series. In ICML/W, 2011.
B. J. Wells, K. M. Chagin, A. S. Nowacki, and M. W. Kattan. Strategies for handling missing data in electronic health record derived data. EGEMS, 1(3), 2013.
Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. Adversarial sparse transformer for time series forecasting. In NeurIPS, volume 33, 2020a.
Yinjun Wu, Jingehao Ni, Wei Cheng, Bo Zong. Dongjin Song, Zhengzhang Chen, Yanchi Liu, Xuchao Zhang, Haifeng Chen, and Susan Davidson. Dynamic gaussian mixture based deep generative model for robust forecasting sparse multivariate time series. In AAAI, 2021.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1):4-24, 2020b.
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In KDD, pp. 753-763, c020c.
Jianing Yang, Yongxin Wang, Ruitao Yi, Yuying Zhu, Azaan Rehman, Amir Zadeh, Soujanya Poria, and Louis- Phiil-p Morency. Mtag: Modal-temporal attention graph for unaligned human multimodal language sequences. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 1009-1021, 2021.
13

---

Published as a conference paper at ICLR 2022
Seongjun Yun, Minbyul Jeong, Rachyun Kim, Jaewoo Kang, and Hyunwoo Kim. Graph transformer networks. In NeurIPS, volume 32. pp. 11983-11993, 2019.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. transformer-based framework for multivariate time series representation learning. In KDD, pp. 2114-2124, 2021.
Daochen Zha, Kwei-Herng Lai, Kaixiong Zhou, and Xia Hu. Towards similarity- time-series classification. SDM, 2022.
Chuxu Zhang, Dongjin Song, Yuncong Chen, Xinyang Feng, Cristian Lumezanu, Wei Cheng, Jingchao Ni, Bo Zong, Haifeng Chen, and Nitesh V Chawla. A deep neural network for unsuper- vised anomaly detection and diagnosis in multivariate time series data. In AAAI, volume 33, pp. 1409-1416 2019.
Li Zhang, Dan Xu, Anurag Arnab, and Philip HS Torr. Dynamic graph message passing networks. In CVPR, pp. 3726-3735, 2020.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: review of methods and applications. AI Open, 1:57-81, 2020.
14

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 ENCODING TIMESTAMPS
For a given time value t, we pass it to trigonometric functions with the frequency of 10,000 (Vaswani al., 2017) and generate time representation p' R$ (omit sample index for brevity) through (Horn et al., 2020):
P4 sin( 10000247E )
P2t+1 cos( 100002
where { is the expected dimension. In this work, we set Ç = 16 in all experimental settings for all models. Please note, encode the time value which is continuous timestamp, instead of time position which is a discrete integer indicating the order of observation in time series.
A.2 ADDITIONAL INFORMATION ON THE CALCCULATION OF TEMPORAL ATTENTION WRIGHT
The Eq. 4 descrihes how we learn the temporal attention weights vector Br, for sensor v, following the self-attention formalism. Different from the standard self-attention mechanism that generates an self-attention matrix, we generate a temporal attention weight vector. The reason is that we only need an attention weight vector (instead of a matrix) to aggregate the observation embeddings into single sensor embedding through weighted sum.
In the standard self-atten elf-attennioon matrix, each element denotes the dependency of an observation em- bedding on another observation embedding. Similarly, each row describes the dependencies of an observation embedding on other observation embeddings (all the observations belong to the same sensor). Our intuition is to aggregate row in the self-attention matrix into scalar that denotes the importance of the observation embedding to the whole sensor embedding.
In practice, we apply the weighted aggregation, parameterized by s, every row in the self-attentoon matrix and concatenate the generated scalars into an attention vector. Next, we give concrete example specifically describe the meaning of Each row, j. of the self-attention matrix captures relationships of observation embedding h to all observation embeddings (h!, k 1,... T). Then, using the learnable weight vector 8, these correlations between observations are aggregated across time to obtain temporal importance weight 8% The 5% represents the importance of the corresponding observation to the whole sensor embedding.
A.3 ADDITIONAL INFORMATION ON SAMPLE EMBEDDING
As we generate sample embedding by concatenating all sensor embeddings, the sample embedding could be relatively long when there is large number of sensors. To alleviate this issue, on one hand, we can reduce the dimension of sample embeddings by adding a neural layer (such as simple fully-connected layer) after the concatenation. On the other hand, when the number of sensors is super large, our model is flexible and can effortlessly switch the concatenation to other readout functions (such as averaging aggregation): this will naturally solve the problem of long vectors. We empirically show that concatenation works better than averaging in our case. We see a boost in the AUROC score by 0.6% using concatenation instead of averaging for generating sample embeddings(P19; Setting 1).
A.4 ADDITIONAL INFORMATION ON SAMPLE SIMILARITIES
In this work, we assume all samples share some common characteristics to some extent. When modeling the similarities across samples, we do not consider the situation where the samples are similar within latent groups and different across groups
Our study focuses on the question of irregularity rather than the question of distribution shifts in time series. To this end, in our experiments, we first rigorously benchmark Raindrop using a standard evaluating setup (Setting 1, which is classification of irregular time series). This is the only setup that most existing methods consider (e.g.. Shukla & Marlin (2021); Che et al. (2018)) and we want to make sure our comparisons are fair. In order to provide more rigorous assessment of Raindrop's performance, we also consider more challenging setups in our experiments (i.e., Settings 2-4) when the dataset is evaluated in nta-standard manner and the split is informed by select data attribute.
15

---

Published as a conference paper at ICLR 2022
Table 3: Dataset statistics. The '# ---mmmtttamp'' refers to the number of all sampling timestamps measured in this dataset. The "i-classes' means the number of categories dataset labels. The 'Static info' indicates if sample's static attributes (e.g.. height and weight) are available. The 'missing ratio' denotes the ratio between the number of missing observations and the number of all possible observations if the dataset is fully-observed.
Datasets
#-samples
#-sensors
# #-timestamps
#-classes
Static info
Missing ratio (%)
P19 38,803
34
60
2
True
94.9
P12
11,988
36
215
2
True
88.4
PAM
5,333
17
600
8
False
60.0
Our results Setting are consistent with those on Settings 2-4. Results on harder Settings 2-4 show that Raindrop can perform comparably better than baselines. Results across these diverse settings increase our confidence that Raindrop is quite flexible and widely applicable.
A.5 FURTHER DETAILS ON DATASETS
P19: PhysioNet Sepsis Farly Prediction Challenge 2019. P19 dataset (Reyna al., 2020) contains 38,803 patients and each patient is monitored by 34 irregularly sampled sensors including 8 vital signs and 26 laboratory values. The original dataset has 40,336 patients, we remove the samples with too short or too long time series, remaining 38,803 patients (the longest time series of the patient has more than one and less than 60 observations). Each patient is associated with static vector indicating attributes: age, gender. time between hospital admission and ICU admission. ICU type, and ICU length of stay (days). Each patient has a binary label representing occurrence of sepsis within the next 6 hours. The dataset is highly imbalanced with only ~4% positive samples.
P12: PhysioNet Mortality Prediction Challenge 2012. P12 dataset (Goldberger et al., 2000) includes 11,988 patients (samples), after removing 12 inappropriate samples following (Horn et al., 2020). Each patient contains multivariate time series with 36 sensors (excluding weight), which are collected in the first 48-hour stay in ICU. Each sample has a static vector with 9 elements including age, gender, etc. Each patient is associated with a binary label indicating length of stay in ICU, where negative label means hospitalization is not longer than days and positive label marks hospitalization is longer than days. P12 is imbalanced with ~93% positive samples.
PAM: PAMAP2 Physical Activity Monitoring. PAM dataset (Reiss & Stricker, 2012) measures daily living activities of 9 subjects with 3 inertial measurement units. We modify to suit our scenario of irregular time series classification. We excluded the ninth subject due short length of sensor readouts. We segment the continuous signals into samples with the time window of 600 and the overlapping rate of 50%. PAM originally has 18 activities of daily life. We exclude the ones associated with less than 500 samples, remaining 8 activities. After modification, PA.M dataset contains 5,333 segments (samples) of sensory signals. Each sample is measured by 17 sensors and contains 600 continuous observations with the sampling frequency 100 Hz. To make time series irregular, we randomly remove 60% of observations. observations are randomly selected but kept the same for all experimental settings and approaches. PAM is labelled by 8 classes where each class represents an activity of daily living. PAM does not include static attributes and the samples are approximately balanced across all 8 categories.
keep fair comparison, the removed
To feed given data into neural networks, we set the input as zero if no value was measured. In highly imbalanced datasets (P19 and P12) perform batch minority class upsampling, which means that every processed batch has the same number of positive and negative class samples. The dataset statistics including sparse ratio are provided in Table 3.
A.6 FURTHER DETAILS ON MODEL HYPERPARAMETER
Baseline hyperparameters. The implementation of baselines follows the corresponding papers including SeFT (Homn et al., 2020), GRU-D (Che et al., 2018), and mTAND (Shukla & Marlin, 2021). We follow the settings of Transformer baseline in (Horn et al., 2020) while implementing Transformer in our work. For average imputation mean, we replace the missing values by the global mean value of observations in the sensor (Shukla & Marlin, 2020). We use batch size of
16

---

Published as a conference paper at ICLR 2022
128 and learning rate of 0.0001. Note that upsample the minority class in each batch to make the batch balance (64 positive samples and 64 negative samples in each batch).
The chosen hyperparameters the same across datasets (P19, P12, PAM), models (both basclines and RAINDROP), and experimental settings. Remarkably, we found that all the baselines make dummy predictions (classify all testing samples the majority label) PAM in Setting 2-3 while RAINDROP makes reasonable predictions. For the comparison to make sense (i.e., the baselines can make meaningful predictions), we use learning rate of 0.001 for baselines on PAM. GRU-D has 49 layers while other models have 2 layers. We run all models for 20 epochs, store the parameters that obtain the highest AUROC in the validation set, and use it to make predictions for testing samples. We use the Adam algorithm for gradient-based optimization (Kingma & Ba, 2014).
RAINDROP hyperparameters. Next, we report the setting of unique hyperparameters in our RAIN- DROP. In the generation of observation embedding, we set R. as 4-dimensional vector, thus the produced observation embedding has dimensions. The dimensions of time representation p' and Tv are both 16. The trainable weight matrix D has shape of 4 x 32. The dimensions of w,,, and w, are the same as the number of sensors: 34 P19, 36 in P12, and 17 in PAM. We set the number of RAINDROP layers L as 2 while the first layer prunes edges and the second layer does not. We set the proportion of edge pruning as 50% (K=50), which means we remove half the existing edges that have the lowest weights. The d2 is set to 20, while the shape of W is 20 x 20. All the activation functions, without specific clarification, are sigmoid functions. The da is set equal to the number of sensors. The first layer of y has 128 neurons while the second layer has C neurons (i.e., 2 for P19 and P12; 8 for PAM). We set X = 0.02 to adjust L, regularization scale. All the preprocessed datasets and implementation codes are made available online. Further details are available through RAINDROP's code and dataset repository.
Readout function. Here we discuss the selection readout function g in section 3.5. Our preliminary experiments show that concatenation outperforms other popular aggregation functions such as averaging (Errica et al., 2021) and squeeze-cxcitation readout function (Kim et al.. 2021; Hu et al., 2018). While any of those aggregation functions can be considered, we used concatenation throughout all experiments in this manuscript.
A.7 PERFORMANCE METRICS
Since P19 and P12 datasets are imbalanced, we use the Area Under ROC Curve (AUROC) and Area Under Precision- Recall Curve (AUPRC) to measure performance. As the PAM dataset is nearly balanced, we also report accuracy, precision, recall and F1 score. We report mean and standard deviation values over 5 independent runs. Model parameters that achieve the best AUROC value on the validation set are used for test set.
A.8 FURTHER DETAILS ON SETUP DETAILS FOR SETTING
In Setting the selected missing sensors are fixed across different models and chosen in the following way. First, we calculate the importance score for each sensor and rank them in descending order. The importance score based on information gain, which we calculate with feeding the observations into a Random Forest classifier with 20 decision trees. In particular, we treat each sample only having one sensor, then feed the single sensor into random forest classifier and record the AUROC. The higher AUROC indicates the sensor provides higher information gain. When we have sensors ranked by their AUROC values, we choose the first n sensors (the ones with highest AUROC values) and replace all observations in these sensors by zeros in all samples in validation and test set. The number of missing sensors is defined indirectly from the user with the sensors' missing ratio which ranges from 0.1 to 0.5.
A.9 ADDITIONAL INFORMATION ON MISSING PATTERN
This work propose RAINDROP which is novel solution for irregularity in multivariate time series through inter- dependencies. RAINDROP is not conflict with other solutions (such as missing pattern and temporal decay) for irregularity. However, the missing pattern is widely discussed in modelling incomplete time series (Che et al., 2018), we explore how to combine the advantages of relational structures and missing pattern. We adopt mask matrix proxy of missing pattern
17

---

Published as a conference paper at ICLR 2022
as in Che et al. (2018). Taking the architecture of RAINDROP, concatenate the observation of,u with binary mask indicator b4 input. The indicator V1,, is set 1 when there is observation of sensor i at time and set as O otherwise, All the experimental settings and hyperparameters are the same as in RAINDROP (P19; Setting 1). The experimental results show that taking advantage of missing pattern can slightly boost the AUROC by 1.2% and AUPRC by 0.9% in P19. This empirically shed the light for future research on integrating multiple characteristics in representation of irregularly time series.
A. COMPARISON BETWEEN TEMPORAL ATTENTION AND LSTM
We conduct extensive experiments to compare the effectiveness temporal attention and LSTM. To this end, replace the temporal attention in sensor embedding generation (Eq 4-5) in RAINDROP by LSTM layer which processes all observation embeddings sequentially. We use zero padding to convert the irregular observations into fixed-length time series the data be fed into LSTM architecture. We regard the last output of LSTT as generated sensor embedding. The number of LSTTM cells equal the dimension of observation embedding. All the model structures identical except in the part of temporal attention and LSTM. We keep all experimental settings (P19; Setting 1) and hyperparameter selections the same. The experimental results show that the temporal self-attention outperform LSTM by 1.8% (AUROC) and additionally saved 49% of the training time. One potential reason that the self-attention mechanism avoids recursion and allows parallel computation and also reduces performance degradation caused by long-term dependencies (Ganesh et al., 2021; Vaswani et al., 2017).
A.11 ADDITIONAL INFORMATION ON METHOD BENCIMARKING
Taking experimental Setting (i.e., classic time series classification) extensive experiments to compare Raindrop with ODE-RNN (Chen et al., 2020), DGM2-O (Wu et al., 2021), EvoNet (Hu al., 2021), and MTGNN (Wu et al., 2020c). As IP-Net (Shukla & Marlin, 2018) and mTAND (Shukla & Marlin, 2021) are from the same authors, we only compare with mTTAND which is the latest model. For the baselines, For methods, which cannot deal with irregular data (e.g., EvoNet and MTGNN), we first impute the missing data using mean imputation and then feed data into the model. For forecasting models (e.g.. MTGNN) which are strictly not comparable with the proposed classification model, we formulate the task as a single-step forecasting, concatenate the learned representations from all sensors and feed into fully-connected layer (work as classifier) to make prediction, and use cross-entropy to quantify the loss.
example, we conduct
follow the settings provided in their public codes.
A. 12 RESULTS FOR P19 (SETTINGS 2-3)
Here report the experimental results for P19 in Setting (Table 4) and Setting 3 (Table 5).
Table 4: Classification on samples with fixed missing sensors (P19; Setting 2)
Missing
Models
10%
30%
40%
AURUC
AUPRC
AUROC
ALPRC
AURUC
AUPRE
AUROC:
AUPRC
ALRUC
AUPRC
AUROC
AUPRC
Transfomeer
$3.2 = 1.3
47.6 3.8
77.4+ 3.5
18.3
1 4.2
35.7
3.4
35.2 5.4
25.1
35.5 +
15.3
36.2 4.2
74.9 1.1
55.5 5550
Trans mezn
$2.1
1.7
47.4
1.4
79.
2.7
40.6
3.7
19.4 25
35.5
2.8
76.9
2.4
37.5
2.59
36.4 2.0
36.5 1.8
74.1
2.3
41.3 4.4.
GRU
838
1.7
46.9
2.1
79.
2.2
37.
25
775
=
35.5
4.6
76.6 2: 1.9
35.1
2.4
14.6
2.7
35.91 2.7
14.1
1.99
55. 1:38
Seift
78.1
31.1
1.8
71.3
25.5
235
655
20
14.0
2 2.1
12.9
12
573
9.
56.0 8:33 31
78 1.3
iITAND
1.8
778
25.5
27.5
1.1
RAINDROP
51.8
5.5
43.
7.
$18
43.8 5.6
Table 5: Classification on samples with random missing sensors (P19; Setting 3)
Missing
Models
20%
40%
AUROC
AUPRC
AUROC
ALPRC
AUROC
AUPRC
AUROC
AUPRC
ALROC
AUPRC
AUROCC
AUPRC
Transfomer
K3.2 +
47.6 1.6
83.2
46.3
3.5
31.6
42.5
8 3.1
42.1 +
50.
41.9 6.6
70.2
1.9
43.3
Trans-mcan
$4.1
1.7
47.4
1.4
53.5
3.7
44.3
6.8
$1.3
-20
45.0
3,6
2 3.2
43.3 6.3
50.
117
41.5 4.6
708
1.1
30.3 5.1
GRU
$3.9
1.7
46.92.11
81.2
344
46.4
2.
78.6 4.1
43.5
2.4
76.3 25
25.5 2.1
742
2:
2.1
14.6 1.35
26.5
4.2
Selft
7887
31.1
2.8
78.8
28.3
25
770 -0
24.1
2.4
75.2
22.5
1:52.00
736
2:
1
72.6
15.7 19
mTAND
50.4
52.4
1.8
28.
=44
26.
22
23
141 2.
25.1 16
286 3:1
RAINDRUP
318
85.5
83.5
47.4
1.5
$2.6
2..9
46.2
18

---

Published as a conference paper at ICLR 2022
Table 6: Comparison of results when excluding dependency graph in RAINDROP (P19; Setting 4). The results are the same as Table except the row 'RAINDROP w/o graph', where we do not consider inter-sensor dependencies and set all sensors as independent in the dependency graph.
Model
Generalizing to new patient group
Trai:: Young + Test: Old
Trair: Old - Test Young
Trin: Male + Test: Female
Train: Female
+ Test: Male
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
Transformer
76.2 + 0.7
30.5 1: 4,8
76.5 à:
33.7 5:7
77.8
1.1
26.0 4: 6.1
75.2 1,0
30,3 5.5
Trans-mean
80. 1.4
39.8
2:9
802
32.5
GRU-D
76.5 - 1.7
29.5 2.3
79.6 = 1.7
78.5 + 1.6
31.9 = 48
76.3
31.1 = 26
S:FT
+.7
26.6 4:
78.9
32.7 2:2.
78.6 0.6
31.1
76.9 0.5
26.4 L.
mTANN
79.0 0.8
28.8 2.3
794 0.6
1.2
78.0 - 0.9
26.5
78.9 1.2
292 200
RAINDROP weo graph
80.5 - 1.1
31.6 +:
78.5 + 0.9
36,7 2:7
81.3 4: 1.5
36.8 # L.7
77.5 4: 1.9
33.4 = 26
RAINDROP
83.2
1.6
43.6 +:
$2.0
44.3
85.0 1.4
45.2
81.2
38
40.7 29
Table 7: Results of ablation study on the PAM dataset (Setting 1).
RAINDROP Model
Accuracy
Precision
Recall
Fl score
W/o weights vector R.
81.1 + 2.6
81.9 + 2.4
80.1
1.6
81.6 + 2.1
W/o éi,wv
82.6 + 1.2
82.9 1.6
84.3 1,4
83.8 + 1.7
W/o ro
86.5 + 2.4
83.3 + 1.9
82.6 1.5
82.9 + 1.4
W/o inter-sensor dependency
W/o p
79.8 + 2.7
80.1 + 3.6
80.6 + 1.7
80.2 + 2.9
W/o ai
85.2 = 2.5
86.4 + 2.7
84.5 = 2.9
85.6 = 2.9
W/o temporal attention
81.5 = 1.9
84.6 1.7
83.9 2.5
84.2 - 2.2
W/o sensor level concatenation
84.4 + 2.1
86.7+ 1.1
85.2+ 1.9
85.8 + 2.6
W/o regularization term C,
87.3 = 2.9
88.6 3.4
87.1 2.8
87.6 + 3.1
Full RAINDROP
1.5
1..
89.9
89.8 1.0
A.13 EVALUATION ON GROUP-WISE TIME SERIES CLASSIFICATION
To understand whether RAINDROP can adaptively adjust its structure and generalize well to other groups of samples which were not observed while training the model. In this setting we split the data into two groups, based on a specific static attribute. The first split attribute is age, where we classify people into young (< 65 years) and old (z 65 years) groups We also split patients into male and female by gender attribute. Given the split attribute, we use group train set and randomly split the other group into equally sized validation and test set.
Taking P19 as example, we present the classification results when the training and testing samples are from different groups As shown in Table 8, RAINDROP achieves the best results over all of the four given cross-group scenarios. For instance, RAINDROP claims large margins (with 4.8% in AUROC and 13.1% % in AUPRC absolute improvement) over the second best model while training on males and testing on female patients.
Although RAINDROP is not designed to address domain adaptation explicitly, the results show that RAINDROP performs better than baselines when transferring from one group of samples another. One reason for our good performance is that the learned inter-sensor weights and dependency graphs are sample-specific and their learning is based on the sample's observations. Thus, the proposed RAINDROP has the power, to some extent, to adaptively learn the inter-sensor dependencies based on the test sample's measurements. RAINDROP is not generalizing to new groups, but generalizing to new samples, which leads to good performance even though our model is not designed for domain adaptation. We validate the reason empirically. We remove the inter-sensor dependencies (set all sensors isolated in the dependency graph; set all af and ef as 0) in RAINDROP and evaluate the model in group-wise time series classification. The experimental results show that the performance drops a lot when excluding dependency graphs and message passing in RAINDROP (Table 6). Without inter-sensor dependencies our model is on par with other basclines and does not outperform them by large margin.
19

---

Published as a conference paper at ICLR 2022
29
30
16
33
13
(a) Negative samples
(b) Positive samples
Figure Learned structure negative and positive samples (P19; Setting 1). The nodes numbered from to 33 denote 34 sensors used in P19 (sensor names are listed in Appendix A.15). To make the visualized structures easier to understand, we use darker green to denote higher weight value and yellow to denote lower weight value. We can observe distinguishable patterss able to adaptively learn graph structures that sensitive the classification task. For example, find that the nodes (pulse oximetry), (diastolic BP), and 12 (partial pressure carbon dioxide from arterial blood) have lower weights in negative samples.
two learned sensor dependency graphs, indicating RAINDROP
Table 8: Classification results when train and test samples originate from different groups (P19).
Generalizing to new patient group
Model
Train: Young + Test: Old
Train: Old - Test: Young
Train: Male + Test: Female
Train: Female + Test Male
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
AUROC
AUPRC
Transformer
76.2 0.7
30.5 +: 4.8
76.5 + 1.1
33.7 4: 5.7
77.8 : 1.1
26.0 6,2
75.2 4 1.0
30.3 5.5
Trans-mean
80.6
39.8 4.2
78.4 +
35.8 2.
80.2 = 1.7
32.1
76.4 + 088
32,5
GRU-D
76.5 1.7
29.5 + 2.3
79.6 + 1.7
4.66
78.5 : 1.6
31,9 11.9::4
76.3 + 2.5
31.1 1.1:::0 2.6
ScFT
77.5
26.6 - 1.2
78.9 1.0
227
78.6 0.6
31.1
76.9 0.5
26.4
ITAND
79.0 0.8
28.8 - 2.3
79.4 +: 0.6
29.8 19.8111
78.0 - 0.9
26.5 - 1.7
78.9 + 1.2
29.2
RAINDROP
44.3
38
40.7
A. 14
FURTHER DETAILS ON ABLATION STUDY
We provide ablation study, taking PAM Setting 1 as example, in Table 7. In the setup of WWo sensor level concatenation', we take the average of all sensor embeddings (in stead of concatenating them together) to obtain sample embedding. Experimental results show that the full RAINDROP model achieves the best performance, indicating every component or designed structure is useful the model. For example, we find that excluding inter-sensor attention weights a will cause decrease of 3.9% in accuracy while excluding edge weights ei (i.e., dependency graphs) will drop accuracy by 7.1%
A.15 VISUALIZATION OF SENEOR DEPENDENCY GRAPHS LEARNED RAINDROP
We visualize the learned inter-sensor dependencies (i.e., Gi before the averaging operation in Eq. 3) on P19 in early sepsis prediction. The visualizations are implemented with Cytoscape (Shannon et al., 2003). The data shown are for testing set of P19 including 3,881 samples (3708 negative and 173 positive). As RAINDROP learns the specific graph for each sample, we take average of all positive samples and visualize it in Figure 4b; and visualize the average of all negative samples in Figure 4b. As we take average, the edges with weights smaller than 0.1 (means they rarely appear in graphs) are ignored. The averaged edge weights range from 0.1 to 1. We initialize all sample graphs as complete graph that has 1,156 = 34 x 34 edges, then prune out 50% of them in training phase, remaining 578 edges. The 34 nodes in figures denote 34 sensors measured in P19, as listed
20

---

Published as a conference paper at ICLR 2022
25
23
22
30
17
33
16
Figure Differential structure dependency graphs between positive and negative samples. The edges are directed. We select the top 50 edges with largest difference (in absolute value) between two patterns. The edges are colored by the divergences. The darker color denotes the connection is more crucial to classification task. Node 0 is not included in this figure as it is not connected with any sensor. We can infer that the heart rate is stable whether the patient will get sepsis node (Oxygen saturation from arterial blood) and the connection from node (Respiration rate) to node 25 (Potassium) are informative for distinguishing sample classes.
Moreover, can the edge from node (systolic BP)
2etee-2019991...00 9/1.0.0/. We list the sensor names here: 0: HR; 1: O2Sat;
2: Temp; 3: SBP; 4: MAP: 5: DBP; 6: Resp; 7: ErCO2: 8: BascExcess; 9: HC03; 10: FiO2; 11:
pH; 12: PaCO2: 13: SaO2; 14: AST; 15: BUN; 16: Alkalinephos; 17: Calcium; 18: Chloride; 19:
Creatinine; 20: Bilirubin direct; 21: Glucose; 22: Lactate; 23: Magnesium: 24: Phosphate; 25:
Potassium; 26: Bilirubin_ |otal: 27: Troponinl; 28: Hct; 29: Hgb; 30: PTT; 31: WBC; 32:
Fibrinogen;
33: Platelets.
We also visualize the differential inter-sensor connections between the learned dependency graphs from patients who are likely to have sepsis and the graphs from patients who are unlikely to suffer from sepsis. Based on the aggregated graph structures of positive and negative samples, we calculate the divergence between two groups of patients and report the results Figure 5. In detail, we sort edges by the absolute difference of edge weights across negative and positive samples. On top of the visualization of the 50 most distinctive edges, we can have a series of concrete insights. For example, the dependency between node 6 (Respiration rate) to node 25 (Potassium) is important to the early prediction of sepsis. Note these data-driven observations could be biased and still need confirmation and future analysis from healthcare professionals. The edges in both Figure 4 and Figure 5 are directed. The edge arrows might be difficult to recognize due to the small figure size. We will provide high-resolution figures to our public repository.
Furthermore, we statistically measure the similarities across samples within the same class and dissimilarities across samples from different classes. Specifically, for every sample, we calculate: 1) the average Euclidean distance between its dependency graph and the dependency graphs of all samples from the same class; the average distance with all samples from the different classes. The P19 dataset has 38,803 samples including 1,623 positive samples and 37,180 negative samples. For a fair comparison, we randomly select 1,623 samples from the negative cohort, then mixed them with equal number of positive samples to measure the averaged Euclidean distances intra- and inter-classes. We select the cohort for 5 independent times with replacement. We find that the distance ((8.6 + 1.7) 10 among dependency graphs of positive samples is smaller than the distance (12..9 + 3.1) x 10-") across samples. The results show that the learned dependency graphs are similar within the same class and dissimilar across classes, which demonstrates RAINDROP can learn label-sensitive dependency graphs.
21


Reviews:
[{"review_id": "wtKl_zJDAI", "reviewer": "Reviewer_Vuyn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is hard to understand the operation for the equation, and the figures in the paper are confusing, and the author does not provide any code to help us to check it.\n\nFor example, \n1) The dot product in Figure 3 a is ambiguous.\nBased on Appendix A.4, it seems the attention α and edge e are all real numbers.\nThus, in Figure 3 a, the dot product for α and e are multiple of the two real numbers without extra weight. However, the dot product of r||p and h is a dot product with matrix D between them. What’s more, the dot product between u and v is a dot product with two vectors between them.\n\n\nWhat’s more, based on equation 3, e is calculated by α. Therefore, it seems there is no need to link u and e in Figure 3 a.\n\n2) The form of β is confusing.\nIn Figure 3 b, the attention weight β seems to be a vector, but it is annotated as a real number instead of a vector symbol.  What’s more, the size of s in Equation 4 is not given; thus, it is hard to say the β is a vector or a real number.\n\n3) The packed observation embedding in Figure 3 b is not mentioned in the paper.\n\n I tried to search the word “packed”. It is only shown in Figure 3 without explanation in any other places.\n", "summary_of_the_paper": "The core idea of the paper is transforming multivariate time series data into a vector z and using z for classification.\nThe transformation has two steps. Firstly, each time record of each sensor is mapped into a  space of 20 dimensions. The first 4 dimensions are used for feature extraction, and the last 16 dimensions are used for positional encoding. Secondly, self-attention is used to transform the embedding into a vector z and use it for classification.\n", "summary_of_the_review": "The presentation of the paper has big space to improve. The source codes are unavailable, and the main results cannot be confidently reproduced.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "pB5GgkKjnv5", "reviewer": "Reviewer_uubC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the authors developed a GNN based methods for modeling irregular multivariate time series (MTS) and their classification. The proposed method draws an analogy between the ripple effects of raindrops and the propagation of intermittent observations of irregular MTS, and learns individual sensor embeddings through a neural message passing paradigm on the dependency graph between sensors. The goal is to learn MTS embeddings, which build upon sensor embeddings, which in turn build upon observation embeddings using temporal attention. During training, the embeddings and the dependency graphs were learned jointly, through the GNN related parameters. The dependency graph was inferred through observation-level attention across sensors. Thus, the whole model embeds the MTS in a hierarchical manner. In the experiments, the proposed method was compared with five related methods on irregular MTS, and was evaluated in several settings of missing patterns. The results indicate the effectiveness of the method. The paper provides a good perspective of applying GNN methods on MTS related tasks.\n\nThe concerns of the paper are summarized as below.\n1. The introduction section claims \"none of the existing methods on irregular MTS considered relational structures in representation learning ... irregular MTS\" (the end of the second paragraph), which may not be true. The following several papers have taken into account of learning correlation parameters between sensors/variables in learning embeddings of irregular MTS.\n\t* Shukla and Marlin. \"Interpolation-Prediction Networks for Irregularly Sampled Time Series.\" In ICLR. 2018. (cited in the paper)\n\t* Wu, et al. \"Dynamic Gaussian Mixture based Deep Generative Model For Robust Forecasting on Sparse Multivariate Time Series.\" In AAAI. 2021.\n\n\tAlthough these papers didn't use GNN, but it is not rigorous to say relational structures have not been considered in the existing methods.\n\n2. From the paper, it is unclear about why assuming inter-sensor dependency is specifically beneficial for dealing the irregularity of MTS, from an intuitive perspective. In other words, it is obscure on how important message passing is for dealing with the irregularity. Considering the dependency is useful for modeling the impacts between sensors, but what's more important to model irregular MTS, such as missing patterns and temporal impact decays (as discussed in many works in this area), are not well elaborated w.r.t. the proposed method.\n\n3. In terms of the technique, it is unclear how could embedding and dependency graph be effectively learned in a joint manner. On one hand, the learning of embedding in Eq (2) depends on meaningful dependency graphs and pairwise attention weights. On the other hand, the learning of dependency graphs in Eq (3) depends on meaningful embeddings and attention weights. This forms a dilemma as neither of the embeddings nor the dependency graphs are available at the beginning. Moreover, the key component, inter-sensor attention in Eq (1) again depends on the embedding of the active sensor, but has little to do with the non-active sensors except for some variable parameters. From the current description, it is hard to understand how could the model training effectively learn both meaningful embeddings and dependency graphs.\n\n4. The proposed method is sort of overcomplicated by introducing many learnable parameters, some are not well justified. For example, in Eq. (4), adding parameter s is for making the output a length-T vector, but what is the meaning of s, and its multiplication with the T-by-T self-attention matrix is unclear.\n\n5. Since Eq (5) doesn't consider temporal order of MTS by using temporal attention, it is better to discuss if there is any impacts by neglecting the temporal structure during the aggregation.\n\n6. The sample embedding is generated by concatenating all sensor embeddings. Since this may generate a long vector, will it limit the application of the proposed method when the number of sensors is large?\n\n7. In the proposed method, many model parameters are shared across all samples for leveraging sample similarities, and there are few parameters for distinguishing different samples. Also, the pair-wise regularization in the loss function encourages similar dependency graphs among all samples. These designs may not be reasonable since not all samples are similar. It may be possible that samples form clusters or groups, some are similar while others are not. Encouraging similarity among all samples may be an unreasonable simplification.\n\n8. In the experiment setting, it is better to summarize the missing data ratio in different datasets to highlight the irregularity in the these datasets. For the compared methods, many important ones were omitted, such as ODE based methods, the above two mentioned papers, GAN-based generative imputation methods (listed below), a memory-based method (listed below), and some GNN-based MTS method (listed below).\n\t* Luo, et al. \"Multivariate time series imputation with generative adversarial networks.\" In NeurIPS. 2018.\n\t* Tang, et al. \"Joint modeling of local and global temporal dynamics for multivariate time series forecasting with missing values.\" In AAAI. 2020.\n\t* Hu, et al. \"Time-Series Event Prediction with Evolutionary State Graph.\" In WSDM. 2021.\n\t* Wu, et al. \"Connecting the dots: Multivariate time series forecasting with graph neural networks.\" In KDD. 2020.\n\n\tAs for the GNN-based MTS methods, although they don't directly model irregular MTS, it is feasible to first do statistics based imputation (mean imputation) and then apply them. This comparison helps understand how could the proposed method better address irregularity. If some of those methods have been compared on the same datasets in the same settings in other papers, it is better to report the numbers from there for a comprehensive understanding.\n\n9. In the results of \"Setting 4\", it is unclear why the proposed method performs better than other methods when transferring from one group of samples to another, and what technical design helps the proposed method be effective in domain adaptation, which seems to be another topic different from modeling irregular MTS.\n\n10. In the experiments, the visualization of the dependency graph cannot support the conclusion that the model learns dependencies for distinguishing classes. To this end, the visualization should demonstrate similarity of dependency graphs within the same class, and the dissimilarity between dependency graphs from different classes. Only observing dissimilarity across different classes provides few information. It is possible that the dependency graphs within the same class are different from each other, but the paper doesn't visualize nor discuss it. Also, considering samples with the same label as similar samples may not be reasonable in medical data because patients from the same class may have different data distribution due to different demographics, regions and population.\n\nIt is good to see the consideration of using visualization, as it facilitates the understanding of what have been really learned by the proposed model, especially considering the complexity of the model design. However, the current results make it difficult to judge whether the performance gains in Table 1, 2, 3 are from the correct learning of the attention and dependency graphs or other factors. Thus it is better to see a thorough visualization and discussion on inter-sensor attention, temporal attention, and the dependency graphs.", "summary_of_the_paper": "This paper introduces a GNN based method for classifying irregular multivariate time series (MTS). The proposed method explicitly models the dependency between sensors, and propagates impacts of intermittent time series observations from sensors to related sensors for learning MTS embeddings. It jointly learns embeddings and dependency graphs through several attentive mechanisms. The paper provides a good perspective of applying GNN methods on MTS related tasks.", "summary_of_the_review": "The paper is meaningful in exploring the new application of GNNs in irregular MTS related tasks. However, the paper is concerned w.r.t. its unclear description in some important concepts, the technical limitations, the lack of discussion and comparison of some related methods, and the non-comprehensive experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "U8myMVwI45E", "reviewer": "Reviewer_cdpu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the model design is reasonable and the results are impressive. It is interesting to see that the dependency between different sensors really helps the predictions in Setting 2,3,4, which will be very useful for many applications. Only thing is that the learned graph (Figure 4) are very dense and it will be interesting to see what happens by making it sparse.", "summary_of_the_paper": "This manuscript describes a method that models multiple time series data (sensors) across different individuals (samples). A graph is used to model the dependency between the sensors, such that the data of one sensor could be used to infer another, which is the key point of this ms. How one sensor affects another is modelled as a message passing problem on the graph. At each time point, an observation embedding is generated, either from a sensor or its neighbours in the graph. Observation embeddings of a sensor is then aggregated into a fixed-length sensor embedding, using temporal attention. These sensor embeddings are concatenated into a sample embedding for an individual for downstream analysis such as classification.", "summary_of_the_review": "I recommend this ms to be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce a GNN based method for classifying irregular multivariate timeseries.
They represent the dependencies among sensors using a graph structure and deploy message passing to  
model the effect of a sensor on another)s=. The approach jointly learns embeddings and the dependency graph. 

The manuscript gathered a clear accept (8) and two marginal below the threshold scores (5). 
I want to accept this work and I explain why. 

The reviews and the ongoing discussion during the rebuttal showed that the work is interesting 
with its main strength being the novel exploration of GNNs application on irregularly samples multivariate 
time series.

There were many concerns raised by a reviewer regarding important  theoretical and methodological issues in the paper. 
During the rebuttal phase, the authors clarified and resolved the majority of the concerns and there was an ongoing discussion among the two sides, authors and reviewer (which I have to admit was a pleasure to watch researchers communicating). 
The authors took into account the feedback and revised the manuscript accordingly. Having read the edits myself, I believe the submission is substantially improved and addressed the concerns sufficiently. 

I expect that this work will stimulate further research in the community and I would like to accept this.
Decision: Accept (Poster)