Set Functions for Time Series
Authors: Max Horn
Michael Moor
Christian Bock
Bastian Rieck
Karsten Borgwardt
Venue:  (2020)
Keywords: Time Series
Set functions
Irregularly sampling
Medical Time series
Dynamical Systems
Time series classification

Abstract: Despite the eminent successes of deep neural networks, many architectures are often hard to transfer to irregularly-sampled and asynchronous time series that occur in many real-world datasets, such as healthcare applications. This paper proposes a novel framework for classifying irregularly sampled time series with unaligned measurements, focusing on high scalability and data efficiency.
Our method SeFT (Set Functions for Time Series) is based on recent advances in differentiable set function learning, extremely parallelizable, and scales well to very large datasets and online monitoring scenarios.
We extensively compare our method to competitors on multiple healthcare time series datasets and show that it performs competitively whilst significantly reducing runtime.

Under review as a conference paper at ICLR 2020
SET FUNCTIONS FOR TIME SERIES
Anonymous authors
Paper under double-blind review
ABSTRACT
Despite the eminent successes of deep neural networks, many architectures are often hard to transfer to irregularl-ssappled and asynchronous time series that occur many real-world datasets, such as healthcare applications. This paper proposes novel framework for classifying irregularly sampled time series with unaligned measurements, focusing on high scalability and data efficiency. Our method SEFT (Set Functions for Time Series) is based recent advances in differentiable set function learning, extremely parallelizable, and scales well to very large datasets and online monitoring scenarios. We extensively compare our method to competitors on multiple healthcare time series datasets and show that it performs competitively whilst significantly reducing runtime.
INTRODUCTION
With the increasing digitalization. measurements over extensive time periods are becoming ubiq- uitous. Nevertheless, in many application domains, in particular healthcare (Yadav et al. 2018). measurements might not necessarily be observed at regular rate or could be misaligned. Moreover, the presence or absence of measurement and its observation frequency may carry information of its own (Little & Rubin 2014), such that imputing the missing values is not always desired.
While some algorithms can be readily applied to datasets with varying length, these methods usu- ally assume regular sampling of the data and/or require the measurements across modalities to be aligned/synchronizze. preventing their application to the aforementioned settings. Existing ap- proaches for unaligned measurements, by contrast, typically rely on imputation to obtain regularly- sampled version of dataset for classification. Learning a suitable imputation scheme, however, requires understanding the underlying dynamics of system; this task is significantly more compli- cated and not necessarily required when classification the main goal. Furthermore, even though a decoupled imputation scheme followed by classification is generally more scalable, it may lose in- formation (in terms of "missingness patterns") that could be crucial for prediction tasks. In addition, the fact that decoupled schemes perform worse than methods that are trained end-to-end has been has been empirically demonstrated by Li & Marlin (2016). Approaches that jointly optimize both tasks also add large computational overhead, thus suffering from poor scalability or high memory requirements.
Our method is motivated by the understanding that, while RNNs and similar architectures are well suited for capturing and modelling the dynamics of time series and thus excel at tasks such as forecasting, retaining the order of input sequence can even be disadvantage in classification scenarios (Vinyals et al., 2015). We show that by relaxing the condition that sequence must be processed order, we can naturally derive an architecture that directly accounts for (i) irregular sampling, and (ii) unsynchronized measurements. Our method SEFT: Set Functions for Time Se- ries, extends recent advances in set function learning to irregular sampled time series classification tasks, yields sate-of-the-an art performance, is highly scalable and improves over current approaches by almost an order of magnitude in terms of runtime.
With SEFT, we propose to rephrase the problem of classifying time series as classifying set of observations. We show how set functions can be exploited to learn classifiers that are naturally applicable to unaligned and irregularly sampled time series, leading to state-of-the-ant performance in irregularly-same sampled time series classification tasks. Our approach can be interpreted as learning dataset-specific summary statistics of time series which are optimized separate instances by class.

---

Under review as a conference paper at ICLR 2020
Furthermore, our method is highly parallelizable and can be readily extended to online monitoring setup with up to thousands of patients.
2 RELATED WORK
This paper focuses on classifying time series with irregular sampling and potentially unaligned mea- surements. We briefly discuss recent work in this field; all approaches can be broadly grouped into the following three categories.
Irregular sampling as missing data While the problem of supervised classification in the pres- ence of missing data is closely related to irregular sampling on time series, there are some core differences. Missing data is usually defined with respect to number of features that could be observed. whereas time series themselves can have different lengths and a "typical" number of ob- served values might not exist. Generally, an arregularly-sappe missing data problem by discretizing the time axis into non-overlapping intervals, and declaring in- tervals in which no data was sampled as missing. This approach followed by |Marlin et al. (2012), where a Gaussian Mixture Model was used to do semi-supervisee clustering on electronic health records. Similarly, Lipton et al. (2016) discretize the time series into intervals, aggregate multiple measurements within an interval, and add missingness indicators to the input of Recurrent Neural Network. By contrast, Che et al. (2018) present several variants of the Gated Recurrent Unit (GRU) combined with imputation schemes. Most prominently, the GRU-model was extended include decay term (GRU-D), such that the last observed value is decayed to the empirical mean of the time series via learnable decay term. While these approaches applicable irreullrly-samplle data, they either rely on imputation schemes or empirical global estimates on the data distribution (our method, by contrast, requires neither), without directly exploiting the global structure of the time series.
time series can be converted into a
Frameworks supporting irregular sampling Some frameworks support missing data. For ex- ample, Lu et al. (2008) directly defined kernel irreuularl--ammple time series, permitting sub- sequent classification and regression with dernel-based classifiers or regression schemes. Further- more, Gaussian Processes (Williams & Rasmussen 2006) constitute a common probabilistic model for time series; they directly permit modelling of continuous time data using mean and covariance functions. Along these lines. Li & Marlin (2015) derived kernel on Gaussian Process Posteriors, allowing the comparison and classification of irreullrly-sampled time series using kernel-based classifiers. Nevertheless, all of these approaches still rely on separate tuning/training of the impu- tation method and the classifier so that structures supporting the classification could be potentially missed in the imputation step. An emerging line of research employs Hawkes processes (Hawkes) 1971 Liniger 2009). i.e. specific class of self-exciting point processes, for time series modelling and forecasting (Mei & Eisner 2017 Yang et al. 2017 Xiao al. 2017). While Hawkes processes exhibit extraordinary performance in these domains, there is no standardised way of using them for classification. Previous work (Lukasik et al. 2016) trains multiple Hawkes processes (one for each label) and classifies time series by assigning it the label that maximises the respective likelihood function. Since this approach does not scale to our datasets, we were unable to perform a fair com- parison. We conjecture that further research will be required to make Hawkes processes applicable to general time series classification scenarios.
End-to-end learning of imputation schemes Methods of this type are composed of two modules with separate responsibilities, namely an imputation scheme and a classifier, where both components are trained discriminatively and end-to-end using gradient-based training. Recently, Li & Marlin (2016) proposed the Gaussian Process Adapters (GP Adapters) framework, where the parameters of a Gaussian Process Kernel are trained alongside a classifier. The Gaussian Process gives rise to a fixed-size representation of the wvegularly-samplee differentiable classification architecture. This approach was further extended to multivariate time series by /utoma et al. (017)) using Multi-task Gaussian Processes (MGPs) (Bonilla et al. 2008). ) which allow correlations between the imputed channels. Moreover, Futoma et al. (2017) made the approach more compatible with time series of different lengths by applying Long Short Term Memory (LSTM) (Hochreiter & Schmidhuber) 1997) classifier. Motivated by the limited scalability of approaches based on GP Adapters, (Shukla & Marlin (2019) suggest an alternative imputation
time series, making it possible to apply any
2

---

Under review as a conference paper at ICLR 2020
1703
(fay zas my)
(Max Ras my s
1722
(fas m2).
(Ray 24, m2) m2).... .... (1ay ma e (f11, 221; ma)
81
E10
Multivariste Series
Encoding
Embedding. Aggregation, and Attention
Classiication
Figure Schematic overview of SEFT's architecture. The first panel exemplifies a potential put, namely a multivariate time series, consisting of 3 modalities miy m22 11.. We treat the yth observation as a tuple (ty, 23, mj), comprising a time tj. a value 2j. and a modality indicator my- summarized
All observations modality is then separately embedded (f') and subsequently aggregated (agg). An attention mech- anism (attn) as described in Section B.3 is then applied to learn the importance of individual and consecutive observations. Respective query matrices for attentions head are illustrated in purple and orange blocks. The results of each attention head then concatenated and used the input for final classification layers.
set of such tuples. Each set of tuples belonging to the same
scheme, the interpolation prediction networks. applies multiple semi-parametric interpolation schemes obtain regularly-sample time series representation. The parameters of the interpola- tion network are trained with the classifier in an end-to-end setup.
3 PROPOSED METHOD
Our paper focuses the problem of time series classification of irregularly sampled and unaligned time series. We first define the required terms before describing our models
3.1 NOTATION & REQUIREMENTS
Definition 1 (Time series). We describe time series of instance as set S; of M == (S,)
observations 8j such that S; == {81,..., }- We assume each observation 8j 1o be represented tuple (tyr Zj, my), consisting of time tj E R++ , observed value 2j E R, and modality indicator my E {1...D) D), where D represents the dimensionality of the time series. We write Ω C. R+ R IN+ denote the domain of observations. An entire time series can thus be represented as
S : ((11, m1, (tM, RM, (tM))M,mM))
(1)
where for notational convenience we omitted the index
We leave this definition very general on purpose, allowing the length of each time series (comprising all channels, such as "heart rate". "respiratory rate", etc. of one instance) to differ, since our models are capable of handling this. Likewise, we neither enforee nor expect all time series be syn- chronized, i.e. being sampled at the same time, but rather we permit unaligned or non-synchronized observations in the sense of not having to observe all modalities at each time point. Time series are collected in a dataset 'D
Definition 2 (Dataset). We consider dataset D to contain time series. Elements of D are tuples, i.e. = {(S), vi)...., (SN, NN)], where S, denotes the 7th time series and y {l...., C} its associated class label.
Figure gives high-level overview of our method, including the individual steps required to per- form classification. To get more intuitive grasp of these definitions, we briefly illustrate our time series notation with example. Let instance i be in-hospital patient, while the time series represent measurements of two channels of vital parameters during a hospital stay, namely heart rate (HR) and mean arterial blood pressure (MAP). We enumerate those channels as modalities
3

---

Under review as a conference paper at ICLR 2020
and 2. Counting from admission time, a HR of 60 and 65 beats per minute measured af- ter 0,5 and h.0h, respectively, whereas MAP values of 80, 85, and 87 mmHg were observed after h.5 1.7h, and 2.5 h. According to Definition [ the time series is thus represented as S = (00.5, 60, 1), (3, 65, 1), (0.5, 80, 2), (1.7, 11.788 85, 2). (3,87, (7,8),,) 2) In this example, observations ordered by modality increase readability; in practice, we are dealing with unordered sets.
Definition 3 -onnsyynchooiized time series). We call
D-dimensional time series non- synchronized if there is at least one time point tj E RR at which at least one modality is not observed, i.e. if there exists t E R+ such that {(ts, 2k, m2) t = t,}] + D.
Furthermore, we assume that no two measurements of the same modality my occur at the same time, L.. t4 # t, for i * j has to be satisfied for all measurements in my- This assumption is not required for technical reasons but for consistency. It also makes it possible to interpret the results later on.
To summarize generic setup, do not require M, the number of observations per time series, to be the same, i.c. len(S,) & len(S,) for + j is permitted, nor do we assume that the time points and modalities of the observations are the same across time series. This setting is common in biomedical time series, for example. Since typical machine learning algorithms are designed to operate on data of a fixed dimension, novel approaches this non-trivial problem are required.
3.2 OUR MODEL
In the following, we describe an approach inspired by differentiable learning of functions that oper- ate on sets (Zaheer et al., 2017 7, Wagstaff et al., 2019). We phrase the problem of classifying time series on irregular grids as learning a function J on a set of arbitrarily many time series observations following Definition [ i.e. S {(11, 21, m.); )..),..... (t 2M, mM) such that f: + R², where S represents generic time series of arbitrary cardinality and IR corresponds to the logits of the C' classes in the dataset. As we previously discussed, we interpret each time series as an unordered set of measurements, where all information is conserved because the observation time is included for each set element. Specifically, we define ʃ to be a set function, i.e. a function that operates on a set and thus has to be invariant the ordering of the elements in the set. Multiple architectures applicable to constructing set functions such as Transformers (Lee et al.) 2019 Vaswani et al.] (017). or Deep Sets (Zaheer al., 2017). Due to preliminary experiments. where Transformers suffered from lower generalization performance in our setting]] we base this work on the framework of Za- heer et al. (2017). Intuitively, this can be seen as computing multivariate dataset-specifie summary statistics, which are optimized to maximize classification performance. Thus, we sum- nam-decoppose the set function f into the form
f(S) =
h(ss) 5;ES
[
where h: Ω + Rd and g: Rd + RC are neural networks, d E N determines the dimensionality of the latent representation, and sj represents a single observation of the time series S. We can view the averaged representations 1/|S| Σr,es h(s;) in general as a -ptaset- summary statistic learned to best distinguish the class labels. Equation 2 also implies the beneficial scalability prop- erties of our approach: each embedding can be calculated independently of the others; hence, the constant computational cost of passing single observation through the function h is scaled by the number of observations, resulting in runtime of (M) for a time series of length M.
Recently, Wagstaff al. (219)) derived requirements for a practical universal function represen- tation of sum cun-deconposable set functions, i.c the requirements necessary for a sum-decomposable function to represent an arbitrary set-function given that h and g arbitrarily expressive. In par- ticular, they show that a universal function representation can only be guaranteed provided that d 2 max; len(S:) is satisfied. During hyperparameter search we thus independently sample the di- mensionality of the aggregation space, and allow it to be in the order of the number of observations that are to be expected in the dataset. Further, we explored the utilization of max, sum, and mean as alternative aggregation functions inspired by Zaheer et al. (017);; Gareeoo et al. (2018).
Please see Section 4.4 for quantification.
4

---

Under review as a conference paper at ICLR 2020
1981 for dy- namical systems: we also observe set of samples from some unknown (but deterministic) dynami- cal process; provided the dimensionality of our architecture is sufficiently large" reconstructing the system up to diffeomorphism. The crucial difference is that we do not have to construct time-delay embedding but rather, let the network leam embedding that is suitable for classification.
Intuition Our method can be connected Takens's embedding theorem (Takens
capable of
Time encoding In order to represent the time point of an observation on normalized scale, we employ variant of positional encodings, introduced by Vaswani al. (2017). Preliminary results indicated that this encoding scheme reduces the sensitivity towards initialization and training hyper- parameters of model. Specifically, the time encoding converts the one- mne-iimnniionll time axis into multi-dimensional input by passing the time t each observation through multiple sine and cosine functions of varying frequencies. Given dimensionality IN of the time encoding, refer to the encoded position as æ E R where
t
(t) == sim
(3)
max t/2
228+ (1) = COS
(4)
1s2k/7
with & E (O...., 7/2} and max is representing the maximal time scale that is expected in the data. Intuitively, we select the wavelengths using geometric progression from 2T max 2n, and treat the number of steps and the maximum timescale max ts as hyperparameters of the model. For all experiments time encodings were used, such that an observation is represented as 83 (ar (tz) 2j. m,).
Loss function If not mentioned otherwise, we choose h and in Equation 2 to be multilayer perceptron deep neural networks, parametrized by weights 0 and vi, respectively. We thus denote these neural networks by ha and ge: their parameters are shared across all instances per dataset. In our training setup, we follow Zaheer et al. (2017) and apply the devised function to the complete time series, i.e. to the set of all observations for each time series. Overall, we optimize a loss function that is defined
C(O, 4) = E(SD))ED
y: Dv
Σ ES
ho(8y)
(5)
S
where E(-) represents a task-specific loss function. In out setup, we either utilize the binary cross- entropy combination with a sigmoid activation function in the last layer for binary classification or multi-label classification tasks and categorical cross-entropy in combination with a softmax acti- vation function in the last layer for multi-class classification tasks.
3.3 ATTENTION-BANE BASED AGGREGATION
So far, our method permits encoding sets of arbitrary sizes into fixed-size representation. For increasingly large set sizes, however, many irrelevant observations could influence the result of the function. The mean aggregation function particularly susceptible this because the influence of an observation to the embedding shrinks proportionally to the size of the set. We thus suggest use a weighted mean order allow the model to decide which observations are relevant and which should be considered irrelevant. This is equivalent to computing an attention a(S,s;) over the set input elements, and subsequently, computing the sum over all elements in the set.
Our approach is based on scaled dot-product attention with multiple heads i E (1...., m} in order be able to cover different aspects of the aggregated se We define al-), i.e. the attention weight function of an individual time series, to depend on the overall set of observations. This is achieved
II Takens's 's embedding theorem, sion (Liebovitch& Toth 1989,, which is typically well below the size typical neural network architectures. Since we are dealing only with single instance (i.e. time series) in this section, use and j to denote head and observation, respectively.
dæ is required, where da refers the fractal box counting dimen-
5

---

Under review as a conference paper at ICLR 2020
by computing embedding of the set elements using smaller set function f', and projecting the concatenation of the set representation and the individual set elements into a d-dimensional space. Specifically, we have ,, [f((),, sy] where W; E (((im((())ss;)) and K E RlSIxd_ Furthermore, we define a matrix of query points Q E Rmxd, which allow the model to summarize different aspects of the dataset via
K3a* Qi vd
exp(esx) Oj. Ex exp(ex,)
Cj
and
where aj.; represents the amount of attention that head gives to set element j. The head-specific row Q; of the query matrix Q allows a head to focus on individual aspects (such as the distribution of multiple modalities) of a time series. For each head, multiply the set element embeddings computed via the set function f with the attentions derived for the individual instances, i.e. ri Esa))[(s))) The computed representation is concatenated and passed to the aggregation network hg as regular set function, i.e. r* [ri ....m] In our setup, we initialize Q with zeros, such that at the beginning of training, the attention mechanism is equivalent to computing the unweighted mean over the set elements.
Overall, this aggregation function is similar to Transformers (Vaswani al. 2017), but differs from them in few key aspects. Standard Transformer blocks would use the information from all set elements in order to compute the embedding of an individual set element, leading runtime and space complexity of O(n)). In contrast, our approach computes the embeddings of set elements independently, leading lower runtime and memory complexity of (n)) Further, observed that computing embeddings with information from other set elements (as the Transformer does) actually decreases generalization performance (see Table for details).
EXPERIMENTS
We executed all experiments and implementations in unified code base, which we also make availableF to the community. While some of the datasets used subsequently have access restrictions, anybody can gain access after satisfying the defined requirements. This ensures the reproducibility of our results. Please consult Appendix A.2 for further details.
4.1 DATASETS
In order to benchmark the proposed method we selected 4 datasets with irregularly-samplee and non-synchronized measurements.
Healing MNIST The H-MNIST dataset was introduced by Krishnan al. (2015) order sim- ulate characteristics which typically occur in medical time series. In our setup, we use variant of this dataset. Every instance of the dataset contains 10 frames, derived from a single instance of MNIST dataset, where the digit is rotated according to an angle uniformly sampled between -90 to 90". Furthermore, randomly-selecte the top left corner of the image in order indicate seasonality in the time series. Finally, 60 % of the data points are randomly discarded in order to yield a final high-dimensional irrgularly-sampled time series with non-synchronized measurements. Using these settings each instance has average 3, 136 observations.
consecutive frames are augmented by a square artefact in
MIMIC-III Tasks MIMIC-III (Johnson et al. 2016) a widely-used, freely-accessible dataset con- taining around 50, 000 distinct ICU stays. The median length of stay is 2.1 and a wide range of physiological measurements (e.g. arterial blood pressure, respiration rate, heart rate) are recorded with a resolution of h. Furthermore, laboratory test results, collected at irregular time intervals are available. Recently, |Harutyunyan et al.) (2019) defined set of machine learning tasks, labels, and benchmarks using subset of the MIMIC-III dataset. We trained and evaluated our method and competing methods on the binary mortality prediction task (M3-Mo rtall tiyy and on the multi- class problem of phenotype classification described in Appendix I The goal of the mortality prediction task is to predict whether a patient will die during his/her hospital stay using only data from the first 48 hours of the ICU stay. This
while applying additional filtering

---

Under review as a conference paper at ICLR 2020
dataset contains around 21, 000 stays of which approximately 10 % result in death. The phenotype classification task consists of 40, 000 patients, each of which can suffer from a multitude of 25 acute care conditions.
Physionet Mortality Prediction Challenge The 2012 Physionet challenge dataset (Goldberger et al. 2000), which we abbreviate P-Mortality contains 12, 000 ICU stays each of which lasts at least h. For each stay, a set of general descriptors (such as gender, age, height, weight) were collected at admission time. Depending on the course of the stay and patient status, up 37 time series variables were measured (e.g. blood pressure, lactate, respiration rate, temperature). While some modalities might be measured regular time intervals (e.g. hourly or daily), some are only collected when required. Not all variables are available for each stay. The goal of the challenge was to predict if and with which certainty -a a patient will die during the hospital stay. The training set consists of 8, 000 stays while the testing set comprises 4, 000 ICU visits. Both datasets are similarly imbalanced, with a prevalence of around 14 %. For simplicity, the general descriptors (such as age and weight), were included as time points with single observation at the beginning of the stay. This treatment similar to the approach by Harutyunyan et al. (2019) in the MIMIC-III benchmarking datasets. Please refer to Table A.I. Table [A.2] and Table A.3 in the appendix for more detailed enumeration of samples sizes and label distributions. The total number of samples may slightly de- viate from the originally published splits, time series of excessive length prevented fitting some methods in reasonable time, and were therefore excluded.
4.2 COMPETITOR METHODS
GRU-simple GRU-SIMPLE (Che et al. 2018) augments the input at time t of a Ratur-Reeurrent- Unit RNN with a measurement mask mi and 8, matrix, which contains the time since the last measurement of the corresponding modality d, such that
8t St-1 + 0
t 1, Lmm = 0
6,
8t
St-1
t >> 1, m = 1
0
t t 0
where 8; represents the time associated with time step t.
Phased-LSTM The PHASED-LSTM (Neil al. 2016) introduced biologically inspired time de- pendent gating mechanism which regulates access to the hidden and cell state of a Long short-term RNN cell (Hochreiter & Schmidhuber 1997). While this allows the network to handle event-based sequences with irregularly spaced observations, the approach does not support unaligned measure- ments. In order still provide the architecture with all relevant information, we augment the input in similar fashion as described for the GRU-SIMPLE approach.
GRU-D GRU-D or GRU-Decay (Che et al. 2018) contains modifications to the GRU RNN cell, allowing it to decay past observations to the mean imputation of modality using a learnable decay rate. By additionally providing the measurement masks as an input the recurrent neural network the last feed in value. Learns how fast to decay back to mean imputation of the missing data modality.
Interpolation Prediction Networks IP-NETWORKS (hukka & Marlin 2019) apply multiple semi- parametric interpolation schemes to irregularly samplee time series to obtain regularly-samplee rep- resentations that cover long-tern trends, transients, and also sampling information. The method combines univariate interpolation step with a subsequent multivariate interpolation; the parame- ters of the interpolation network are trained with the classifier in an end-to-end fashion.
Transformer In the TRANSFORMER architecture (Vaswani et al. 2017) the elements of a sequence are encoded simultaneously and information between sequence elements is captured using Multi- Head-Attention blocks. In our case, an individual sequence element corresponds to all measurements available at a given time point, augmented with a measurement indicator. Transformers are normally modelling tasks and in our setup adapted to classification tasks by mean- aegeeggggggg the final representation. This representation is then fed into a one-layer MLP to predict logits for the individual classes
used for sowuencc-tt-sqqueee
7

---

Under review as a conference paper at ICLR 2020
K
Lactate
SysABP
Urine
mmig
100
50
10
20
30
40
10
20
30
40
Figure 2: Visualizations of single attention head on an instance of the i-Mortality dataset. We display set of variables relevant for assessing patient stability and organ failure: Serum Potassium (K), Lactate, Systolic Arterial Blood Pressure (SysABP), and Urine output. Darker colors represent higher attention values.
4.3 EXPERIMENTAL SETUP
To permit a fair comparison between the methods, we executed hyperparameter searches for each model on each dataset, composed of uniformly sampling parameters according Appendix [A.3] Training was stopped after 20 epochs without improvement of the validation loss, the hyperparam- eters with the best overall validation performance were selected for quantifying the performance on the test set. The train, validation, and test splits were the same for all models and all evaluations. Final performance on the test set was calculated by 3 independent runs of the models; evaluation took place after the model was restored to the state with the best validation loss. In all subsequent benchmarks, we use the standard deviation of the test performance of these runs as generalization performance estimates.
4.4 RESULTS
The results are shown Table [I] Overall, our proposed method exhibits the lowest per-epoch runtime most datasets, while either yielding competitive state-the-art performance. Further, the trade-off between runtime and performance of the proposed method is very good on all datasets (see Figure A.I Figure A.2 in the appendix for visualization of this argument). In order to elucidate the contribution of individual model components, we also provide an ablation study in Table A.4 Here we see that the attention mechanism contributes more the model performance, while the positional encoding seems to be beneficial for datasets with highly-varying time series lengths, in particular M3-Phemee 133-hhnotyppnng,
Opening the black box In the medical domain, it is of particular interest to understand the deci- sions a model makes based on the input is provided with. The formulation of our model and its per observation perspective which extent an individual observation contributed to the output of the model. We exemplify this in Figure with a patient time series that was combined with our models attention values, displayed for a set of clinically relevant variables. After reviewing these records with our medical expert, we find that our model able to pick up regions with drastic changes in individual modalities. Moreover, it is able to inspect other modalities at the same associated time (for instance, at hour 20). This is behaviour similar to what one would expect from an alerted clinician reviewing the logged medical records. Interestingly, we observe that the model attends to known trends (that are consisting with domain knowledge about patient deterioration ultimately resulting in death) such as increase lac- tate hemodynamic instability, indicated by drops in blood pressure. Furthermore, the model appears to be alerted by persisting low urine output. After several hours, this can indicative of kidney failure.
time series gives it the unique property of being able quantify to
8

---

Under review as a conference paper at ICLR 2020
Table 1: Performance comparison of methods on benchmarking datasets. Performance metrics have been rescaled to 100 for readability reasons. "AUC" denotes the area under the Receiver Operating Characteristic (ROC) curve; "PR AUC" denotes the area under the precision recall curve. "MICRO" refers to evaluating the metric globally by treating each entry of the label indicator matrix as a label. For "MACRO", the metric is computed for each class and then averaged. whereas in "WEIGHTED" the class-wise metrics are weighted by class imbalance. Values denoted with "OOM" were not obtainable due to restrictions in GPU memory.
DATASET
MODEL
MICRO AUC
MACRO AUC
WEIGHTED AUC
RUNTIME
SIMPLE
99.09 0.05
99.01
99.03 0.05
11.43 0.47
PHASED-LSTM
98.63 + 0.13
98.50 = 0.15
98.52 + 0.14
33,93 = 1.11
GRU-D
99.42 + 0.01
99.37 =
0.02 0.02
99.38 + 0.02
11.81 + 0.44
E-MNIST
IP-NETS
99.06 +
98.96 = 0.03
98.98
127.76 = 0.95
TRANSFORMER
99.59 0.05
99.55 0.05
99.56 0.05
21.62 0.90
SEFT*
99.76 - 0.01
0.01 99.75
99.75 + 0.01
4.05 - 0.35
GRU-SIMPLE
79.89 0.14
73.91 0.19
72.55 + 0.16
112.58 2.03
PHASED-LSTM,
80.00 0.06
73.91 = 0.09
72.65 + 0.08
400.41 = 14.14
GRU-D
82.16 0.04
77.14 - 0.03
76.08 + 0.01
288.70 + 16.66
M3- M3-Phenotyping
IP-NETS
-00M-
-0OM-
-0MM
-0OM-
TRANSFORMER
-00M-
-0O--
-0M-
-0OM-
SEFT
81.22 0.12
74.90 + 0.11
56.27 2.14
SEFT-ATTN
82.00 0.06
76.95 = 0.09
75.88 + 0.09
52.32 = 0.74
ACCURACY
AUC
AUC
RUNTIME
GRU-SIMPLE
88.24 0.38
36.36 1.31
79.36 + 0.26
22.80 0.56
PHASED-LSTM
88.32 0.31
35.30
1.38
80.16 + 0.22
25.54 0.26
GRU-D
89.56 + 0.38
46.76
0.65
83.73 + 0.21
31.85 + 0.86
M3-Mortality
IP-NETS
89.73 + 0.16
45.88 1 0.87
83.30 1 0.56
101.12 4.52
TRANSFORMER
89.14 1 0.15
42.32 - 0.41
82.60 1 0.55
4.79 1 0.02
SEFT
88.65 0.49
36.18
79.15 1 3.00
3.72 0.11
SEFT-ATTN
89.48 + 0.16
0.96 45.25
83.79 + 0.59
16.64 = 0.20
GRU-SIMPLE
85.66 0.14
39.43 0.71
79.79 + 0.16
0.06
PHASED-LSTM
85.57 0.11
39.55 - 0.62
78.71 + 0.76
18.59 + 1.15
GRU-D
87.19 0.30
54.95 = 0.54
86.58 + 0.32
14,08 = 0.38
E-Mortality
IP-NETS
87.23 0.18
54.87 = 0.41
86.42 + 0.18
7.21 0.46
TRANSFORMER
86.47 0.08
48.72 0.61
83.49 + 0.46
2.69 0.43
SEFT
87.11 0.32
52.07 0.41
84.12 + 0.32
3.07 0.03
SEFT-ATTN
87.62 0.16
54.05 0.27
85,50 + 0.13
7.54 + 0.08
*: Due the high dimensionality of H-MNIST and associated memory issues, the elements constructed by concatenating the observation time with values associated with the time point and measurement indicators. Furthermore, as this dataset features only 10 time steps and missingness induced randomly, we refrained from applying the aitention-basee aggregation.

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
In this work, we presented a novel approach for classifying time series with iregularly-sampled and unaligned, that is non-synchronized, observations. Our approach yields state-of-the-art strongly competitive performance on numerous simulated and real-world datasets, while reducing runtime by almost half. Moreover, we demonstrated that combining the perspective of individual observations with an attention mechanism permits increasing the interpretability of the model. This is particularly relevant for the medical and healthcare applications.
For future work, we reserve more extensive exploration of the learned latent representation to evaluate its utility for clustering of time series or visualization of their similarity.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Edwin V. Bonilla, Kian M Chai, and Christopher Williams. Multi-task gaussian process prediction. In Advances Neural Information Processing Systems (NeurIPS), pp. 153-160, 2008.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):6085, 2018.
Joseph Futoma, Sanjay Hariharan, and Katherine Heller. Learning to detect sepsis with Multitask Gaussian Process RNN classifier. In International Conference on Machine Learning (ICML), pp- 1174-1182, 2017.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison. Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pp. 1690-1699, 2018.
Ary L. Goldberger, Luis A.N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung-Kan Peng, and H. Eugene Stanley. Phys- iobank, physiotoolkit, and physionet: components of a new research resource for complex physi- ologic signals. Circulation, 101(23):c215 e220, 2000.
Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, and Aram Galstyan. Mul- titask learning and benchmarking with clinical time series data. Scientific Data, 6(1):96, 2019.
ISSN 2052-4463.
Alan G. Hawkes. Spectra of some self-exciting and mutually exciting point processes. Biometrika, 58(1):83-90, 1971.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Alistair E. W. Johnson, Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark. MIMIC-III, freely accessible critical care database. Scientific Data, 3, 2016.
Rahul G. Krishnan, Uri Shalit, and David Sontag. Deep Kalman Filters. arXiv e-prints, art. arXiv:1511.05121, Nov 2015.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: A framework for attention- based permutation innuutatiooniivriiant neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3744-3753, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Steven Cheng-Xina Xian Li and Benjamin M Marlin. Classification of sparse and irregularly sampled time series with mixtures of expected gaussian kemelss and random features. In UAI, pp. 484- 493, 2015.
Steven Cheng-Xine Xian Li and Benjamin M. Marlin. A scalable end-to-end Gaussian process adapter for irregularly sampled time series classification. In Advances In Neural Information Processing Systems 29, pp. 1804-1812, 2016.
Larry S. Liebovitch and Tibor Toth. A fast algorithm to determine fractal dimensions by box count- ing. Physics Letters A, 141(8):386-390, 1989.
Thomas Josef Liniger. Multivariate Hawkes processes. PhD thesis, ETH Zurich, 2009.
Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with mns: Improved classification of clinical time series. In Machine Learning for Healthcare Conference, pp. 253-270, 2016.
Roderick JA Little and Donald Rubin. Statistical analysis with missing data, volume 333. John Wiley & Sons, 2014.
11

---

Under review as a conference paper at ICLR 2020
Zhengdong Lu, Todd K Leen, Yonghong Huang, and Deniz Erdogmus. A reproducing kernel hilbert space framework for pairwise time series distances. In Proceedings of the 25th International Conference Machine learning, pp. 624-631, 2008.
Michal Lukasik, P. K. Srijith, Duy Vu, Kalina Bontcheva, Arkaitz Zubiaga, and Trevor Cohn. Hawkes processes for continuous time sequence classification: an application to rumour stance classification in Twitter. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 393-398, Berlin, Germany, August 2016. Association for Computational Linguisties. doi: 10. 181558/11/P16-20666 URL https: 7/ //www. ac 1 web. org/ antho www.thhoooy//Pee 6-64
Benjamin M. Marlin, David C. Kale, Robinder G. Khemani, and Randall C. Wetzel. Unsupervised pattern discovery in electronic health care data using probabilistic clustering models. In Proceed- ings of the 2nd ACM SIGHIT International Health Informatics Symposium, pp. 389-398. ACM, 2012.
Hongyuan Mei and Jason M. Eisner. The neural Hawkes process: A neurally coff-modultiing mul- tivariate point process. In I. Guyon, U. V. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp- 6754-6764. Curran Associates, Inc., 2017.
Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. Phased Istm: Accelerating recurrent network train- ing for long event-based sequences. In Advances in Neural Information Processing Systems 29, pp. 3882-3890, 2016.
Satya Narayan Shukla and Benjamin Marlin. Interpolation-prediciiinn networks for irregularly sam- pled time series. In International Conference on Learning Representations, 2019.
Floris Takens. Detecting strange attractors in turbulence. In David Rand and Lai-Sang Young (eds.), Dynamical Systems and Turbulence, pp. 366-381, Heidelberg, Germany, 1981. Springer.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NeurIPS), pp. 1998-60088 2017.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence sequence for sets. arXiv preprint
2015.
Edward Wagstaff, Fabian B. Fuchs, Martin Engelcke, Ingmar Posner, and Michael Osborne. On the Limitations of Representing Functions on Sets. 2019.
Christopher K. Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. .ITT Press, 2(3):4, 2006.
Shuai Xiao, Junchi Yan, Mehrdad Farajtabar, Le Song, Xiaokang Yang, and Hongyuan Zha. Joint modeling of event sequence and time series with attentional twin recurrent neural networks. arXiv e-prints, art, arXiv:1703.08524 2017.
Pranjul Yadav, Michael Steinbach, Vipin Kumar, and Gyorgy Simon. Mining electronic health records (EHRs): survey, ACM Computing Surveys, 50(6):85, 2018.
Yingxiang Yang, Jalal Etesami, Niao He, and Negar Kiyavash. Online learning for multivariate hawkes processes. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Gameet (eds.), Advances in Neural Information Processing Systems 30, pp- 4937-4946 Curran Associates, Inc., 2017.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R. Salakhutdi- nov, and Alexander Smola. Deep sets. In Advances in Neural Information Processing Sys- tems (NeurIPS), pp. 3391-3401, 2017.
12

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
Table A.I: M3 -Mortality prevalence of labels for the binary classification task
Training Prevalence
Testing Prevalence
In-hospital deaths
0.135
0.116
Table A.2: E-Mortality prevalence of labels for the binary classification task
Training Prevalence
Testing Prevalence
In-hospital deaths
0.140
0.
A.1 DATA FILTERING
Due to memory requirements of some of the competitor methods, it was necassary to excluded time series with extremly high number of measurements. For the M3 M3-Phenatyping patients with more than 2000 distinct time points were discarded from training. For 113-Mortalis patients with more than 1000 time points were discarded as they contained dramatically different measuring frequencies compared to the rest of the dataset.
A.2 IMPLEMENTATIONAL DETAILS
All experiments were run using tensorflow 0.15. 0rco and training was performed on NVIDIA Geforce GTX 1080 GPUs. In order allow a fair comparison between methods, the input processing pipeline cached model specific representations and transformations of the data. To further increase efficiency of the RNÑs, sequences were binned in to buckets of jointly trained instances depending their sequence length. The buckets were determined according to (0.25, 0.5, 0.75) quantiles of the length distributions of the datasets.
A.3 TRAINING, MODEL ARCHITECTURES AND HYPERPARAMETES SEARCH
General All models were trained using the Adam optimizer, while randomly sampling the learning rate from (0.001, 0.0005, 0.00025, 0.0001). Further, the batch size of all methods was sampled from the values (32, 64, 128, 256).
Recurrent neural networks For the RNN based methods (GRU-SIMPLE, PHASED- LSTM, GRU-D and IP-NETS), the number of units was sampled in from the values (16, 32, 64, 128, 256, 512). Further, recurrent dropout and input dropout were sampled from the values (0.0, 0.1, 0.2, 0.3). Solely, for the PHASED-LSTM method, we did not apply dropout to the recurrent state and the inputs, as the learnt frequencies were hypothesized to fulfill similar function as dropout (Neil et al. 2016).
We vary the number of layers, dropout in between the layers and the number of nodes per layer for both the encoding network he and the aggregation network ge from the same ranges. The number of layers is randomly sampled between 1 and 5. the number of nodes in layer are uniformly sampled from the range (16, 32, 64, 128, 256, 512) and the dropout fraction is sampled from the values (0.0, 0.1, 0.2, 0.3). The width of the embedding space prior to aggregation sampled from the values (32, 64, 128, 256, 512, 1024, 2048). The aggregation function selected to be one of mean, sum and mar. The number of dimensions used for the positional embedding 7 is selected uniformly from (4, 16) and max_ts selected from the values (10, 100, 1000).
SEFT
SEFT-Attn The parameters for the encoding and aggregation networks are sampled in a similar fashion as for SEFT. In contrast set the aggregation function to be sum described in the text.
13

---

Under review as a conference paper at ICLR 2020
Table A.3: M3-Pherotyping ng prevalence of labels for the multi label classification task
Phenotype
Training
Validation
Testing
Acute and unspecified renal failure
0.216
0.207
0.211
Acute cerebrovascular disease
0.0746
0.0753
0.0662
Acute myocardial infarction
0.103
0.103
0.108
Cardiac dysrhythmias
0.322
0.317
0.323
Chronic kidney disease
0.135
0.131
0.132
Chronic obstructive pulmonary disease
0.132
0.128
0.126
and bronchiectasis
Complications of surgical procedures or medical care
0.207
0.201
0.213
Conduction disorders
0.0726 0.268
0,07
0.0704
Congestive heart failure; nonhyperten-
0.264
0.268
sive
Coronary atherosclerosis and other
0.323
0.317
0.331
heart disease
Diabetes mellitus with complications
0.0955
0.0945
0.094
Diabetes mellitus without complication
0.194
0.187
0.192
Disorders of lipid metabolism
0.291
0.287
0.289
Essential hypertension
0,421
0.41
0.424
Fluid and electrolyte disorders
0.267
0.276
0.265
Gastrointestinal hemorrhage
0.0715
0.0747
0.0788
Hypertension with complications and secondary hypertension
0.133
0.131
0.13
Other liver diseases
0.0884
0.0904
0.0883
Other lower respiratory disease
0.0514
0.0484
0.0565
Other upper respiratory disease
0.0408
0.0371
0.0429
Pleurisy; pneumothorax; pulmonary
0.0858
0.09
0.0905
collapse
Pneumonia (except that caused by berculosis or sexually transmitted dis- ease)
0.14
0.135
0.135
Respiratory failure; insufficiency; arrest
0.18
0.184
0.177
(adult)
Septicemia (except in labor)
0.142
0.145
0.138
Shock
0.0783
0.0745
0.0811
Total samples
29 208
6359
6266
Further we use a constant architecture for the attention network f' with 2 layers, 64 nodes per layer, heads and a dimensionality of the dot product space of 128. We solely sample the amount of attention dropout uniformly from the values (0.0, 0.1, 0.25, 0.5).
Transformer We utilize the same model architecture as defined Vaswani et al. (2017), where one hidden layer MLP feed-forward network, with dimensionality of the hidden layer selected to be twice the model dimensionality. The parameters for the Transformer network were sampled according the following criteria. The dimensionality of the model was sampled uniformly from the values (64, 128, 256, 512, 1024), the number of attention heads per layer from the values (2, 4, 8) and the number of layers from the range 6] I.. Further, of dropout of the residual connections and the amount of attention dropout uniformly from the values (0.0, 0, 0.2, 0.3, 0.5), and the maximal timescale for the time embedding from the values (10, 100, 1000) (similar to the SEFT approach).
sampled the amount
14

---

Under review as a conference paper at ICLR 2020
Table A.4: Ablation study of individual components SEFT. "AUC" denotes the area under the Receiver Operating Characteristic (ROC) curve; "PR AUC" denotes the under the precision recall curve; "RUNTIME" denotes the runtime one training epoch in seconds.
DATASET
MODEL
MICRO AUC
MACRO AUC
WEIGHTED AUC
RUNTIME
B -N-MIIST
SEFT
99.76 = 0.01
99.75 = 0.01
99.75 = 0.01
4.05 + 0.35
SEFT (NO ATTENTION)
81.22 = 0.12
75.95 = 0.09
74.90 = 0.11
56.27 + 2.14
M3 Phenootttyi
SEFT-ATTN (NO TIME ENC.)
80.46
0.86
74.70 = 1.12
73.48 1.18
50.17 + 0.84
SEFT-ATTN
82.00 0.06
76.95
0.09
0.09 75.88
52.32 0.74
ACCURACY
PR AUC
AUC
RUNTIME
SEFT (NO ATTENTION)
88.65 = 0.49
36,18 :: 5.07
79.15 1 3.00
3.72 +: 0.11
M3-Mortality
SEFT-AITN TIME ENC.)
89.31
0.08
44.12 - 0.06
83.72 1 0.34
17.60 0.43
SEFT-ATTN
0.16
45.25
0.96
83.79 0.59
16.64 0.20
SEFT (NO ATTENTION)
87.11
0.32
= 0.41
84.12 = 0.32
3.07 0.03
P-Mortality
SEFT-AFTH TIME ENC.)
87.03
0.06
1.04
84.91
0.04
SEFT-ATTN
87.62
0.16
54.05 + 0.27
85.50 1 0.13
7.54 0.08
M3 M3-Mortality
P P-Mortality
84
+
86
82
AUC'nn %
iuC'n %
80
82
78
80
76
78
0
20
40
60
80
100
5
10
15
20
Runtime in S
Runtime in
SEFT-ATTN, SEFT, GRUD-D, GRU-SIMPLE, IP-NETS, PHASED-LSTM, TRANSFOMER
Figure A.1: visualisation of the runtime of all methods and their AUC for datasets with binary classification scenario.
15

---

Under review as a conference paper at ICLR 2020
H-MNISS ST
M3 13-Phenotypping
Weighted AUC in %
Weighted AUC in %
99.5
75
99
74
73
98.5
50
100
100
200
300
400
Runtime in s
Runtime in
SEFT-ATTN, SEFT, GRUD-D, GRU-SIMPLE, IP-NETS, PHASED-LSTM. TRANSFOMER
Figure A.2: visualisation of the runtime of all methods and their AUC for datasets with a multil- abel classification scenario. Please note that the model definition for SEFT changes between the left and the right column; please see Table for more details.
16


Reviews:
[{"review_id": "Hygu7HYM9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of supervised classification of time-series data that are irregularly sampled and asynchronous, with a special focus on the healthcare applications in the experiments. Inspired by the recent progress on differentiable set function learning, the paper proposes an approach called Set Functions for Time Series (SEFT), which views the time series as sets, and use a parametrized sum-decomposing function f as the model for representing the probabilities of different classes, with the sets as the inputs. The problem then reduces to learning the finite dimensional parametrization of the function f under a given loss, which is a differentiable optimization problem that can be learned via standard optimization methods. Together with a positional embedding of the timestamps and an attention-based aggregation, the paper reports improved performance of the proposed approach on a few healthcare time series with asynchronous and irregularly sampled data. In particular, the runtime is largely shortened, while the final accuracy remains competitive to other methods compared in the paper. \n\nThe idea of SEFT is novel and the results are also showing its promise. In addition, the interpretability shown in section 4.3 is also attractive. However, there are several issues that limit the contribution and maturity of this paper. \n\nFirstly, the paper proposes to model time series as a set. But this loses the information of the order of the time series, which can be extremely important in those datasets with long history dependence. In such cases, I'm not convinced that the set modeling would work. The authors should double check the characteristics of the datasets that are used, and see if they lack long history dependence properties in intuition. If so, this should be mentioned clearly. The authors should also make a more fair comparison with other approaches (like those based on RNN) on datasets with strong history dependence, e.g., Memetracker datasets of web postings and limit-order books datasets. Otherwise, it would be not clear whether this set modeling is generally applicable for general time series data.\n\nSecondly, the authors missed a large amount of related literature for approaching asynchronous and irregularly sampled time series, namely (marked) point-process based approaches. See papers like [1, 2, 3], to name just a few. The authors should at least include some of the recent approaches in this direction for comparison before claiming the superiority of SEFT.\n\nThirdly, there are a few parts that are not very clear. 1) The discussion about complexity (order m and m\\log m) at the bottom of page 1 is weird -- what does this complexity refer to? Does it include the learning of the unknown parameters in the models (like training of the neural networks in this paper)? 2) The loss function in formula (5) is not specified later in the paper (at least hard to find). 3) The Table 1 should be explained in much more details. In particular, why don't we include SEFT-ATTN for H-MNIST? The comment after * is also not clear to me -- is it relevant to why SEFT-ATTN is not included? And what are MICRO/MACRO/WEIGHTED AUC? And why are we using different sets of performance criteria for the first two and last two datasets?\n\nFinally, some minor comments: 1) On page 2, \"the following methods\" should be \"the above methods\"; 2) on page 3, the meaning of \"channels\" should be specified clearer; 3) on page 4, in formulae (3) and (4), should there be \\pi or 2\\pi in the formula?\n\n[1] Mei, Hongyuan, and Jason M. Eisner. \"The neural hawkes process: A neurally self-modulating multivariate point process.\" Advances in Neural Information Processing Systems. 2017.\n[2] Xiao, Shuai, et al. \"Joint modeling of event sequence and time series with attentional twin recurrent neural networks.\" arXiv preprint arXiv:1703.08524 (2017).\n[3] Yang, Yingxiang, et al. \"Online learning for multivariate Hawkes processes.\" Advances in Neural Information Processing Systems. 2017.\n\n############## post rebuttal ###############\nAfter reading the authors' rebuttal, I decide to improve the rating to 5 (reflected as 6 due to the ICLR rating system limitation this year).\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1l8MVt9FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper idea of this paper is straightforward and clear: treat the irregular time series as a bag of events, augment them with time information using positional encoding, and process the events in parallel. The idea is certainly faster than the sequential algorithms such as RNNs and their extensions. However, as shown in the experiments, because it does not encode the \"sequential-ness prior\" to the model, it is less accurate. Compared to RNNs, the proposed model has better access to the entire length of sequences and does not suffer from the limited memory issues of RNNs and variants.\n\nThe proposed idea in this paper can be considered a simplified version of the Transformers. Like transformers, the time and order are only provided to the model using the positional encoding and attention is central to aggregation over the sequence. Realizing the relationship with the Transformers not only decreases the novelty degree for this paper but also requires the authors to include the Transformers in the baselines.\n\nFinally, the results reported in the experiments are nice, especially for the baseline GRU-D! However, the MIMIC-III Mortality benchmark has a lot more than 21,000 stays to the best of my recollection. Can you please elaborate on how the number of data points has decreased?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJeBhVSAYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe work is focused on classification of irregularly sampled and unaligned multi-modal time series. Prior work has primarily focused on imputation methods, either end-to-end or otherwise. This paper approaches the problem as a set function mapping between the time-series tuples to the class label. The proposed method is uses a set encoding of a multi-modal time series input, followed by mode-specific encoding of the tuples which are then aggregated in multiple ways prior to classification. An attention mechanism is attached in order for the model to automatically weigh the relevance of tuples for the classification. The model is compared to imputation based baselines on clinical ICU time series classification tasks. The performance mostly appears comparable across baselines but the proposed method has much better run-times. \n\nThe paper is for the most part well written, and related work well characterized. The formulation is interesting and clinically relevant as well so the choice of data-sets makes some sense. I have a few concerns about the architecture formulation and lack of clarification and intuition in what appears to be the main contribution of the paper (Sec 3.2 and 3.3) which I will detail below:\n\na. In the evaluation, I really want to see a decoupling between the \"time encoding step\" and \"attention based aggregation\" on the performance to figure out to isolate different sources of performance improvements. That is can there be a SEFT without time encoding? If not, why not? I encourage more ablation like studies that look at different sources of performance gains and demonstrate them in experiments.\n\nb. The description of Sec 3.3. is really missing key motivation for the choices made around how the attention formulation is designed. For example why does the dot produce include the set elements? What if it doesn't? What is Q supposed to capture? \n\nc. Is a_{j,i} shared across instances? Then irrespective of the number of observations per instance, the $j^{th}$ tuple gets similar weights? If not appropriate indexing will help clarify this.\n\nd. It would be useful to provide how exactly a label is inferred for a *new* test instance. \n\nI have some minor additional feedback (just for presentation and motivation purposes):\n\n1. Authors make a claim in the introduction which should likely be qualified with a citation - \"Furthermore, even though a decoupled imputation scheme followed by classification is generally more scalable, it may lose information that is relevant for prediction tasks\". How does decoupled imputation imply loss of relevant information? By losing information about which observations are missing and relying on that for prediction? Does this clinically make sense? Or even generally? \n\n2. In Sec 3.3, you probably mean $W_i \\in R^{(im(f') + |s_j|) \\times d}$. That is parenthesis are missing?\n\n3. What are the +- std errors indicating? Is it cross validation error on a held-out test set? \n\n4. Initially $i$ is indexing samples and by equation (3), (4) $i$ indexes time(?) and in Sec 3.3 $i$ indexes observations? How are observations defined here? is it measurement of specific modality at a specific time instance? Can you clear this in the introduction itself? \n\n-----------------------------------------------------------------------------------------------------------------------------------------------------------------------\nI have read the authors updated draft and response. The experiments section looks much better now. \n\n1. The overall contribution has less clinical utility in my opinion as generally a patient likely deteriorates over time before an adverse outcome and therefore -- to give the model too much flexibility w.r.t. time ordering doesn't make quite as much sense. This is reflected in the fact that experimental results are not drastically better than other baselines. The authors might be able to show the utility of the method on other time series classification datasets where this is not a limitation of the data itself. However in those settings, it may be a bit hard to beat transformers. Do the authors have a sense of where the benefits of this method really are?\n\n2. Mortality tasks are generally on the simpler side of clinical prediction problems as well. Nonetheless I think the contribution has some utility to the community. I do encourage the authors to try non--clinical datasets for a comparison\n\n3. Please have a discussion that includes limitations and to discuss where the benefits of your methods really lie. A clear and thoughtful discussion is currently missing in your conclusions. \n\nWith that said, I am updating my score to a 6.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates a new approach to classification of irregularly sampled and unaligned multi-modal time series via set function mapping. Experiment results on health care datasets are reported to demonstrate the effectiveness of the proposed approach. 

The idea of extending set functions to address missing value in time series is interesting and novel. The paper does a good job at motivating the methods and describing the proposed solution. The authors did a good job at addressing the concerns of the reviewers. 

During the discussion, some reviewers are still concerned about the empirical results, which do not match well with published results (even though the authors provided an explanation for it). In addition, the proposed method is only tested on the health care datasets, but the improvement is limited. Therefore it would be worthwhile investigating other time series datasets, and most important answering the important question in terms of what datasets/applications the proposed method works well. 

The paper is one step away for being a strong publication. We hope the reviews can help improve the paper for a strong publication in the future. 
Decision: Reject