Probabilistic Imputation for Time-series Classification with Missing Data
Authors: Hyunsu Kim
SeungHyun Kim
Eunggu Yun
Hwangrae Lee
Jaehun Lee
Juho Lee
Venue: Submitted to ICLR 2023 (2023)
Keywords: Missing Data Imputation
Multivariate Time Series
Classification
Generative Model
Variational Autoencoder
Dropout

Abstract: Multivariate time series data available for real-world applications typically contain a significant amount of missing values. A dominant approach for the classification with such missing values is to heuristically impute the missing values with specific values (zero, mean, values of adjacent time-steps) or learnable parameters. However, these simple strategies do not take the data generative process into account, and more importantly, do not effectively capture the uncertainty in prediction due to the multiple possibilities for the missing values. In this paper, we propose a novel probabilistic framework for classification with multivariate time series data with missing values. Our model consists of two parts; a deep generative model for missing value imputation and a classifier. Extending the existing deep generative models to better capture structures of time-series data, our deep generative model part is trained to impute the missing values in multiple plausible ways, effectively modeling the uncertainty of the imputation. The classifier part takes the time series data along with the imputed missing values and classifies signals, and is trained to capture the predictive uncertainty due to the multiple possibilities of imputations. Importantly, we show that na\"ively combining the generative model and the classifier could result in trivial solutions where the generative model does not produce meaningful imputations. To resolve this, we present a novel regularization technique that can promote the model to produce useful imputation values that actually help classification. Through extensive experiments on real-world time series data with missing values, we demonstrate the effectiveness of our method.

Under review as a conference paper at ICLR 2023
PROBABILISTIC IMPUTATION FOR TIME-SERIES CLAS- SIFICATION WITH MISSING DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Multivariate time series data available for real-world applications typically contain a significant amount of missing values. A dominant approach for the classification with such missing values is to heuristically impute the missing values with specific values (zero, mean, values of adjacent time-steps) or learnable parameters. How- ever, these simple strategies do not take the data generative process into account, and more importantly, do not effectively capture the uncertainty in prediction due to the multiple possibilities for the missing values. In this paper, we propose novel probabilistic framework for classification with multivariate time series data with missing values. Our model consists of two parts; a deep generative model for missing value imputation and a classifier. Extending the existing deep generative models better capture structures of time-series data, our deep generative model part is trained to impute the missing values in multiple plausible ways, effectively modeling the uncertainty of the imputation. The classifier part takes the time series data along with the imputed missing values and classifies signals, and is trained to capture the predictive uncertainty due to the multiple possibilities of imputations. Importantly, show that naïvely combining the generative model and the classi- fier could result trivial solutions where the generative model does not produce meaningful imputations. To resolve this, we present a novel regularization tech- nique that can promote the model to produce useful imputation values that actually help classification. Through extensive experiments on real-world time series data with missing values, we demonstrate the effectiveness of our method.
INTRODUCTION
Multivariate time-series data are universal; many real-world applications ranging from healthcare, stock markets, and weather forecasting take multivariate time-series data inputs. Arguably the biggest challenge in dealing with such data is the presence of missing values, due to the fundamental difficulty of faithfully measuring data for all time steps. The degree of missing is often severe, in some applications, more than 90% of data are missing for some features. Therefore, developing an algorithm that can accurately and robustly perform predictions with missing data is considered an important problem to be tackled.
In this paper, we focus on the task of classification, where the primary goal is to classify given mul- tivariate time-series data with missing values, simply imputing the missing values with heuristically chosen values considered be strong baselines that are often competitive or even better than more sophisticated methods. For instance, one can fill all the missing values with zero, the mean of the data, or values from the previous time steps. GRU-D (Che et al., 2018) proposes a more elaborated imputation algorithm where the missing values are filled with mixture between the data means and values from the previous time steps with the mixing coefficients learned from the data. While these simple imputation-based methods work surprisingly well (Che et al., 2018; Du et al., 2022), they lack fundamental mechanism to recover the missing values, especially the underlying generative process of the given time series data.
Dealing with missing data is deeply connected to handling uncertainties originating from the fact that there may be multiple plausible options for filling in the missing values, it is natural to analyze them with the probabilistic framework. There have been rich literature on statistical analysis for missing data, where the primary goal is to understand how the observed and missing data

---

Under review as a conference paper at ICLR 2023
generated. In the seminal work of Little and Rubin (2002), three assumptions for the missing data generative process were introduced, including Missing Completely At Random (MCAR), Missing At Random (MAR), and Missing Not At Random (MNAR). While MCAR MAR simplifies the modeling and thus makes the inference easier, they may be unrealistic for real-world applications, because they assume that the missing mechanism is independent of the missing values (MAR) or both missing and observed values (MCAR). MNAR, the most generic assumption, assumes that the missing mechanism depends on both missing and observed values, so the generative model based on the MNAR assumption should explicitly take the missing mechanism into account. Based on this framework, Mattei and Frellsen (2019) presented deep generative models for missing data under MAR assumption, and this was later extended to MNAR in Ipsen et al. (2021). Combining the deep generative model and classifier, Ipsen al. (2022) proposed a hybrid model that can classify missing data with problematically imputed values generated under MAR assumption.
Still, in our opinion, there no satisfactory work combining probabilistic generative models for multivariate time-series data with missing values and classification models, so that the classifier could consider the uncertainty in filling in the missing values when making predictions. The afore- mentioned probabilistic frameworks are not designed for classification (Mattei and Frellsen, 2019; Ipsen et al., 2021), and more importantly, not tailored for time series data (Ipsen et al., 2022). A naive extension of Ipsen et al. (2022) for time series is likely to fail; putting the obvious difference between the static and time series data aside, the fundamental difficulty of learning the generative models for missing is that there are no explicit learning signals that could promote the model to generate "meaningful" missing values. Since we don't have ground truth for the missing values, in principle, the generative model can generate arbitrary values (c.g., zeros), and the combined classi- fier can still successfully classify time series data, which is a critical problem that is overlooked in the existing works.
To this end, we propose a hybrid model combining the deep generative models for multivariate time series data and the classification models for them. The generative model part is built under the MNAR assumption, and borrowing the structure of GRU-D (Che et al., 2018), the generative model is designed to naturally encode the continuity of the multivariate time series data in the gen- erative process. The classifier then takes the missing values generated from the generative model and executes classification, and unlike the algorithms based on heuristic imputations, it takes multi- ple feasible options for the missing values and computes predictions based them. To tackle the difficulty in guiding the generative model to generate "meaningful" missing values, we introduce a novel regularization technique that deliberately erases observed values during training. As conse- quence, the classifier is forced to do classification based more on the generated missing values, so the generative model is encouraged to produce missing values that are more advantageous for the classification. Using the various real-world multivariate time series benchmarks with missing val- ues, we demonstrate that our approach outperforms baselines both in terms of classification accuracy and uncertainty estimates.
2 BACKGROUND
2.1 SETTINGS AND NOTATIONS
Let æ = [Ii, T.] T E Rd be d-dimensional vector, along with the mask vector 8 (#1, E (0, 13a, where 8n 1 xy is observed and 8j = 0 otherwise. Given a mask 8, we can split x into the observed part ahs := {-, 88 = 1} and the missing part mis == {xr s, = 0). For collection of data, the 7th instance is denoted as x; = [ are defined similarly. For a multivariate time-series data, we denote the vector of 4h time step as He = [ or E Rd, and the corresponding mask as 8, = [[st......, The tth time step of 7thh instance of a collection is denoted as Zt,5, which is split into x.th and x according to 82.i.
I.]T
and s,, x:th, and x:""
Following Mattei and Frellsen (2019); Ipsen al. (2021), we assume that the joint distribution of an input x and a mask is factorized as pe (x, = Pe (z) (t)P (s) z). The conditional distribu- tion Pv (s x>) plays an important role for describing missing mechanism. Under MCAR assump- tion, we have p(s|æ) = p(s). under MAR we have p (s x)) = Py (8) (s) and under MNAR we have (s(#) = ppps bbs mis)). The likelihood for the observed data obss is thus computed as Po obs 8) = po ( (xz, mid. =
2

---

Under review as a conference paper at ICLR 2023
2.2 MISSING DATA IMPORTANCE- WEIGHTED AUTOENCODER AND ITS EXTENSIONS
AutoEncoder (MI- WAE) (Mattei and Frellsen, 2019), deep generative model for missing data, and its extensions to MNAR and supervised settings. Similar to variational autoencoder (VAE) (Kingma and Welling, 2014), MIWAE assumes that data is genearted from a latent representation z. but only ob- serve x with s generated from the missing data model (s(s)2) æ). MIWAE assumes MAR, so we have Pup (8) (8)))=
In this section, we briefty review the Missing data
Pv (a )b) The fog-likelihood for (x 8) is then computed as
log p ha 8) log / Pio (sl "o (o ar = (2)d()ddd "dzda:
= loggge Pv (s/2x + log / po (2)0 12)
(1)
=log po (axto))
For the missing data imputation, Pup (8)x not necessary, so we choose to maximize only the log (arbb)) The integral intractable, so we consider the Importance Weighted AutoEneoder (IWAE) lower bound (Burda et al., 2015),
K po (a: (z)))((x)) 4 (2nhs)
1 log Po (2: 2 EZnK log
= LMIWAE (0, Φ).
(2)
K
Here, 4s ( for Is = 1...., are i.dd copies of the variational distribution (encoder) (h) approximating the true posterior (z) Ez,x denotes the expectation w.rt. I (=x )bs), K is the number of particles, and the lower bound gets tighter as K increases and converges to the upper- bound as K + 00.
Ipsen al. (2021) presented not-MIWAE, an extension of MIWAE with MNAR assumption. The log-likelihood for (wtt,, under the MNAR assumption is,
log Pe, s (x s) = log / Pus (s/x mi )pe (x 1 ()p (xi po (z dzda.
(3)
where we are assuming that (w "ii) are independent given 2. The corresponding IWAE lower- bound with the variational distribution go mi xb/ ) po mms |=) ( is,
1
K
pe th ax )e bb (x)po(=x))
(B,M (b, = E 1TT
log
(4)
K
2 (z anobs)
where Ex aik denotes the expectation w.r.t. II
pe (x (=)) (=x (2o
On the other hand, Ipsen et al. (2022) extended MIWAE to a supervised learning setting, where the goal is to learn the joint distribution of an observed input ahs mask s, and corresponding label y,
log Pe, vb (y abs
= log
px (x
x mis (w mis " pe ths mix (z)dddd
+ log / a
= log Pop (s/
mi )o ( (x (2) (z) (2)
(5)
=log (1).
The term Pes (b) is irrelevant to the prediction for so we choose to maximize log Pa.A (y, obt)), which again can be lower-bounded by IWAE bound with the variational distri- bution yol 22, mis) obs) = po mis 2) ob)2
1
K
PA PA obs
.o (o obs 2x #x)p(zz)) ( 2 (onth))
SamMMIWAE (0 Φ) = FF snx T.K
log
(6)
K
k=1
where E. Ex an denotes the expectation w.r.t.
l po po mis
1 (2 (2000
3

---

Under review as a conference paper at ICLR 2023
2.3 GRU FOR MULTIVARIATE TIME SERIES DATA AND IMPUTATION METHODS
We briefly review GRU (Cho et al., 2014) as it is being used as a building block for our method. Given a multivariate time series (x,){=1- GRU takes a vector of one time step time and accu- mulates the information into a hidden state h4- Specifically, the forward pass at 1 time step takes at and updates the hidden state h, as follows:
a, - o (W at + Ualu-1 + ba), T- Fo (W + U -, + br)
h: = tanh W + Ur, @ h2-1) + b). h = (1 - ai) @ h + 3 hi,
where @ denotes the element-wise multiplication. We also review the heuristical imputation meth- ods described in Che et al. (2018), which are for GRU-based multivariate time-series classifiers and commonly used as baselines.
GRU-zero: simply put zero for all missing values. That is, R..j St.j
GRU-mean: imputes the missing values &t St.Jiet + (1 st,r where žJ Σ Et St.y Σx= Σ. St,i.j is the empirical mean observed values for jth fea- ture of a given collection of time series data ((=),))==
GRU-forward: imputes the missing values as œt.5 = St.FL, + (1 - (11)))) where the last time when j"h feature was observed before
GRU-simple:
along with the imputed vector 2; (either by GRU-mean or GRU-forward), con- catenate additional information. Che et al. (2018) proposed concatenate 1) the mask 8t, and the time-interval 6, saving the length of the intervals between observed values (see Che et al. (2018) for precise definition). The concatenated vector [ 8t, 6r] is then fed into GRU.
GRU-D: introduces learnable decay values for the input æ, and hidden state he follows:
nx = exp(- max W 7 5: by O)),
Yn cxp( max (W 7h. 5₁ b 0)).
Given a vector 2; with mask st. GRU-D imputes the missing values as
it = $t, + (1 - st.5) + (1 - )g)))
(8)
That is, the missing is imputed as a mixture of the last observed Te and the empirical mean Tj with the mixing coefficient set the learned decay. The hidden state from the previous time step h,-1 is decayed as 7 @ h,-1 and passed through GRU with the imputed i..
METHODS
In this section, describe method, probabilistic framework for multivariate time series data with missing values. Our method is an extension of Sup.IWAE to time series data under MNAR assumption, but the actual implementation In Section 3. we first present supnotMIWAE, an MNAR version of supMIWAE, with the encoder and decoder architectures designed for time series data with missing values. In Section 3.2, we show why the sup(not)MIWAE for data with missings may fail, and propose a novel regularization technique to prevent that.
merely naïve composition of the existing models.
3.1 SUPNOTMIWAE FOR MULTIVARIATE TIME SERIES DATA
Given a multivariate time series data #1.7 == (a+) with observed L and missing mis missing mask 81:7 == (8t) and label y. we assume the following state-space model with latent vectors 21.7" = (2
Po 70. x 10 $1:1)
= f Px ( Tii po (1 2 po mis (1:1) )oo (Z1:T P (S1:7 (err (aterdddaii di
(9)
Below describe each component more in detail.
4

---

Under review as a conference paper at ICLR 2023
Prior (x(T)) assume autoregressive prior for 21:7,
I
po(Z1:T) = (0, 1) II (Zzi diag( (21:t-1));
(10)
t=
h = PrUpp h1-1); where (Mpr(
(pr( opr
computed as
(p((21:), (zr(= = MLP (h
(11)
Here, GRU,r(Zz, 1) is GRU cell that takes 2, and the hidden state h4-1 and update to h.-
Decoders po (21.7 (1)) and po (2TT (111)
The decoder for the observed pe pe( x1:7) is de-
fined in autoregressive fashion,
T
po (o obs (1:)) = II (axt Made (Z1:)) diag dee (z1; 1=1
(12)
where (Jose (Zia), a dee (21)))=_ are defined as in (11) using GRU. The decoder for the missing ( shares the same model; that is, both observed value decoder pe (w (1:)) and the missing decoder po (ammis
p ( (z.r) share the same model.
Missing model Pwl (11.T FIT)) The missing model is simply assumed to independent Bernoulli distributions over the time steps and features.
P
II 1= 5-1
Bern mii. ((112))
(13)
where Tmis (F1.T) is computed
mis (TTT))
MLP mis (xLT).
(14)
Classifier px px(3) bet xeiiit)) We simply use common GRU-based time series classifier for this. Let ht be the hidden state from GRU after consuming (2 xiii)) Then the conditional distri- bution defined as
a xx())) Categorical(y/ Softmax(Lincaraa (hr)).
(15)
During the forward pass, the classifier takes the observed input 1 and the missing values gen- erated from the decoder po ( [=1)). We find beneficial to adopt the idea of GRU-D, where instead of directly putting the generated missing values a.. putting the decayed missing values as follows:
x; == (x) ohs mis where xi ~ (p
> [21:))
(16)
Lt. St + (1 (Ta, + (1 )ot.))
(17)
where Vols = exp(- max(0, W asd, + ba.)) is learnable decay. We find this stabilizes the learning when the generated missing values - are inaccurate, for instance, in the early stage of learning.
Note also the difference between (16) and the original GRU-D imputation (8). In GRU-D, the last
observed values are mixed with the mean feature, while ours mix them with the generated values.
Encoder qo (2 1:7 ( Given the generative model defined as above, we introduce the variational distribution for (a: (m:177> 21:T) for lower-bounding the liglliiel
98 o (1rr)
Z1:T ah pe ( 12:TT (a (Z1-7 (
(18)
Here, the encoder do 1:7 1.2) is defined autoregressive model as before,
T
q (21.7
II N (ZelMeen (z diag(o? (z t=1
(19)
1:T
Similar to the decoder, we use GRU to compute (Mone (x Tene (w for 1,.., T. However, since I; includes many missing values, rather than putting only the observed values, we find beneficial to put the imputed value & input to the encoder. For the imputation, we adopt GRU-D. To summarize, the encoder parameters are constructed from GRU-D outputs, with the inputs imputed with learnable decay values.
5

---

Under review as a conference paper at ICLR 2023
Objective
Having all the ingredients defined, the IWAE bound for supnotMIWAE is computed as
follows:
1
log px. (1. obs $1:T) 2 SappnotMTAAE 0.
E E EKA.T
log
Wk
(20)
K
where
Po
(3l 1.7 2 k.1:T Pas 11: 1:7: a mis
mis
1:T Po Po
ohs (
1:T
Wk
(21)
l (=
11
Here, (20(2R, 1.T (.rr)p )o ( are i.i.d. copies of the variational distribution, and denotes the expectation w.r.t. those i.i.d. copies.
B Mor L.R
3.2 OBSDROPOUT: REGULARIZING SUPNOTMIWAE FOR BETTER IMPUTATION
The problem with (20) is that there is no clear supervison for the missing values #1:2" Obviously, if we had an access to the missing values, the conditional probability Pe ( the model learn to correctly impute those missing values. Without such true values, we can only encourage the model to impute the missing values with some indirect criteria. In the objective (20), there are two terms that the model hinges on for this matter.
mis (= would guide
The missing model Pe (117 2..r) (177) this term encourages the model to reconstruct the missing mask 8t from the imputed value an so in principle, the model should impute the missing values in way that they are distinguishable from the observed values. However, in general, the distributions of the observed and the missings are not necessarily different, and more importantly, the model can easily cheat the objective. For instance. consider a trivial case where the model imputes all the missing values with zero. The conditional probability Pe ($1:1 ( a:mis can still be maximized by setting mii (11.5) = st. (unless there are not many observed with nhs 0).
The classifier poly)e p :: this term expects the model generate meaningful imputa- tions so that they are helpful for the classification. However, as shown in prior works (Che et al., 2018), the classifier can achieve decent classification accuracy without meaningful impu- tations, for instance, it will still be able to classify the signals while all the missing values are imputed with zeros. Hence, in the current form, there is no strong incentive for the model to learn non conviaivial imputations that will bring significant accuracy gain over the zero imputations.
To summarize, in the current form, the objective (20) is not likely to generate realistic missing values. To resolve this, we may introduce a missing model pe (1:77 obs mis much more claborated than the simple i.i.d. model that we using right now, but that may require some dataset-specific design. Instead, we present a simple regularization technique that can effectively enhance the quality of the imputed values.
Our idea is simple; when passing the observed inputs क and the imputed missing values ae (i.e.. imputed by (16)) to the classifier, deliberately drop some portion of the observed inputs. Without dropping the observed inputs, the classifier may heavily rely on the observed inputs to do the classi- fication, but if some of the observed inputs are dropped out during training, the classifier can focus more on the imputed missing values ini..... As a result, the model is encouraged to generate more "useful" missing values that are beneficial for classification. More specifically, let & be predefined dropout probability. Then we construct the imputed input ã; to the classifier as follows:
Mi,j Bern(1 - 13); St. == Mt,
ã == (x. mis.) where (xity miis po (x mis (
Gr.j = St.j + (1 $e, (Ta (1 Yok, ).,5)
(22)
That is, when an observed Tt,j is dropped out, we put generated value with the decay applied as in (16), so that the classifier could focus more on the values generated by the decoder as we intended. We call this idea OhsDropout, since are dropping out the observed values during the training.

---

Under review as a conference paper at ICLR 2023
encoder (=(urra1))))
decoder Pe(af a(h)=
obsdropout me Beru(1
classifier
pxtM (ur)
St
77
Ct
0.2
0.18
0.2
deGay
0.17
0
N/A
>>
GRU-D
Z1:T7
GRU
0.14
8
0.13
GRU
D.5
0.49
1
0.5
0.5
M/A
0.22
6
0.21
Figure An overview of our model with obsdropout.
With the mask variables mA.T, the likelihood is obtained by marginalizing mar from the joint distribution with m..r-
ff po, Po, wi sur, ma- marr)mm Jdmis
Pe, P (y, tb $1:)
(23)
where the joint distribution is decomposed as
obs Po,b.A F1:T; 81.T, m1::)
(y obs mis LTT #1:7" m1.7 (P ( ( (1:1//11))) poo (xft)-1 11.r )Po (2f (po)) (24)
Consequently, the IWAE objective slightly modified follows:
K
1
C Sa (A, (0,, == E K aa
log
Wk
(25)
K
1=
where
PA (y 1
mis k.i:: MA.1: (ap (81:7) lng 1 ax,1 )o (m I )oo (2
Wg
(26)
I (2k, (
where the expectation over K i.i.d. copies of the variational distribution,
g(Z1:21 aiier mi.r (x (0(21:1111)) obs )po (= (2
(27)
with (s(m):r) IT Bern(ms,)18).
3.3 PREDICTION
Similar to SupMIWAE, exploit Self-Normalized Importance Sampling (SNIS) to approximate the predictive distribution for new input obs With the model trained with obsdropout, we have
s
K
plyla M
ohe
(x)
P>(y to
K.T:T m m..r)
(28)
where
(z (e)
1:7, . (x mis (o)
1.T m k.
Rid. ~
go g0 (z w (ar
Wk po (x 2k. )po (2k,1:))
Wk
We
(29)
1sb 1kT (
Σk Wi
RELATED WORKS
There two lines of literature closely related to our method. First line consists of work dealing with the problem missing data imputation based on Importance Weighted AutoEncoder (IWAE) and another consists of work designing tailored neural network architecture for supervised learning on multivariate time series data with missing values.
Mattei and Frellsen (2019) proposed MIWAE bound for training DLVMs in the presence of missing data under MAR assumption. Ipsen et al. (2021) modified MIWAE bound suitable to MNAR see- nario. Ipsen al. (2022) extended MIWAE bound supervised learning task. This line of work

---

Under review as a conference paper at ICLR 2023
Table 1: Classification performances of baseline methods and ours on PhysioNet 2012 dataset.
Method
AUPRC: (1)
AUROCC()
(CF
bFCC(() (J)
(B)
CE(())
ECE (1)
BS (1)
GRU-Mean
0:524 =0.006
0.843 +0.05
0.498 +0.009
0.475 #0.016
0.164 +0.002
0.459
=0,020
0.200 +0.014
0.152 =0.008
GRU Siiple
0.426 20.012
0.810 20.008
0.621 10.062
0,400 20.042
0.201 20.017
0.450 =0.027
0.138 10.033
0,147 =0,011
GRU-Forward
0.526 00.007
0.853 10000
0:422 100006
0.476 10.015
0.157 10.002
0.450 20.025
0.200 1:0 10.025 025
0.147 =0.007
GRU-D
0.553 -0.004
0.863 +0.01
0.74 +0.010
0.473 +0.034
0.152 +0.004
0.434 =0.040
0.184 +0.033
00.41-
PTM
0.412 0.000
0.801
+0.022
0,333
+0.628
0.332 00.001
0.234 10.059
0,477
=0.142
0.121 10.036
0.132
=0.014
IP-Ners
0:527 =0.006
0.859
+0002
+0.014
0:453 +0.023
0.156 +0.00
0.416
=0.008
0.165 +0.028
0.136 00.000
Ours
0.557 =0.007
0.870
+0.00
0.466
+0.011
0.442
#0.011
0.152
#0.004
0.388
=0.033
0.153
+0.027
0.127
=0.012
w/o obsdropout
0,538
10.011
0.859
10.000
0.188
10.002
0.472
10.050
0.157 10.007
0.415
20.057
0.184 10.057
0,149 00.02
w/ MLP encldee
0.537 00.006
0.859 100002
0.495 +00087
0.445 +0.047
0.160 10.011
0,410 =0.044
0.157 +0.048
0.135 =0.016
Table 2: Classification performances of baseline methods and ours on MIMIC-III dataset.
Method
AUPRC (?)
AUROC (f)
bCE.
bЕCE (J)
bBS (1)
CE)
ECE
BS ())
GRU-Mean
0.469 10.006
0.841 200002
0.503 1:0004
0.490 1.0008
0.163 10.001
0.481
=0.010
0.226 10.007
0,161 =0.004
GRU-Simple
0,392
0,815
10.007
0.471 10:020
0.181 10.002
0.219
0,168 20.012
GRU--ar
0.519 -0.005
0,887 +0.002
0.471 +0.003
0.497 +0.011
0.155 +0.001
0.468
=0.018
0.230 +0.016
0.154 -0.000
GRU-D
0.498 -0.006
0.851 20:003
0.511
+0.011
0.465 10.019
0.161 40.002
0.438
=0.028
0,186 10.019
0.145 =0.110
PhasedLSTM
0.780 10.001
0.445 10.028
0.199 10:007
0.222 00.000
0,166
IP-Ners
0.486 -0.002
018477 +0.002
0.494
+0.010
0.478 +0.025
0.162 +0.003
0.458
+0.055
0.208 +0.024
0.153 -0.013
Ours
0.519 10.007
0.858 10.003
0.470
1.0005
0:490 10.007
0.155 10.000
0.157
20.013
0.217 10.010
0,152 20.005
wro obsdropouu
0.515
0.857
#0.03
0.492
00:016
0.156
+0.002
0.221 10.018
0.154 00,00
w/ MLP encidec
0.509 =0.007
0.857 +0.003
0.471 +00005
0.485 +00.08
0,155 +0.002
0.450
=0.011
0.217 +0.008
0.149=0.000
provides useful framework to train DLVMs under missingness. However, is not directly appli- cable for time series data because it cannot model the temporal dependency within series. There exists previous work to make Deep latent variable models suitable for multivariate time series. For example, Fortuin et al. (2020) proposed VAE architecture which aims to impute multivariate time series data, using Gaussian process prior to encode the temporal correlation in the latent space.
Researchers have developed deep neural network architectures customized multivariate time ries classification task. There have been several architectures showing competitive empirical per- formance in multivariate time series classification task. Che et al. (2018) modified the architecture of GRU intending to perform supervised learning with sparse covariates by introducing learnable temporal decay mechanism for input and hidden state of GRU. This mechanism has been applied to further research. For example, Cao et al. (2018) employed temporal decay in hidden states of their bidirectional-Re RNN-based model to capture the missing pattern of irregularly sampled times se- ries. Shukla and Marlin (2019) presented hybrid architecture of interpolation network and classifier. Interpolation network returns fully observed and regularly sampled representation of original time series data. Taking this representation as an input, even common deep neural network model makes good predictive performance.
EXPERIMENTS
In this section, we demonstarte our method on real-world multivariate time series data with missing values. We compare ours the baselines on three datasets: PhysioNet 2012 (Silva et al., 2012), MIMIC-III (Johnson et al., 2016) and Human Activity Recognition (Anguita et al., 2013). 3). PhysioNet 2012 and MIMIC-III datasets contain Electronic Health Records of patients from Intensive Care Units (ICU). Human Activity Recognition dataset consists of the 3D coordinate of sensors mounted on the people doing some daily activities such as walking, sitting. See Appendix for the details of datasets. For all three datasets, we compare classification accuracy and the uncertainty quantification performances. For PhysioNet 2012, we also compare the missing value imputation performance of methods the baselines.
For the baselines, we considered GRU classifiers with various imputation methods, and few other deep neural network based methods that are considered to be competitive in the literature. See Appendix A for detailed description of the baselines. For the uncertainty quantification metrics, we compared cross-entropy (CE, equals negative log- ligeliiol,,,,d,, expected calibration error (ECE), brier score (BS). Especially, for PhysioNet 2012 and MIMIC-III, we also considered balanced versions of them (with "b" in front of the metric names), since those datasets are largely imbalanced so the
8

---

Under review as a conference paper at ICLR 2023
Table 4: Imputation performance on
Table 3: Classification performance of baseline methods and ours on Human Activity Recognition dataset.
PhysioNet 2012 dataset.
Method
Method
(TC
(C)
BCE (P)
BS (J)
Moun
0.696 20.001
0,994 =00000
GRU-Mean
0,780 +0.005
0.163
+0.0121
0.019
=0.006
0.046 +00002
Forward
0.399 10.001
0.574 =0.000
GRU Simple
0.767 10.008
0.161 10.0029
0.015 -0.003
0.047 10.001
GRU-D
0.406 1:0000
0.586 =0.001
GP-VAE SAITS
GRU--Forwrd
0.798 =0.007
0.152 #0 0038
0.020
00.003
0.043
=0.001
0.694 10.001
0.995 =0.000
GRUAD
0.789 +0.004
0. 0.150 +0.0046
0.018
00.044
0.044
+0.001
Ours
0,391 1:0.002
0.564 =0.002
Ours
0.798 10.004
0.141 10.0028
0.005
10.001
0.042 1.0.01
wio supervision
0.400 1:0.002
0.573 =0.009
obsurooouu
03
03
0.
-28
Figure 2: Plots of ddec 21:t), d d (= (left) Our model which architecture of encoder and de- coder MLP. (right) Our model trained with obsdropout with rate 0.5.
usual uncertainty quantification metrics may be biased. Please refer to Appendix A for the detailed description of the metrics.
5.1 CLASSIFICATION RESULTS
We summarize the classification results in Table 1, Table 2, and Table 3. general, ours acheive the best performance among the competing methods both in terms of prediction accuracy and uncer- tainty quantification. We also provide an ablation study for our model to see the effect of 1) using time-aware architecture (GRU) for the encoder and decoder of supnotMIWAE, and 2) obsdropout. The results clearly show that both components play important roles for our model. In Appendix B, we provide further results showing the effect of dropout rate 3 for the performance.
5.2 IMPUTATION RESULTS
We quantitatively check the imputation performance of our model on PhysioNet 2012 dataset in Ta- ble 4, and visually check the imputation quality by changing our model settings in Fig. 2. Although our model is designed for the classification, ours achieved the lowest MAE and MRE, outperforming the baseline (SAITS) specifically designed for the imputation. Especially, the ablation study on the class supervision part Px (y) #1:T) and the obsdropout implies that the imputation values generated by model which was trained to better classify the signals more "realistic". Fig. 2 highlight the effect of using GRU based encoders and decoders and obsdropout. The values imputed with those techniques form smoother trajectories and better capture the uncertainties in the intervals without observed values.
6 CONCLUSION
In this paper, we presented novel probabilistic framework for multivariate time series classifica- tion with missing data. Under the MNAR assumption, we first developed deep generative model suitable for generating missing values in multivariate time series data. Then we identified an im- portant drawback the naïve combination of the deep generative models with the classifiers and proposed a novel regularization technique called obsdropout to circumvent that. We demonstrated that ours could classify real-world multivariate time series data more accurately and robustly than existing methods. In this paper. we focused GRU-based architectures for both generative model and classifier. An interesting future work would be extending our methods with other architectures such transformers (Vaswani al., 2017).

---

Under review as a conference paper at ICLR 2023
Reproducibility statement Please refer to Appendix A for full experimental detail including datasets, models, and evaluation metrics.
REFERENCES
D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. Reyes-Ortiz. A public domain dataset for human activity recognition using smartphones. 2/th European Symposium on Artificial Neural Networks, 2013. 8
Y. Burda, Grosse, and Salakhutdinov. Importance weighted autoencoders. arXiv preprint 2015. 3
W. Cao, Wang. J. Li, Zhou, L. Li, and Y. Li. BRITS: bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems 3/ (NeurIPS 2018), 2018. 8
Z. Che, S. Purushotham, Cho, D. Sontag, and Y. Liu. Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 8(1), 2018. 1, 2, 4, 8,8
K. Cho, B. van Merrienboer, G. Gulcehre, Bahdanau, Bougares, H. Schwenk. and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing (EMNLP), 2014. 4
W. Du, D. Cote, and Y. Liu. SAITS: self-attention-hant -based imputation for time series. arXiv preprint orXiv:2202.085/6 16, 2022. 1, 15
V. Fortuin, Baranchuk, G. Raetsch, and Mandt. GP-VAE: Deep probabilistic time series imputation. In Proceedings of The 23rd International Conference on Artificial Intelligence and Statistics (AISTATS 2020), 2020. 8, 15
H. Harutyunyan, H. Khachatrian, D. C. Kale, G. Ver Steeg, and A. Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):1-18, 2019. 12
M. Horn, M. Moor, C. Bock, B. Rieck, and K. Borgwardt. Set Functions for Time Series. In Proceedings The 37th International Conference on Machine Learning (ICML 2020), 2020. 12
N. B. Ipsen, P. Mattei, and J. Frellsen. not-MIWAE: deep generative modelling with missing not at random data. In International Conference on Learning Representations (ICLR), 2021. 2, 3,
N. B. Ipsen, P. Mattei, and J. Frellsen. How to deal with missing data in supervised deep learning?
In International Conference on Learning Representations (ICLR), 2022. 3,7
A. Johnson, T. Pollard, and R. Mark. MIMIC-III clinical database (version 1.4), 2016. URL sttps://don orgg10. httgg//....////2/22... 13026/
D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations (ICLR), 2014.
R. Little and D. Rubin. Statistical analysis with missing data. Wiley, 2002. 2
Mattei and J. Frellsen. MIWAE: deep generative modelling and imputation of incomplete data sets. In Proceedings of The 36th International Conference on Machine Learning (ICMI. 2019), 2019. 2, 3,
Y. Rubanova, R. T. Q. Chen, and D. Duvenaud. Latent ODEs for Iregularly-Sampled Time Series. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019. 12, 13
H. Schütze, C. D. Manning, and P. Raghavan. Introduction to information retrieval, volume 39.
Cambridge University Press Cambridge, 2008. 15
S. N. Shukla and M. Marlin. Interpolation-Preete Prediction Networks for Irregularly sampled Time Series. In International Conference on Learning Representations (ICLR), 2019.
10

---

Under review as a conference paper at ICLR 2023
1. Silva, G. Moody, D. J. Scott, L. A. Celi, and R. G. Mark. Predicting in in-hospital mortality of patients in icu: The physionet/computngg in cardiology challenge 2012. Computing in Cardiology, 2012. 8
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo- sukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017.

---

Under review as a conference paper at ICLR 2023
Table 5: Statistics of each dataset.
PhysioNe:2012
MIMIC-III
Human Activity
Number of samples
11,971
21,107
6554
Number variables
16
Maximum numher of time steps
215
292
221
Mean of number of time steps
74
78
162.4
Total missing rale
0.843
0.655
0.923
A EXPERIMENTAL DETAILS
A.1 DATASETS
A.1.1 DATASET DESCRIPTION
PhysioNet Challenge 2012 (PhysioNet 2012) This dataset contains approximately 12,000 Hlec- tronic Health Records of adult patients who were admitted intensive care unit (ICU). Each record contains to 37 time series variables including vital signs such as heart rate, temperature. All of the variables are measured during the first 48 hours of each patient's admission ICU and the sampling rate of times series varies among variables. After preprocessing, we have 37 features and 11, 1,971 data point. On this dataset, we conduct missing data imputation task and mortality prediction task, which aims to predict in-hospital mortality of ICU patients using information collected during first 48 hours ICU.
MIMIC-III MIMIC-III dataset is frrely accessible and widely used database which includes de- identified Electronical Health Record of patients who stayed in ICU of Beth Israel Deaconess Med- ical Center from 2001 to 2012. It originally consists of approximately 57,000 records of patients stayed in ICU. Records including various variables such as medications, in-hospital mortality and vi- tal signs. Harutyunyan al. (2019) set variety of benchmark tasks based subset of this database. Among them, we conduct binary in-hospital mortality prediction task. After preprocessing, our dataset contains 16 features and 21,107 data points. For this dataset, we conduct mortality predic- tion task which is identical to PhysioNet2012 classification task.
UCI Localization Data for Person Activity (UCI Human Activity) This dataset includes records of five people doing some usual activities such as walking or sitting. All people wear sensors on their right ankle,left,bell and chest. During activities, and the sensors record their position in the form of three-dimensional coordinates at very short intervals. Activities of each people at certain time point classified into one of 11 classes and recorded with the position of sensors. After preprocessing, we have total 6554 time series with 12 preprocessed data, we conduct maline-prediction task. The objective of this task is to classify each individual's 's activity per time point based on the position of sensors.
coordinates of 4 devices). Using this
A.1.2 DATA PREPROCESSING
For all dataset, we basically standardize the numerical covariates so that all features have zero mean and unit variance, respectively.
PhysioNet2012 and MIMIC-II Since there is no fixed rule for preprocessing Physionet2012 and MIMIC-III database, researchers usually preprocess the raw data on their own so that there are countless possibilities for the form of preprocessed dataset. Therefore, it is difficult for practitioners to compare experimental results with other works. For the comparability, we employ python package medical-tss datassssss (Horn et al., 2020) which provides the unified data preprocessing pipeline for Physionet2012 and MIMIC-II III datasets. For both dataset, patients who have more than 1000 time steps or have no observed time series data were excluded from the dataset. Also, discretizing the time step of data by hour and aggregate the measurement is frequently used to preprocess Physione12012 in previous work (Rubanova et al., 2019), but this package preserves much more original time series variables while preprocessing than hourly based aggregation preprocessing for both dataset. We follow the preprocessing of the medical. ts_dat as..tsssssss library.
datasets
12

---

Under review as a conference paper at ICLR 2023
Table 6: The number parameters of baseline models and our model on each dataset.
Method
PhysioNet 2012
MIMIC-III
Human Activity
GRU-Mean
226,817
210,689
84,647
GRU-Simple
351,745
301,569
85,117
GRU-Forward
294.145
276,225
85,687
GRU-D
332.363
292,897
86,566
PhasedLSTM
511.745
445,441
IP-Nets
353,151
301,841
Ours
262,980
224,305
84,251
UCI Human Activity For comparability, we decide to preprocess this dataset based on (Rubanova al., 2019). However, we modify some part of dataset apply to our code implementation. We standardize the dataset while (Rubanova et al., 2019) did not standardize the dataset. Also, we make new variable that records the time index of lastly observed value of each data point to distinguish between missingness and meaningless padding.
A.2 DETAILS FOR CLASSIFICATION EXPERIMENTS
For all experiments, we five different seeds to conduct experiments.
A.2.1 BASELINE METHODS
GRU-mean: Missing value is simply replaced with the empirical mean of each variable.
GRU-forward: Missing entries are filled with previously observed value.
concatenate the mask 8t, and the time-interval 8, along with the imputed vector it The concatenated vector [ 8t, 6e] is then fed into GRU.
GRU-simple:
Missing values are imputed as weighted mean of the last observed æes and the mean &, with the learnable weight.
GRU-D:
Phased-LSTM: This model is LSTM variants designed to deal with long sequence input by introducing time gate in their cell to prevent memory decay when useful information is absence for a long time.
Interpolation- Predicction Network(IP-Netss Instead of directly imputing missing values, this model employed semi-parametric interpolation network that makes regularly spaced representation of irregularly sampled time series data. Then, this representation is fed into prediction network such as GRU.
Since we conduct online- preddicion task on Human Activity dataset, we do not consider IP-Nets as baseline models because this model use future information when conducting interpolation.
A.2.2 TRAINING DETAILS
In order to conduct experiments fairly, we fix the parameters of every model similar, or at least set our model have relatively small number of parameters compared other baseline. Also, we use Adam optimizer with learning rate 0.0001 and batch size 128 for all models for MIMIC-III in-hospital mortality prediction task. In Physionet2012 experiment, we use Adam optimizer with learning rate 0.001 and batch size 128 for all models. For online-prediction task, we also employ Adam optimizer with learning rate 0.001 and batch size 128.
We employ early stopping for all classification experiments. For the mortality prediction tasks, set early stopping patience to 10 epochs and set the area under the precision-recall curve (AUPRC) of the validation data the early stopping criterion. For Human activity prediction task, used validation accuracy as stopping criterion and set early stopping patience to 20 epochs. Since the label imbalance of Physsinet2012 and MIMIC-III is extreme, oversample the mortality class to train models on the balanced batches.
13

---

Under review as a conference paper at ICLR 2023
Table 7: Hyperparameter settings for classification experiments
Dataset
Method
Hyperparameters
GRU-Mean
n_umits: 256, dropout: 0.0, recurrent,dropout: 0.0
GRU-Simple
n_units: 256, dropout: 0.0, recurrent eccurret_drroouut 0.0
GRU-Forward
n.units: 256, dropout: 0.0, droppott 0.0
GRU-D
n_units: 256, dropout: 0.0, accurrent_dropoude 0.0
PhysioNei2012
PSasedLSTM
n.units: 256, use-peepholes: False, leak: ptt.ppediiiittiiaii... 1000.0
IP-Nets
n.units: 256, imputation stepsize: reconst fraction: 0.5
n_train Jatents: 10, n_train_samples: n_jestJaatent:: n_lest_samples: 30,
Ours
hidden: 128, z.di:: 32, uniits:: 128, observe observe.dropout 0.5
GRU-MMca
256, dropout:
accurrent.ropouut:
dropout:
GRU-Simple
units: 256, dropout:
dropout:
GRU-Forward
n.units: 256, dropout:
recurrent.rroourt
GRU-D
n_umits: 256, dropout:
sccureet.dropout: 0.0
MIMIC-III
Phasedl STM IP-Nets
n_units: 256, use see_peepholes: False,
period_jnir_maax 1000.0
n units: 256, imputation stepsize: reconst. fraction:
n.train latents: n.train sumples:: 1, n.test w.test.lateeeen latents:
samples: 30,
Ours
n hidden: 128, z dim: 32, n units: 128, observe dropout:
GRU-Mean GRU-Simple BRRU-Forwand
n_umits: 160, dropout: 0.0, sourrent_drrpour 0.0
n.umits: 132, ,2 dropout: 0.0, ecrurent.dropout: 0.0
n_umits: 140, dropout: 0.0, recureet serurent_drpout: 0.0
Human Activity
GRU-D
n units: 135, dropout: 0.0, recurrent drppout 0.0
n_train latents: 10, w.ttain..umplle I, n.test 10, simples 10, n hidden: 128, z z.dim: 32, n. units: 128, observe dropout: 0.2
Ours
A.2.3
HYPERPARAMETERS
See Table and Table 7 for hyperparameter settings of our model and baseline methods for all classification experiments.
A.2.4 EVALUATION METRICS
For the classification task, we evaluate all models in terms of both predictive accuracy and predictive uncertainty. We use the area under precision recall curve (AUPRC), the area under receiver oper- ating characteristic (AUROC), and the accuracy (ACC) to evaluate the predictive performance. To measure the uncertainty calibration of the model, we use cross entrophy (CE), expected calibration error (ECE) and brier score (BS) for comparing calibration. In addition, also check the balanced versions of uncertainty metrics due to severe class imbalance of datasets.
Balanced metric supervised dataset D, which contains input data æ and corresponding label we simply re-weight each uncertainty metric by class ratio.
Mhal (D) = - Σ M(D))
(30)
Here, D. is subset of the dataset 'D which only contains the label
Accuracy Metrics Accuracy metrics are defined using the following terms, where tp, tn, fn, and fp denote true positive, true negative, false negative, and false positive respectively.
tp + in
accuracy tp fp+ + fn + tn
(31)
tp
precision
(32)
tp + fp
4p
recall
(33)
tp + fn
tp
sensitivity = tp + fn
(34)
14

---

Under review as a conference paper at ICLR 2023
Table 8: Hyperparameter settings for imputation experiments
Dataser
Mathod
Hyperpurametons
Mean
Forward
units: dropout: recureent dropout:
n.irain Intents: 10, n. train samples: n.lest
units: lssentt Insentc:
n. n.test.smpples 30,
Ours w/ obudropout
PhysioNet2012
128,
dropout
Ours wio supervision
train latents: trin samples: I. lncets: 20, n. samples: 30,
hidden: 12K. Jiim 32, n.mmis: 128.
obstropout
samples:
Insents:
samples: 30,
nidddeen: 128, z.lime 32, n.amis: 128, deserve_drspout 0.0
GP-NAE
kernel scale: I. Libeta: n. 5.grropee 5. n.heul n. n.group inner.lyyrrss 255 2im.rr imput with mask: Trae, MIT missing rate012, units: 128. a hidden:126
hiddem: 128, dim: 35,
SAITS
AUPRC (Schütze et al., 2008): area under precision recall curve.
AUROC: area under receiver operating characteristic, area under sensitivity curve.
A.3 DETAILS FOR IMPUTATION EXPERIMENTS
Basically, we perform imputation experiments on the testdataset of Physionet2012. We randomly delete 10% of observed data for testing imputation performance of models. We measure the perfor- mance for five different seeds.
A.3.1 BASELINE METHODS
Mean: Replace missing values with global mean.
Forward: Impute missing value with previously observed value
Missing values are imputed as weighted mean of the last observed æv and the mean T, with the learnable weight.
GRU-D:
GP-VAE: This model VAE-based probabilistic imputation method proposed by Fortuin et al. (2020). This method employ GP-prior to encode the temporal correlation in the latent space.
SAITS:
This model is self-attention based imputation model which Du et al. (2022) pro- posed.
Ours w/o supervision: To analyze the effect of supervised signal to the imputation perfor- mance, we just remove supervised term from training objective and train only genera- tive part of our architecture.
Ours w/o dropout: Our model without obsdropout.
A.3.2
EVALUATION METRICS
For the imputation task, evaluate all methods in terms of MRE(Mean Relative Error) and MAE(Mcan Absolute Error).
A.3.3 TRAINING DETAILS
For GRU-D and Our model, we apply early stopping rule identical mortality prediction task since imputation performance of our model is deeply related to classification error. For GP-VAE,SAITS and Our model w/o supervised signals, do not early stopping and train for 50 epochs for each model.
A.3.4 HYPERPARAMETERS
15

---

Under review as a conference paper at ICLR 2023
Table Effect of obsdropout rate for ours on PhysioNet 2012 dataset.
Drapout
AUPRC (?)
AUROC (f)
(CF())
MECF (U)
(1)
CE())
BCB()
(1)
0.0
0.538 10.011
0.859 =00006
0.488 10.032
0.472 =0.050
0.157 10.007
0.445
=00057
0.184 1.0057
0.149 =0.021
0.1
0,545 20.014
0.862 10.007
0.473 10.023
0.474 =0.023
0.153 10.006
0.437 20.023
0.184 20.029
0.146 =0.008
10.005
10.025
20.033
0.154
10.037
0.161
0.138
0.3
0.554 20,008
0,868 00.001
0.464 10.006
0,477
=0.039
0.149 10.002
0,434 =0.056
0.181 10.040
0.145 =0.021
0.4
0.538 10.005
0,868 1.00066
0.470 10.014
0.452 20.026
0.152 1.00004
0.401 1.0.03
0.155 10.027
0.132 10.013
0.871
0.462
0.149
0,400 =0.025
0.132
0.6
10.005 0.556
0.869 00.002
0.458 10.008
0.474
10.021
0.149 100002
0.425 00.027
0.179 10.018
0.141 10.011
0.7
0.554 +0.014
0.867 +0.00
0.462 +0.013
0.471
00002
0.151 #0.001
0.425 02.029
0.179 +20022
0.140 011
0.8
0.558
10.010
0,864 00.009
0.472 10.031
0,489 20.03
0.153 10.008
0.455 00.02
0.194 1:0053
0.152 =0.012
0.9
0,546
+0.010
0.859 00000
0.469 +0.015
0.486 0000
0.154 +0.003
0.453 023
0.1 0.198 +0.020
0.152 =0 009
Table 10: Effect of obsdropout rate for ours on MIMIC-III dataset.
Drapout
AUPRC (?)
AUROC (f)
PCF
bECE
bBS (1)
CE.()
ECB
BS (L)
0.857
0.492
0.156
10.023
0.221 10.018
0.154
10.000
0.1
0.519 +0.007
0,858 +0.003
0.470 +0.005
0.490 -0.007
0.155 +0.002
0.457 +0.013
0.217 +0.010
0.152 -0.003
0.2
0.519 10.008
0,839 100003
0.468 10.004
0.501 10.000
0.155 100001
0.472 10.014
0.228
1.0000
0.158
_
0.3
0.518
+0,004
0.859 +0.002
0.469 +0.002
0.494 +0.008
0.155 +-0.001
0.462 +000013
0.220 +00009
0.154
-0.005
0.4
0,518
10.004
0,858
10.003
0.470 10.003
0.493 10.015
0.155 100001
0.461 10.025
0.220 100017
0.153 10.009
+0.010
0,855 +00.03
+0.010
0.157
+00.013
+-0013
0.152 =0.005
0.6
0.504
10,010
0,851 10.003
0.483 10.005
0,501 20.011
0.160 10,002
0,488 10.01
0.229 1.0010
0.163 10,006
0.7
0.480
+0.036
0,840 00.017
0.499 +0.020
0.497
00.010
0.166 #0.007
0.495 =0.00
0.236
+0.024
0.163 =0 006
0,171
0.235
0.163
0.9
0.419 +0,024
0,810 00.01
0.543 +0.011
0,496 029
0,181 +0.001
0,534 0000
0.268 +0.030
0.174 =0.019
See Table 8 for detailed hyperparameter settings for imputation experiment.
B ADDITIONAL EXPERIMENTS
We conduct numerous ablation experiments to analyze the effect of obsdropout method. In Phys- ioNet2012 dataset, our models with obsdropout method outperforms at all rates. Also, in MIMIC-III datset, our technique works well reasonable rates. Although there is much room for further anal- ysis of effect of obsdropout on the predictive performance, these results are sufficient to show that our technique is generally effective.
16


Reviews:
[{"review_id": "vychxsf5vh", "reviewer": "Reviewer_For5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on imputation and prediction for time series with missing data. It extends an existing deep generative model to impute the missing values in the time series and make predictions given both the observations and imputations. A regularization method \"obsDropout\" is proposed to randomly drop some observed entries to force the prediction model to focus more on the imputed entries to drive the generation of good imputations.", "summary_of_the_review": "Overall, this paper presents some interesting extensions of the existing deep generative model for time series missing data imputation. The formulations are sound. However, some methodological design needs stronger justification and the empirical evaluation seems weak.", "strength_and_weaknesses": "Strengths:\n- The proposed model is reasonable and sound. Detailed analysis of the problem as well as existing work is provided.\n- Although the proposed supnotMIWAE is largely a variant of supMIWAE, it adds new elements to tackle the MNAR and prediction and the formulation is overall sound.\n- Overall, the paper is organized and easy to follow.\n\nWeaknesses:\n- My main concern regarding the method is that the prediction task may not be enough to drive the learning of imputation, especially MNAR. Even with the obsDropout regularization incorporated, the model only generates imputation that helps most the prediction but may not be the most accurate imputation.\n- The empirical evaluation, compared against existing methods like GRU-forward and GRU-D, seems weak. The improvement over the SOTA baselines is quite marginal.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nEwjpTNwa3", "reviewer": "Reviewer_1NJg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a deep generative model for multivariate time series with missing data (in the most general setup of missing-not-at-random, MNAR). The development of the model follows the same core design principles presented in [1], [2], and [3], extending it and proposing a joint hybrid model for the time series, the missing mechanism, and a classifier. The inference of such a model can be intractable in general, thus an approximation for the log-likelihood is proposed, adapting the Importance Weighted AutoEncoder (IWAE) lower bound for the proposed model (which defines the loss function to be optimized). Furthermore, it is proposed a regularization for this lower bound by dropping more portions of the observations, encouraging imputations that contribute towards the classification tasks. The empirical validation is done using multiple datasets and comparing with multiple baselines and state-of-the-art.\n\n[1] P. Mattei and J. Frellsen. MIWAE: deep generative modelling and imputation of incomplete data sets. ICML 2019.\n[2] N. B. Ipsen, P. Mattei, and J. Frellsen. not-MIWAE: deep generative modelling with missing not at random data. ICLR 2021.\n[3] N. B. Ipsen, P. Mattei, and J. Frellsen. How to deal with missing data in supervised deep learning?. ICLR 2022.\n\n\n", "summary_of_the_review": "Overall, this a solid and well-motivated work, tackling an relevant problem (NMAR imputation for time-series). My only concern is the limited technical novelty. That being said, I recommend acceptance.", "strength_and_weaknesses": "## Strength\n* The modeling choices and inference method is well motivated and principled\n* The problem of imputation with NMAR in the multivariate time-series setup with a hybrid deep generative model is novel and potentially impactful\n* Empirical results demonstrate the strength of the proposal\n\n## Weakness:\n* The methodology is incremental\n\ntypos:\nIrrelvent -> irrelevant (page 3)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Cw4yNQHsODT", "reviewer": "Reviewer_wRvk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors suggest a sequential VAE/IWAE framework for imputation and classification of time series data that is particularly applicable\n under the missing not at random (MNAR) setting. To improve the imputation performance, the authors suggest to mask some observed values and subsequently reconstruct them. The method is compared agains imputation approaches using GRUs that are not fully probabilistic. \n\n\n---\nUpdate after the authors' response:\nThe authors have added an additional probabilistic imputation benchmark model which has been found to be outperformed by the approach suggested here. This improves the empirical validation of the new approach.\n\n---", "summary_of_the_review": "The proposed modelling setup is new and addresses a relevant question that has not yet been addressed previously. The model (encoder, decoder) is somewhat standard. Including the missingness model is new, as well as the dropout procedure, along with the variational bound. The empirical performance appears favourable compared to imputations with GRUs, but in my opinion, a more detailed empirical comparison with other probabilistic imputation approaches would improve the paper.", "strength_and_weaknesses": "Positives:\n- The paper proposes a probabilistic model for time series data that allows for very flexible assumptions for the missing values. Imputation for the time series setting/for non-linear latent state-space models has not been explored in the literature, as far as I am aware. \n- The method appears to yield better empirical performance on two benchmark datasets, compared to imputation approaches using GRUs.\n\nNegatives:\n-There appears to be a lack of empirical comparison to other probabilistic imputation approaches, such as Fortuin et al., 2020. Even if such approaches make strong assumptions on the missingness patterns, it would be insightful to see how this translates into the empirical forecasting performance.\n- The missingness model appears relatively simple compared to the other components (encoder, classifier) in that it does not depend on the full history of the observations via the RNN latent state. A more flexible model should not require 'data-specific design' choices. \n\nComments/Actionable feedback:\n-For the suggested bound with the dropout variables in (22), does this constitute a lower bound on something sensible?\n- My feeling is that even without dropout, there should be an incentive to learn non-trivial imputations, because I am not only minimising the classification error, but also the log-probability of the observed time series.\n- A semi-supervised approach to regularise the missing data has been suggested in Sahra Ghalebikesabi, Rob Cornish, Chris Holmes, Luke Kelly: Deep Generative Missingness Pattern-Set Mixture Models. AISTATS 2021. I was wondering how this relates to the dropout procedure suggested here?\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4SSxSZuu9J", "reviewer": "Reviewer_ssPS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "THis paper tackles time series classification problems with missing data. A deep generative model is used to impute missing data first before classification. The classifier is trained to capture the predictive uncertainty due to the multiple possibilities of imputations. A novel regularization technique is proposed that can promote the model to produce useful imputation values that actually\nhelp classifcation. ", "summary_of_the_review": "Can't handle irregularly sampled data well, improvement is marginal", "strength_and_weaknesses": "Pros:\n1. This paper is probabilistic and uncertainty is considered for classification\n2. The regularization techniques makes sense\n\nCons: \n1. Several recent works have shown that GRU-D is not good at handle irregularly sampled data compared to ODE based methods, is it possible to extent this paper to handle irregularly-sampled data well?\n2. In Table1 and Table2, the improvement over GRU-D is marginal, so I am not sure this method is as effective as claimed.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject