Conditional Information Bottleneck Approach for Time Series Imputation
Authors: MinGyu Choi
Changhee Lee
Venue: ICLR 2024 poster (2024)
Keywords: time series imputation; time series interpolation; information bottleneck
Primary area: general machine learning (i.e., none of the above)

Abstract: Time series imputation presents a significant challenge because it requires capturing the underlying temporal dynamics from partially observed time series data. Among the recent successes of imputation methods based on generative models, the information bottleneck (IB) framework offers a well-suited theoretical foundation for multiple imputations, allowing us to account for the uncertainty associated with the imputed values. However, directly applying the IB framework to time series data without considering their temporal context can lead to a substantial loss of temporal dependencies, which, in turn, can degrade the overall imputation performance. To address such a challenge, we propose a novel conditional information bottleneck (CIB) approach for time series imputation, which aims to mitigate the potentially negative consequences of the regularization constraint by focusing on reducing the redundant information conditioned on the temporal context. We provide a theoretical analysis of its effect by adapting variational decomposition. We use the resulting insight and propose a novel deep learning method that can approximately achieve the proposed CIB objective for time series imputation as a combination of evidence lower bound and novel temporal kernel-enhanced contrastive optimization. Our experiments, conducted on multiple real-world datasets, consistently demonstrate that our method significantly improves imputation performance (including both interpolation and extrapolation), and also enhances classification performance based on the imputed values.

Published as a conference paper at ICLR 2024
CONDITIONAL INFORMATION BOTTLENECK APPROACH FOR TIME SERIES IMPUTATION
MinGyu Choi
Changhee Lee Chung-Ang University, Korea changheelee@can kr
Massachusetts Institute of Technology, USA chemgyuêmi @m e
ABSTRACT
Time series imputation presents significant challenge because it requires captur- ing the underlying temporal dynamics from partially observed time series data. Among the recent successes of imputation methods based generative models, the information bottleneck (IB) framework offers a well-suited theoretical foundation for multiple imputations, allowing us to account for the uncertainty associated with the imputed values. However, direct application of IB framework to time series data without considering their temporal context can lead a substantial loss of temporal dependencies, which, in turn, can degrade the overall imputation performance. To address such a challenge, we propose a novel conditional information bottleneck (CIB) approach for time series imputation, which aims to mitigate the potentially negative consequences of the regularization constraint by focusing on reducing the redundant information conditioned on the temporal context. We provide theoreti- cal analysis its effect by adapting variational decomposition. We use the resulting insight and propose novel deep learning method that approximately achieve the proposed CIB objective for time series imputation as a combination of evidence lower bound and novel temporal kernel-enhanced contrastive optimization. Our experiments, conducted on multiple real-world datasets, consistently demonstrate that our method significantly improves imputation performance (including both interpolation and extrapolation), and also enhances prediction performance based on the imputed values.
INTRODUCTION
Multivariate time series data often includes missing features, with diverse missing ratios and patterns depending on distinct sampling periods or measurement strategies (Johnson et al., 2016). Since these missing features significantly impair the performance of downstream tasks and comprehension of the temporal dynamics, time series imputation, which aims to reconstruct the missing features, has become pivotal and pervasive topic across numerous practical domains, including healthcare, environmental science, and various other fields. What makes time series imputation challenging is that imputation method must satisfy two essential requirements: i) it must account for underlying temporal dependencies, and ii) it should allow for multiple imputations to facilitate uncertainty quantification for real-world decision-making.
Generative models, particularly variational autoencoders (VAEs) (Kingma & Welling, 2014), have been employed in the context of multiple imputation tasks due their capability generate samples imputation methods primarily focus on defining the evidence
in probabilistic manner. lower bound, where the reconstruction error is computed only over the observed part of the incomplete data (Sohn et al., 2015; Nazabal et al., 2020). These methods can be naturally interpreted under the information bottleneck (IB) principle (Tishby & Zaslavsky, 2015), providing an iiformatioon-hhoreeee understanding of what constitutes imputation-relevan representation. This understanding is based on the fundamental trade-off between maintaining concise representation (i.e., regularization) and preserving good representation power (i.e., reconstruction) (Voloshynovskiy al., 2019).
However, a direct application of the IB principle to time series imputation struggles with capturing the underlying temporal dependencies. Our motivating examples in Figure 1(B) show that imputation methods trained with the conventional IB framework lose a significant amount of information about temporal dynamics relevant for imputing missing values. In this paper, theoretically analyze that

---

Published as a conference paper at ICLR 2024
the overly striet regularization in the conventional IB may force the encoder to rely solely the observed features at a particular time point, rather than learning the underlying temporal dependencies from the remaining observations from other time steps. To overeome such issue, we propose novel conditional information bottleneck (CIB) framework for time series imputation. Our framework structure of the IB principle while preserving temporal
adopts the information through conditional regularization, allowing us to circumvent the strict regularization constraints of the conventional IB. Throughout the experiments conducted on multiple real-world datasets including image sequences, weather measurements, and electrical health records, our pro- posed method consistently outperforms the of imputation methods with respect both imputation performance and prediction performance based on the imputed values.
2 PRELIMINARIES: INFORMATION BOTTLENECK APPROACH TO IMPUTATION
In this section, we first formally describe the information bottleneck (IB) principle (Tishby & Zaslavsky, 2015; Alemi al., 2017), which provides an information-thhoom understanding of what a task-relevant representation is in terms of the fundamental trade-off between having a concise representation and good representative power. Then, we present generative model for imputing missing features under the IB principle.
Let X and Y be random variables for the input feature and the target label, respectively. The IB principle aims to find the bottleneck random variable Z that compresses the information in X while keeping the information relevant for predicting Y as the following (Tishby & Zaslavsky, 2015),
min 0.8
T [2 x) (Y;
where E is Lagrangian multiplier that balances the two mutual information terms, and and 0 correspond to learnable parameters that define probabilistic mappings ( (Z and go (Y Z). respectively. The core motivation of (1) to find the optimal distribution of latent representation Z and the corresponding inference model parameters & that removes label-irrelevant information from X while preserving the information about the class label Y.
This offers information thheorctic perspective generative model-based imputation methods which generate missing observations from the observed features.
Definition 1. (Imputation) Let X and XT
T be random variables for the partially observed features and missing features of X. respectively, such that X = X U " Then, we define imputation as an unsupervised IB as follows:
min 1a (Z; 3la(X: 0,0
(2)
where B &Ry is a Lagrangian maltiplier, and á and 0 correspond to learnable parameters that define probabilistic mappings go (o)ZZ) x") and go [X] Z), respectively.
The above definition in (2) aims at finding the distribution of latent representation Z and the corre- sponding parameters é that preserves the core information for accurately reconstructing the (complete) original input X while suppressing redundant information from its incomplete observation, X
METHOD
3.1
PROBLEM FORMULATION
We consider a general temporal dynamics setting in which each instance of (discrete) time series input comprises sequence of measurements (i.e., observations), denoted as x1:7 def [ XT), collected during the time interval (r). Here, X, Rd is the complete input vector measured time [Tt-1, Te) and is realization of a random variable X,.' However, in practice, time series data often contains missing features with arbitrary patterns such that z is not observed during [T--1, Te) for any feature 1 E {1,..., d} any time step E (1...., T). This phenomenon is particularly common in domains such as healthcare (Johnson et al., 2016) where each feature may
'Throughout the paper, will often use upper-case letters to represent random variahles and lower-case letters represent their corresponding realizations, Please refer Appendix E for notation table.
2

---

Published as a conference paper at ICLR 2024
(B)
MEXI
Mimmnn
9
7
7
7
9
0
6
6
6
6
Ground
Macinica
9
7
9
9
6
6
6
6
Inpui
9
9
9
9
>
6
6
6
i
GB (Ours)
9
9
9 9
6
6
V
(OR-KE)
800
9
9
9 9
6
6
6
6
(H-WAE)
CB (Dura]
Intepplation
Extrapadation
Figure 1: (A) Conceptual illustration of the and CIB principles. By conditioning regularization on the remaining input time steps, the latent representation can better preserve the underlying temporal dependency. (B) Motivating experimental results on interpolation (left) and extrapolation (right). Because features in single time step completely missing, model must collect information from other time steps. The conventional IB approach HAE) shows deteriorating performance in both cases. Another IB approach (AE) using a Gaussian process prior demonstrates enhanced perfor- mance for interpolation but often significantly loses time series characteristics for extrapolation (i.e., the writing style is corrupted). The CIB approach (Ours) exhibits improved imputation performance for both cases. Complete quantitative results are available in Table 1.
have a distinct sampling period or when non-unifome sampling strategies are employed. To denote missing observations, we partition the input vector x; at each time step into observed features x; and missing features x," such that X; = x U x!"
Objective. Our aim is reconstruct the complete time series input X1:7 by filling the missing features from the observed time series input xi.r- Formally, we seek to generate x from the conditional distribution pl (X!"" Xi:r). By modeling the conditional distribution p(X: (X Xir) instead of using deterministic mapping, uncertainty associated with the imputed values.
generate multiple imputations, allowing to capture the
What makes this problem challenging is that we must account for the underlying temporal dynamics represented by x when imputing missing features x," for t E (1.....77 T}. We can straightfor- wardly apply the unsupervised IB described in (2) to obtain latent representations Z, by discarding information from the observed time series input Xi that is redundant for reconstructing X,-
Formally, this can be achieved by minimizing To (Z.; xir) (X, Z.) with comprehensive encoder (e.g.. RNN or Transformer) capable of effectively modeling the temporal dependencies within the observed time series observations, i.c., go (Z, (xir). However, enforcing such strict regularization constraints on the encoder may lead to a significant loss of information regarding the temporal context that can be achieved by observations
different time steps, which we denote as X (X; T E (I,.... 7) (). This may cause the imputation of X." at time step / to heavily rely on the observed features at that particular time point, i.e., X;, rather than being able to learn from temporal dependencies present in other observations, i.e., X (as shown in Figure (BB).
To tackle this issue, alleviate the potentially negative consequences of the regularization constraint by directing our attention to the redundant information of the observed input at time step when it conditioned on its temporal context represented by the remaining observed time series X.,- This offers novel information-tiee thhorretc rationale for time series imputation, as defined below: Definition 2. (Time Series Imputation) Let X and X be random variables for the partially observed features and missing features of X, at time step t. Then, given the observed time series input X i:T we define time series imputation at time step t as unsupervised CIB as follows:
min 6,9
310 Ze)
Conditional Regularization
Reconstruction
where X represents the random variables for the remaining input observations, excluding X
By conditioning on X (3) guides us to find latent representations Z, and the corresponding infer- ence model parameter which encompass all retrievable information from the entire observed input time series X (reconstruction)) while discarding information that is redundant for capturing X!" given the available temporal context from the remaining observed time series X3, (conditional regu- larization). Overall, the proposed objective in (3) enables us more effectively utilize information
3

---

Published as a conference paper at ICLR 2024
from X for imputing X compared to other IB-related alternatives, whose conceptual illustration can be seen in Figure 1(A).
3.2 DEEP VARIATIONAL CONDITIONAL INFORMATION BOTTLENECK ON TIME SERIES
In this subsection, we will transform our objective (3) into learnable form by utilizing variational decomposition. (3) is represented as a combination of the traditional BL with mutual information along the time axis, which can be approximately achieved by minimizing the contrastive loss.
3.2.1 MAXIMIZING RECONSTRUCTION: min (Xx Ze)
0.9
Following the derivations introduced in (Voloshynovskiy et al., 2019), we can find lower bound of the reconstruction term the following:
(x;; - + (x(x, (x)ll(( Rxg / Pur
( (zj.)
Dlog (x
Exi Pen (x [log (x; [z:]] dal Cb200
where H(-) is the entropy and the last inequality holds due to the non-negativity of entropy and -iveeieeee Here, we introduce a feature estimator, denoted as pa po(X.|Z,). as varia- tional approximation (z,, We model the feature estimator as an isotropic Gaussian, i.c.. (x(X; N (Ns Ze), diag(a (Z))) where (e)()) and (+ are implemented by neural networks parameterized by 0.
In many practical scenarios, the ground-truth values for missing features are unknown during training. Thus, to accurately learn the reconstruction process given the latent representation of the observed time series, we apply (4) only to the features observed at each time point, similar to the approach in (Nazabal et al., 2020).
Discussion on the Conditional Reconstruction. One might question why the reconstruction term not conditioned on X as given by an alternative form of the CIB, i.e., ming. a (Z,: X; x (I; ((((,) Applying the chain rule of mutual information? decomposes the conditional reconstruction as the follows: [X;; Z, I(X,; Z,, X - I(X,; ((,) ). turns out that the first term can be bounded by a mathematically equivalent expression as shown in (4), suggesting that this term encourages mitigating constraints temporal context for reconstruction (see Appendix A.1). However, minimizing the second term attempts to eliminate information about the target X, at time point t, which can be achieved from the observation other than time point t, i.e., X,,- This contradicts the goal of time series imputation where we aim to capture temporal context from the remaining observed time steps. Our empirical results also support that minimizing / [(x;; X(;) deteriorates the model performance (see Appendix A.2 for derivation, B.2 for experimental results).
3.2.2 MINIMIZING CONDITIONAL REGULARIZATION: min Le (Z;:
0.0
We employ the chain rule for mutual information on the conditional regularization term as follows:
min I(Z;; X; x min [ZZ; Xi - I(, 0,0
6.8
It is worth highlighting that the application of the chain rule decomposes the conditional regularization into two components: (i) minimizing the information between the latent representation Z, and the entire observed time series input X that encourages the latent representation to be concise, while (ii) maximizing the information from X to capture the underlying temporal dynamics provided by the observations at the remaining time steps. This prevents a significant loss of temporal context in the IB and, in turn, enhances the utilization of temporal dependencies from the remaining time steps.
Minimizing I (Z, X The first term in (5) can be bounded as follows (see Appendix A.3):
(Zz Exe Ex [DxL (go (2,) x (pp(() C3
LLtt V. W, and Y random variables, then the chain rule gives T(Y; W = I(Y: -(Y; Conditional reconstruction can be appropriate for capturing information that exclusively depends the corresponding input, introduced in (Fischer, 2020; L.ce al., 2023).
4

---

Published as a conference paper at ICLR 2024
where utilize the unit isotropic Gaussian the prior distribution, i.e., p(Z,) = VV., We model the stochastic encoder as a multivariate Gaussian distribution defined as go (2, 1:7 (uup(Xi diag(o (Xii)) where (-(+) and (+(+) are implemented neural networks parame- terized by Φb. This explains why the (unconditional) IB struggles with modeling temporal dynamics, as discussed in Section 3.1. That is, (6) forces the encoder mappings that depend on time series inputs to converge to the unit Gaussian, imposing overly strict regularization. This hinders capturing of temporal dependencies present in other observations, motivating us to explicitly capture temporal dynamics by maximizing (Z;; X rather than relying solely on the reconstruction signal in (4).
Maximizing XZ, To bound the second term in (5), adopt the InfoNCE minimization from the contrastive learning on latent representations that approximately achieves maximizing the corresponding mutual information (Oord et al., 2018 Tian et al., 2020). Suppose we have the latent representation at time step f given an observed time series xi.r as a reference, i.c., Z, ~ 9o (Z, (Z())) Since our aim is to maximize information from X we intentionally make missing observations at time step from the reference, and employ latent representations of time steps other than & positive pairs. For negative pairs, we consider latent representations at different time steps from other time series observations. Finally, define our novel contrastive learning loss with cosine similarity of latent representations along the time axis as follows (see Appendix A.4):
Σ
exp (zi 2.z2e/1 /T)
''E(L....I..
X()) 2 Exq
log
Σ
Σ
exp (zi 2.T22 (i/ZT/T)
deft,...,T 2 "/
where is the temperature parameter. Here, 2n' 2o (Ze (x()) denotes the positive pair obtained by masking the reference time series, such that x is created by replacing x with zeros from x².7- We regard such positive pairs as augmentations of a given time series since latent representations with missing values at time step 1. share task-relevant information about the underlying temporal dynamics of given time series. We denote Xi. set of negative samples comprising other time series in the same mini-batch, where x indicates an observed time series from Xi. This makes our encoder capture time series-level semantics such as underlying disease progression patterns that can be distinguished from others - by pushing these samples from the reference. Such an attribution is necessary for reconstructing missing values (and associated downstream tasks in the experiments) specific to the input time series. Please refer to Appendix C for implementation details.
3.2.3 OPTIMIZATION
Now, we introduce a novel imputation method, which we refer to as Time-series Imputation using Conditional Information Bottleneck (TimeCIB), that consists of the stochastic encoder, qo, and the feature estimator, Po. introduced above. Please see Figure C1 for a schematic illustration of our framework. Overall, optimize our method based on the following objective by combining all the loss functions that allows us to approximately achieve time series imputation defined in (3):
min 0,0
BC 9 + C² + C
(8)
where 7 E Ryo is balancing coefficient that trades off the impact of CJ. We provide sensitivity analysis on B, y in Appendix B.4.
3.3 INTRODUCING INDUCTIVE BIAS ABOUT TEMPORAL DYNAMICS
Now, we illustrate how we can inject inductive bias about the underlying temporal dynamics by employing temporal kernels further improve the expressive power of TimeCIB.
The alignment of the latent representation (Wang & Isola, 2020), attained through contrastive learning based on (7), renders the similarity between latent representations at two adjacent time points indistinguishable from the similarity between those at two distant time points. This phenomenon appears to contradict real-world temporal dynamics, such as gradually deteriorating or periodic behavior of disease progression patterns. To address this, we employ conditional aligument (Dufumier et al., 2021) that introduces inductive bias about the underlying temporal dynamics with temporal kernels as the following:
5

---

Published as a conference paper at ICLR 2024
E rE(1..... Crp
exp (ze Te.22 Ze Zx/T)
I(Z;; 2 Exp.
log
det
C
Σ xi Exir ximulleliss))
E
exp (z, 2 TT)
t'e(L...TT
where Cr, E IR is a kernel coefficient as a function of two time points and
Incorporating prior knowledge of underlying similarity into contrastive learning is not novel concept. Several previous works have leveraged supervisory information adapt contrastive learning. including semantic similarities for text classification (Suresh & Ong, 2021). angle similarity for gaze estimation (Wang et al., 2022), and Gaussian priors for time series and video representation learning (Tonekaboni et al., 2021; Chen et al., 2022). In this paper, we evaluate the following two representative temporal kernels to evaluate our method on data with different temporal behaviors and will leave the choice of the kernel as thyerpaaaameter:
Crauchy Mouchy(T,77) (T, = (1+ (T (T-T)²
2sinn (πT(T - r')/p) (10)
Cperiodic (T, 7') = = exp
12
12
Cauchy Smoothing. Under the assumption that two nearby time points should be more similar than those far away, we smooth the latent representations of time series by assigning higher weights to nearby time points when pulling the representations, utilizing the Cauchy kernel defined as Ceauchy (T, r') in (10) which is the mixture of infinite RBF kernels with different time scales (Rasmussen, 2004). This is generalized form that reduces to uniform weights as in (7), i.e., / = 3o gives c(T, r') =?.
Periodic Smoothing. Unfortunately, Cauchy smoothing may not be appropriate when the underlying temporal dynamics exhibit periodic behavior (e.g., seasonality). To incorporate domain knowledge about periodic time series data, we utilize the exponentiated sine-squared kernel given Cperiadic (T, r') in (10) where IR corresponds to the length scale and p IR reflects the periodicity.
4 RELATED WORKS
Time Series Imputation: Predictive Methods. Earlier works on time series imputation have been proposed for single imputation utilizing predictive models. M-RNN (Yoon et al., 201 8b) and BRITS (Cao et al., 2018) are representative RNN-based methods that predict missing observations by employing bidirectional RNNs to capture both past and future temporal dependencies. Inspired by the recent success of transformers modeling time series data over conventional RNN architectures, recent predictive methods for time series imputation employ self-attention mechanism to enhance imputation performance (Bansal et al., 2021; Du et al., 2023; Shan et al., 2023). However, these methods cannot provide multiple imputations and, therefore, fail to incorporate the uncertainty associated with imputed values.
Time Series Imputation: Generative Methods. Two main strands of generative models VAFs (Kingma & Welling, 2014) and generative adversarial networks (GANs) (Makhzani et al., 2015) have been introduced for multiple imputation due their ability stochastically generate samples. Here, we describe VAE-based methods. See Appendix D.3 for more related works.
VAE-based imputation methods primarily focus on defining the evidence lower bound, where the reconstruction error is computed only over the observed part of the incomplete data while missing values are filled with arbitrary values (c.g., zeros) during inference (Sohn et al., 2015; Nazabal et al., 2020). Fortuin et al. (2020) proposed GP-VAE which adopts a similar approach to efficiently handle incomplete (missing) data in temporal setting by assuming that the latent representation of input time series evolves smoothly over time according to Gaussian process (GP). While introducing the GP prior improves the ability to capture the underlying temporal dynamics, GP-VAE still cannot capture shared temporal structures across time series data, as it employs an independent GP prior for each time series. More recently, L-VAE (Ramchandran et al., 2021) and its conditional extension (Ramchandran et al., 2022) further improve the GP prior of GP- VAE by utilizing auxiliary covariates information. We focus our comparison on VAE-based models since these models can be information cramattion-thoorticaalll theoretically interpreted optimizing the IB Voloshynvvskiy et al., 2019).

---

Published as a conference paper at ICLR 2024
It is worth clarifying that TimeCIB can be distinguished from GP-VAE in terms of how they achieve time series imputation and leverage temporal kernels. Under the IB framework, GP-VAE models the smooth temporal evolution of latent variables by replacing the traditional unit Gaussian prior with GP prior specified by temporal kernels. However, TimeCIB is motivated by the inherent limitation of the IB in discarding temporal information (see Section 3.1) and proposes a novel CIB principle that alleviates the strict regularization of IB. Temporal kernels are optionally adopted to introduce an inductive bias the underlying temporal dynamies.
Information Bottleneck with Conditional Information. Several works have tailored the IB principle (Tishby & Zaslavsky, 2015; Alemi et al., 2017) by introducing conditional reconstruction or applying conditional regularization to extract information in alignment with their specific objectives. (Gondek & Hofmann, 2003) proposed conditional reconstruction to discover a new meaningful set of clusters that is orthogonal to the known class labels. (Fischer, 2020; Tezuka & Namekawa, 2021) introduced conditional regularization for supervised learning, which minimizes only redundant information given label information, thereby preventing the loss of label-related information due to overly strict regularization in the conventional IB principle. More recently, (Lee et al., 2023) utilized both conditional reconstruction and regularization discover a label-related core subgraph from pair of two molecular graphs.
From this perspective. work aligns with conditional regularization approaches. While previous works have primarily focused on mitigating regularization concerning target label information, our method aims to alleviate overly strict regularization that can hinder the learning of underlying temporal dynamics. Moreover, to the best of our knowledge, this is the first work that presents an inbommttion-teorretii definition for time series imputation and proposes a novel conditional IB that can effectively preserve temporal dynamics for better imputation.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Evaluation Metries. We evaluate the imputation performance from two perspectives: i) Imputation performance which measures feature-wise (pixel-wise) reconstruction. Specifically, we assess the negative log- likelihood (NLL) and mean squared error (MSE) of the imputed values on artificially missing features. ii) Prediction performance, which indirectly measures how well the imputed values preserve task-relevant information, which is crucial aspect of imputation methods in practice. Following the experimental setup in (Fortuin et al., 2020) and (Yoon et al., 2019), we train separate classifiers or predictors with imputed values to predict the target labels. Then, we evaluate the area under the receiver operating characteristic (AUROC) for classification tasks and the MSE of the forecast (ForecastMSE) for forecasting tasks to measure the discriminative and predictive performance of imputation methods, respectively.
Baseline Models. We focus our comparison on VAE-based models since these models can be interpreted under the IB principle as suggested in (Voloshynovskiy al., 2019). Moreover, these multiple imputation methods can provide uncertainty of the imputed values, which is often crucial to support maaiiinnnmmkingggg processes such clinical interventions in healthcare. Hence, for baseline models, we compare our proposed method with the following: i) GP-VAF (Fortuin et al., 2020) which utilizes the Gaussian process (GP) prior to model time dependency, ii) HI-VAE (Nazabal et al., 2020) and iii) VAE (Kingma & Welling, 2014), both of which use an autoencoder architecture and are capable of imputing values at each time step. addition to the above baselines, we compare with cutting-edge predictive methods: an RNN-based method, BRITS (Cao et al., 2018 20, and a transformer-basee methods, SAITS (Du et al., 2023). Moreover, we also compare with otate-of- the-art generative imputation methods, attention-based autoencoder approach, mTANs (Shukla & Marlin, 2021). and diffusion-basee approach, CSDI (Tashiro et al., 2021).* 4 For a fair comparison, the magnitudes of the number of parameters are set to be equivalent among the evaluated methods. More detailed explanations about bascline models are provided in Appendix D.
We compare NLL only with two VAE-based methods, since NLL cannot be measured for predictive methods; m TAN is probabilistic hut uses fixed variance; and NI.I. of CSDI affected by the noise schedule. (please refer to Appendix F.2 of Tashiro al. )). We mark asterisks (*) in Table 2 and Table 3 specify this.
7

---

Published as a conference paper at ICLR 2024
Table 1: Imputation and prediction performance on the image sequence datasets.
HealingMNIST (missing with MAAR portzrn))
RotatedMNIST (nterppolation & extrapolation
Methods
NLL())
MSB(])
AUROC(T)
NLL())
MSB(]]
No
0.293 +: 0.000
0.920 0.000
0.133 +: 0.000
Mean Imp.
0.168 = 0.000
0.938 0.000
0.085 = 0.000
Forward Imp.
+: 0.000
0.946 0.000
0.080 +: 0.000
VAE
0.480 0.002
0.232 + 0.000
0.922 0.000
1.773 0.127
0.133 I 0,000
HI VAE
0.290 0.001
0.134 1: 0.003
0.962 1: 0.001
0.207 1 0.007
0.087 1: 0.001
SP-VAE
0.261 0.001
+ 0.002
0.960 0.002
0.190 0.001
0.080 + 0.004
Ours(Uni Uniform)
Ourst
0.204 0.002
0.090 1. 0.001
0.967 1: 0.001
0.184 0.001
0.077 1: 0.001
Ours(Canchy)
0.202 + 0.004
0.088
0.002
0.967 + 0.000
184 0.001
0.076
0.5
Vi
unoa
0.4
0U:S
0L15
0.3
Imputatton
c.15
0.
0.05
Random
Bempora
Terporral
Soetisl
MWAR
0.8
0.7
Missing Pattern
(a) Robustness on missing patterns.
(b) Robustness on missing ratios.
Figure Robustness analysis for missing patterns and missing ratios on HNAIT
5.2 MAIN RESULTS
Imputation on image sequences. To evaluate imputation performance diverse missing scenarios, we assess imputation performance on two MNIST sequence benchmarks with various missing patterns. HealingMNIST (Krishnan et al., 2015) has approximately 60% of missing pixels under missing- not-at-random (MNAR) pattern on every time step, where the missing probability of white pixels is twice larger than that of black pixels. Given that the model is not provided with information about the underlying missing mechanism, this task is particularly challenging, yet it mirrors many practical scenarios. For example, in healthcare. patients with depression are more likely refuse answers about the severity of their condition (Gliklich et al., 2014). RotatedMNISTT (Ramchandran et al., 2021) evaluates performance on interpolation time step are completely missing. This makes imputation more challenging since the model must reconstruct all the missing values at a given time step solely based on the temporal dependency. Table I demonstrates that TimeCIB provides state-of-the- art imputation and prediction performance both datasets, and the application of the Cauchy kernel (10) can further improve the performance.
extrapolation, where all features at an arbitrary
We further evaluate the robustness of our model on HealingMNIST with four additional missing patterns with 60% of missing ratio (Figure B2a) and that with different missing ratios ranging from 10% 90% with the random missing pattern (Figure B2b). To assess the robustness of missing patterns, we employ Random, Spatial (i.e., neighboring pixels have correlated missing probabilities). and Temporal+/- (i.c., positive/negativy temporal correlation). The imputation performance of our proposed method is most robust on diverse missing patterns and missing ratios. Please refer to Appendix B for more experimental details and results.
Imputation for weather forecasting. Weather forecasting is one of the representative fields where we can observe diverse scales of seasonality such as daily, weekly, monthly, or yearly basis which is one of the most important characteristics of time series data. In this experiment, we focused on two weather forecasting datasets Beijing Air Quality (Zhang et al., 2017) and US Local whose time series measurements are collected every hour. Our model is capable of using the prior knowledge on the periodicity of the target data, by applying conditional alignment with temporal kernels (Section 3.3); assume daily periodicity by setting outlined in (Yoon et al., 901 we also evaluate the utility of imputation methods on forecasting by assessing the prediction performance (ForecastMSE) of separately trained LSTM using time series data with imputed values. As shown in Table 2, TimeCIB outperforms the benchmarks on both weather forecasting datasets in terms of both imputation and prediction performance. The
24. Inspired by the experiments
8

---

Published as a conference paper at ICLR 2024
Table 2: Imputation and prediction performance on the weather forecasting datasets.
Beijing (7=24)
US Local (7=766)
Methods
NLL())
MSE(J)
(orccastMSE(L)
NEL(J)
MSE(1)
ForCeas(MSB(U)
No Imp.
1,015 + 0.000 0.539 4: 0.000
1.1 + 0.000
0.610 0.000
Mcan Imp.
0.460 = 0.000
0.517 + 0.000
0.509 + 0.000
0.432 = 0.000
Tmp.
0.399 0.000
0.502 4:
0.391 + 0.000
0.401
BRITS
0.390 + 0.002 0.490 4 0.006
0.384 + 0.001
0.398 = 0.027
SAITS
0,283 + 0.013 0.430 + 0.005
0.273 = 0.002
0.350 = 0.067
mTAAs
0.287 0.005 0.436 - 0.005
0.268 : 0.018
0.337 0.003
CSDIn=5))
0.287 + 0.003 0.423 = 0.003
0.378 + 0.001
0.364 = 0.036
CSD(
0,270
VAB
1.427
0.001
1.016 1 0.002
0.524 1 0.006
1.462 1 0.002
1.086 1 0.004
0.467 1 0.034
HI-VAE
1.081
0.003
0.321
0.008
0.464
0.008
1.078 + 0.005
0.317 +: 0.010
0.380 + 0.060
GP VAEE
1.077
0.006
0.316
0.011
0.463
1 0.008
1.078 1 0.005
0.318
1 0.010
0.385 0.051
Ourst Unitoem)
1.063
= 0.001
0.291
+ 0.004
0.445
4 0.003
1 1.052 + 0.001
0.265
+ 0.002
0.351
= 0.060
Ouns(Prriodic)
1.060
0.283
0.443
0.004
0.002
0.260
0.003
0.327 0.022
-12
-12
0.80
24
(a) Beijing
US Local
(c) Physionet2012
Figure 3: Comparison of the imputed values for examples in (a) Beijing, (b) US Local, and (c) Phys-
ionent2012 datasets, highlighting that TimeCIB provides more accurate imputations by considering temporal dependencies. Dots and crosses are observed and missing ground-truth values, respectively.
performance of our method is further enhanced when equipped with a temporal periodic kernel (10), highlighting our model's ability to incorporate the correct inductive bias.
Imputation for electrical health records. Time series imputation is of special impor- tance in healthcare where each feature may have a distinct sampling period and strategies. In this context, evaluate imputation meth- ods on Physionet2012 - Mortality Prediction Challenge (Silva et al., 2012), which aims to predict in-hospital mortality of intensive care unit (ICU) patients from 48 hours of records with roughly 80% of missing features. Fur- thermore, we conduct additional evaluations to assess whether the imputation methods pre- serve the critical characteristics of a given time series i.c., whether patient's 's status is deteriorating or not - after replacing the missing features with imputed values. Table 3 shows that TimeCIB provides imputation performance comparable to the best benchmark while outperforming the VAE-based methods by a great margin. Furthermore, it achieves the best classification perfor- mance, successfully capturing information about the temporal dynamics of patients' status. Note that while the mean imputation provides better imputation performance, the imputed values drastically lose the crucial information for discriminating patient's 's status.
Table 3: Imputation and prediction performance on the clinical dataset.
Physiomet2012 (mortality prediction)
Methods
NLL())
MSE(L)
AUROC(1)
No Imp.
0.962 + 0.000
0.692 +: 0.000
Mean Imp.
0.511 0.000
0.703 + 0.000
Forward Imp.
0.613 1: 0.000
0.710 1: 0.000
BRITS
0.529 + 0.004
0.700 + 0.005
SAITS
0.501 0.021
0.007
mTANs
0.499 + 0.008
0.721 + 0.004
CSDI(n=5)
0.548 0.014
0.705 + 0.005
CSDRin=25)
0.478 + 0.002
0.683 + 0.033
VAE
1.400 = 0.000
0.962 + 0.000
0.691 + 0.001
HL-VAE
1.315
VAE
1.227 1 0.007
0.616 1 0.013
0.730 1 0.006
Unitorm))
1.883 = 0.007
0.528 = 0.011
0.744 0.009
Ours(Cauchy)
1.179 1 0.006 0.521 + 0.012
0.744 0.009
6 CONCLUSION
In this paper, we have presented TimeCIB, a novel cntemmttion-hhooretii approach for time series imputation. While inheriting the multiple imputation and uncertainty measurement properties of the IB, TimeCIB addresses the limitation of the IB principle in capturing underlying temporal dynamics by replacing conventional regularization with conditional regularization. Our variational decomposition showed that CIB could be approximated by optimizing the evidence lower bound (ELBO) and the contrastive objective. We also demonstrated that introducing inductive bias based on temporal kernels further enhance expressive power, acting a form of conditional alignment. Our empirical results on image sequences, weather forecasting, and electrical health records prove that TimeCIB is effective in wide range of practical cases.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We thank anonymous reviewers for many insightful comments and suggestions. CL was supported through the IITP grant funded by the Korea (MVIM (No. 2021-0-0134,, Al Graduate School Program, CAU).
CODE AVAILABILITY
Codehase used in this paper is available httpt:tpp///c htttps://gthuub htwp:///bb.co///cenngg////eeee
REFERENCES
Alexander A. Alemi, lan Fisher, Joshua V. Dillon, and Kevin Murphy. Deep Variational Information Bottleneck. 2017.
Parikshit Bansal, Prathamesh Deshpande, and Sunita Sarawagi. Missing Value Imputation on Multidimensional Time Series. Proceedings of the VLDB Endowment, (4(11):2533-254) 2021.
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. BRITS: Bidirectional Recurrent Imputation for Time Series. Advances in neural information processing systems, 31, 2018.
Minghao Chen, Fangyun Wei, Chong Li, and Deng Cai. Frame-wise Action Representations for Long Videos via Sequence Contrastive Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13801-13810, 2022.
Wenjie Du, David Côté, and Yan Liu. SAITS: Self-attention-Bauts Imputation for Time Series. Expert Systems with Applications, 219:1196199 2023.
Benoit Dufumier, Pietro Gori, Julie Victor, Antoine Grigis, and Edouard Duchesnay. Conditional Aligmment and Uniformity for Contrastive Learning with Continuous Proxy Labels. arXiv preprint arXiv:2111.05643. 2021.
Ian Fischer. The Conditional Entropy Bottleneck. Entropy, 22(9):999, 2020.
Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt. GP-VAE: Deep Probabilistic Time Series Imputation. In International conference artificial intelligence and statistics, pp.
1651-1661. PMLR, 2020.
Richard E Gliklich, Nancy A Dreyer, Michelle B Leavy, et al. Registries for Evaluating Patient Outcomes: a User's Guide. 2014.
David Gondek and Thomas Hofmann. Conditional Information Bottleneck Clustering. In 3rd ieee international conference on data mining, workshop on clustering large data sets, pp. 36-42, 2003.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta- VAE: Learning Basic Visual Concepts with Constrained Variational Framework. In International conference on learning representations, 2017.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. MIMIC-III, Freely Accessible Critical Care Database. Scientific data, 3(1):1-9, 2016.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised Contrastive Learning, Advances in neural information processing systems, 33:18661-18673 2020.
Diederik P Kingma and Max Welling. Auto-encoding Variational Bayes. In Proceedings of the International Conference on Learning Representations (ICLR), volume 2014.
Rahul G Krishnan, Uri Shalit, and David Sontag. 151 2015.
Deep Kalman Filters. arXiv preprint
10

---

Published as a conference paper at ICLR 2024
Namkyeong Lee, Dongmin Hyun, Gyoung S Na. Sungwon Kim, Junseok Lee, and Chanyoung Park. Conditional Graph Information Bottleneck for Molecular Relational Learning. 2023.
Yonghong Luo, Xiangrui Cai, Ying Zhang, Jun Xu, et al. Multivariate Time Series Imputation with Generative Adversarial Networks. Advances in neural information processing systems, 31, 2018.
Yonghong Luo, Ying Zhang, Xiangrui Cai, and Xiaojie Yuan. E2GAN: End-to-end Generative Adver- sarial Network for Multivariate Time Series Imputation. In Proceedings of the 28th international joint conference artificial intelligence, pp. 3094-3100. AAAI Press Palo Alto, CA, USA, 2019.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial Autoencoders. arXiv preprint arXiv:151 2015.
Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Isabel Valera. Handling Incomplete Heterogeneous Data Using VAEs. Pattern Recognition, 107:107501, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding, arXiv preprint arXiv:1807.05745 2018.
Siddharth Ramchandran, Gleb Tikhonov, Kalle Kujampää, Miika Koskinen, and Harri Lähdesmäki. Longitudinal Variational Autoencoder. In International Conference on Artificial Intelligence and Statistics, pp. 1898-3906. PMLR, 2021.
Siddharth Ramchandran, Gleb Tikhonov, Otto Lönnroth, Pekka Tiikkainen, and Harri Lähdesmäki. Learning conditional variational autoencoders with missing covariates. arXiv preprint arXiv:2203.01218, 2022.
Carl Edward Rasmussen. Gaussian Processes in Machine Learning. In Advanced Lectures on Machine Learning: ML Summer Schools 2003, Canberra, Australia, February 2-14, 2003, Tübingen, Germany, August 4-16, 2003, Revised Lectures, pp. 63-71. Springer, 2004.
Siyuan Shan, Yang Li, and Junier B Oliva. NRTSI: Non-recurrent Time Series Imputation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Satya Narayan Shukla and Benjamin Marlin. Multi-Time Attention Networks for Irregularly Sampled Time Series. In International Conference on Learning Representations, 2021. URL https: // //opereevieww... net.ieewwweeeeeeeeeee..ee.eeeeee forum? :000mmm11000000000._._00
Ikaro Silva, George Moody, Daniel Scott, Leo Celi, and Roger G Mark. Predicting in-hospital Mortality of ICU Patients: The physionet/computing in Cardiology Challenge 2012. In 2012 Computing in Cardiology, pp. 245-248. IEEE, 2012.
Kihyuk Sohn. Improved Deep Metric Learning with Multi-class N-pair Loss Objective. Advances in neural information processing systems, 29, 2016
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning Structured Output Representation using Deep Conditional Generative Models. Advances in neural information processing systems, 28, 2015.
Varsha Suresh and Desmond C Ong. Not All Negatives Are Equal: Label-aware Contrastive Loss for Fine-grained Text Classification. arXiv preprint arXiv:2109.05422 2021.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional Score-Base
Diffusion Models for Probabilistic Time Series Imputation. Advances in Neural Information Processing Systems, 34:24804-2444 2021.
Taro Tezuka and Shizuma Namekawa. Information Bottleneck Analysis by a Conditional Mutual Information Bound. Entropy, 23(8)9974, 2021.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive Multiview Coding. In Computer AisionEECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776-794. Springer, 2020.
11

---

Published as a conference paper at ICLR 2024
Naftali Tishby and Noga Zaslavsky. Deep Learning and the Information Bottleneck Principle. In 2015 ieee information theory workshop (itw), pp- 1-5. IEEE, 2015.
Sana Tonekaboni, Danny Eytan, and Anna Goldenberg. Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding. In International Conference on Learning Representations, 2021. URL https: / opwnrevieweweeeeeeeeeeeeee net/ d
Slava Voloshynovskiy, Mouad Kondah, Shideh Rezaeifar, Olga Taran, Taras Holotyak, and Danilo Jimenez Rezende. Information Bottleneck Through Variational Glasses. In NeurIPS Workshop Bayesian Deep Learning, 2019.
Tongzhou Wang and Phillip Isola. Understanding Contrastive Representation Learning Through Alignment and Uniformity on the Hypersphere. In International Conference on Machine Learning. pp. 9929 9939. PMLR, 2020.
Yaoming Wang. Yangzhou Jiang, Jin Li, Bingbing Ni, Wenrui Dai, Chenglin Li, Hongkai Xiong, and Teng Li. Contrastive Regression for Domain Adaptation on Gaze Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19376-19385, 2022.
Jinsung Yoon, James Jordon, and Mihaela Schaar. GAIN: Missing Data Imputation using Generative Adversarial Nets. In International conference machine learning, pp. 5689-5698, PMLR, 2018a.
Jinsung Yoon, William R Zame, and Mihaela der Schaar. Estimating Missing Data in Tem- poral Data Streams Using Multi-directional Recurrent Neural Networks. IEEE Transactions on 2018b.
Biomedical Engineering,
Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. Time-series Generative Adversarial Networks. Advances in neural information processing systems, 32, 2019.
Shuyi Zhang, Bin Guo, Anlan Dong, Jing He, Ziping Xu, and Song Xi Chen. Cautionary Tales on Air-quality Improvement Beijing. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 473(2205):00100457, 2017
12

---

Published as a conference paper at ICLR 2024
A DERIVATIONS
A.1 VARIATIONAL APPROXIMATION OF FIRST TERM IN CONDITIONAL RECONSTRUCTION,
Here, show that the first of the conditional reconstruction / (Xi; Ze, X²,) can be approxi- mated as in (4) based on the following derivation:
p(xe(z(xx(1)
I 1X,; Ze x,,) Ex(x)
log
p(x))
pog p me, x(,) x(e)
Ep(x)) px))
(11)
p(x,, Ze, log le, x ) dx H(x,)
22 xx
p(x,, Re x log d
Xxe
The last inequality holds because of the non-negativity of entropy. Note that p(x, Ze (x)) can be marginalized by introducing X;:
pX,,, Ze X(+)= /
p(X., Xix )4 Z Xir)dX
(12)
Then, putting (12) into the (11) gives
1Xx;; Z, xx.)
p(xe, 40 (ze (x::) log d
xx.
E (x x
logp(x, 2r x x((1
log 2e, x(e po (x, (z)) po X (2))
= (p(
E (x.(x)))))))
p(x, / Iz,, x(,)
E (oggo((x The last term can be expressed in form of the KL-divergence:
+ Exxx.exxx
log
po (x, (Z.)
(13)
p(x: (ix, x(.)
E Ep(x,axx()))
log
Ep(x,x())
log po (x(x)(
po (xe(2e))
(14)
Ep(an)q))
DKL (p(x, 1e, x 1lpo (x+ (x)l2e))
Due to the non- non-negativity of the KL-divergence, plugging (14) into (13) gives (4) as
I(X;; Z, X,)) Ep(x,
(Z, (Xo,
(15)
13

---

Published as a conference paper at ICLR 2024
A.2 VARIATIONAL APPROXIMATION OF SECOND TERM CONDITIONAL RECONSTRUCTION,
Analogous to the derivation in (Alemi et al., 2017), I(Xe; X(,) is upper bounded as follows:
log p(x,(xx)) (x(
1 (Xx; (1)) Ep(x)
p(x:)
log p(x, (x(()))
((x1)
(16)
Bpixcog [oo p(x,) g(xx))
log g(xx))
(xa(((xx)lla(xx))
S Po(t)))))
where we set ((x)) = No 1) (Alemi et al., 2017: Lee et al., 2023). Specifically, our imple- mentation, we deterministically encode 2, by taking na (x(,) and then calculate the KI.-divergence between the decoder distribution pa (X, Z.)) with (Xx.).
A.3 VARIATIONAL APPROXIMATION OF FIRST TERM IN CONDITIONAL REGULARIZATION.
log Cx(x).T Zt)
1(Z xir)
Ego (g((xxx
To (Ze Pdtnaa (onn(xx))) (xi:r)
In (Ze (x:T) p(2t)
Boolan
log
(Z()
p((t)
(17)
log (2r
log p(zz)
Egg (2n x(er)
+ Egs (m. x
p(z:)
(o(z))
(x DKL (ze (Zo (z)xxx:))ll((x)))) - (xl(go(z))ll((zz))
E Epau(xi DKI. (go
The last inequality holds because of the non-negativity of KL-divergence.
14

---

Published as a conference paper at ICLR 2024
A.4 CONTRASTIVE APPROXIMATION OF SECOND TERM CONDITIONAL REGULARIZATION,
In Section 3.2.2, we optimize / (Z,; x by approximate the mutual information into a contrastive form, which is similar to (Oord et al., 2018; Tian et al., 2020).
p(X(,)
IZ x -Ex log
p( (z.)
p(X),)
-Ex log
N p p(X;,1 (Z.)
+ log(NN
p(X(.)
Exx log
N
p X {e (Z,)
p(X;,)
2
Ex log
(N (N-1) - 1)
p(X [z,)
p(X))))
p(X 2. [Z,)
-x log
p(x: (((1,))))) (N - 1)Bxx
X
(18)
,(
p(X;,,) (e,)
Ex log
1
x Zep
X,E p(X;))
p(x, Z.)) p(X;1)
Exx log
p(X. I.) ((X.))
ExTEx
p(X)))
((X. (Z,) D(X).)
Ex log
Σxi x Zu.)
p(X..))
S(Z,, X(.)
Ex log
Exi J(Zz, Xi.T)
There remain two design choices: i) selection of X and ii) formulation of function f. For i), we use mini-batch approach that X chosen from other time series inputs in the same mini-batch. For ii), we adopt the average of cosine similarities along the time axis. Specifically, we define the function f using an alternative representations Zμ which is obtained by inputting the masked input qol Z2 X
f (Z)
Σ f(z, Zv)
Σ exp Zu/T//
(19)
L'ELTTM
L'E(TT
where R is temperature hyperparameter.
Alternatively, we can also define the function for negative samples in similar way.
(T.))
Σ
f(Ze, Z,)
Σ
exp Ze "zz /T)
(20)
('E(L:T)
(E(1:T)
Then (Z, (,, is lower bounded by
Σ exp Z Zx//1
1(Z,; X 2 log
(21)
Σ
Σ exp Zz "Z,TZ Zu.ZZi11 ZT/TT (E(ITT
We can observe that [(Z,; X.,) is lower bounded by the similar form of NT-Xent objective function in (Sohn, 2016) and specifically in-version of the supervised contrastive loss in (Khosla et al., 2020).
15

---

Published as a conference paper at ICLR 2024
B ADDITIONAL RESULTS
B.1 MORE ON ROBUSTNESS
Missing Patterns. To assess the robustness of our model on diverse missing patterns, we tested our model on five different types of missingness generated by the missing mechanism outlined in (Fortuin et al., 2020). Here, we describe the lata-genrrating process of each missing pattern as the folowing: Spatial: Define spatial Gaussian process with RBF kernel, and draw the missingness patterns by sampling from this process. Temporal+: Define temporal Gaussian process with RBF kernel for each feature dimension, and draw from this process. Temporal-: Implement with determinantial point process for each feature dimension.
Table Bl: Robustness on missing patterns, MNIST (NLL)
Pattern
Mean imp.
Forward imp.
VAE
HI-VAE
GP-VAE
Ours
Kandom
0.3300 = 0.0010
0.2218 + 0.0008
0.1957 = 0.0006
0,0887 = 0.0010
Spatial
0,0016
0.2259 + 0.0010
0.1779
0.1509 + 0.0038
Temporal-
0.1940 1 0.0006
0.1349
1 0.0005
0.1175
0.0004
0.0879 1 0.0004
Temporal+
0.1918 0,0006
0.1332
+ 0.0005
0.1206
0.0004
0.0927 + 0.0006
MNAR
0,4798 0.0016
0.2896
1 0.0010
0,2606
0.0008
0. 0,2017
1 0.0041
Table B2: Robustness on missing patterns, HealingMNIST (MSE)
imp.
Forward imp.
HI.VAR
GP-VAR
Ours
Rundon
0.069 1 0.000
0.099 0.000
0.100
0.000
0.046 1 0.001
0.036 1 0,000
0,036
0,000
Spatial
0.090 + 0.000
0.099 1: 0.000
0.122 1: 0.000
0.097 1: 0.000
0.071 4: 0.001
0.059
0.000
Temporal-
0.086 0.000
0.093 0.000
0.101
0.000
0.046 0.001
0.037 0.001
0.036
0.000
Temporal+
0.107 = 0.000
0.117 + 0.000
0.101
+ 0.000
0.017 = 0.001
0.038 + 0.001
0.037 + 0.000
MNAR
0.168
0.177 0.000
0.232
0.000
0.134 0.003
0.114 0.002
0.002
Missing Ratio.
Analogous to (Fortuin et al., 2020), we validate the robustness of our model on variety of different missing ratios using random missing patterns, ranging from 10% to 90%.
Table B3: Robustness on random missingness, HealingMNIST (NLL)
Missing
VAE
HI-VAE
GP-VAB
Ours
10% %
0.1319 0.0039
0.0783 1 0.0022
0.0695 1 0.0010
0.0666 1 0.0004
20% %
0.1506 0.0054
0.0844 1 0.0009
0.0744 1 0.0009
0.0688
0.0004
30 %
0.1637 + 0.0042
0.0874 + 0.0057
0.0799 = 0.0008
0.0734 +: 0.0005
%
0.1950 0.0061
0.0974 0.0102
0.0861
0.0782 0. 0.0003
50 %
0.2164 1 0.0046
0.1172 1 0.0127
0.0937 1 0.0012
0.0849 1 0.0005
0.2634 0.0108
0.1264 + 0.0070
0.1040 0.0008
0.0944
0.0001
70 %
0,2800 + 0.0051
0.1489 + 0.0034
0.1 = 0.0003
0.1088 + 0.0002
80 %
0.2928 0.0058
0.1698 1 0.0177
0.1408 1 0.0002
0.1317 1 0.0009
90 %
0.3071 0.0054
0.2290 + 0.0042
1852 0.0003
0.1691 0.0006
Table B4: Robustness random missingness, HcalingMNIITT (MSE)
Missing
VAE
HI-VAE
GP-VAE
Ours
10% %
0.0582
0.0023
0.0336 : 0.0009
0.0296
0.0004
0.0274 : 0.0008
20 %
0.0700 + 0.0030
0.0367 + 0.0000
0.0311 - 0.0004
0.0283 4: 0.0001
%
0.0857
0.0373 0.0029
0.0336
0.0305
0. 0.00
40 %
0.1185 1 0.0033
0.0420 1 0.0051
0.0362 1. 0.0000
0.0328 1 0.0001
%
0.1317 0.0006
0.0517 4 0.0063
0.0390
0.0005
0.0356
0.000
60 %
0.1328 0.0000
0.0561 1 0.0036
0.0436
0.0001
0.0402 1 0.0000
70 %
0.1328 1 0.0000
0.0681 + 0.0018
0.0505 + 0.0001
0.0470 1: 000000
80 %
0.1328 0.0000
0.0787
= 0.0108
0.0594 0.0001
0.0584
0.0008
90% %
0.1328 + 0,0000
0.1088 + 0.0076
0.0826 - 0.0009
0.0755 + 0.0002
16

---

Published as a conference paper at ICLR 2024
B.2 ABLATION STUDY THE CONDITIONAL RECONSTRUCTION.
As derived in Section 3.2, the conditional reconstruction term is given as the follows:
[(x;; Zx(X;,) - ((x;; Z, { I(x;; ()) To support claim that the effect of minimizing
[X;;; x deteriorates the model performance, we introduce an auxiliary weight w and make weighted version of the conditional reconstruction term as
T(X; Z, x [(X; Z, x (Xx;
(22)
Then, we can see the effect of I (X;; x by varying the value of w from by assessing the imputation performance on the Physioner2012 dataset. Figure shows that, starting from =
which is equivalent to the CIB objective in (8), focusing more on T(X;; x²,) gradually deteriorates
both the imputation performance (MSE) and the classification performance (AUROC).
0.75
0.74
0.73
-0.5
Dlsssiication
Imputation
0.5
Velg coofficlant
Figure Bl: The effect of w in the weighted conditional reconstruction in (22).
B.3 QUANTITATIVE ASSESSMENT (ZZ X(,)
In addition to the qualitative assessment on I(Z,; x..) shown in Figure and empirical performances in Section 5, compare the approximated quantity I(Z;; X using (9) to assert that CIB objective indeed preserves more information. Please note that, derived in (A.4), the lower bound is given by TZ; x 2 log N L. which depends on the batch size N and the objective L3 Table B5 shows the values of L3 indicating that a lower value corresponds more preserved information. This result demonstrates that our CIB objective preserves more information in [(Z;; x based on approximation.
Table B5: Quantitative results on (Z, X.)). Lower value indicates more preservation of information.
Dataset
VAE
HI-VAE
GP-YAE
Unitooo)))
Ours(Kemel)
RNII
3.619 + 0.005
3.077
0.008
3.435 4 0.001
3.428 0.000
3.418 = 0.002
Beijing
3.950 + 0.015
3.822 + 0.012
3.762 1 0.012
3.700 + 0.008
3.686 + 0.006
17

---

Published as a conference paper at ICLR 2024
B.4 SENSITIVITY ANALYSIS
Here, we provide the sensitivity analysis on 3 and y in (8). Before presenting results, we rewrite our objective follows analyze the effects of individual regularization terms given fixed reconstruction,
min 6.0
C3.0 + B C3 + +2
(23)
where 3' -1/3 and 7 7/B. Now, (23) shares the same notational convention used in 3-VAE (Higgins et al., 2017), where we can systematically assess the sensitivity of TimeCIB on the two regularization objectives.
032
SIR
(C_
P
TL'
(a) Sensitivity on
(b) Sensitivity on
Figure B2: Sensitivity analysis the regularization objectives, 3' and 7 7 0.5 for (a), and 3' 0.1 for (b).
Sensitivity on 3r. 8 is related to the regularization trade-off that low-8' (weak regularization) cause lack of timeseries wise information and eventually overfitting observed values, while high-3' (strong regularization) can cause loss of timepoint-wise information since all distribution would become unit Gaussian, well-studied by previous works.
Sensitivity on o. Since 7 also works as a regularization term, sensitivity on y is similar to 8", but the cause of behaviour slightly different. For low-y' (weak regularization), there no driving force on conserving temporal dynamics (see Section 3.1), thereby cause loss of temporal dynamics and would eventually become HI-VAE (Nazabal et al., 2020) when 7 = (strong regularization) cause loss of timepoint-wise information, since T. will map all z, from the same time series into single point, which means that we cannot distinguish individual timepoints. Figure B2 illustrates quantitative and qualitative results on the HealingMNIST.
the other hand, high-y'
Optimization Strategy. Simultaneous optimization of 3 and > could be difficult. However, motivated from the above sensitivity analysis, one can first set 7/=0 and find optimal 3r (which is HI-VAE) Second-stage parameter search within the close range of order of magnitude of 3 could help finding optimal 7/
18

---

Published as a conference paper at ICLR 2024
C IMPLEMENTATION DETAILS
Here, we provide implementation details, including illustration of overall frameworks, algorithms, and hyperparameters. The overall framework illustrated in Figure Cl.
Calculation of 402
96
min IN
4
9
Ni.
Vedimbe
Vaimbs
Figure CI: Illustration of TimeCIB framework.(A) Conceptual illustration TimeCIB framework. CIB maximizes [x(x;; z:), while minimizing Lø x² (x))) (B) Implementation of optimizing 1o(z),()).
C.1 TRAINING AND INFERENCE.
At training stage, input time sequences are first fed into the stochastic encoder go (Z. = (Xa(Xi (ii,)) diag(as (Xi,))) where
2 layers of bidirectional LSTM. For image dataset, we used CNN preprofessor before the encoder module. We then sample 2, from resulting latent (14 (z,), dgT) reconstructs the complete observation from the latent vector. In addition to previous conventional ELBO procedures, we also encodes X which are generated by masking X; with zeros. These masked input series are also mapped by the same encoder and latent vectors are obtained. Consequently, we use two sets of time series representations: Z1rr and æ1. for calculating (7). For training efficiency, we optimized (7) stochastically, which is, we randomly sampled at every epoch and minimized contrastive loss on that time step. At single prediction, we use the mean of the latent distribution na J (Xi (:T)) the corresponding latent representation and also reconstruct the observation from the mean the decoder distribution μu (zz). All experiments were conducted using 48GB NVIDIA RTX A6000.
distribution, and decoder pe (X, (X,
19

---

Published as a conference paper at ICLR 2024
C.2 ALGORITHM
We provide the pseudo-algorithm of TimeCIB as below:
Algorithm Conditional Information Bottleneck on Time Series
input batch size B, time length T for sampled minibatch (Xi.7 LB do
sample & {1: 7
generate masked input (
for {L...., 33} in parallel do
for t' E {1:7} in parallel do
encode and sample Z 2 (Z) 1
1.T = (N
1:7 o (x(()))) (T)
encode and sample Z qo Z
X -N(uo( (x.).) ool (x()) (Xook))))
decode po po X Zz) N Ze (iag((o(Zee)
L. L. log(po (x::)) [zr)) 2 C2 KL
25 X (0, [)]
end for
end for for
(I..... B in parallel
33 C3 + Contrastive Loss ( it: Z 2 (9)
end for
C = 3L 0.8 + 23 + +
update o to minimize
end for
C.3 HYPERPARAMETER SPECIFICATION.
provide hyperparameter settings used for our experiments.
Table Cl: Hyperparameter specifications.
Hidden Dim
Batch Size
Epochs
Leamingg Rate
Temperature 1.0
Kernel parameter
MNIST
128
64
30
2.0
RotasedMNIST
Beijing
128
64
100
le
1.0
4.0
Local
2.0
Physioner2012
16
256
50
1.0
32
For fair comparison, the magnitudes the number of parameters are set be equivalent among the evaluated deep learning methods.
Table C2: Number of learnable parameters.
GP-VAL HI-VAE
HealingMNIST
1.2.7
1.2M
1.2M
1.2M
RotatedMNIST
1.2M8
1.2M
1.2M
1.2M
Beijing
368
368K
368
368K
343K
362K
398
375
US Local
68K
G8K
68K
68K
51K
S7K
63K
62K
Physiome12012
29K
20

---

Published as a conference paper at ICLR 2024
D DATASET, BASELINES, AND MORE RELATED WORKS
D.I DATASET STATISTICS
We provide baseline statistics of the benchmark datasets.
Table D1: Data Statistics.
# Samples
Lea (2)
Feature Dim
# Classes
Missing Ratio (orslart)
HealingMNISS
60000/ 10000/ 10000
28 x 28
/607
RoiatedMNIST
50000/ 1000000000011111 /10000
28 x 28
/60%
Beijing
851 /304/ /306
24
132
258/60%
USLocal
8-77 /29/298/ 298
168
-160%
Pbysione:2012
3997 3997/39988 3993/3997
48
35
2
80%// 2%
D.2 BASELINE MODELS
HI-VAE (Nazabal al., 2020) HI-VAE proposes general generative framework that extends the VAE structure to be suitable for fitting incomplete heterogeneous data. Nazabal et al. (2020) proposed likelihood models for diverse data real-valued, positive real-valued, interval, categorical, ordinal, and count data and also expanded the conventional VAE to handle missing features. In this paper, we followed their approach to calculate reconstruction only to the observed features.
GP-VAE (Fortuin et al., 2020) GP-VAE expands information bottleneck time series impu- tation, by using Gaussian process prior in the latent space. To use the Gaussian process, GP- VAE's inference model encodes the input with respect to each latent dimension, (Z -((xxr))
(m,,,A,)) where j indexes the dimensions in the latent space. The latent prior is formalized by z (P(m= .)) where m and & are the mean and covariance functions, respectively. GP- VAE also utilizes the Cauchy kernel as a covariance function to capture temporal dynamics.
D.3 MORE RELATED WORKS
GAN-based imputation methods generate imputed values adversarially, assisted by masked re- construction loss, with the goal of making the generated samples closely resemble the original ones based on their observed values and indistinguishable from the original incomplete data by discriminator. GAIN (Yoon al., 2018a) uses a random masking and hinting mechanism to aid the discriminator in better distinguishing real and generated samples, thereby improving imputation performance. GRUI-GAN (Luo al., 2018) and extension EAGAN (Luo et al., 2019) adopted the GAN framework into temporal setting by enhancing the GRU cell to account for temporal dynamics, considering time lag influences related to observations and missing values.
21

---

Published as a conference paper at ICLR 2024
E
TABLE OF NOTATIONS
Notations
X1T
Complete data features of one time series
X
Observed features of one time series
XTT
Missing features of one time series Complete data features of one time series, except the time
X
X
Observed features of one time series, except the time t Missing features of one time series, exept the time Complete data features at time
X,
X;
Observed features at time I Missing features at time Representation features at time t, generated from X Representation features at time t, generated from X Set of parameters of an encoder
X
Z
Z.
8
Set of parameters of a decoder
22


Reviews:
[{"review_id": "eDYPKYlddy", "reviewer": "Reviewer_Vm5N", "summary": "The paper *Cconditional Information Bottleneck Approach for Time Series* extends the information bottleneck (IB) to deal with time series imputation by proposing a novel conditional information bottleneck. The authors also propose a novel deep learning method that can approximately achieve the proposed CIB objective for time series imputation using the lower bound and a novel temporal kernel. The proposed model is compared against some state-of-the-art VAE-based models for time series imputations under three different datasets, showing promising results and outperforming the other baselines.", "questions": "Here are some questions that I would like the authors to address:\n\n1. In Equation 7, in the denominator, why do you have a summation over X when you don’t have any variable x, just z? This also holds for Equation 9. I believe you mean that you sample from the encoder to get z out of x. However, I think the equation could be improved. \n2. The samples from Equation 4, how are they taken? How many samples? Is the method sensible to the number of samples? Do the results improve by taking more samples? Did you consider the following approaches such as MIWAE? \n3. Why do you always outperform all baselines, in all datasets under any missing data assumption? I find it rather surprising given that you have a loss (Equation 8) composed of three terms, which I believe must be difficult to optimize. Besides, there is not much information about the $\\beta$ and $\\gamma$ parameters. How do they influence the optimization? There is no discussion about this. \n4. What do black crosses and black dots represent in Figure 3? I believe dots are observations and crosses missing points, but this is not commented on. \n5. Looking at Figure 3a), for example: how is it possible that HIVAE and GP-VAE obtain such poor results compared to your proposed method? I assume this is the case for HIVAE since it is not designed to deal with temporal data, but for GP-VAE this is very surprising. As described in [2], I would expect this to happen in the missing parts, where the GPVAE would produce a more “mean” solution less correlated to the original signal. However, in the observations, I would expect the GP-VAE to obtain better results. \n6. In Table 3 you have an interesting result. In terms of RMSE, it can be observed that mean imputation (one of the most naive approaches) outperforms any baseline, even state-of-the-art methods. What do you think about this result? As observed in [2], sometimes error metrics are inconclusive when evaluating temporal scenarios. Did you find this behaviour in other datasets? Did you think about the possibility of instead of using missing data randomly or following other mechanisms (which you withdrew from GPVAE and other papers I believe) using missing data in burts or sequences, where actually it is the standard scenario you would find in a temporal scenario such as in healthcare? This could be an interesting point to be analyzed here: check whether the solutions from baselines and proposed methods are more correlated.  \n\n[1] Mattei, P. A., & Frellsen, J. (2019, May). MIWAE: Deep generative modelling and imputation of incomplete data sets. In *International conference on machine learning* (pp. 4413-4423). PMLR.\n\n[2] Barrejón, D., Olmos, P. M., & Artés-Rodríguez, A. (2021). Medical data wrangling with sequential variational autoencoders. *IEEE Journal of Biomedical and Health Informatics*, *26*(6), 2737-2745.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-structured and easy to follow. The motivation and the proposed method are very well described in the manuscript and nicely derived in the Appendix. The more tricky points are also well described (e.g. the application of the chain rule in Equation 5). The related work section is also nicely depicted, making clear that the authors are aware of the most recent works for missing data imputations in temporal series. Finally, the results are concise and clear to read. It is evident from the results that their proposed method opens very prospective lines for future research in missing data imputation in time series. \n\nFurthermore, the paper holds interest from an information bottleneck perspective in two significant ways. Firstly, it extends the information bottleneck concept to a conditional information bottleneck, representing a novel contribution in its own right. Secondly, the introduction of a fresh objective for VAE-based models to address missing data concerns is noteworthy. The authors follow the standard assumptions for handling missing data using VAEs, such as evaluating the objective on the observations of filling the missing points with zeros. However, the paper sets itself apart not solely due to its new objective based on the conditional information bottleneck but also because of its capacity to generate fresh avenues of research and stimulate discussions during the conference", "weaknesses": "I would like to point out some weaknesses that I consider the paper exhibits and that could be further analyzed by the authors:\n\n1. As many papers dealing with temporal data, sometimes notation becomes a bit complex and makes the reading more difficult. \n2. I believe more information about the experimental section could be provided. That, is, for HealingMNIST and RotatedMNIST, which kind of CNN-based encoder/decoder architecture is being used; for Physionet, which kind of networks are used? There is no information about this, especially for the proposed method, since for baselines this can be accessed in their referred papers. And this is quite important when asking why the proposed method outperforms any other baseline. This leads me to the following point. \n3. I am a bit hesitant about the superior performance of the proposed method in (almost) every dataset and performance metric. This creates some doubts that will be asked in the question section.\n4. There is no reference for code availability to reproduce the experiments. Will you upload the code to Git Hub or any available repository? This is rather important for the sake of reproducibility. And given that the results are (fairly always) better than baselines, this should be a must-have for this paper. \n5. I find the paper rather novel in terms of the proposal of the CIB objective. However, in terms of the VAE-based model, I believe there is much inspiration coming from GPVAE [1], due to the idea of including an extra kernel to further enhance the learning of the temporal dynamics. Is this the case? Should the authors be more worried about properly referencing this work?\n6. An ablation study should be included to determine the importance of each contribution in the paper. That is, i) how the different terms in the proposed objective influence the optimization ii) how important is it to introduce the kernel — without the kernel, would the results approach the ones from GPVAE, or does the proposed model still outperform other baselines? I think it is important to analyze each contribution of the paper to obtain a clear and detailed view of the behaviour of the proposed model. \n\n[1] Fortuin, V., Baranchuk, D., Rätsch, G., & Mandt, S. (2020, June). Gp-vae: Deep probabilistic time series imputation. In *International conference on artificial intelligence and statistics* (pp. 1651-1661). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lY5p4nZOM9", "reviewer": "Reviewer_mVWF", "summary": "This paper focuses on time series imputation. The authors believe that information bottleneck (IB) is a well-suited theoretical foundation for imputation task. While directly applying conventional IB framework may results in substantial loss of temporal dependencies, the authors propose a novel conditional information bottleneck (CIB) approach. The proposed CIB suggests the model to learn more information about latent representation from temporal contexts. The idea is novel and interesting, which is supported by some theoretical analysis and experimental result.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed TimeCIB is novel and technical sound.\n2. The theoretical analysis is solid, making the whole framework more persuasive and reliable.\n3. Smoothing the latent representation is interesting and novel.", "weaknesses": "1. The motivation of using conditional term $X^o_t | X^o_{\\\\t}$ should be further explained. Why using the conditional term can tackle the issue of conventional IB?\n2. A figure of the whole framework will be helpful for understanding this work.\n3. The compared baselines are out of date. There exist massive works on time series imputation, the authors should compare their method with more recent works.\n4. It would be appreciated if the authors could provide code.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EKAXv9PksL", "reviewer": "Reviewer_yyLt", "summary": "Time series with missing values are prevalent in real-world applications. The paper proposes the use of an improved information bottleneck approach for irregular time series imputation, achieving superior performance compared to other VAE-based methods, as demonstrated in the experiments.", "questions": "1. What are the unique aspects of the CIB approach compared to other methodologies, including ODE, transformers, and diffusion models? Can you provide a more comprehensive evaluation, including comparisons with state-of-the-art models like ODE-based models and diffusion models, to demonstrate the performance of your model?\n\n2. The paper claims that the CIB approach can mitigate the excessive regularization associated with IB. However, there seems to be a lack of quantitative analysis to substantiate this claim.  It would be beneficial if the authors could provide explicit metrics or criteria to measure the extent of regularization and demonstrate how CIB effectively addresses this issue.\n\n3. Mitigating regularization is a double-edged sword, as it might lead to overfitting, especially in the context of learning temporal dependencies in time series data.  The paper should address how the CIB approach ensures an optimal balance, preventing the model from overly adapting to the training data and subsequently degrading its performance on unseen data.  Is there any empirical evidence or theoretical analysis to showcase this balance?\n\n4. Using MINIST in the illustration of temporal dependency (Figure 1) is dubious compared to other irregular time series datasets (e.g., weather, traffic). It cannot demonstrate how the model handles temporal dependencies between different dimensions in multi-variate situations. How does the CIB approach model handle the multi-variate dependencies exactly?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The model effectively addresses the issue of temporal information loss in the latent space of VAEs.", "weaknesses": "1. The paper exclusively focuses on Variational Autoencoder (VAE) models grounded in Information Bottleneck (IB) theory.  While this approach is well-articulated, it is notable that other kinds of generative models, for instance, ODE-based models and diffusion models, have demonstrated remarkable performance in the field of time series imputation.  The absence of a discussion or comparison with SOTA models in the related work and experimental sections is conspicuous, making it less convincing regarding its contribution to the irregular time series imputation task.\n\n2. While the paper introduces a novel regularization method in the context of time series imputation, the innovation appears to be somewhat incremental. The approach can be perceived as a clever technique or ‘Trick’ rather than a substantial paradigm shift.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OUTa0jdnGJ", "reviewer": "Reviewer_wRxf", "summary": "This paper propose a novel time series imputation method inspired from the perspective of information bottleneck (IB). To solve the temporal context loss issue of vanilla IB, they propose a conditional IB to better consider temporal context. Several lower bound and upper bound are deducted for optimization. In a specific contrast learning loss, a kernel trick is used to reweright contribution of data from different time steps. Experiments results show that the proposed method achieve SOTA performance.", "questions": "None", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The method is inspired from a theoritical background of information bottleneck and can impute stochastically. \n2. The bound derivations are interesting.", "weaknesses": "1. In equation 4, both the entropy term and the KL term are dropped to compute the lower bound. Will this bound be too loose? After dropping, the lower bound looks like a reconstruction loss rather than mutual information.\n\n2. The paper doesn't mention the number of learnable parameters for each methods.\n\n3. I'd like to see comparison with another two baselines mTAN [1] and CSDI [2].\n\n[1] Multi-Time Attention Networks for Irregularly Sampled Time Series\n[2] CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)