Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models
Authors: Juan Lopez Alcaraz
Nils Strodthoff
Venue: Accepted by TMLR ()

Abstract: The imputation of missing values represents a significant obstacle for many real-world data analysis pipelines. Here, we focus on time series data and put forward SSSD, an imputation model that relies on two emerging technologies, (conditional) diffusion models as state-ofthe-art generative models and structured state space models as internal model architecture, which are particularly suited to capture long-term dependencies in time series data. We demonstrate that SSSD matches or even exceeds state-of-the-art probabilistic imputation and forecasting performance on a broad range of data sets and different missingness scenarios, including the challenging blackout-missing scenarios, where prior approaches failed to provide meaningful results.

Published in Transactions on Machine Learning Research (02/2023)
Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models
Juan Miguel Lopez Alcaraz
www.looper.alcaroo@@oo..dd
Division AeşHea
Oldenburg University
Nils Strodthoff
mils.srrodthoff@wol.ddd de
Division AIfHealth
Oldenburg University
Reviewed on OpenReview: https: // openreview. net/ net/forum? id= hHbIbbAPp
Abstract
The imputation of missing values represents a significant obstacle for many real-world data analysis pipelines. Here, we focus on time series data and put forward SSSD, an imputation model that relies on two emerging technologies, (conditional) diffusion models as state-of- the-art generative models and structured state space models as internal model architecture, which are particularly suited capture long-term dependencies time series data. We demonstrate that SSSD matches or even exceeds stale-of-the-at probabilistic imputation and forecasting performance broad range of data sets and different missingness scenarios, including the challenging Bacckout-missing scenarios, where prior approaches failed to provide meaningful results.
1 Introduction
Missing input data is a common phenomenon in real-world machine learning applications, which have many different reasons, ranging from inadequate data entry over equipment failures file losses. Handling missing input data represents a major challenge for machine learning applications as most algorithms require dala without missing values to train. Unfortunately, the imputation quality has a critical impact on downstream tasks, as demonstrated in prior work (Shadbahr et al., 2022); : and poor imputations can even introduce bias into the downstream analysis (Zhang et al. 2022] which can polentially call into question the validity of the results achieved in them.
In this work, focus on time series data modality, where missing data is particularly prevalent, for example, due to faulty sensor equipment. We consider range of different missingness scenarios, see Figure [ for visual overview, where the example of faulty sensor equipment former example already suggests that not-at-random missingness scenarios are significant for real-world scenarios. Time series forecasting is naturally contained in this approach as special case of blackout missingness, where the location of the imputation window at the end of the sequence. We also stress that the most realistic scenario to address imputation as an underspecified problem class is the use of probabilistic imputation methods, which do not provide only a single imputation but instead allow samples of different plausible impulations.
There is a large body of literature time series imputation, see (Osman et al. 2018 for review, ranging from statistical methods (Lin & Tsai, 2020) to autorogressive models (Atyabi et al. 2016 Bashir & Wei 2018). Recently, deep generative models started to emerge as a promising paradigm to model time series imputation of long sequences or time series forecasting problems at long horizons. However, many existing models remain limited to the random missing scenario or show unstable behavior during training. In addition, we demonstrate that state-of-the-a approaches even fail deliver qualitatively meaningful imputations in blackout missing scenarios certain data sets.

---

Published in Transactions on Machine Learning Research (02/2023)
RM
RBM
BM
TF
Color scheme introduction. The proposed model SSSD*4 provides imputations for different missingness scenarios that not only quantitatively but qualitatively superior, see below, different data sets for different missingness scenarios (RM: random missing, RBM: random block missing, BM: blackout missing TF: time series forecasting). The signal is in blue, where the white background represents the conditioned ground truth, whereas the gray background represents time steps in specific channels to imputed. Prediction bands derived from 100 imputations represent quantiles from 0.05 to 0.95 in light green and from 0.25 0.75 in dark groen. As these bands do not allow visually assessing the quality of individual imputations, always additionally show randomly selected single sample in orange.
Figure 1:
In this work, we aim to address these shortcomings by proposing a new generative-model-baare approach for time series imputation. We use diffusion models as the current statc-of-the-art in terms generative modeling in different data modalities. The second principal component of our approach the use of structured state- space models (Gu et al., (2022a) instead of dilated convolutions or transformer layers main computational building block of the model, which particularly suited to handling tung-termmddeeendeniee in time series data.
To summarize, our main contributions are as follows: (1) We put forward combination of state-space models as ideal building blocks to capture long-term dependencies in time series with (conditional) diffusion models as the current state-of-the-art technology for generative modeling (2) We suggest modifications to the contemporary diffusion model architecture DifnWave[Kong et al. 2021] to enhance
capability for time series modeling. In addition, we propose a simple yet powerful methodology in which the noise of the diffusion process is introdueed just. to the regions to be imputed, which turns out to be superior to approaches proposed in the context of image inpainting (Lugmayr et al.) (022). (3) We provide extensive experimental evidence for the superiority of the proposed approach compared to stato-of-tho-art approaches on different data sets for various missingness approaches, particularly for the most challenging blackout and forecasting scenarios.
2 Structured state space diffusion (SSSD) models for time series imputation
2.1 Time series imputation
Let Zo be a data sample with shape Lx where L represents the number of time steps, and K represents the number of fcatures or channels. Imputation targets are then typically specified terms of binary masks that match the shape of the input data, i.., mamp E {0, 1) 1x where the values be conditioned on are marked by ones and zeros denote values to be imputed. In the case where there also missing values in the
2

---

Published in Transactions on Machine Learning Research (02/2023)
imput, one additionally requires mask Mavi of the same shape to distinguish values that are present in the input data (1) from those that are not (0).
One approach of categorizing missingness scenarios in the imputation literature (Rubin 1976 Lin & Tsai 2020 Osman et al. 2018) is according to missing completely at random, missing at random and random block missing, where the first case the missingness pattern does not depend on feature values, in the second case it may depend on observed features and in the last case also on ground truth values of the features to be imputed. Here, focus the first category and follow Khayati et al. (2020) for different popular missingness scenarios in the time series domain: Rondom missing (RM), corresponds to situation where the zero-entries of an imputation mask are sampled randomly according to uniform distribution across all channels of the whole input sample. In contrast to Khayati et al. (2020), we consider single time steps for RM instead of blocks of consecutive time steps. This is different for the remaining missingness scenarios considered in this work, where we first partition the sequence into segments of consecutive time steps according to the missingness ration. Firstly, for random block missing (RBM) we sample one segment for each channel imputation larget. Secondly, blackoul. missing (BM) we sample single segment across all channels, i.e., the respective segment is assumed to be missing across all channels, and lastly, time-series forecasting (TF) which can be seen particular case of BM imputation, where the imputation region spans consecutive region of t time steps, where t denotes the forecasting horizon across all channels located at the very end of the sequence. The missingness scenarios are described in more detail in Appendix A.3
2.2 Diffusion models
Diffusion models (Sohl-Dickstein et al., 2015 represent a class of generative models that demonstrated
state-of-the-art performance on a range of different data modalities, from image (Dhariwal & Nichol 2021
Ho et al. 2020 2022ah over speech (Chen el al., 2020 Kong el al. 2021 to video data (Ho et al. b022b Yang et al. 2022).
Diffusion models learn mapping from latent space to the original signal space by learning to remove noise in backward process that was added sequentially in Markovian fashion during so-called forward process. These two processes, therefore, represent the backbone of the diffusion model. For simplicity, we restrict ourselves to the unconditional case at the beginning of this section and discuss modifications for the conditional case further below. The forward process is parameterized as
(( IT ((110)
II c=1
(1)
where (())(x;,1)== = B, and the (fixed or learnable) forward-process variances 3r adjust the noise level. Equivalently, x, can be expressod in closod form as 24 / Vaizo + (1 at)e for c e~ N00 1), where 04 = Ei= - (.). The backward process is parameterized as
T
Po ... Fo- = p(æT) II (o(
(2)
where IT ~ NO 1). Again, (= is assumed as normal-distrrbbteed (with diagonal covariance matrix) with learnable parameters. Using a particular parametrization of po( Ho et al. (2020) showed that the reverse process can be trained using the following objective,
L
so(varo t)),
where D refers to the data distribution and c0(#,, t) is parameterized using a neural network, which is oquivalent earlier score-matching tochniques (Song Ermon 2019 Song et al.) 2021 This objective can be seen a weighted variational bound on the negative log-likelihood that down- weights the importance of terms at small I, i.e., at small noise levels.
3

---

Published in Transactions on Machine Learning Research (02/2023)
Extending the unconditional diffusion process described so far, one can consider conditional variants where the backward process is conditioned on additional information, i.e. co c0( L, where the precise nature of the conditioning information с depends on the application at hand and ranges from global to local information such as spectrograms <Kong et al. 2021), see also (Pandey et al. 2022) for different ways of introducing conditional information into the dillusion process. In our case, is given by the concatenation of input (masked according to the imputation mask) and the imputation mask itself during the backward process, i.e., c= Con O (Mimp O Mmvi), (Min C mmvi), where @ denotes point-wise multiplication, Mimp is the imputation mask and Mavi is the missing value mask. In this work, we consider two different setups, denoted as Do and D1, respectively, where we apply the diffusion process to the full signal or to the regions to be imputed only. In any case, the evaluation of the loss function in Table 3 is only supposed to be on the input values for which ground truth information is available, i.e,, where Mmvi = 1. For Do. this can be seen as a reconstruction loss for the input values corresponding to non-zero portions of the imputation mask (where conditioning is available) and an imputation loss corresponding to input tokens at which the imputation mask vanishes, c.f. also (Du al.] 2022). For Di, the reconstruction loss vanishes by construction. Finally, we also investigate an approach using model trained in unconditional fashion, where the conditional information is only included during inference (Lugmayr et al. 2022). In Appendix [A.1 provide detailed comparison to
et al., 2021). as the only other diffusion fifussioo-based imputer in the literature, which crucially relates to the design decision whether to introduce a separate diffusion axis.
2.3 State space models
The recently introduced structured state-space model (SSM) (Gu et al. 2022a) represents a promising modeling paradigm to efficiently capture long-term dependencies in time series data. At its heart, the formalism draws on linear state space transition equation, connecting a onc-dimensional input sequence u(t) to one-dimensional output sequence y()) via N-dimensional hidden state sx((). Explicitly, this transition equation reads
(1(1) Ax(t)= + Bu(t) and v(t) = Cx((t) + Du(t),
(4)
where A, B, C, D are transition matrices. After discretization, the relation between input and output can
be written as a convolutionn operation that can be evaluated efficiently on modern GPUs (Gu et al 2022a).
The ability to capture long-term dependencies relates to a particular initialization of A E R according to HiPPO theory (Gu et al.] 2020 2022b). In (Gu et al.) 2022a), the authors put forward a Structured State Space sequence model (S4) by stacking several copies of the above SSM blocks with appropriate uormalization layers, point-wise fully-connected layers in the style of transformer layer, demonstrating excellent performance on various sequence classification tasks. In fact, the resulting S4 layer parametrizes a shapo-preserving mapping of data with shape (batch, model dimension, length dimension) and can therefore be used as drop-in replaccment for transformer, RNN, or one-dimensional convolution layers (with appropriate padding). Building on the S4 layer, the authors presented SaShiMi, a generative model architect for sequence generation (Goel et al.) 2022) obtained by combining S4 layers in a U-Net-inspired configuration. While the model was proposed as an autoregressive model, the authors already pointed out the ability to use the (non-causal) SaShiMi as component in stale-of-the-art non-auloregressive models such as DiffWave (Kong et al. 2021
These models as well as the proposed approaches from Section 2.4 stand in a longer tradition of approaches that rely on combinations of state space models and deep learning. These have been used for diverse tasks, such as for the modeling of high-dimensional video data, which also might allow for imputation Fraccaro et al. (2017: Klushyn et al. (2021), , for causal generative temporal modeling Krishnan et al. (2015), and for forecasting Rangapuram et al. (2018); Wang et al. (2019); Li et al. (2021).
2.4 Proposed approaches
We propose and investigate a number of different diffusion imputer architoctures. It is worth stressing that there is no prior work on direct applications of conditional diffusion models for time series imputation except.
4

---

Published in Transactions on Machine Learning Research (02/2023)
Input
ID Cuny Rea
ID Cony
ota
of))
(B.C.L)
A.R.C.T.)
BeC
(R,C..))
Conditional
Si
ID Conv (B,C.L)
(B.C.L)
(moup
11 Come
84
Addition
Diffusion embedding
(R.2"C.J)
(B.2
(32
ID Cony
T-Emb
C.F.
Sigmeid
(U,128)
(M.512)
(B,512)
(B.C.)
(B.C.L)
Rescched byee
Tunh
Rk
RoIn
$4
S4 layer
Residual lorer
Sta
Swish
T-Emb
Diffiasion T embbedinng
Conv
Ria
Cocy
Skip commettion
(3, Skip C.L)
Multiplication
Fully connected
(BKL)
Output
+
Addition Assignment
ID Conv
ID convolution
(B,K,I)
Figure 2: Proposed SSSD model architecture.
for (Tashiro al. 2021), which is based DiffWave, which was proposed in the context of speech synthesis. Thus, propose direct adaptation of the DiftWave-architeccure (Kong et al.) 2021 baseline for time series imputation, see Appendix C for details. In particular, we use the proposed setup where the diffusion process is only applied to the parts of the sequence that are supposed to imputed. Based on this model, we put forward SSSD*4 the main architectural contribution of this work, where instead of bidirectional dilated convolutions, we used S4 layer as a diffusion layer within each of its residual blocks after adding the diffusion embodding. As a second modification, we include a second S4 layer after the addition assignment with the conditional information, which gives the model additional flexibility after joining processed inputs and the conditional information. The effectiveness of this modification is demonstrated in an ablation study in Appendix B.1. The architecture is depicted schematically in Figure Second, under the name SSSD we explore an extension of the non-autoregressive of the SaShiMi architecture for time series imputation through appropriate conditioning. Third, we investigate CSDIS1 modification of the CSDI (Tashiro et al. 2021) architecture, where we replace the transformer layer operating in the time direction by an S4 model. this way, we aim assess potential improvements of an architecture that is more adapted to the domain of time series. A more detailed discussion of the model architectures including hyperparameter settings can be found in Appendix C By construction, though this could be circumvented by the use of causal S4 layers). There are situations where this use of bidirectional context could lead to data leakage, e.g. when using the imputed data as input for a forecasting task as downstream application.
imputation models always use bidirectional context (even
At this point, we recapitulate the main components the proposed approach: The model architecture was described in this very section, which crucially relies on the ability to use state-space model layers, see Section 2.3 as a direct replacement for convolutional layers in the original Diffwave architecture. The loss function is described in Equation
(3). As described in Section 2.2 only apply noise the portions of the time series to be imputed with training and sampling procedures described explicitly along with pseudocode in Appendix A.4
3 Related Work
Decp-learning based time scries imputation Time series imputation a very rich topic. A complete discussion- even of the deep learning literature alone- clearly beyond the scope of this work, and we refer the reader recent review the topic (Fang & Wang 2020 Deep-learning-base based time series imputation

---

Published in Transactions on Machine Learning Research (02/2023)
methods can be broadly categorized based on the technology used: (1) RNN-based approaches such BRITS (Cao et al., 2018), GRU-D (Che et al. 2016), NAOMT (Liu et al., 2019), and M-RNN (Yoon et al. 2019) use single or multi-directional RNNs to model the time series. However, these algorithms suffer of diverse training limitation, and as pointed out in recent works, many of them might only show sub-optimal
performance across diverse missing scenarios and at different missing ratios (Cini et al. 2022) (Du el al.
2022). (2) Generative models represent the second dominant approach in the field. This includes GAN-basod
approaches such as E2-GAN (Luo al. 2019), and GRUI-GAN (Luo et al. 2018) or VAE-based approaches such GP-VAE (Fortuin et al. 2020). Many of these were found to suffer from unstable training and failed to reach state-of-the-art performance (Du et al. 2022). Recently, also diffusion models such CSDI (Tashiro let al. 2021), the closest competitor to our work, were explored with very strong results. (3) Finally, there is a collection of approaches relying in modern architectures such as graph neural networks (GRIN) (Cini el al., 2022), and permutation equivariant networks (NRTSI) (Shan et al. 2021), self-attention to capture temporal and feature correlations (SAITS) (Du al. 2022), controlled differential equations networks (NouralCDE)
(Morrill et al. 2021) and ordinal differential equations networks (LatenI-ODE) (Rubanova et al., 2019
Conditional generative modeling with diffusion models Diffusion models have been used for related tasks such impainting, in particular in the image domain (Saharia et al. 2021 Lugmayr al. 2022 With appropriate modifications, such methods from the image domain are also directly applicable in the time series domain. Sound is a very special time series, and diffusion models such as DiffWave (Kong ct al. 2 conditioned on different global labels or Mel spectrograms showed excellent performance in different speech generation tasks, Returning to general time series, as already discussed above, the closest competitor CSDI (Tashiro et al. 2021). CSDI and this work represent diffusion models and can be seen as viiaavv-vvrrants. The main differences between the two approaches are (1) using SSMs instead of transformers (2) the conceptually more straightforward setup of a diffusion process in time direction only opposed to feature and time direction, see a further discussion of it in Appendix A.1 (3) different training objective to denoise just the segments to be imputed (D1). In all of our experiments, we compare to CSDI to demonstrate our approach's 's superiorily and that exchanging single components (as in CSDI$A) is not. sufficient to reach a qualitative improvement of the sample quality in certain BM scenarios.
Time series forecasting The literature on time series forecasting is even richer than the literature on time series imputation. One large body of works includes recurrent architectures, such as LSTNet (Tai et al. 2018), and LSTMa (Bahdanau al.] 2015). Recently, also modern transformer-based architectures with encoder-decoder design such Autoformer (Wu et al.] 2021), and Informer (Zhou et al. 2021), showed excellent performance on long-sequence forecasting. The range of methods that have been applied to this task is very diversc and includes for example GP-Copula (Salinas et al. 2019). Transformer MAF (Rasul et al. 2021b), and TLAE (Nguyen & Quanz 2021). Finally, with Time- Gran (Rasul et al. 2021ab there is another diffusion model that showed good performance on various forecasting tasks. However, due to its autoregressive nature its domain of applicability cannot be straightforwardly extended to include time scries impulation.
4 Experiments
4.1 Experimental protocol
As already discussed above, we do not keep the (input) channel dimension as an explicit dimension during the diffusion process but only keep it implicitly by mapping the channel dimension the diffusion dimension. This is inspired by the original DiffWave approach, which was designed for single-channel audio data. As will demonstrate below, this approach leads to outstanding results in scenarios where the number of input channels remains limited to less than about 100 input channels, which covers for example typical single-patient sensor data such as electrocardiogran (ECG) or electroencephalograa (EEG) in the healthcare domain. For more input channels, the model often shows convergence issues and one has to resort to different training strategics, c.g., by splitting the input channels. As we will also demonstrate below, this approach without any further modifications already leads to competitive results on data sets with few hundred input channels, but can certainly be improved by more elaborate procedures and is therefore not within the main
6

---

Published in Transactions on Machine Learning Research (02/2023)
scope of this work. For this reason also the mains experimental evaluation of our work focuses primarily on data sets with less than 100 input channels, Section A.2 in the supplementary material contains a more detailed overview of the channel split approach and provide further research directions on the same matter.
Throughout this section, we always train and evaluate imputation models on identical missingness scenarios and ratios, e.g., we train on 20% RM and evaluate based on the same setting. The training on the models was performed on single NVIDIA A30 cards. We use for all of our experiments MSE as loss function, similarly, different batch sizes for each dataset implemented, see details Appendix for details. In our experiments, we cover very broad spectrum of popular data sets both for imputation and forecasting along with a corresponding diverse selection of bascline methods from different communitics, to demonstrate the robustness of our proposed approach. There are diverse performance metrics utilized in this work (where in all cases, lower scores signify better imputation results), most of them involve comparing single imputations to the ground truth, others, incorporate imputations distribution and are therefore specific to probabilistic imputers. We refer to the supplementary material for discussion on them. Similarly, we present small and concisc tables in the main text. to support our claims. We refer to the supplementary material for more results, including additional baselines, further details on data sets, and preprocessing procedures.
As general remark, we try to quote baseline results directly from the respective original publications to avoid misleading conclusions due to sub-optimal hyperparameter choices while training baseline models ourselves. Quite naturally, this leads to a rather in-homogeneous set of baseline models available for the different datasets/taske taaks. To ensure better comparability, we nevertheless decided to train a single strong baseline across all tasks for both diffusion and imputation, where we explicitly mark baseline results obtained by ourselves. We chose CSDI (Tashiro al. 2021 for this purpose, which is, on the one hand, our closest competitor the only non-autoregressive diffusion model and, on the other hand, which represents strong baseline both for imputation and forecasting, see the original publication. In terms of metrics, we report the commonly used metrics for the respective datasets.
4.2 Time series imputation
SSSD outperfors state-of-the-art imputers on ECG data As first data set, we consider ECG data from the PTB-XL data set (Wagner al.) 2020a by Goldberger et al., 2000). ECG data represents an interesting benchmark case as producing coherent imputations beyond the random missing scenario requires to capture the consistent periodic structure of the signal across several beats. We preprocessed the ECG signals at sampling rate of 100 Hz and considered L = 250 time steps (248 in the case of SSSDSA). We considered three different missingness scenarios, RM, RBM, and BM. We present the investigated diffusion model variants, such as and conditional adaptation of the original DiffWave model, CSDIS1 SSSDSA, and SSSDS4 where applicable both for training objectives Dy and D1. As basclines we consider the deterministic LAMC (Chen el al. 2021) and CSDI (Tashíro el al.] 2021) as a strong probabilistic baseline. We report. averaged MAE, RMSE for 100 samples generated for each sample in the test set.
Table I contains the empirical results of the time series imputation experiments on the PTB-XL dataset. Across all model types and missingness scenarios, applying the diffusion process to the portions of the sample to be imputed (Dt) consistently yields better results than the diffusion process applied to the entire sample(Do). Also the unconditional training approach from Lugmayr et al. (2022) proposed in the context of a state-of-the-an diffusion- based method for image inpainting, lead to clearly inferior performance, see Appendix
B.1 In the following, we will therefore restrict ourselves to the D2 setting. The proposed SS outperforms the rest of the imputer models by a significant margin in most scenarios, in particular for BM, where we find reduction in MAE of more than 50% compared to CSDI. Similarly, we note that DiffWave baseline shows very strong results, which are in some scenarios on par with the technically more advanced SSSDSA nevertheless, the proposed SSSDS4 demonstrate a clear improvement for time series imputation and generation across all the settings. Also, there is a clear improvement. that the S4 layer on CSDIS4 provides to CSDI across RM and BM settings, especially for the RM scenario where CSDI outperforms the rest of methods with lower MAE. We hypothesize that the CSDI-approach is helpful in the RM sctting, where consistency across features and time has to be reached, whereas $ its variants show clear advantages for RBM and BM (and TF as discussed below) where modeling the time dependence is of primary importance.
7

---

Published in Transactions on Machine Learning Research (02/2023)
Table 1: Imputation for RM, RBM and BM scenarios on the PTB-XT. data set (extracts, see Table 24 for further bascline and scttings comparisons, and Table 7 for missingness ratio 50%). All results (including baselines) were obtained from models trained by us.
Model
MAE
RMSE
20% RM on PTB-XI
LAMC
0.0678
0.1309
CSDI
0.0038=2e-6
0.018945e-5
DiffWave
0.0043+40-4
0.0177++4c44
CSDI
0.0031+1e-7
0.0171=66-4
SSSD
0.0045=3e-7
0.0181+44-6
SSSD
4e00
0.0119:1e-4
20% RBM on PTB-XL
LAMC
0.0759
0.1498
CSDI
1e0186=1e-5
0.0435++ee4
DiffWave
0.0
0.080815e-3
CSDI
0.0222-20-5
0.0573+1c-3
SSSDSA
0.0170=1e-4
0.0492+1e-2
SSSD
3e0
0.0226119-4
20% BM on PTB-XI.
LAMC
0.0840
0.1171
CSDI
0.1054=4e-5
0.2254:1e-55
DiffWave
0.0451=7e-4
0.137815e-3
CSDr
0.0792-20-4
0.1879+1c-4
SSSD
0.0435=3e-3
0.1167+1e-2
SSSD
0.0
0.0832:18-3
Target
Target
0.5
value
0.5
0.0
0.0
150
250
0
50
100
150
200
250
Target
Target DirtWave
Rt Unconditional training
value
value
0.5
0.0
100
150
200
250
50
100
150
200
250
Target 5SSO
Target S$50FF
value
value
0.5
100
150
2000
250
Figure 3: PTB-XL BM imputations for the V5 lead of an ECG from healthy patient.
Existing approaches fail produce meaningful BM imputations Figure shows imputations on a BM scenario for a subset of models from the PTB-XL imputation task. The main purpose is demonstrate that the achieved improvements through the proposed approaches lead to superior samples in way that is

---

Published in Transactions on Machine Learning Research (02/2023)
even apparent to the naked eye. The top figure demonstrates that the state-of-the-art imputer is unable to produce any meaningful imputations (as visible both from the shaded quantiles well as from the exemplary imputation). As an example, the identification of a QRS complex with a duration inside the range of 0.08 sec to 0.12 see is considered as normal signals. However, the model fails to detect the complex and rather shows a misplaced peak. The impulation quality improves qualitatively with the proposed CSD1 variant. but still misses essential signal features. Only the proposed DiffWave imputer, SSSD and SSSD models capture all essential signal features. The qualitatively and quantitatively best result is achieved by SSSD$1 which excels at the task and shows well-controlled quantile bands as expected for normal ECG sample. We present additional experiments at larger missingness ratio of 50% in Appendix B.2 to underline the robustness of our findings. In addition, we investigate in Appendix B.4 the robustness of the SSSD model against mismatcces between training and test time in terms of missingness scenario and missingness ratio, which is of crucial importance for practical applications. The established benchmarking tasks for imputation tasks are very much focused on the RM scenario.
SSSD S4 shows competitive imputation performance compared to state-of-the-art approaches on other data sets and high missing ratios To demonstrate that the excellent performance of SSSD extends to further data sets, we collected the MuJoCo data set (Rubanova et al. 2019) from Shaa et al. (2021) to test SSSD on highly sparse RM scenarios such as 70%, 80%, and 90%, for which we compare performance against the baselines RNN GRU-D (Che et al.] 2016), ODE-RN ENN- (Rubanova et al. 2019). NeuralCDE (Morrill et al., (021), Latent- (Rubanova et al. 2019), NAOMI (Liu al., 2019). NRTST (Shan et al., 2021), and CSDI (Tashiro et al., 2021). We report an averaged MSE for single imputation per sample the lest set over 3 trials. All baselines results were collected from Shan al. (2021 except. for CSDI, which trained ourselves.
Table 2: Imputation MSE results for the MuJoCo data set. Here, we use a concise error notation where the values in brackets affect the least significant digits e.g. 0.572(12) signifies 0.572 + 0.012. Similarly, all MSE results are in the order of le-3.
Model
70% RM
80%
RM
90% RM
RNN GRU-D
11.34
14.21
19.68
ODE-RNN
9.86
12.09
16.47
ODE-RNN
9.86
12.09
16.47
NeuralCDE
8.35
10.71
13.52
Latent- ODE
3.00
2.95
3.60
NAOMI
1.46
2.32
4.42
NRTSI
0.63
1.22
4.06
CSDI
0.24(3)
0.61(10)
4.84(2)
SS
0.59(8)
1.00(5)
1.90(3)
Table 2 shows the empirical R.M results on the MuJoCo data set, where SSSD outperforms all baselines except for CSDI on the 70% and 80% missingness ratio. For the scenario with highest RM ratio, 90%, SSSD outperforms all the baselines, achieving an error reduction of more than 50% with a MSE of 1.90e-3. is worth stressing that among the three considered missingness scenarios for imputation CSDI is best-suited for RM, see Table I and the discussion in Appendix A.1
high-dimensional data sets We also explore the potential of SS data sets with more than 100 channels following the simple but potentially sub-optimal channel splitting strategy described above. We implemented the RM imputation task on the Elcetricity data set (Dua & Graff
S4 84 performance
2017) from Du el al. (2022) which contains 370 features at different missingness rat such as 10%, 30% and
50%. As baselines, we consider M-RNN (Yoon et al., 2019), GP- VAE (Fortuin et al. 2020), BRITS (Cao ct al. 2018). SAITS (Du al.) 2022). transformer variant from Du al. (2022), and CSDI Tashiro et al.
2021). We report an averaged MAE, RMSE, and MRE from one sample generated per test sample over 3
trial period.
9

---

Published in Transactions on Machine Learning Research (02/2023)
Table 3: RM Imputation results for the Electricity data set. Overall, CSD1 outperformed all the R.M.
scenarios' baselines metrics even at high levels of missing data. All baseline results were collected from Du
et al. 2022).
Model
MAE
RMSE
MRE
10% RM Electricity
Median
2.056
2.732
110%
M-RNN
1.244
1.867
66.6%
GP-VAE
1.094
1.565
58.6%
BRITS
0.847
1.322
45.3%
Transformer
0.823
1.301
44.0%
SAITS
0.735
1.162
39.4%
CSDI
3e-3
15.012--4e22
81.10% 1e-3
SSSD
0.345+1c-4
0.554+50-5
18.4%+50--
30% RM on Electricity
Median
2.055
2.732
110%
M-RNN
1.258
1.876
67.3%
GP-VAE
1.057
1.571
56.6%
BRITS
0.943
1.435
50.4%
Transformer
0.846
1.321
45.3%
SAITS
0.790
1.223
42.3%
CSDI
0.921
7--2
49.27%
S4
0c4
0.625#1c-4
21.8+0%
50% RM on Electricity
Median
2.053
2.728
109%
M-RNN
1.283
1.902
68.7%
GP-VAE
1.097
1.572
58.8%
BRITS
1.037
1.538
55.5%
Transformer
0.895
1.410
47.9%
SAITS
0.876
1.377
46.9%
CSDI
4e2
2.371+-3e4
14.93%=1e-3
S4
0.532=1c-
0.821+1c-
28.5%+1c4
Table 3 contains the empirical results of the imputation task, where for the 10% and 30% scenarios. Overall, SSSD excelled at. the imputation task, demonstrating significant error reductions against. the strongest baseline SAITS and CSDI with outstanding error reductions of more than 50% and almost 50% for each metric on the 10% and 30% scenarios respectively. For the 50% scenario, SSSD outperforms all the basclines on RMSE, while CSDI shows better performance in terms of MAE and MRE. Similarly, we tested on a 25% RM task from Cini et al. (2022) on the PEMS-Bay and METR-LA data sets (Li et al. 2018) which contains 325 and 207 features respectively. On the PEMS-Bay data set, SSSD outperformed well-establisheed baselines such MICE (White et al.) 2011)), rGAIN (Miao al.] (021), BRITS (Cao al., 2018). and MPGRU (Huang et al. 2019) in terms of all three metrics MAF, MSE, and MRE, while was only superseded
by the recently proposed GRIN Cini et al. 2022). On the METR-LA data set, SSSDS4 S4 again outperformed by GRIN but par with the remaining models. We refer to Table 25 in the supplementary material for a more in-depth discussion of the results.
4.3 Time series forecasting
S4 S4 on the proposed data set We implemented the CSDL, CSDI SASDSA and SSSD models on two data sets For both, we report MAE, and RMSE as metrics for hundred samples generated per test sample in three trials. As first application, we reconsider the case of ECG data from the PTB-XL data set. As before, work at sampling frequency of 100 Hz, however, for this task L 1000 time steps per
10

---

Published in Transactions on Machine Learning Research (02/2023)
value
value
Target
Target
2
CSD1
-2
CSD
50
100
150
200
250
50
100
150
200
250
value
value
2
Target SSS0
-2
get SSSD
50
100
150
200
250
50
100
150
200
250
Figure 4: PTB-XL TF for the V1 lead of an ECG from a patient with a complete left bundle branch block (CLBBB).
sample, for which condition 800 time steps and forecast 200. SSSD outperformed MAE with 0.087, while SSSD achieved slightly larger error of 0.090. SSSD outperformed on RMSE with 0.219. CSDIS4 achieved smaller errors than CSDI on the three metries. Again, the samples presented in Figure demonstrate a clear improvement that is again even visible to the naked eye.
SSSD S4 shows competitive forecasting performance compared to state-of-the-art approaches on various data sets We test on the Solar data set collected from GluonTS (Alexandrov et al. 2020 a forecasting lask where the conditional values and forecast horizon are 168 and 24 time sleps respectively. As baselines, we consider CSDI (Tashiro et al. 2021), GP-copula (Salinas et al. 2019), Transformer MAF (TransMAF) (Rasul et al. 2021b), and TLAE (gguyen & Quanz, 2021). All baseline results were collected from its respective original publications. We report averaged MSE of 100 samples generated per test sample over three trials.
Table 4: Time series forecasting results for the solar data set.
Model
MSE
GP-copula
9.8e2 + 5.2e1
TransMAF
9.3e2
TLAE
6.8c2 + 7.5cl
CSDI
9.0e2 + 6.1e1
S4
2.7e2 + 0.43e1
Table 4 contains the empirical results for the solar data set, which demonstrate the excellent forecasting capabilities of S4 It achieved a MSE of 2.7e2 corresponding to a 60% error reduction compared to TLAE as strongest bascline and also considerably stronger than CSDI, which serves as consistent baseline across all experiments.
Finally, we demonstrate SSSDS*'s forecasting capabilities on conventional benchmark data sets for long- horizon forecasting. To this end, collected the preprocessed ETTml data set from Zhou et al. (0021) and used it for forecasting at five different forecasting settings, where the forecasting length is of 24, 48, 96, 288 and 672 time steps, and the conditional values are 96, 48, 284, 288, and 384 time steps, respectively.
We compare to LSTnet (Lai et al. 2018)) LSTMa (Bahdanau el. al., 2015), Reformer (Kitaev et al. 2020)) LogTTaan Li et al.) 2019). Informer (Zhou et al. 2021), one Informer baseline called Informer Autoformer (Wu al. 2021), , and CSDI (Tsshiro et al. 2021 We report an averagod MAE and MSE for a single sample
generated for each test sample over trials. All bascline results were collected from Zhou al. (0021): Wu
let al. (021)) except. for CSDI, which we trained ourselves.
11

---

Published in Transactions on Machine Learning Research (02/2023)
Table 5: Time series forecasting results on the ETTml data set.
Model
24
48
96
288
672
on ETTml (MAE)
LSTNet
1.170
1.215
1.542
2.076
2.941
LSTMa
0.629
0.939
0.913
1.124
1.555
Reformer
0.607
0.777
0.945
1.094
1.232
LogTri LogTrans
0.412
0.583
0.792
1.320
1.461
Informert
0.371
0.470
0.612
0.879
1.103
Informer
0.369
0.503
0.614
0.786
0.926
CSDI
0.370(3)
0.546(2)
0.756(11)
0.530(4)
0.891(37)
Autoformer
0.403
0.453
0.463
0.528
0.542
SSSD
0.361
0.479(8)
0.547(12)
0.648(10)
0.783(66)
TF on ETTml (MSE)
LSTNet
1.968
1.999
2.762
1.257
1.917
LSTMa
0.621
1.392
1.339
1.740
2.736
Reformer
0.724
1.098
1.433
1.820
2.187
LogTTrass
0.419
0.507
0.768
1.462
1.669
Informert
0.306
0.465
0.681
1.162
1,231
Informer
0.323
0.494
0.678
1.056
1.192
CSDI
0.354(15)
0.750(4)
1.468(47)
0.608
0.946(51)
Autoformer
0.383
0.454
0.481
0.634
0.606
SSSD
0.351(9)
0.612(2)
0.538(13)
0.797(5)
0.804(45)
Table contains forecasting results on the ETTml dataset. The experimental results confirm again SSSD Ss robust forecasting capabilities, specifically for long-horizon forecasting, where also the conditional time steps increases. For the first setting, SSSD outperformed the rest of the basclines on MAE. For the second set setting, on shorter forecast conditional and target lengths SSSD scores are comparable with Autoformer and Informer, while on the remaining three settings, S4 outperformed all baselines with in parts significant error reductions but does not reach the performance of Autoformer. SSSD performs consistently better than CSDI except for forecasting length of 288, where CSDI even outperforms Autoformer. It is worth stressing that Autoformer represents very strong baseline, which was tailored to time series forecasting including decomposition into seasonal and long-term trends, which is perfectly adapted to the forecasting task on the ETTm1. It is an interesting question for future rescarch if SSSD can also profit from a stronger inductive bias and/or if there are forecasting scenarios, which do not follow the decomposition into seasonal and long-term trends so perfectly, where such kinds of inductive bias can even be harmful. To summarize, as very encouraging sign that versatile method such as SSSDS4, which capable of performing imputation as well forecasting, is clearly competitive with most of the forecasting baselines.
5 Conclusion
In this work, we proposed the combination of structured state space models as emerging model paradigm for sequential data with long-term dependencies and diffusion models as the current state-of-the-art approach for generative modeling. The proposed * outperforms existing state-of-the-art imputers on various data sets under different missingness scenarios, with particularly strong performance in blackout missing and forecasting scenarios, provided the number of input channels does not grow too large. In particular, we present examples where the qualitative improvement in imputation quality is even apparent to the naked eye. We see the proposed technology as a very promising technology for generalive models in the time series domain, which opens the possibility to build generative models conditioned on various kinds information from global labels local information such as semantic sogmentation masks, which turn enables a broad range of further downstream applications. The source code underlying our experiments is available under http::///ggttaaa..cco////aaaaata
12

---

Published in Transactions on Machine Learning Research (02/2023)
References
Alexander Alexandrov, Konstantinos Benidis, Michael SchheeeSchneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C. Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, Lorenzo Stella, Ali Caner Türkmen, and Yuyang Wang. Gluonts: Probabilistic and neural time series modeling in python. Journal of Machine Learning Research, 21(116):1-6 6,, 2020.
Adham Atyabi, Frédérick Shie, and Adam Naples. Mixture of autoregressive modeling orders and its implication on single trial eeg classification. Expert systems with applications, 65:164-180) 180,, 2016. doi:
/tl.0///.eewaa.216.... 08.044.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Faraj Bashir and Hua-Liang Wei. Handling missing data in multivariate time series using a vector autoregressive model-impunne ionelliimputation (var-im) algorithm. Neurocomput. 276(C):233 feb 2018. ISSN 0925-2312. doi:
N0.1016/j.neucom..2 201..0..0......00777
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems, volume 31, 2018.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values, 2016.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2020.
Xinyu Chen, Mengying Lei, Nicolas Saunier, and Lijun Sun. Low-rank autoregressive tensor completion for spatiotemporal traffic data imputation. IEEE Transactions on Intelligent Transportation Systems, pp. 1-10, 2021. doi: 10.1109/TIISS WTTTS.2021.33 2021.311360.
Andrea Cini, Ivan Marisca, and Cesare Alippi. Filling the g ap 8: Multivariate time series imputation by graph neural networks. In International Conference on Learning Representations, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems, volume 34, pp. 8780-8794, 2021.
Wenjie Du, David Cote, and Yan Liu. SAITS: Self-Attemtion-baseel Imputation for Time Series. arXiv preprint 2202.000000 08516, 2022.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
Chenguang Fang and Chen Wang. Time series data imputation: survey on deep learning approaches, arXiv preprinl ar Xrx::2201 11347, 2020.
Vincent Fortuin, Dmitry Baranchuk, Gunnar Ractsch, and Stephan Mandt. Gp-vac: Deep probabilistic time series imputation. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 1651-1661, 26 28 Aug 2020.
Marco Fraccaro, Simon Kamronn, Urrih Paquet, and Ole Winther. disentangled recognition and noulinear dynamics model for unsupervised learning. Advances in neural information processing systems, 30, 2017.
Karan Goel, Albert Gu, Chris Donahue, and Christopher Re. It's raw! Audio generation with state-space models. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp. 7616 7633, 17 23 Jul 2022.
Ary L. Goldberger, Luis A. N. Amaral, Leon Glass, Jeffrey M. Hausdorff, Plamen Ch. Ivanov, Roger G. Mark, Joseph E. Mietus, George B. Moody, Chung-Kang Peng, and H. Eugene Stanley. PhysioBank, PhysioToolkit, and PhysioNet. Circulation, 121 e22000 2000. doi: 10.1161/01 100..................
13

---

Published in Transactions on Machine Learning Research (02/2023)
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. In Advances in Neural Information Processing Systems, volume 33, pp- 1474 1487, 2020.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. International Conference on Learning Represenlations, 2022a.
Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra, and Christopher Ré. How to train your hippo: State space models with generalized orthogonal basis projections. arXiv preprint 2206. 10037, 2022b. doi: 10.48550/ARXIV, 2...222....222
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840 0001,85111 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23:47:1 23:47:1-47:33, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusson models. Xieee preprint 2204. 03458, 2022b.
Yujun Huang, Yumpeng Weng, Shuai Yu, and Xu Chen. Diffusion convolutional recurrent neural network with rank influence learning for traffic forecasting. In 18th IEFE International Conference On Trust, Security And Privacy In Computing And Communications, pp. 678 685, 2019. doi: Teet/rrrustttiieeeeeeee 2019.00096.
Mourad Khayati, Alberto Lerner, Zakhar Tymchenko, and Philippe Cudré-Mauroux. Mind the gap. Proceedings of the VLDB Endowment, 13(5):768-788 782, January 2020. doi: 10. 000144//3377369.33373
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference Learning Representations, ICLR 2020, 2020.
Alexej Klushyn, Richard Kurle, Maximilian Soelch, Botond Cscke, and Patrick van der Smagt. Latent matters: Learning deep state-space models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 10234 10245. Curran Associates, Inc., 2021.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: versatile diffusion model for audio synthesis. In 9th International Conference Learning Representations, ICLR 2021, 2021.
Rahul G. Krishnan, Uri Shalit, and David Sontag, Deep kalman filters. orXie preprint 1511.05121, 2015. doi: 10.48550/ARXIV15511.. 151.V.01111.011111
Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks. In The 4 1ss International ACM SIGIR Conference on Research & Development Information Retrieval, pp. 355104, 2018.
Longyuan Li, Junchi Yan, Xiaokang Yang, and Yaohui Jin. Learning interpretable deep state space model for probabilistic time scries forecasting. arXio preprint ArX:::2102.000 202.0077 2021.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer time series forecasting. In Advances in Neural Information Processing Systems, volume 32, 2019.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In International Conference on Learning Representations (ICLR '18), 2018.
W.C. Lin and C.F. Tsai. Missing value imputation: review and analysis of the literature. Artificial Intelligence Review, 53(2):14877133)) 2020. doi: 10. 10.10077/11446--00--09700
14

---

Published in Transactions on Machine Learning Research (02/2023)
Yukai Liu, Rose Yu, Stephan Zheng, Eric Zhan, and Yisong Yue. Naomi: Non-antoregrresiive multiresolution sequence imputation. In Advances in Newral Information Processing Systems, volume 32, 2019.
Andreas Lugmayr, Martin Danelljan, Andrés Romero, Fisher Yu, Radu Timofle, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, pp. 11451-11461, IEEE, 2022.
Youghong Luo, Xiangrui Cai, Ying Zhang, Jun Xu, and Yuan xiaojie. Multivariate time series imputation with generative adversarial networks. In Advances in Neural Information Processing Systems, volume 31, 2018.
Yonghong Luo, Ying Zhang, Xiangrui Cai, and Xiaojie Yuan. E'gan: End-to-end generative adversarial network for multivariate time scries imputation. In Proceedings of the Suenty-Eightt International Joint 7 2019. doi: 10.24963/ijcaii 2019/429.
Conference on Artificial Intelligence, IJCAI-19, pp.
Chenlin Mong, Ruiqi Gao, Diederik P Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. arXiv preprint (xXio:2200001144 2022.
Xiaoye Miao, Yangvang Wu, Jun Wang, Yunjun Gao, Xudong Mao, and Jianwei Yin. Generative semi- supervised learning for multivariate time series imputation. Proceedings of the AAAI Conference on Artificial Intelligence, 35(10):8983 8991, May 2021.
James Morrill, Cristopher Salvi, Patrick Kidger, and James Foster. Neural rough differential equations for long time series. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 7829 7838, 18 24 Jul 2021.
Nam Nguyen and Brian Quanz. Temporal latent auto-encoder: A method for probabilistic multivariate time series forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, (5(10):9117-9129) May 2021.
Muhammad Osman, Adnan M. Abu-Mahfouz, and Philip R. Page. A survey on data imputation techniques: Water distribution system as use case. IEEE Access, 6:63279- 63291, 2018. doi: 10.1109/ACCESS.220 2877269.
Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar. DiffuseVAF: Efficient, controllable and high-fidelity generation from fow-dimensional latents. Transactions on Machine Learning Research, 2022.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In Advances in Neural Information Processing Systems, volume 31, 2018.
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. In International Conference on Machine Learning, pp. 8857-8868. PMLR, 2021a.
Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs M. Bergmann, and Roland Vollgraf. Multivariate probabilistic time scries forecasting via conditioned normalizing flows. In Ith International Conference on Learning Represenlations, ICLR 2021, 2021b.
Yulia Rubanova, Ricky T. Q Chen, and David K Duvenaud. Latent ordinary differential equations for irregularly-sampled time series. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fax, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Donald B. Rubin. Inference and missing data. Biometrika, 63(3))581 592 12 1976. ISSN 0006-3444. doi: 00.1088/biomeet/633..35
Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lec, Jonathan Ho, Tim Salimans, David J. Floet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In NewrIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
15

---

Published in Transactions on Machine Learning Research (02/2023)
David Salinas, Michael Bohlke-Schnnider, Laurent Callot, Roberto Medico, and Jan Gasthaus. High- dimensional multivariate forecasting with low-rank gaussian copula processes. In Advances in Newral Information Processing Systems, volume 32, 2019.
Tolou Shadbahr, Michael Roberts, Jan Stanezuk, Julian D. Gilbey, Philip Teare, Sören Dittmer, Matthew Thorpe, Ramón Viñas Torné, Evis Sala, Pietro Lio', Mishal N. Patel, AIX-COVNET Collaboration, James H. F. Rudd, Thomas Mirtti, Antti S. Rannikko, John A. D. Aston, Jing Tang, and Carola-Bibiane Schonlicb. Classification of datasets with impuled missing values: does imputation quality matter? orXiv preprinl 2206.08477 78, 2022.
Siyuan Shan, Yang Li, and Junier B. Oliva. Nrtsi: Non-recurrent time series imputation. aiv preprinl 2102.03340 2021.
Jascha Sohl-Dickstein Eric Weiss, Niru Maheswaranathann and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256 2265, 07 09 Jul 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Scoro-based generative modeling through stochastic differential equations. In International Conference Learning Representations, 2021.
Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34: 14804-24816, 2021.
Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I. Lunze, Wojeiech Samek, and Tobias Schaeffter. PTB-XT, large publicly available electrocardiograply dataset. Scientific Data, 7 (1):154, 2020a. doi: 10.1038/ 10.1038/s411977001 /00////19-000004955555
Patrick Wagner, Nils Strodthoff, Ralf- Dieter Bousseljot, Wojeiech Samek, and Tobias Schaeftter PTB-XL, a large publicly available electrocardiograply dataset, 2020b.
Yuyang Wang, Alex Smola, Danielle Maddix, Jan Gasthaus, Dean Foster, and Tim Januschowski. Deep factors for forecasting, In International conference on machine learning, pp. 6607-66 6617. PMLR, 2019.
Ian R White, Patrick Royston, and Angela M Wood. Multiple imputation using chained equations: Issues and guidance for practice. Stat. Med., 30(4):377-33)) February 2011.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 22419 22430, 2021.
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation. arXiv preprinl 2203.09481, 2022.
Jinsung Yoon, William R. Zame, and Mihaela van der Schaar. Estinating missing data in temporal data streams using multi-directional recurrent neural nerworks. IEEE Trunsactions on Biomedical Engineering, 66(5):1477-1499) 1490, 2019. doi: 10.1009/TBME:221
Zhihui Zhang, Xiangjun Xiao, Wen Zhou, Dakai Zhu, and Christopher I Amos. False positive findings during genome-wide association studies with imputation: influence of allele frequency and imputation accuracy. Human Molecular Genetics, 3111):146-155, 2022. doi: 10.1093, 10.1109//mg/// http//hgg/d/// /dda2233
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forccasting. In The Fhitty-Fifth AAAT Conference on Artificial Intelligence, AAAI 2021, volume 35, pp. 11106 11115, 2021.
16

---

Published in Transactions on Machine Learning Research (02/2023)
A Technical and implementation details
A.1 Time and feature diffusion
In this section compare to the different design choices between the proposed approach and CSDI
Tashiro
et al. 2021). as the only other diffusion-based imputer in the literature. As in Tashiro et al. our parametrization of ea E c) on the DiffWave architecture (Kong et al. 2021 a versatile diffusion model architecture proposed in the context of audio generation.
2021
base
CSDI relics on introducing a separate diffusion dimension into the original three-dimensional input repre- sentation, i.e. working on an extended four-dimensional internal representation of shape (batch dimension, diffusion dimension, input channel dimension, time dimension). This necessitates processing time and feature dimensions alternatingly since many modern architectures for sequential data, such as transformers structured state space modcls, are only able to process sequential, i.c. throe-dimensional, imput batches.
We take the conceptionally simpler path of mapping the input channels into the diffusion dimension and performing only diffiusio along the time dimension, i.e. processing batches of shape (batch dimension, diffusion dimension, time dimension), which implics that no separate diffusion process along time and feature axis are required as a single diffusiin along the time direction is sufficient. This choice also corresponds to the common approach when applying diffusion models for example to image or sound data.
We see this design choice as central explanation for the superior performance of SSSD 84 compared to CSDI (as well CSDIS1 which should perform on par with SSSDS1 if the improved handling of the time diffusion was the major issuc) in the regime of small number of input channels. We hypothesize that for a large number of input channels it becomes increasingly difficult for SSSD to reconstruct the original input channels from the internal diffusion channels. We see the central strength of the proposed model in its ability to capture long-term dependencies along the time direction, as required for example to perform imputations in the BM scenario. The approach taken by CSDI of disentangling feature and diffusion axes, seemss to provide advantages in situations with many input channels and/or situations, such as RM imputation, where the temporal consistency is less important than the ability of inferring relations between different channels at given time step.
A.2 Channel splitting
As described in the main text, we found convergence issues during training our proposed models which are direct variants from DiffWave such as SSSDSA and SSSD S4 using more than about 100 input channels, which we attribute to the handling of the diffusion axis, see the discussion in Appendix [A.1] The simplest way to generate predictions also for cases with more input channels is by splitting up original batches of shape (batch dimension, input channel dimension, time dimension) along the channel dimension into several batches of shape (batch dimension, reduced input channel dimension, time dimension), where we just used consecutive channels in the original channel order provided by the data set. This approach can be implemented straightforwardly but comes with the obvious disadvantage that identical channels in these input batches with reduced channel dimension can correspond to different channels in the original input, which might lead to suboptimal prediction performance.
the extension of the approach to include more claborate channel splitting procedures as an interesting direction for future research. First, batches with reduced channels could be defined based on channel correlations instead of on the original channel order based on the assumption that the model should be able to leverage information most effectively from correlated channels. Second, the information which of the different reduced channel batches was passed could be incorporated as additional information in order to learn channcl-specific representations across batches that originate from different channels in the original input.
Apart from addressing this issue by keeping separate channel axis, see Appendix A.1
17

---

Published in Transactions on Machine Learning Research (02/2023)
A.3 Implementation details on missingness scenarios
As described in Section 2 our research focuses on three different masking strategies, RM, RBM, BM, as well as TF, which can be seen as a special case of BM. For RM, we randomly sample a number of missingness ratio x total time steps as imputation target, separately for each channel. For RBM and BM we start by partitioning the sequence into segments of length missingness ratio x total time steps, where a potential remainder to reach the full sequence length assigned to final segment. For RBM, we randomly sample one of these segments per channel as imputation target. For BM we only sample single segment, which is used as imputation target across all channels.
A.4 Training and Sampling Code
Algorithm Training step
Require: Diffusion hyperparameters {T, So. }1), Net (model), Mimp (imputation mask), Mavi (missing value mask), D1 (use mode D1), zo (ground truth signal)
3 LinSpace(start = Bo, end 31, steps T)
Compute diffusion process hyperparameters
B
3: = B
for t in range(1, (1, T) do
ā. II 1 02
3, = B (1 0 11-11/(1 - āx)
of B
end for
T std (0, 1)
Training
DS RandInt (T) Crlee Cr)
M Mamp @ Mmmi
C' Concat(zo @ M, M)
if D1 then
Eoid To @ M + Fstd @ (1 MM
end if
I = aa DSS x xo + - (I-aa, x Fstd
y = Net( C, DS)
if D1 then
loss (y @ M Fo ① M)
else
loss (y- x
end if
update_ (oasttt__aaam
Algorithm 1 contains pseudocode for the training step in our experiments. Tt starts with the computation of the diffusion process hyperparameters 8, a, ae, and of given the number of diffusson steps T' and the beta schedule start 3y and beta schedule end 31 values between which we interpolate lincarly. For the training step, firstly, we instiate random Gaussian noise of the same shape as the input. batch (0, and compute diffusion steps DS as random integers of the size of the conditional input batch. Secondly, we create the combined mask M obtained via point-wise multiplication of imputation mask Mimp and missing value mask Mavi. Thirdly, the conditional information C' is obtained by concatenating the input To (masked according to the imputation mask) and the imputation mask itsolf. Then, if our diffusion setting is D1, the conditional information gets indexed into the noisy batch given M. Then, we update Ty as T from ((x]((zo)) Afterwards, the diffuuson model outputs a generated time series given T. G, and DS. Finally, we use mean squared error as loss function and update the network parameters using backpropagation.
Algorithm 2 shows pseudocode for the sampling procedure applied during our experiments. First, we again compute the diffusion process hyperparameters B, a, ãt, and of given the number of diffusion steps T and the
18

---

Published in Transactions on Machine Learning Research (02/2023)
Algorithm Sampling algorithim
Require: Diffusion hyperparameters {T, fo. }1 Net (model), C (conditioning: observed signal), M (combined mask M mimp O mmvi), D1 (use mode D1)
3 LinSpace(start Bo, end B1, steps 7)
Compute diffusion process hyperparameters
1 - B
3. = B
for in range(1, 1, T) do
0: = II 0x
8. 81 (1 0 11-1)/(11 )+)
of Br
end for
C = Con M)
D. Sampling
Istd = (0, 1) for in range(T-1, (T-1, -1, -1) do
if D1 then
= Fotd ① (1- M) - COM
end if
DS = x ones([ (niie (M [0]. 1])
60 Net (r. C, DS)
T (In- (1 a|t))/ atll) E0)/ Valt)
if t 0 then
== [ x (00 1)
end if
end for
beta schedule start 8n and bela schedule end 81 values between which we interpolate linearly. For sampling, we start from random Gaussian noise of the same shape as batch No,, 1) 1) then, it gets updated, first, by the masking strategy in case the diffusion is applied only to the segments to be imputed, second, by the model generation which also takes input the observed series to condition on C, the mask M, and diffusion steps DS, and lastly updated from to (xx-1 to Ho (xx)) given the diffusion hyperparameters, where if the diffusion step is not the last, we add a variance term to Tt-1-
Ablation studies
B.1 Model architecture
Table contains the results obtained within ablation study done aiming to find the best configuration for an S4 layer in our proposed model. We present three different variants with capital letter notations. (A) represents our proposed DiflWave imputer with dilated convolutional layers. (B) represents a DiffWave
imputer with a S4 layer in replacement of the dilated convolutional layer. (C) represents setting B and an
extra S4 layer after the conditional data addition (SSSD$4). Lastly, setting (D) represents the testing of our proposed SSSDS1 however, with an unconditional training, following the procedure proposed (Lugmayr ct al. 2022 for image inpainting.
The experiment
carried out in the three investigated missing scenarios RM, RBM, and BM at two different missingness ratios, 20% and 50%BM on the PTB-XL data set. Metrics were obtained at 150,000 training iterations. We report an averaged MAE, and RMSE, over three trials for 10 samples generated for each sample of the test set. The results in Table t shows the metrics obtained for the four different model variants considered in our ablation study. The replacement the dilated convolutions by S4 layers leads a significant crror reduction (sctting vs. sctting A) for most of the scenarios. The introduction of a second S4 layer again leads to a smaller but still consistent improvement over the setup with single S4 layer (setting C vs. setting B) for all of the scenarios. The unconditional training procedure proposed in (Lugmayr
19

---

Published in Transactions on Machine Learning Research (02/2023)
Table 6: Results of the ablation study on model architecture.
Model
MAE
RMSE
MAE
RMSE
20% RM on PTB-XL
50% RM on PTB-XL
A: Difiwave
4.
1.77e-2+4e-4
9.10e-3:55-5
4.2
B: SSSDD* (single S4 layer)
1.75
2.01e-2tle-4
8.15e-3:14-5
3.64e-2+5e-4
C: SSSD $4
4e-
1.19
8.00e-3-88-5
3.5---2-t6--4
D: RePaint (uncond. training)
1.3-e-2t9-55
1e-
2.31e-2448-5
6.1
20% RBM on PTB-XL
50% RBM on PTB-XL
A: Diflwave
2.50e-2+1e-3
8.08e-2+:5e-3
4.84e-2-136-3
1.40-el+t4e-3
B: (ASD**(singlee S4 layer)
2.42e-245e-4
1-25e-166e:
4.49e-2:22-3
1.36e- 1=5e-3
C: SSSD $4
1.0-e-2+te--
2.26e-2-t9es
3.73--2--44-4
1.31e-1-3-93
D: RePaint (uncond. training)
6.40e- 2
1e-
1.26e- 1+3--3
2.5le- 1#le-2
20% BM on PTB-XL
50% BM on PTB-XL
A: Diflwave
4.35e-2.1:e-3
1i6el1lles
1.23e-1
2.76e-1+:3e-
B: $SSD (single S4 layer)
3.67e-2+2e-3 3.24e-2+te-- 1.23e-1+40-3
9.29e-2:t2e22
1.46--1+e-
2.81e-1+4e-3
C: SSSD S4
8.32e-2-8e-3
1.16e-1=3e-3
2.66e-1+3e-
D: RePaint (uncond. training)
2.13--1486-3
1.67e-1+1e-
3.10e-1+2ec
ct al. 2022) is clearly inferior to the proposed conditional training (setting D vs. sctting C), see also Figure for a qualitative impression of the generated samples. This leads us to propose SSSD (setting C) as default. variant for further experiments.
Target Unconditional training
0.5
0.50
value
value
0.25
0.00
-0.5
Target Unconditional training
0025
-1.0
50
100
150
200
250
100
150
200
250
1.0
Target
0.75
Target
Unconditional training
Unconditional training
value
0.5
0.0
0.5
50
100
150
200
250
50
100
150
200
250
Figure 5: PTB-XL. BM imputations for the leads I, V1, V4, and aVF of an ECG from a patient with simus arrhythmia obtained from unconditional training (sctting D).
B.2 Additional missingness ratio on PTB-XL
Table we present additional empirical results the PTB-XL dataset at 50% of missigness across all investigated scenarios for selected set of models (CSDI, DiffWave, and SSSD$4), which complement the corresponding results in Table for a missingness ratio of 20%. We observe that DiflWave (with dilated convolutional layers) outperforms CSDI on the RBM and BM missing scenarios, however, not in the RM sconario, which best suited for the approach taken by CSDI, see the discussion in Appendix A.1 SSSD outperforms both CSDI and DiffWave for all three scenarios and both metrics in line with its strong performance demonstrated for smaller missingness ratios.
20

---

Published in Transactions on Machine Learning Research (02/2023)
Table Results for selected models at a missingness ratio of 50% on PTB-XT.
Model
MAE
RMSE
50% RM on PTB-XL.
CSDI
9.00
7.20e-2-2.30e-3
DifeWave
9.10e-3t5.77c-5
4.25c-2-4.58o-4
SSSD S4
8.00 3.58--226.22c-4
50% RBM on PTB-XL
CSDI
6.25e-2+1.60e-3 4.84e-243.11o-
1.5de-1=4.50es
DiffWave
1.40-c1=4.88--3
SSSD S4
3.73e-2+t4911e 2.91ee
1.31e-1+3.45e-3
50% BM on PTB-XL
CSDI
1.49e-1+2210e-3 2.95e-1=5.60r
DifFWave
1.23e-146.80e4
2.76e-1=3.28e-
SSSD 54
1.16e- 1.3.5--3 33.2e-3
B.3
Dependence number of diffusion steps
Table 8: Imputation for BM scenario on the PTB-XL data set at 50% BM with 50 and 200 diffusion steps.
Model Diffusion steps
Training time
Sampling time
MAE
RMSE
SSSD 84
50
31 hours
627 seconds
1.88--17.000-- 1.16e-1#3.00e-- 0.267-3..23--
0.350=1.23e-2
SSSD
200
36 hours
3,740 seconds
Table 8 demonstrates the impact of reducing the number of diffusion steps from the default value 200 used in this work to 50. It leads to slightly faster training time (86% of the original training time) and to considerably faster sampling times (16% of the original sampling time), however, comes at the cost of considerably worse target metrics (166% and 131% of the original metrics). It is worth mentioning that
alternative methods, such as (Meng el al. 2022 have been proposed to reduce the sampling time, which do not come with such disadvantages.
B.4 Robustness analysis
In this ablation study study the robustness of our models with respect mismatch of either missingness ratio missingness scenario between train and test. While the latter is rather relevant for identifying optimal training scenarios, the former is also very relevant for the practical applicability of imputation models as for real-world scenarios is not realistic to assume a constant missingness ratio.
Table 9: Imputation for BM scenario on the PTB-XL data set for the benchmarking of different missingness ratios such as 20%, and 50%.
Setting
Test 20%
Test 50%
MAE
RMSE
Train 20% BM
3.24e-243..0e--3
8.32e-218..m 000e
1.29e- 1.0-11 2.7le-117.89e-3
Train 50% BM 8.99c-241.040-- 2.26e-1+4.40e-3
1116c-13.056 2.66e-143.23e-3
We carry out the experiments on the effect of a mismatch in missingness ration on PTB-XL in the BM scenario for missingness ratios 20% and 50%. The results are compiled in Table g They show that the model trained on smaller missingness ratio but evaluated on a larger missingness ratio takes a penality of 11% (MAE) in comparison model that was trained and evaluated on the high missingness ratio. In the opposite case where trained larger missingness ratio but evaluated at smaller reveals larger
21

---

Published in Transactions on Machine Learning Research (02/2023)
performance malus with metrics that are almost three times larger compared to the performance of model trained at the matching small missingness ration. This limited experiment seems to indicate that models are significantly more robust against testing at higher missingness ratios compared to the missingness ratio seen at training time. However, is also worth stressing that the mismatch between the missingness ratios is already quite large this experiment.
Table 10: Imputation 50% missingness ratios on the PTB-XL XLL data set for the benchmarking of different sccnarios such as RM, RBM, and BM.
MAE
Test RM
Test RBM
Test BM
50% MAE
Train RM
8.00e-3118.81e- 1.40c-2:44.90-55
8.11e-212.49e-3
1.60e-127.59--
Train RBM
3.73e-2=+4.91--4
8.79c-2+2.25c-8
Train BM
1.499-2+44.990-5
8.41e-2+ 4.4e-33
1.16e-1+3..5e-3
50% RMSE
Train RM
3.53e-2:16.82e.
1778e-117.50e-
2.69e-1#1.840
Train RBM
1.94
1.31e-1#3.45e-3
2.17e-1=6.350
Train BM
5.20e-2+1.00e-
2.16e-123.50e-
1.66e-133.2ee.
The second experiment addresses the robustness with respect to mismatches in missingness scenarios between train and test time. This seems like a less realistic scenario for practical applications where the missingness scenario is unlikely to change but touches charactoristics expected during test time is really the optimal approach. In Table 10 carry out such experiment again on the PTB-XL dataset comparing RM, RBM and BM as missingness scenarios, for simplicity at a fixed missingness ration of 50% in all cases. For testing on RM and RBM, the respective matching training conditions clearly lead to the best results. Interestingly, for testing on BM, training on RBM leads to considerable reductions (25% in terms of MAE and 19% terms of RMSE) compared to training on BM. This represents an interesting finding worth pursuing in future research.
the important question if training on identical missingness
C Training hyperparameters and architectures
Table 11: LAMC hyperparameters.
Hyperparameter
Value
Rhos
0.001, 0.01, 0.1, 1, 2, 3, 5, 10, 20.
Lambdas
0.001, 0.01, 0.1, 1, 2, 3, 5, 6, 8, 10.
Epsilons
1e44, le-3, le-2, 1, 2, 5, 8, 10.
Ranks
2, 5, 10, 20, 50, 80, 100.
Iterations
100
LAMC For the LAMC algorithm (Chen et al. 2021 defined a range of hyperparameterss to be chosen by an exhaustive grid search given the best test metrics. Table [1 contains the LAMC hyperparameter implemented through all the experiments. We decided to implement this algorithm as a baseline as it represents a qualitatively different methodology for time series imputation compared to our probabilistic diffusion models, which relies on matrix completion approach.
CSDI and S4 We use the official implementation of CSDI Tashiro et al. 2021 | as a baseline with the authors' default hyperparameters settings. The only modification for CSDI* was to replace the temporal transformer layer by S4 layer. Table 12 contains all the hyperparameters that for CSDI and CSDI training and architecture. For the C $4 setting, the S4 model implemented in all experiments is a bidirectional layer with layer normalization, dropout, and N = 64 as internal state dimensionality.
22

---

Published in Transactions on Machine Learning Research (02/2023)
Table 12: CSDI hyperparameters.
Hyperparameter
Value
Residual layers
Residual channels
64
Diffusion embedding dim.
128
Schedule
Quadratic
Diffusion steps T
50
Bo
0.0001
B₁
0.5
Feature embedding dim.
128
Time embedding dim.
16
Self-attention layers time dim. Self-attention heads time dim. 8 Sell-attention layers urature dim. Self-attention heads feature dim. Optimizer
1
I
8
Adam
Loss function
MSE
Learning rate Weight docay
1 x 10-3
1 x 10-6
Table 13: DiffWave hyperparameters.
Hyperparamcter
Value
Residual layers
Residual channels
256
Skip channels
256
Diffusion embedding dim. 1
128
Diffusion embedding dim. 2
512
Diffusion embedding dim. 3
512
Schedule
Lincar
Diffusion steps
200
Bo
0.0001
B
0.02
Optimizer
Adam
Loss function
MSE
Learning rate
2 x 10-4
DiffWave Table 13 contains all the hyperparameters for training and architecture of the DiffWave imputer. As previously discussed, DiffWave was released by its authors (Kong et al. 2021) as diffusion model for speech synthesis. Here, we are referring to DiffWave model as a custom implementation the context of time series aopputationfforecasttto
Some of the architectural differences are a large number of residual layers and channels, but most importantly our proposed training procodure which involves a different use of conditional information, such as the concatenation conditioning information and binary mask.
SA Table 14 contains the SSSD SA hyperparameters for all our experiments. SSSD is variant of the SaShiMi (Goel et al. 2022) model, 128-dimensional U-Net model with six residual layers, consisting of an S4 and feed-forward layers in a block manner al each pooling level, where the pooling factor in decreasing the sequence length is 2 and and its respective feature expansion of 2. There is a single three-layer diffusion embedding with dimensions 128, 512, and 512, respectively for all the residual layers. As for $4, we use a bidirectional layer with layer normalization, no dropout, and an internal state dimension of N 64, but with gated linear unit in each layer.
SA
23

---

Published in Transactions on Machine Learning Research (02/2023)
Table 14: SSSD hyperparameters.
Hyperparameter
Value
Residual layers
Pooling factor
[2.2]
Feature expansion
2
Diffusion embedding dim. 1
128
Diffusion embedding dim. 2
512
Diffusion embedding dim. 3
512
Schedule
Linear
Diffusion steps T
200
Bo
0.0001
B1
0.02
Optimizer
Adam
Loss function
MSE
Learning rate
x 10
Table 15: SSSD hyperparameters.
Hyperparameter
Value
Residual layers
36
Residual channels
256
Skip channels
256
Diffusion embedding dim. 1
128
Diffusion embedding dim. 2
512
Diffusion embedding dim. 3
512
Schedule
Linear
Ditfusion steps T
200
Bo
0.0001
Bi
0.02
Optimizer
Adam
Loss function
MSE
Learning rate
x 10-4
S4 94 We present in Table the proposed SSSD hyperparameters and training settings used throughout all our experiments. The SSSD) a model builds on DiffWave (Kogg et al. 2021) and consists of 36 stacked residual layers with 256 residual and skip channels. As for SSSD** SSSD uses a three-layer diffusion embedding of 128, 256, and 256 hidden units dimensions with swish activation function after the second and third layer. After the addition of diffusion embedding, implemented convolutional layer to double the channel dimension of the input before computing the first S4 diffusion. Similarly, after similar expansion of the conditional information and its addition to the input, there is the application of a second S4 layer. Then the output is passed through a gated tanh activation function as non-linearity, from which we then project back from residual channels to the channel dimensionality with a convolutional layer. We used 200 time steps on a linear schedule for diffusion configuration from a beta of 0.0001 to 0.02. We utilized Adam optimizer with learning rate of 2 10-1, For the S4 model, similar to the other approaches, used bidirectional layer with layer normalization, dropout, and an internal state dimension of N = 64.
S4 In Table 16 we present the hyperparameter settings for the state space S4 model GGu et al., 2022a) used in this work. Overall, we utilize single S4 layer with a bidirectional setting for time dependencies learning of series in both directions. Similarly, we applied layer normalization and utilize internal state of dimensionality N 64 used in prior work (Gu al. 2022a
24

---

Published in Transactions on Machine Learning Research (02/2023)
Table 16: S4 hyperparameters.
Hyperparameter
Value
Layers
State dimensions
64
Bidirectional
Yes
Layer normalization
Yes
Drop-out
0.0
Maximum length
as required
Hyperparameter tuning In this paragraph, we briefly describe the strategy that led to the hyperparameter selections discussed in the previous paragraphs: As first remark, we focus on SSSD in this section. These settings were applied to SSSD without further experimentation. We only adapted the number of residual layers to obtain a model with comparable computational complexity as SSSDS1 For CSDI we left the hyperparameters as close to the original default hyperparameters as possible. As general remark, we point out that the hyperparameter optimization was carried out based on validation data set at 248 time steps. We did not adjust the hyperparameters for other data sets, which can be seen hint for the robustness of the proposed method. Below, briefly comment specific aspects of the hyperparameter selection.
scores on the PTB-XL
Diffusion hyperparumeters: We implemented a linear schedule with a minimum noise level Bo of 0.001 and a maximum noise level of B1 as 0.5, based on trial and error. We found that the fewer diffusion steps, the faster the network converges during training, however, the cost of less accurate results. In this respect, using 200 diffusion steps represented a reasonable compromise. Supporting results can be found in Appendix [3. Three-layer diffusion embedding: From the beginning of our experiments, we implemented three-layer embedding with 128, 512, and 512 hidden units, respectively. While reducing the residual layers in some experiments, we also experimented with reducing those dimensionalities to 16, 32, 32, or 64, 64, 256. However, we found that the former choice led to better results.
Residual channels and skip channels: we introduced large number of channels (256) to avoid the degradation problem.
Residual layers: We set the number of residual layers to 36 after the experimental phase of our ablation study. In our previous experiments with single S4 layer, we worked with 48 layers but reduced it 36 to increase the speed of convergence during training and to reduce the computational complexity of the overall model, which in variant C requires two S4 layers.
D Data set descriptions
Table 17: PTB-XL data set details.
Description / Setting
PTB-XL 248
PTB-XL 1000
Train size
69,764
17,441
Validation size
8,772
2,193
Test size
8,812
2,203
Training batch
32
Sample length
248
1000
Sample features
12
12
Conditional values
198
800
Target values
52
200
PTB-XL data set Table 17 contains the PTB-XL data set details (Wagner et al. 2020a b Goldberger
et al. 2000). The PTB-XL ECG data set consists of 21837 clinical 12-lead ECGs, each lasting 10 seconds, from 18885 patients.
The data set was collected and preprocessed as in the Physionet repository. We collected
25

---

Published in Transactions on Machine Learning Research (02/2023)
for all experiments the ECG signals at sampling rate of 100 Hz. For the three imputation and forecasting scenarios, we utilize 20% as target values. In all of the settings, the number of input channels is 12 as it is a 12-lead electrocardiogram. For the 248 time steps setting, the data set was preprocessed on crops, which corresponds to 69,764 training and 8,812 test samples.
Table 18: Electricity data set details.
Description
Value
Train size
817
Test size
921
Training batch
43
Sample length
100
Data set features
370
Sample features
37
Conditional values
90, 70, 50
Target values
10, 30, 50
Electricity data set Table 18 contains details on the electricity data set, which was used for RM imputation. The electricity data set from the UCI repository (Dua Grafi] 2017) contains electricity usage data (in kWh) gathered from 370 clients which represent 370 features every 15 minutes. The data set was collected and preprocessed in (Du al. 2022). As the data set not contain missing values, we collected the complete data set and in our experiments and randomly dropped the values for the computation of targets according to the RM scenario. The data is already normalized and present results in this setting, The first 10 months of data (2011/01- 2011/100 are the test set, the following 10 months of data (2011/11- 2012/08) the validation set and the left (2012/09 - 2014/12) the training set. We directly utilize the training and test set. leaving the validation set out. We consider this a challenging task due to the fact that the test set contains many clients that were not present in the training set. The data set contains 817 samples of a length of 100 time steps with the 370 mentioned features. However, we observed faster convergence when applying feature sampling, specifically, splitting the 370 channels into 10 batches of 37 features each, then, passing to the network mini-batches of 43 samples each with 37 features and its respective length of 100 to ensure that we do not drop any data during training.
Table 19: MuJoCo data set details.
Description
Value
Train size
8000
Test size
2000
Training batch
50
Sample length
100
Data set features
Conditional values
50, 30, 20, 10
Target values
50, 70, 80, 90
MuJoCo data set Table 19 contains details of the MuJoCo data set, which data set for physical simulation, created by the authors at (Rubanova et al. 2019) using the "Hopper" model from the Doepmind Control Suite. The hopper's initial placements and speeds are randomly sampled in such way that the hopper rotates in the air before crashing to the earth. There are 10,000 sequences of 100 regularly sampled time points for cach trajectory in the 14-dimensional data set. By convention, there is 80/20 random split for training and lesting. Both data sets were already preprocessed by the NRTSI authors in (Shan el al.) 2021 to ensure a fair comparison.
PEMS-BAY data set Table 20 contains details on the PEMS-Bay data set (Huang et al. 2019). which compiled for The Performance Measurement System (PeMS) of the California Transportation Agencies
26

---

Published in Transactions on Machine Learning Research (02/2023)
Table 20: PEMS-Bay data set details.
Description
Value
Train size
1200
Test size
50
Training batch
40
Sample length
200
Data set features
325
Sample features
65
Conditional values
150
Target values
50
(CalTrans). It represents network of 325 traffic sensors in the California Bay Area. It contains traffic readings every five minutes for six months, from January 1st 2017 to May 31st 2017. The data set was collected from Cini et al. (2022). The methodology of Ci et al. (2022) required the creation of an adjacency matrix from the original data set due to the use of a graph neural network. We work directly with the data set, which has 52,116 time stops and 325 features. However, in order to obtain samples for training, we considered 200 time steps over the first 50,000 to get 250 samples, we set the first 240 for training and the 10 rest for testing. Then, we fit standard scaler on the training set and transform both the training and the test sets. Finally, we feature sampled obtain batches by a factor of 5, where for training set we obtained 5 batches of 240 samples with 200 time steps and 65 channels each. Finally, we iterate over the second dimension to pass batches of 40 for training.
Table 21: METR-LA data set details.
Description
Value
Train size
750
Test size
100
Training batch
50
Sample length
200
Data set features
207
Sample features
40
Conditional values
150
Target values
50
METR-LA data set Table 21 contains details the METR-LA data set, which represents the traffic data from road network consisting of 207 loop detectors in period between the 1st of March 2012 to the 30th of June 2012. Similar to the preprocessing of the PEMS-Bay data set, we selected from the original set the first 34,000 time steps and the first 200 features, we obtained 170 samples of 200 time steps each. We used the first 150 as training set and the remaining 20 for testing. Finally, we sampled features by factor of 5, obtaining for training batches of 150 samples with 200 time steps and 40 features cach. For training we iterate over the first dimension to obtain batches of 50 samples.
Solar data set Table 22 presents details on the solar data set. The original data set contains the solar power production records in the year 2006, sampled every 10 minutes from 137 photovoltaic power plants in Alabama State (Lai et al.
conventional benchmark, GluoTTS (Alexandrov et al. 2020) which represents hourly sampled data. The task is to condition on 168 time sleps to forecast the following 24. The whole data set contains 73 samples of 192 lime steps and 128 features each in a chronological manner, we use the first 65 as training set and the remaining 8 as the set. Similar to the preprocessing applied other data sets describod above, we feature sample this data set by a factor of 2, where for the training set, for example, we obtained batches of 65 samples with 192 time steps and 64 features each. We standard scale the train and test sets for training using training set statistics.
2018). However,
collected the data set from
27

---

Published in Transactions on Machine Learning Research (02/2023)
Table 22: Solar data set details.
Description
Value
Train size
130
Test size
16
Training batch
65
Sample length
192
Data set features
128
Sample features
64
Conditional values
168
Target values
24
Table 23: ETTml data set details.
Description
Value
Train size
33,865, 34,417, 34,000, 33,600, 33,200 11,490, 10,000, 11,420, 10,000, 10,000
Test size
Training batch
65, 127, 17, 14, 4
Sample length
120, 96, 480, 576, 1,052
Data set features
7
Conditional values
96, 48, 384, 288, 384
Target values
24, 48, 96, 288, 672
contains the ETTml data set details. This data set was created to investigate the amount of detail required for long-time series forecasting based on the Electricity Transformer Temperature (ETT), which an important measure for the long-term deployment electric power. The data set contains information from compilation of 2-year data from two distinct Chinese counties. Here, we work with ETTml which covers data at 15-minute level. The data is composed of the target value oil temperature and six power load features. We collected and preprocossod the data directly from Zhou et al. (2021). ). we forecast in each of the benchmarking horizons, utilizing train and test sets, coming from the original split of /val/test set, which was for 12/4/4 months, respectively. As seen in Table 23 there are five different preprocessing scttings implemented with regard to the forccasting horizon, the first table contains information on three and the second for the remaining two, where the main difference the nunber of values used to condition on, and the targets to forecast. Similarly, there are some differences with respect to the batch size used during training. As the samples get longer, we utilize smaller batch sizes. Finally, for samples generation, the test sets are very large with more than 10,000 samples each, subset the test set in order to obtain batches.
ETTml data set Table 23
E
Additional technical information
E.1
S4 Model
ifn > (2m + 1)2/2((2(2 1
Ank
n+1
ifn = k
(5)
0
if < k
S4 is a clep structured stale space model, which is built with four ideal components for time series analysis, specifically, for the handling of long sequences. First of all, it contains the continuous representation of SSMs previously introduced Eq. 4 for which with the help of HiPPO matrices (Gu et al., 2020). provides an importance score for each past time step through an online compression of signals and novel and robust updating system (HiPPO-LegS) that scales for long sequences. In nutshell, HiPPO aims to overcome the
28

---

Published in Transactions on Machine Learning Research (02/2023)
issue of time series of different lengths and the vanishing gradient problem. HiPPO is referenced in 5 which compresses the scaled Legendre measure (TegS) operator for uniform system update on the SMM vector A.
= (1 (1-1-11 A/2 A) (1 A/2
B -1-1//2 -A
(6)
C=C
Additionally, the HiPPO architecture provides the advantage of being able to handle irregularly sampled data with the help of recurrent discretization procedure by a step size A using the bilinear method. The discretization procodure in Eq. crcates sequenceeto-ssquenee mapping that yiclds recurrent state space; here, the SSMs can be computed like an RNN. Concretely, it can be viewed as hidden state with a transition matrix A.
yk = CAB uu CAB 1-
U1 +... CABu:-1+ CBup y=
(7)
(K, = (CAB JiEL = (CB, CAB, CAB"
(8)
The RNN described previously can be transformed into a CNN by unrolling in the discrete convolution step, which is shown in 7 where single convolution can be computed efficiently with fast Fourier transform given the SSM convolution kernel K represented in Eq. We strongly encourage the reader to refer (Gu et al. 2022a and (Gu et al. 2020) for further investigation of the S4 model and its components. From an external perspective, however, the $4 layer that is composed of multiple S4 blocks can be considered as drop-in replacement for a one-dimensional (potentially dilated) convolutional layer, RNN, or transformer layer, which is particularly adapted to the requirements of time scries analysis.
E.2 Performance metrics
This section describes the performance metrics used throughout this work. We use wide range of performance metrics for the computation of errors between largets and imputed values (er = f), where for and we apply the respective conditional masking for metrics computation (Meval = mavi O (1 mimp)).
MAE
(y 1) O Mevalla.k
(9)
n1
1
MSE
(y - () 3 meval L.k
(10)
1=1 k=1
K
RMSE
(y- - 1 C Meval)?.
(11)
K ( 00,
MRE
meval 1,k
(12)
Dt.k
Firstly, from a deterministic perspective, specifically for the computation of absolute errors, we implemented mean absolute error (MAF) equation which determined by dividing the total absolute errors by the
29

---

Published in Transactions on Machine Learning Research (02/2023)
total sample size of observations For the account of larger error sensitivity we implemented mean squared error (MSE) equation 10 which is the sum Σ of squared errors ea divided by the total sample size of
observations n, similarly, implemented root mean squared error (RMSE) equation 11 to account for larger errors while preserving the error ranges in proportion to the observed mean, and to measure the precision of our imputations mean relative error (MRE) which as percentage representation computes the mean of differences between the absolute errors y il divided by their target values y.
E.3
Training times
Table 27
Table 29 and Table 28 describe the number of training iterations and epochs used while training
each of the data sets.
Table 24: Imputations for RM, RBM, and BM scenarios the PTB-XL data set.
Model
MAE
RMSE
20% RM on PTB-XL
Median
0.1040
0.2071
LAMC
0.0678
0.1309
DillWaveDo
0.004742e-5
0.0175+:3e4
DiffWaveD;
4.0043++4-4
4.0177+4e-4
CSDI
0.0038+20-6
0.0189+56-5
CSDI S4
0.0031+1e-7
0.0171166-4
SSSD SA Do
0.005243e-5
0.022914e-6
SSSD SA Di
0.0045+3e-7
0.0181+46-6
SSSD Do
0.004412e-5
0.0137+1e-
SSSD
0.0034
0.0119+1e-4
20% RBM on PTB-XT.
Median
0.1074
0.2157
LAMC
0.0759
0.1498
DiffWaveDo
0.0
0.120918e-3
DiffWaveD;
1c-
0.0808
CSDI
1.0186+1e-5
0.043512e-4
CSDIS4
0.0222+20-5
0.0573+1e-3
SSSD SA Do
0.0
0.0612t1e-2
SSSDS Di
0.0170+1e-4
0.0492+1e-2
SSSDS1D Do
0.011642e-4
0.025117e-4
SSSD
0.0103-:3--3
0.0226-99-4
20% RM on PTВ-XL
Median
0.1252
0.2347
LAMC
0.0840
0.1171
DiffWaveD,
0.0492+1e-3
0.140518e-3
DillWaveD,
0.045147e-4
0.137815e-3
CSDI
0.105414e-5
0,225417e-5
S4
0.0792+20-4
0.1879+164
SSSD SA Do
0.0493+1e-3
0.119247e-3
SSSD D
0.0435436-3
0.1167+10-2
SS Do
0.0415+1e-3
0.107315e-3
SS
0.032413e-3
0.0832188-3
30

---

Published in Transactions on Machine Learning Research (02/2023)
Table 25: MAE, MSE, MRE for PEMS-BAY and METR-LA data sets, All baseline results were collected from Cini et al 2022
Model
MAE
MSE
MRE
25% RM on PEMS- PEMS-Bay
Mean
5.42+00.00
86.59+0..00
8.67=0.00
KNN
4.30::0.00
49.80+1.000
6.88=0.00
MF
3.29+4000
51.39+0.64
5.27=0.02
MICE
3.09:0.02
31.4340.41
4.95=0.02
VAR
1.30+0.00
6.52=0.01
2.07=0.01
GAIN
1.88+0.02
10.37++0.00
3.01
BRITS
1.47:0.00
7.94=0.03
2.36=0.00
MPGRU
1.11#00000
7.59-0.02
1.77=0.00
GRIN
0.67
1.55
1.08+0.00%
SSSD
0.97++0.01
2.98+0.03
1.42-0.01
25%
RM
on PEMS-Bay
Mean
7.56+0.00
142.22:0.00
13.10#0.00
KNN
7.88+0.00
129.29+0.00
13.65+00000
MF
5.56:L0.03
113.46#1.08
9.62
MICE
4.42++0.07
55.07+11.66
7.65+
VAR
2.69:00.00
21.10:10.0
4.66=0.00
GAIN
2.83+40.01
20.03:+0.09
4.91=0.01
BRITS
2.34
16.46+10.05
4.05
MPGRU
2.44+0.00
22.17
4.22=0.00
GRIN
1.91 10.41:10.03
3.30
SSSD
2.83+0.02
21.95+:.14
5.59=0.08
Table 26: Forecasting results for the PTB-XL data set.
Model
MAE
RMSE
Median
0. 131
0.273
CSDI
0.10590000009
0.302=0.0004
S4
0.120:0.0002
0.246=0.0001
SSSD
0.087
0.220-0.012
SSSD S4
0.090+0.003
0.2194400006
Table 27: Training on the proposed PTB-XI. data sets. training epochs (e) and iterations (i) on the proposed
PTB-XL data set.
Model
PTB 248
PTB 1000
CSDI
200 (c)
200 (c)
SS
200 (e)
200 (e)
DiffWave
150,000 (i)
150,000 (i)
SSSDSA
150,000
150,000 (i)
SS
150,000 (i)
150,000 (1)
31

---

Published in Transactions on Machine Learning Research (02/2023)
Table 28: CSDI training benchmarking data sets. contains the CSDI training epochs (e) the bench- marking data sets.
Dataset
Setting
Epochs
Flectricity
Imputation 10%
200 (e)
Electricity
Imputation 30%
200 (e)
Electricity
Imputation 50%
200 (c)
MuJoCo
Imputation 70%
200 (e)
MuJoCo
Imputation 80%
200 (c)
MuJoCo
Imputation 90%
200 (e)
ETTml
Forecast 24
200 (e)
ETTml
Forecast 48
200 (c)
ETTml
Forecast 96
200 (c)
ETTml
Forecast 288
200 (e)
ETTml
Forecast 672
200 (e)
Table 29: SS training on benchmarking data sets. contains the SSSD training iterations (i) on the benchmarking data sets across diverse baselines.
data set
Setting
Iterations
Electricity
Imputation 10%
150,000 (i)
Electricity
Imputation 30%
150,000 (i)
Electricity
Imputation
50%
150,000
PEMS-BAY
Imputation 25%
350,000
METR-LA
Imputation 25%
250,000
MuJoCo
Impulation 70%
232,000
MuJoCo
Imputation 80%
160,000
MuJoCo
Imputation 90%
150,000
(1)
Solar
Forecast 24
100,000
(1)
ETTml
Forecast 24
212,000
ETTml
Forecast 48
150,000
ETTml
Forecast 96
250,000
ETTml
Forecast 288
250,000 (i)
ETTml
Forecast 672
250,000 (i)
32

---

Published in Transactions on Machine Learning Research (02/2023)
F
Multimedia Appendix
value
value
-1
Target
CSDI
-1
Tar Taiget
CSDI
50 100
150
200
250
50
100
150
200
250
value
Target
Tar Target
SSSD
SSSD
150
Figure 6: PTB-XL RM imputations of V3 lead for an ECG from patient with anterior myocardial infarction (AMI). The figure shows 100 RM imputations the PTB-XL data set. As clearly seen from the plots above, all the diffusion models are highly capable of reconstructing time series, even when there is high ratio of missing values in RM scenario. Quantitatively, as seen in the main paper, we observed that CSDI outperforms the rest of the models on this data set. The differences in terms of imputation quality can hardly be observed visually as the rest of the models present. slight green shaded areas surrounding the orange imputed areas which represent the imputation quantiles.
single sample tested at 50% for four imputers
1.0
Target CSDI
1.0
Target CSDI
value
0.5
value
0.0
0.5
-0.5
50
100
150
200
250
100
150
200
250
Target SS
Target 5550
value
0.5
0.5
0.0
005
100
150
200
250
Figure 7: PTB-XL RBM imputations of V4 lead for ECG from patient with an AV block (AVB). The figure shows 100 RBM imputations single sample tested at 50% for four imputers on the PTB-XL data set. As previously discussed, RBM assumes that the missing blocks are located at different time steps on cach feature. Technically, in this setting, the diffusion models are capable inferring the missing segments in a given channel from neighboring channels at the same time steps, which empirically seems to be a factor that enables an overall very good reconstruction across all models.
33

---

Published in Transactions on Machine Learning Research (02/2023)
value
value
Target
Target
C5D1
C5D
-2
1
value
value
0
Target SS
1
Target
SSSD
-2
200
250
50
100
150
200
250
Figure PTB-XL BM imputations for lead V2 for an ECG collected from a healthy patient. The figure shows four BM imputations tested 30% on the PTB-XL data reiterating the qualitative differences imputation quality already demonstrated in the main text.
0.6
Target CSDI
0.6
Target
CSD
value
0.4
value
0.2
0.0
100
200
0.6
Target 5S50
0.6
Target S
value
value
0.2
0.2
0.0
250
50
100
150
200
250
Figure PTB-XL BM imputations for lead V6 for a normal heart condition. The figure shows four BM imputations tested at 30% on the PTB-XL dala set.
34

---

Published in Transactions on Machine Learning Research (02/2023)
1.0
Target
0.5
0.5
CSD
value
value
0.0
Target
-0.5
CSDI
-0.5
50
100
150
200
250
value
value
0.0
0.0
Target
SSSD54
S5SD#4
100
150
200
250
Figure 10: PTB-XL TF for lead V3 for an ECG from a patient with left ventricular hypertrophy. The figure shows 100 BM imputations on a single sample tested at 50% for four imputers on the PTB-XL data set. CSDI model is overall learning the trend that the scrics has, with a few correct feature generations as seen the plot, nevertheless, we observe that the impulation falls outside the 0.05 and 0.95 quantile range, which is basically an outlier. CSDI improves the quality of the generations, as its quantiles are less diverse and start to follow the signals patterns, however, it seems that after certain number of steps, the learning decrease dramatically. On the contrary, SSSD and SSSD correctly capture the characteristics of the signal. In particular, SSSDS1 maintains a tight interquartile range even at longer forecasting horizons.
Target
0.5
0.5
SSSD
value
value
0.0
0.0
-0.5
Target
SSSD
-1.0
-1.0
20
40
100
120
20
40
60
80
100
1.0
Target 5550
Tar
S SSD
value
value
0.0
-1.0
500
100
200
300
400
500
600
Target
SSSD14
value
1
200
400
600
800
1000
Figure 11: ETTml TF for the five different forecasting settings. From top to bottom 24, 48, 96, 288, 384 forecasting horizon targets. These plots display the complete sample including the conditional part.
35

---

Published in Transactions on Machine Learning Research (02/2023)
150
value
100
Target SSSD
value
100
Target
SSSD
50
50
25
50
100
125
150
175
200
100
150
150
value
100
Target
100
Target
SSSO
SSSDRF
50
100
200
150
Target
150
S
value
value
100
Target SSSD
50
50
25
50
75
100
125
150
175
200
Figure 12: TF for six different channels of the Solar data set. These plots display the complete sample including the conditional part, see Figure 13 for more detailed plots of the imputed area only.
Target S
Target 5550
value
value
50
50
10
15
20
25
30
10
15
20
25
30
150
Target
100
SSSD24
555D
value
10
15
20
30
15
20
25
Target
Target SsSD
5S5D
100
volua
value
0
10
15
30
Figure 13: TF for six different. channels of the Solar data set. These plots display only the imputed area.
36


Reviews:
[{"review_id": "RacVuEhjGn", "reviewer": "Reviewer_vTws", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n---\n1. As many architectures are only compatible with regular discretely sampled data, data imputation techniques are important to allow for common deep neural networks to be trained on data with missingness. \n\n2. The strong experimental results provide strong support for the benefits of using SSSD for data imputation, despite lower performance in the forecasting scenario.\n\nWeaknesses\n---\n1. The paper is mostly well written, but I did find it a little difficult to understand how the model was concretely implemented. In particular, 1) how the model utilizes state space components, 2) what loss function and batching procedure was used in training, and 3) how imputation is performed at run-time. It would be great to get all architectural components laid out in equation form for detailed analysis.\n\n2. While cited literature on imputation models is comprehensive, there is a lot of prior work on hybrid deep learning models that incorporate state space components which should be referenced – e.g. Kalman VAE, Deep Kalman Filters, Deep State Space Models, Deep Factors for forecasting etc to name a few.\n\n3. Are all models compatible with backward imputation, and does the SSSD use backward and forward smoothing in all imputation tasks? \n\n4. Could the authors describe imputation applications where a backward step might be useful? Where imputation as a pre-cursor to train other models, would there be any concerns about leaking future information into the imputed data if a backward smoothing step is used?\n\n5. For consistency, all models should use a common set of benchmarks and metrics for better comparisons.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new hybrid model for time series imputation -- which combines a structured state-space models together with deep learning components. This allows the model to pick up longer-term dependencies more easily, while still learning complex patterns from data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think a much more detailed description of the model architecture and training would be crucial to include -- along with concrete answers to the questions above.", "broader_impact_concerns": "-", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GVDRrI5p56", "reviewer": "Reviewer_kCUh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strong Aspects:**\n\n1) The motivation behind the proposed methodology is clear. The introduction of S4 layers adapted in the UNet architecture to handle imputation scenarios in time-series data is novel and makes sense. The authors also provide a nice ablation experiment in the Appendix which justifies the use of S4 layers as architectural components.\n\n2) The methodological contributions are supported by sound empirical results (both qualitatively and quantitatively). The proposed method seems to perform better (or at par) than competing SoTA baselines over multiple benchmarks across missingness settings like RM, MNR, BM, or Forecasting. Moreover, the quantitative results have been discussed in detail in the main text while most of the training details for SSSD and other baselines have been covered in significant detail in the Appendix.\n\n3) With relevant adjustments, the authors show that the proposed method can also adapt to datasets with more than 100 channels on the Electricity dataset with significant improvements over other baselines which is interesting!\n\n**Weaknesses**: Given the strong aspects, I do have some concerns/questions about the experiments in general, which are discussed in more detail in the \"Requested Changes\" section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present SSSD (Structured State Space Diffusion), a model which combines recent innovations in sequence modeling (i.e. S4 layers) with conditional diffusion models for time-series imputation. The main methodological contributions are as follows:\n\n1) The authors propose to augment the DiffWave model architecture by introducing two S4 layers in each residual block in the UNet-based denoiser (Fig. 2 in the manuscript) to enhance its capability for time series modeling.\n2) Diffusion is either applied on the entire input signal ($D_0$) or in the regions of the input which need to be imputed only ($D_1$). The choice of $D_1$ is justified using empirical results.\n3) The conditioning signal to the reverse process is the concatenation of the imputation masks and the imputed signal.\n\nThe authors justify the choice of S4 layers using a small ablation experiment while also presenting empirical results on various benchmarks (like PTB-XL, MuJoCo etc.) in tasks across settings like RM, MNR, BM and Forecasting (which is a special case of BM-missingness).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Requested Changes (Major):**\n\n1) The authors have provided some ablation results related to S4 layers in the Appendix. However, it appears that these results are based on the 20% BM scenario on the PTB-XL dataset. Given that, the introduction of S4 layers in the UNet model is one of the primary contributions of this work, I would expect these ablations across multiple imputation settings (including RM, MNR) and missingness ratios (20%, 30% etc.). It would suffice to show these results for at-least one dataset (maybe PTB-XL?).\n\n2) Diffusion and Training Hyperparameters: It would help to observe the impact of using larger number of timesteps during training and inference on the imputation performance. The authors do mention about this briefly in Appendix A.1 as: \"We found that the fewer diffusion steps, the\nfaster the network converges during training, however, at the cost of less accurate results\". It would be great if authors can present some quantitative results about the same on a dataset (maybe PTB-XL) since the Inference speed vs performance tradeoff is central to diffusion models.\n\n3) Did the authors experiment with a setting where there is a mismatch between the missingness ratios during training and inference? For instance, a setting when the model is trained at 20% RM but is evaluated at 30% RM (and vice-versa)? Another possibility could be training for one missingness scenario (maybe 20% RM) and evaluating for MNR or BM? From my understanding of Section 4.1, the authors always train and evaluate across the same setting. It would be interesting to benchmark the generalization capabilities of the imputation model.\n\n4) For the PTB-XL dataset results in Section 4.2, the authors consider a 20% missingness ratio across all settings (RM, MNR and BM) (Table 1 and 20). However, for other datasets (like MuJoCo (Table 2), Electricity (Table 3 and 21)) the authors consider multiple sparsity ratios for only the RM scenario. Is there a specific reason for the selection of specific settings for specific datasets? If possible, I would strongly recommend the authors to present comparisons across multiple imputation ratios (20%, 30% etc.) across multiple settings for atleast one dataset (maybe PTB-XL or whichever the authors deem fit).\n\n**Requested Changes (Minor)**\n\nThe following changes are mostly meant to improve the readability of the paper:\n\n1) Consider adding a more (formal) explanation of different missingness scenarios in Section 2 which is consistent with existing survey literature [1,2]. For instance does the Random Missing (RM) scenario correspond to MCAR or MAR as described in [1]? Similarly for other missingness scenarios considered in the paper.\n\n2) The authors mention about a channel splitting approach for datasets with more than 100 channels. However, I could not find an explanation of the same in the main text or the Appendix. If already discussed, can the authors point me to a relevant section in the paper. If not, I would strongly recommend the authors to explain the approach. Moreover, the authors hint that the method might not perform well on datasets with more than 100 channels per time point without modifying the approach with suitable approaches. I would encourage the authors to mention this as possible future directions to the proposed approach (among others).\n\n3) In Section 2 (Diffusion models), consider citing [3,4] since Score matching with gaussian perturbation kernels $p(x_t|x_0)$ is equivalent to the “epsilon”-prediction framework in SoTA Diffusion models (as also shown in Ho et al.). Moreover, for Video Diffusion models consider citing [5] which is one of the first papers to propose diffusion for Video prediction. Lastly, for Conditional Diffusion models, consider citing [6] which discusses different conditioning mechanisms for forward and reverse processes in the Diffusion backbone. It is worth noting, that list is in no way exhaustive, and the authors might want to look into relevant literature.\n\n4) There are several instances of typos and missing references in the paper. For example, references to relevant tables in Sections 4.2 and 4.3 on the ECG data are missing. In Section 4.3, “table 4” should be “Figure 4”. I would highly recommend the authors to fix these (but not limited to) minor issues in subsequent revisions.\n\nReferences:\n\n[1] A survey on data imputation techniques: Water distribution system as a use case\n\n[2] Missing value imputation: a review and analysis of the literature (2006–2017)\n\n[3] Generative Modeling by Estimating Gradients of the Data Distribution\n\n[4] Score-Based Generative Modeling through Stochastic Differential Equations\n\n[5] Diffusion Probabilistic Modeling for Video Generation\n\n[6] DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents", "broader_impact_concerns": "The authors present a generative model in this work, so Broader concerns regarding generation of fake data apply to this method as well.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lG5yYGOhcG", "reviewer": "Reviewer_5Tb4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Time series imputation is an important problem.\n- The proposed model seems to exhibit strong empirical performance on many tasks.\n\nWeaknesses:\n- The combination of the two existing DiffWave and S4 models seems slightly incremental, but that should normally not be a problem for TMLR.\n- The selection of baselines in the experiments is sometimes a bit unclearly motivated.\n\nMajor comments:\n- In the imputation literature, the different types of missingness are usually called missing completely at random (MCAR), at random (MAR), and not at random (MNAR) [1]. It seems like what you call RM is MCAR, while MNR, BM, and TF are all types of MAR. MNAR would imply that the missingness actually correlates with the underlying true value, which I don't think you consider. Especially your MNR nomenclature is therefore a bit confusing and it would probably be better to use the standard terms or at least clearly explain their correspondence to your settings.\n- There are many baselines that seem pretty strong, but only show up in selected experiments, such as NAOMI, NRTSI, SAITS, and the Autoformer. It is a bit unclear to the reader how these sets of baselines are selected and why these strong baselines would not be used in the other experiments. Could for instance the Autoformer not also be used on the solar data, or electricity, or ECG? I would generally expect at least the strongest baselines to be used in all experiments where they can plausibly be applied.\n\nMinor comments:\n- In the figures, why is always a random sample plotted instead of the mean of the imputation distribution? Generally, I would find the mean more informative, since the quality between samples can presumably vary a lot.\n- In Sec. 2, should it say $m_{imp} \\in \\lbrace 0,1 \\rbrace^{L \\times K}$ or can the indicators indeed take values in the interval between 0 and 1, as written in the paper?\n- Please use \\citep{} for parenthetical citations.\n- Could you discuss the tradeoff of mapping the input channels to the diffusion dimension in a bit more detail? Especially to readers that are not perfectly familiar with the related literature, this could help to assess what the implied design decisions entail.\n- Fig. 2: Tahn -> Tanh\n- Fig. 2: $\\sigma2$ -> $\\sigma^2$\n- Tab. 2: maybe you could just write in the caption that all values are on the order 1e-3, to remove clutter from the table\n- Tab. 4: The error bars seem quite large. Is the proposed method's performance really 0.027 +/- 4.32 or should it be 0.027 +/- 0.0432 or something? If the error bars are really that large, shouldn't all methods be boldened in the table, because their error bars would overlap? Does this imply anything about the robustness of the proposed method?\n\n[1] Donald B. Rubin, Inference and missing data, Biometrika, Volume 63, Issue 3, December 1976, Pages 581–592, https://doi.org/10.1093/biomet/63.3.581", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose to combine the DiffWave diffusion model for time series with structured state-space models (S4) as internal layers to yield a conditional diffusion model that can be used for time series imputation. They show that this outperforms many existing time series imputation baselines on different real-world datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As mentioned above, the relation of the studied missingness settings to the known types from the literature (MCAR, MAR, MNAR) should be made more clear. Also, the strong baselines, such as the Autoformer, should be compared against in all experiments where they can plausibly be used. Besides, there are a few typos and minor issues that should be fixed.", "broader_impact_concerns": "There are no boarder impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a conditional diffusion model for time series imputation. Their model relies on combining DiffWave and structured state-space models. They argue that this approach addresses some of the shortcomings of existing models and can handle long-term dependencies in time series data. The authors stress the importance of using probabilistic imputation methods that provide samples of different plausible imputations instead of a single imputation.

Time series imputation is an important practical problem, e.g., in healthcare data analysis. Strengths of the paper include systematic ablations of their approach, as well as good practical performance, especially in high-dimensional time series. 

The authors successfully resolved minor issues relating to the implementation (e.g., uni- vs. bidirectional context), their used nomenclature regarding different missingness patterns, added a consistent baseline method, and improved the paper structure for better readability. 

Overall, I recommend the paper for acceptance. 
Decision: Accept (poster)