CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation

1 Introduction . Multivariate time series are abundant in real world applications such as finance , meteorology and healthcare . These time series data often contain missing values due to various reasons , including device failures and human errors [ 1 , 2 , 3 ] . Since missing values can hamper the interpretation of a time series , many studies have addressed the task of imputing missing values using machine learning techniques [ 4 , 5 , 6 ] . In the past few years , imputation methods based on deep neural networks have shown great success for both deterministic imputation [ 7 , 8 , 9 ] and probabilistic imputation [ 10 ] . These imputation methods typically utilize autoregressive models to deal with time series . Score-based diffusion models – a class of deep generative models and generate samples by gradually converting noise into a plausible data sample through denoising – have recently achieved state-ofthe-art sample quality in many tasks such as image generation [ 11 , 12 ] and audio synthesis [ 13 , 14 ] , outperforming counterparts including autoregressive models . Diffusion models can also be used to impute missing values by approximating the scores of the posterior distribution obtained from the prior by conditioning on the observed values [ 12 , 15 , 16 ] . While these approximations may work well in practice , they do not correspond to the exact conditional distribution . In this paper , we propose CSDI , a novel probabilistic imputation method that directly learns the conditional distribution with conditional score-based diffusion models . Unlike existing score-based approaches , the conditional diffusion model is designed for imputation and can exploit useful information in observed values . We illustrate the procedure of time series imputation with CSDI in 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Figure 1 . We start imputation from random noise on the left of the figure and gradually convert the noise into plausible time series through the reverse process pθ of the conditional diffusion model . At each step t , the reverse process removes noise from the output of the previous step ( t+ 1 ) . Unlike existing score-based diffusion models , the reverse process can take observations ( on the top left of the figure ) as a conditional input , allowing the model to exploit information in the observations for denoising . We utilize an attention mechanism to capture the temporal and feature dependencies of time series . For training the conditional diffusion model , we need observed values ( i.e. , conditional information ) and ground-truth missing values ( i.e. , imputation targets ) . However , in practice we do not know the ground-truth missing values , or training data may not contain missing values at all . Then , inspired by masked language modeling , we develop a self-supervised training method that separates observed values into conditional information and imputation targets . We note that CSDI is formulated for general imputation tasks , and is not restricted to time series imputation . Our main contributions are as follows : • We propose conditional score-based diffusion models for probabilistic imputation ( CSDI ) , and implement CSDI for time series imputation . To train the conditional diffusion model , we develop a self-supervised training method . • We empirically show that CSDI improves the continuous ranked probability score ( CRPS ) by 40-65 % over existing probabilistic methods on healthcare and environmental data . Moreover , deterministic imputation with CSDI decreases the mean absolute error ( MAE ) by 5-20 % compared to the state-of-the-art methods developed for deterministic imputation . • We demonstrate that CSDI can also be applied to time series interpolations and probabilistic forecasting , and is competitive with existing baselines designed for these tasks . 2 Related works . Time series imputations with deep learning Previous studies have shown deep learning models can capture the temporal dependency of time series and give more accurate imputation than statistical methods . A popular approach using deep learning is to use RNNs , including LSTMs and GRUs , for sequence modeling [ 17 , 8 , 7 ] . Subsequent studies combined RNNs with other methods to improve imputation performance , such as GANs [ 9 , 18 , 19 ] and self-training [ 20 ] . Among them , the combination of RNNs with attention mechanisms is particularly successful for imputation and interpolation of time series [ 21 , 22 ] . While these methods focused on deterministic imputation , GP-VAE [ 10 ] has been recently developed as a probabilistic imputation method . Score-based generative models Score-based generative models , including score matching with Langevin dynamics [ 23 ] and denoising diffusion probabilistic models [ 11 ] , have outperformed existing methods with other deep generative models in many domains , such as images [ 23 , 11 ] , audio [ 13 , 14 ] , and graphs [ 24 ] . Most recently , TimeGrad [ 25 ] utilized diffusion probabilistic models for probabilistic time series forecasting . While the method has shown state-of-the-art performance , it can not be applied to time series imputation due to the use of RNNs to handle past time series . 3 Background . 3.1 Multivariate time series imputation . We consider N multivariate time series with missing values . Let us denote the values of each time series as X = { x1 : K,1 : L } ∈ RK×L where K is the number of features and L is the length of time series . While the length L can be different for each time series , we treat the length of all time series as the same for simplicity , unless otherwise stated . We also denote an observation mask as M = { m1 : K,1 : L } ∈ { 0 , 1 } K×L where mk , l = 0 if xk , l is missing , and mk , l = 1 if xk , l is observed . We assume time intervals between two consecutive data entries can be different , and define the timestamps of the time series as s = { s1 : L } ∈ RL . In summary , each time series is expressed as { X , M , s } . Probabilistic time series imputation is the task of estimating the distribution of the missing values of X by exploiting the observed values of X . We note that this definition of imputation includes other related tasks , such as interpolation , which imputes all features at target time points , and forecasting , which imputes all features at future time points . 3.2 Denoising diffusion probabilistic models . Let us consider learning a model distribution pθ ( x0 ) that approximates a data distribution q ( x0 ) . Let xt for t = 1 , . . . , T be a sequence of latent variables in the same sample space as x0 , which is denoted as X . Diffusion probabilistic models [ 26 ] are latent variable models that are composed of two processes : the forward process and the reverse process . The forward process is defined by the following Markov chain : q ( x1 : T | x0 ) : = T∏ t=1 q ( xt | xt−1 ) where q ( xt | xt−1 ) : = N ( √ 1− βtxt−1 , βtI ) ( 1 ) and βt is a small positive constant that represents a noise level . Sampling of xt has the closed-form written as q ( xt | x0 ) = N ( xt ; √ αtx0 , ( 1 − αt ) I ) where α̂t : = 1 − βt and αt : = ∏t i=1 α̂i . Then , xt can be expressed as xt = √ αtx0 + ( 1− αt ) where ∼ N ( 0 , I ) . On the other hand , the reverse process denoises xt to recover x0 , and is defined by the following Markov chain : pθ ( x0 : T ) : = p ( xT ) T∏ t=1 pθ ( xt−1 | xt ) , xT ∼ N ( 0 , I ) , pθ ( xt−1 | xt ) : = N ( xt−1 ; µθ ( xt , t ) , σθ ( xt , t ) I ) . ( 2 ) Ho et al . [ 11 ] has recently proposed denoising diffusion probabilistic models ( DDPM ) , which considers the following specific parameterization of pθ ( xt−1 | xt ) : µθ ( xt , t ) = 1 αt ( xt − βt√ 1− αt θ ( xt , t ) ) , σθ ( xt , t ) = β̃ 1/2 t where β̃t = { 1−αt−1 1−αt βt t > 1 β1 t = 1 ( 3 ) where θ is a trainable denoising function . We denote µθ ( xt , t ) and σθ ( xt , t ) in Eq . ( 3 ) as µDDPM ( xt , t , θ ( xt , t ) ) and σDDPM ( xt , t ) , respectively . The denoising function in Eq . ( 3 ) also corresponds to a rescaled score model for score-based generative models [ 23 ] . Under this parameterization , Ho et al . [ 11 ] have shown that the reverse process can be trained by solving the following optimization problem : min θ L ( θ ) : = min θ Ex0∼q ( x0 ) , ∼N ( 0 , I ) , t|| − θ ( xt , t ) || 2 2 where xt = √ αtx0 + ( 1− αt ) . ( 4 ) The denoising function θ estimates the noise vector that was added to its noisy input xt . This training objective also be viewed as a weighted combination of denoising score matching used for training score-based generative models [ 23 , 27 , 12 ] . Once trained , we can sample x0 from Eq . ( 2 ) . We provide the details of DDPM in Appendix A . 3.3 Imputation with diffusion models . Here , we focus on general imputation tasks that are not restricted to time series imputation . Let us consider the following imputation problem : given a sample x0 which contains missing values , we generate imputation targets xta0 ∈ X ta by exploiting conditional observations xco0 ∈ X co , where X ta and X co are a part of the sample space X and vary per sample . Then , the goal of probabilistic imputation is to estimate the true conditional data distribution q ( xta0 | xco0 ) with a model distribution pθ ( x ta 0 | xco0 ) . We typically impute all missing values using all observed values , and set all observed values as xco0 and all missing values as x ta 0 , respectively . Note that time series imputation in Section 3.1 can be considered as a special case of this task . Let us consider modeling pθ ( xta0 | xco0 ) with a diffusion model . In the unconditional case , the reverse process pθ ( x0 : T ) is used to define the final data model pθ ( x0 ) . Then , a natural approach is to extend the reverse process in Eq . ( 2 ) to a conditional one : pθ ( x ta 0 : T | xco0 ) : = p ( xtaT ) T∏ t=1 pθ ( x ta t−1 | xtat , xco0 ) , xtaT ∼ N ( 0 , I ) , pθ ( x ta t−1 | xtat , xco0 ) : = N ( xtat−1 ; µθ ( xtat , t | xco0 ) , σθ ( xtat , t | xco0 ) I ) . ( 5 ) However , existing diffusion models are generally designed for data generation and do not take conditional observations xco0 as inputs . To utilize diffusion models for imputation , previous studies [ 12 , 15 , 16 ] approximated the conditional reverse process pθ ( xtat−1 | xtat , xco0 ) with the reverse process in Eq . ( 2 ) . With this approximation , in the reverse process they add noise to both the target and the conditional observations xco0 . While this approach can impute missing values , the added noise can harm useful information in the observations . This suggests that modeling pθ ( xtat−1 | xtat , xco0 ) without approximations can improve the imputation quality . Hereafter , we call the model defined in Section 3.2 as the unconditional diffusion model .

Summary: This work proposes a probabilistic data imputation model for multivariate time series. The proposed method uses a conditional diffusion model to estimate the (masked) conditional distribution of missing data given observed values. To utilise information from observed values, the authors employ a neural attention mechanism to capture spatiotemporal dependencies of time series. The authors use self-supervised learning for training the model.  In experiments, the author demonstrate competitive performance of their method against a number of comparable methodologies on time series imputation and forecasting tasks.