Generative Time-series Modeling with Fourier Flows

1 INTRODUCTION . Lack of access to data is a key hindrance to the development of machine learning solutions in application domains where data sharing may lead to privacy breaches ( Walonoski et al . ( 2018 ) ) . Areas where this problem is most conspicuous include medicine , where access to ( highly-sensitive ) clinical data is stringently regulated by medical institutions ; such strict regulations undermine scientific progress by hindering model development and reproducibility . Generative models that produce sensible and realistic synthetic data present a viable solution to this problem—artificially-generated data sets produced by such models can be shared widely without privacy concerns ( Buczak et al . ( 2010 ) ) . In this paper , we focus on the time-series data setup , where observations are collected sequentially over arbitrary periods of time with different observation frequencies across different features . This general data setup is pervasive in the medical domain—it captures the kind of data maintained in electronic health records ( Shickel et al . ( 2017 ) ) or collected in intensive care units ( Johnson et al . ( 2016 ) ) . While many machine learning-based predictive models that capitalize on such data have been proposed over the past few years ( Jagannatha & Yu ( 2016 ) ; Choi et al . ( 2017 ) ; Alaa & van der Schaar ( 2019 ) ) , much less work has been done on generative models that could emulate and synthesize these data sets . Existing generative models for ( medical ) time-series are based predominantly on implicit likelihood modeling using generative adversarial networks ( GANs ) , e.g. , Recurrent Conditional GAN ( RCGAN ) ( Esteban et al . ( 2017 ) ) and TimeGAN ( Yoon et al . ( 2019 ) ) . These models apply representation learning via recurrent neural networks ( RNN ) combined with adversarial training in order to map noise sequences in a latent space to synthetic sequential data in the output space . Albeit capable of flexibly learning complex representations , GAN-based models can be difficult to train ( Srivastava et al . ( 2017 ) ) , especially in the complex time-series data setup . Moreover , because they hinge on implicit likelihood modeling , GAN-based models can be hard to evaluate quantitatively due to the absence of an explicitly computable likelihood function . Finally , GANs are vulnerable to training data memorization ( Nagarajan et al . ( 2018 ) ) —a problem that would be exacerbated in the temporal setting where memorizing only a partial segment of a medical time-series may suffice to reveal a patient ’ s identify , which defeats the original purpose of using synthetic data in the first place . Here , we propose an alternative explicit likelihood approach for generating time-series data based on a novel class of normalizing flows which we call Fourier flows . Our proposed flow-based model operates on time-series data in the frequency-domain rather than the time-domain—it converts variable-length time-series with varying sampling rates across different features to a fixed-size spectral representation using the discrete Fourier transform ( DFT ) , and then learns the distribution of the data in the frequency domain by applying a chain of data-dependent spectral filters to frequency-transformed time-series . Using the convolution property of DFT ( Oppenheim ( 1999 ) ) , we show that spectral filtering of a timeseries in the frequency-domain—an operation that mathematically resembles affine transformations used in existing flows ( Dinh et al . ( 2016 ) ) —is equivalent to a convolutional transformation in the timedomain . This enhancement in the richness of distributions learned by our flow comes at no extra computational cost : using Fast Fourier Transform ( FFT ) algorithms , we show that the entire steps of our flow run in O ( T log T ) time , compared to the polynomial complexity of O ( T 2 ) for a direct , time-domain convolutional transformation . We also show that , because the DFT is a linear transform with a Vandermonde transformation matrix , computation of its Jacobian determinant is trivial . The zero-padding and interpolation properties of DFT enables a natural handling of variable-length and inconsistently-sampled time-series . Unlike existing explicit-likelihood models for time-series data , such as deep state-space models ( Krishnan et al. , 2017 ; Alaa & van der Schaar , 2019 ) , our model can be optimized and assessed through the exact likelihood rather than a variational lower bound . 2 PROBLEM SETUP . We consider a general temporal data setup where each instance of a ( discrete ) time-series comprises a sequence of vectors x = [ x0 , . . . , xT−1 ] , xt ∈ X , ∀ 0 ≤ t ≤ T − 1 , covering a period of T time steps . We assume that each dimension in the feature vector xt is sampled with a different rate , i.e. , at each time step t , the observed feature vector is xt = [ xt,1 [ r1 ] , . . . , xt , D [ rD ] ] , where rd ∈ N+ is the sampling period of feature dimension d ∈ { 1 , . . . , D } . That is , for a given sampling period rd , we observe a value of xt , d every rd time steps , and observe a missing value ( denoted as * ) otherwise , i.e. , xt , d [ rd ] , { xt , d , t mod rd = 0 , ∗ , t mod rd 6= 0 . ( 1 ) The data setup described above is primarily motivated by medical time-series modeling problems , wherein a patient ’ s clinical measurements and bio-markers are collected over time at different rates ( Johnson et al . ( 2016 ) ; Jagannatha & Yu ( 2016 ) ) . Despite our focus on medical data , our proposed generative modeling approach applies more generally to other applications , such as speech synthesis ( Prenger et al . ( 2019 ) ) and financial data generation ( Wiese et al . ( 2020 ) ) . Each realization of the time-series x is drawn from a probability distribution x ∼ p ( x ) . In order to capture variable-length time-series ( common in medical problems ) , the length T of each sequence is also assumed to be a random variable—for notational convenience , we absorb the distribution of T into p. One possible way to represent the joint distribution p ( x ) is through the factorization:1 p ( x ) = p ( x0 , . . . , xT−1 , T ) = p ( T ) · T−1∏ t=0 p ( xt |x0 , . . . , xt−1 , T ) . ( 2 ) We assume that the sampling period rd for each feature d is fixed for all realizations of x . The feature space X is assumed to accommodate a mix of continuous and discrete variables on its D dimensions . 1Our proposed method is not restricted to any specific factorization of p ( x ) . Key objective . Using a training data set D = { x ( i ) } ni=1 comprising n time-series , our goal is to ( 1 ) estimate a density function p̂ ( x ) that best approximates p ( x ) , and ( 2 ) sample synthetic realizations of the time-series x from the estimated density p̂ ( x ) . When dealing with data sets with variable lengths for the time-series , we model the distribution p ( T ) independently following the factorization in ( 2 ) . We model p ( T ) using a binomial distribution . Throughout the paper , we focus on developing a flow-based model for the conditional distribution p ( x0 , . . . , xT−1 |T ) . 3 PRELIMINARIES . Let z ∈ RD be a random variable with a known and tractable probability density function p ( z ) , and let g : RD → RD be an invertible and differentiable mapping with an inverse mapping f = g−1 . Let x = g ( z ) be a transformed random variable—the probability density function p ( x ) can be obtained using the change of variable rule as p ( x ) = p ( z ) · | detJ [ g ] |−1 = p ( f ( x ) ) · | detJ [ f ] | , where J [ f ] and J [ g ] are the Jacobian matrices of functions f and g , respectively ( Durrett ( 2019 ) ) . Normalizing flows are compositions of M mappings that transform random draws from a predefined distribution z ∼ p ( z ) to a desired distribution p ( x ) . Formally , a flow comprises a chain of bijective maps g = g ( 1 ) ◦ g ( 2 ) ◦ · · · ◦ g ( M ) with an inverse mapping f = f ( 1 ) ◦ f ( 2 ) ◦ · · · ◦ f ( M ) . Using the change of variables formula described above , and applying the chain rule to the Jacobian of the composition , the log-likelihood of x can be written as ( Rezende & Mohamed ( 2015 ) ) : log p ( x ) = log p ( z ) + M∑ m=1 log |detJ [ fm ] | . ( 3 ) Existing approaches to generative modeling with normalizing flows construct composite mappings g with structural assumptions that render the computation of the Jacobian determinant in ( 3 ) viable . Examples of such structurally-constrained mappings include : Sylvester transformations , with a Jacobian corresponding to a perturbed diagonal matrix ( Rezende & Mohamed ( 2015 ) ) , 1× 1 convolutions for cross channel mixing , which exhibit a block diagonal Jacobian ( Kingma & Dhariwal ( 2018 ) ) , and affine coupling layers that correspond to triangular Jacobian matrices ( Dinh et al . ( 2016 ) ) . 3.1 FOURIER TRANSFORM . The Fourier transform is a mathematical operation that converts a finite-length , regularly-sampled time domain signal x to its frequency domain representationX ( Bracewell & Bracewell ( 1986 ) ) . A T -point discrete Fourier transform ( DFT ) , denoted asX = FT { x } , transforms a ( complex-valued ) time-stamped sequencex , { x0 , . . . , xT−1 } into a length-T sequence of ( complex-valued ) frequency components , X , { X0 , . . . , XT−1 } , through the following operation ( Oppenheim ( 1999 ) ) : Xk = T−1∑ t=0 xt · e−2πj· kt T , ∀1 ≤ k ≤ T − 1 , ( 4 ) where j corresponds to the imaginary unit of a split-complex number . Using Euler ’ s formula , the complex exponential terms in ( 4 ) can be expressed as e−2πj· kt T = cos ( 2π · ktT ) − j · sin ( 2π · ktT ) . Thus , the Fourier transform decomposes any time-series into a linear combination of sinusoidal signals of varying frequencies—the resulting sequence of frequency components , X , corresponds to the coefficients assigned to the different frequencies of the sinusoidal signals constituting the time domain signal x . The DFT is a key computational and conceptual tool in many practical applications involving digital signal processing and communications ( Oppenheim ( 1999 ) ) . 3.2 FOURIER TRANSFORM PROPERTIES . We will rely in developing our model on various key properties of the DFT . These properties describe various operations on the time-domain data and their dual ( equivalent ) operations in the frequency domain . The DFT properties relevant to the development of our model are listed as follows : Convolution : x1 ⊗ x2 ⇐⇒ X1 X2 . Symmetry : If x is real-valued⇒ Xk = X∗k−m , ∀m ∈ N. Even/Odd Transforms : F { Even ( x ) } = Re ( X ) , F { Odd ( x ) } = Im ( X ) , where denotes element-wise multiplication , ⊗ denotes circular convolution , Re ( . ) and Im ( . ) denote the real and imaginary components , Even ( x ) = ( x+ x− ) /2 , Odd ( x ) = ( x− x− ) /2 , where x− signifies the reflection of x with respect to the x = 0 axis , and x = Even ( x ) + Odd ( x ) . Another property that is relevant to our model is the interpolation property , which posits that zero-padding of x in the time domain corresponds to an up-sampled version ofX in the frequency domain .

Summary: The paper presents Fourier Flows (FF), which is a time series generative model in the frequency domain. It shows that the Jacobian of the DFT is equal to 1, which means that DFT does not add too much overhead. The results on the real-world datasets are encouraging and expected because the predictive results mainly rely on the overall trend of the time series. By analysis in the frequency domain, we usually can capture the main trend accurately. The main concerns for the paper are the computational overhead on the proposed algorithm on non-periodic, long, and variable-length time series.