Generative Time-series Modeling with Fourier Flows

1 INTRODUCTION . Lack of access to data is a key hindrance to the development of machine learning solutions in application domains where data sharing may lead to privacy breaches ( Walonoski et al . ( 2018 ) ) . Areas where this problem is most conspicuous include medicine , where access to ( highly-sensitive ) clinical data is stringently regulated by medical institutions ; such strict regulations undermine scientific progress by hindering model development and reproducibility . Generative models that produce sensible and realistic synthetic data present a viable solution to this problem—artificially-generated data sets produced by such models can be shared widely without privacy concerns ( Buczak et al . ( 2010 ) ) . In this paper , we focus on the time-series data setup , where observations are collected sequentially over arbitrary periods of time with different observation frequencies across different features . This general data setup is pervasive in the medical domain—it captures the kind of data maintained in electronic health records ( Shickel et al . ( 2017 ) ) or collected in intensive care units ( Johnson et al . ( 2016 ) ) . While many machine learning-based predictive models that capitalize on such data have been proposed over the past few years ( Jagannatha & Yu ( 2016 ) ; Choi et al . ( 2017 ) ; Alaa & van der Schaar ( 2019 ) ) , much less work has been done on generative models that could emulate and synthesize these data sets . Existing generative models for ( medical ) time-series are based predominantly on implicit likelihood modeling using generative adversarial networks ( GANs ) , e.g. , Recurrent Conditional GAN ( RCGAN ) ( Esteban et al . ( 2017 ) ) and TimeGAN ( Yoon et al . ( 2019 ) ) . These models apply representation learning via recurrent neural networks ( RNN ) combined with adversarial training in order to map noise sequences in a latent space to synthetic sequential data in the output space . Albeit capable of flexibly learning complex representations , GAN-based models can be difficult to train ( Srivastava et al . ( 2017 ) ) , especially in the complex time-series data setup . Moreover , because they hinge on implicit likelihood modeling , GAN-based models can be hard to evaluate quantitatively due to the absence of an explicitly computable likelihood function . Finally , GANs are vulnerable to training data memorization ( Nagarajan et al . ( 2018 ) ) —a problem that would be exacerbated in the temporal setting where memorizing only a partial segment of a medical time-series may suffice to reveal a patient ’ s identify , which defeats the original purpose of using synthetic data in the first place . Here , we propose an alternative explicit likelihood approach for generating time-series data based on a novel class of normalizing flows which we call Fourier flows . Our proposed flow-based model operates on time-series data in the frequency-domain rather than the time-domain—it converts variable-length time-series with varying sampling rates across different features to a fixed-size spectral representation using the discrete Fourier transform ( DFT ) , and then learns the distribution of the data in the frequency domain by applying a chain of data-dependent spectral filters to frequency-transformed time-series . Using the convolution property of DFT ( Oppenheim ( 1999 ) ) , we show that spectral filtering of a timeseries in the frequency-domain—an operation that mathematically resembles affine transformations used in existing flows ( Dinh et al . ( 2016 ) ) —is equivalent to a convolutional transformation in the timedomain . This enhancement in the richness of distributions learned by our flow comes at no extra computational cost : using Fast Fourier Transform ( FFT ) algorithms , we show that the entire steps of our flow run in O ( T log T ) time , compared to the polynomial complexity of O ( T 2 ) for a direct , time-domain convolutional transformation . We also show that , because the DFT is a linear transform with a Vandermonde transformation matrix , computation of its Jacobian determinant is trivial . The zero-padding and interpolation properties of DFT enables a natural handling of variable-length and inconsistently-sampled time-series . Unlike existing explicit-likelihood models for time-series data , such as deep state-space models ( Krishnan et al. , 2017 ; Alaa & van der Schaar , 2019 ) , our model can be optimized and assessed through the exact likelihood rather than a variational lower bound . 2 PROBLEM SETUP . We consider a general temporal data setup where each instance of a ( discrete ) time-series comprises a sequence of vectors x = [ x0 , . . . , xT−1 ] , xt ∈ X , ∀ 0 ≤ t ≤ T − 1 , covering a period of T time steps . We assume that each dimension in the feature vector xt is sampled with a different rate , i.e. , at each time step t , the observed feature vector is xt = [ xt,1 [ r1 ] , . . . , xt , D [ rD ] ] , where rd ∈ N+ is the sampling period of feature dimension d ∈ { 1 , . . . , D } . That is , for a given sampling period rd , we observe a value of xt , d every rd time steps , and observe a missing value ( denoted as * ) otherwise , i.e. , xt , d [ rd ] , { xt , d , t mod rd = 0 , ∗ , t mod rd 6= 0 . ( 1 ) The data setup described above is primarily motivated by medical time-series modeling problems , wherein a patient ’ s clinical measurements and bio-markers are collected over time at different rates ( Johnson et al . ( 2016 ) ; Jagannatha & Yu ( 2016 ) ) . Despite our focus on medical data , our proposed generative modeling approach applies more generally to other applications , such as speech synthesis ( Prenger et al . ( 2019 ) ) and financial data generation ( Wiese et al . ( 2020 ) ) . Each realization of the time-series x is drawn from a probability distribution x ∼ p ( x ) . In order to capture variable-length time-series ( common in medical problems ) , the length T of each sequence is also assumed to be a random variable—for notational convenience , we absorb the distribution of T into p. One possible way to represent the joint distribution p ( x ) is through the factorization:1 p ( x ) = p ( x0 , . . . , xT−1 , T ) = p ( T ) · T−1∏ t=0 p ( xt |x0 , . . . , xt−1 , T ) . ( 2 ) We assume that the sampling period rd for each feature d is fixed for all realizations of x . The feature space X is assumed to accommodate a mix of continuous and discrete variables on its D dimensions . 1Our proposed method is not restricted to any specific factorization of p ( x ) . Key objective . Using a training data set D = { x ( i ) } ni=1 comprising n time-series , our goal is to ( 1 ) estimate a density function p̂ ( x ) that best approximates p ( x ) , and ( 2 ) sample synthetic realizations of the time-series x from the estimated density p̂ ( x ) . When dealing with data sets with variable lengths for the time-series , we model the distribution p ( T ) independently following the factorization in ( 2 ) . We model p ( T ) using a binomial distribution . Throughout the paper , we focus on developing a flow-based model for the conditional distribution p ( x0 , . . . , xT−1 |T ) . 3 PRELIMINARIES . Let z ∈ RD be a random variable with a known and tractable probability density function p ( z ) , and let g : RD → RD be an invertible and differentiable mapping with an inverse mapping f = g−1 . Let x = g ( z ) be a transformed random variable—the probability density function p ( x ) can be obtained using the change of variable rule as p ( x ) = p ( z ) · | detJ [ g ] |−1 = p ( f ( x ) ) · | detJ [ f ] | , where J [ f ] and J [ g ] are the Jacobian matrices of functions f and g , respectively ( Durrett ( 2019 ) ) . Normalizing flows are compositions of M mappings that transform random draws from a predefined distribution z ∼ p ( z ) to a desired distribution p ( x ) . Formally , a flow comprises a chain of bijective maps g = g ( 1 ) ◦ g ( 2 ) ◦ · · · ◦ g ( M ) with an inverse mapping f = f ( 1 ) ◦ f ( 2 ) ◦ · · · ◦ f ( M ) . Using the change of variables formula described above , and applying the chain rule to the Jacobian of the composition , the log-likelihood of x can be written as ( Rezende & Mohamed ( 2015 ) ) : log p ( x ) = log p ( z ) + M∑ m=1 log |detJ [ fm ] | . ( 3 ) Existing approaches to generative modeling with normalizing flows construct composite mappings g with structural assumptions that render the computation of the Jacobian determinant in ( 3 ) viable . Examples of such structurally-constrained mappings include : Sylvester transformations , with a Jacobian corresponding to a perturbed diagonal matrix ( Rezende & Mohamed ( 2015 ) ) , 1× 1 convolutions for cross channel mixing , which exhibit a block diagonal Jacobian ( Kingma & Dhariwal ( 2018 ) ) , and affine coupling layers that correspond to triangular Jacobian matrices ( Dinh et al . ( 2016 ) ) . 3.1 FOURIER TRANSFORM . The Fourier transform is a mathematical operation that converts a finite-length , regularly-sampled time domain signal x to its frequency domain representationX ( Bracewell & Bracewell ( 1986 ) ) . A T -point discrete Fourier transform ( DFT ) , denoted asX = FT { x } , transforms a ( complex-valued ) time-stamped sequencex , { x0 , . . . , xT−1 } into a length-T sequence of ( complex-valued ) frequency components , X , { X0 , . . . , XT−1 } , through the following operation ( Oppenheim ( 1999 ) ) : Xk = T−1∑ t=0 xt · e−2πj· kt T , ∀1 ≤ k ≤ T − 1 , ( 4 ) where j corresponds to the imaginary unit of a split-complex number . Using Euler ’ s formula , the complex exponential terms in ( 4 ) can be expressed as e−2πj· kt T = cos ( 2π · ktT ) − j · sin ( 2π · ktT ) . Thus , the Fourier transform decomposes any time-series into a linear combination of sinusoidal signals of varying frequencies—the resulting sequence of frequency components , X , corresponds to the coefficients assigned to the different frequencies of the sinusoidal signals constituting the time domain signal x . The DFT is a key computational and conceptual tool in many practical applications involving digital signal processing and communications ( Oppenheim ( 1999 ) ) . 3.2 FOURIER TRANSFORM PROPERTIES . We will rely in developing our model on various key properties of the DFT . These properties describe various operations on the time-domain data and their dual ( equivalent ) operations in the frequency domain . The DFT properties relevant to the development of our model are listed as follows : Convolution : x1 ⊗ x2 ⇐⇒ X1 X2 . Symmetry : If x is real-valued⇒ Xk = X∗k−m , ∀m ∈ N. Even/Odd Transforms : F { Even ( x ) } = Re ( X ) , F { Odd ( x ) } = Im ( X ) , where denotes element-wise multiplication , ⊗ denotes circular convolution , Re ( . ) and Im ( . ) denote the real and imaginary components , Even ( x ) = ( x+ x− ) /2 , Odd ( x ) = ( x− x− ) /2 , where x− signifies the reflection of x with respect to the x = 0 axis , and x = Even ( x ) + Odd ( x ) . Another property that is relevant to our model is the interpolation property , which posits that zero-padding of x in the time domain corresponds to an up-sampled version ofX in the frequency domain .

Summary: The paper introduces a new convolutional flow architecture that uses the DFT to convert the generated time series to the frequency domain. Convolutions are performed by multiplication in the frequency domain through a spectral affine layer that transform the even or odd part of the signal using a data dependent filter. The resulting time-domain convolution has input dependent weights, an interesting and original approach clearly different from other convolutional flow such as [1].