Generative Time-series Modeling with Fourier Flows
Authors: Ahmed Alaa
Alex James Chan
Mihaela van der Schaar
Venue: ICLR 2021 Poster (2021)

Abstract: Generating synthetic time-series data is crucial in various application domains, such as medical prognosis, wherein research is hamstrung by the lack of access to data due to concerns over privacy. Most of the recently proposed methods for generating synthetic time-series rely on implicit likelihood modeling using generative adversarial networks (GANs)—but such models can be difficult to train, and may jeopardize privacy by “memorizing” temporal patterns in training data. In this paper, we propose an explicit likelihood model based on a novel class of normalizing flows that view time-series data in the frequency-domain rather than the time-domain. The proposed flow, dubbed a Fourier flow, uses a discrete Fourier transform (DFT) to convert variable-length time-series with arbitrary sampling periods into fixed-length spectral representations, then applies a (data-dependent) spectral filter to the frequency-transformed time-series. We show that, by virtue of the DFT analytic properties, the Jacobian determinants and inverse mapping for the Fourier flow can be computed efficiently in linearithmic time, without imposing explicit structural constraints as in existing flows such as NICE (Dinh et al. (2014)), RealNVP (Dinh et al. (2016)) and GLOW (Kingma & Dhariwal (2018)). Experiments show that Fourier flows perform competitively compared to state-of-the-art baselines.

Published as a conference paper at ICLR 2021
GENERATIVE TIME-SERIES MODELING WITH FOURIER FLOWS
Ahmed M. Alaa University of California, Los Angeles, USA ahmedmal laa@ucla a. edu
Alex J. Chan
University of Cambridge, UK
e Jo3 400cam. ac. ac.uk .uk
Mihaela van der Schaar University of Cambridge, UK University of California, Los Angeles, USA Cambridge Center for Al in Medicine, UK The Alan Turing Institute, UK mv47 cam. uc.uk
ABSTRACT
Generating synthetic time-series data is crucial in various application domains, such medical prognosis, wherein research is hamstrung by the lack of access to data due to concerns over privacy. Most of the recently proposed methods for generating synthetic time- see-series rely on implicit likelihood modeling using generative adversar- ial networks (GANs)-bhe but such models can be difficult to train, and may jeopardize privacy by "memorizing" temporal patterns in training data. In this paper, pro- pose an explicit likelihood model based on a novel class of normalizing flows that view time-series data in the frequency-domain rather than the fime-domain. The proposed flow, dubbed a Fourier flow, uses a discrete Fourier transform (DFT) to convert variable- length time-series with arbitrary sampling periods into fixed- length spectral representations, then applies (data-dependent) spectral filter filttr to the rrequency-transfoomee time-series. We show that, by virtue the DFT analytic properties, the Jacobian determinants and inverse mapping for the Fourier flow can be computed efficiently in linearithmic time, without imposing explicit structural constraints as in existing flows such as NICE (Dinh al. (2014)), RealNVP (Dinh et al. 6)) and GLOW (Kingma & Dhariwal (2018)). Experiments show that Fourier flows perform competitively compared to state-of-the-an baselines.
INTRODUCTION
Lack of access to data is a key hindrance to the development of machine learning solutions in applica- tion domains where data sharing may lead privacy breaches (Walonoski al. (2018)). Areas where this problem is most conspicuous include medicine, where access to (highly-sensitie stringently regulated by medical institutions; such strict regulations undermine scientific progress by hindering model development and reproducibility. Generative models that produce sensible and realistic synthetic data present viable solution to this problem seafiiiially-genrrate data sets produced by such models can be shared widely without privacy concerns (Buczak et al. (2010)).
clinical data is
In this paper, we focus on the time-series data setup, where observations are collected sequentially over arbitrary periods of time with different observation frequencies across different features. This general data setup pervasive the medical domain-in captures the kind of data maintained electronic health records (Shickel et al. (2017)) collected in intensive care units (Johnson et al. (2016)). While many machine sarring--bsee predictive models that capitalize on such data have been proposed over the past few years (Jagannatha & Yu (2016); Choi et al. (2017); Alaa & van der Schaar (2019)), much less work has been done on generative models that could emulate and synthesize these data sets.
Existing generative models for (medical) time-series based predominantly implicit likelihood modeling using generative adversarial networks (GANs), e.g., Recurrent Conditional GAN (RCGAN) (Esteban et al. (2017)) and TimeGAN (Yoon al. (2019)). These models apply representation learning

---

Published as a conference paper at ICLR 2021
via recurrent neural networks (RNN) combined with adversarial training in order to map noise sequences in a latent space to synthetic sequential data in the output space. Albeit capable of flexibly learning complex representations, GAN-based models can be difficult to train (Srivastava et al. (2017)), especially in the complex time-series data setup. Moreover, because they hinge on implicit likelihood modeling, GAN-based models can be hard to evaluate quantitatively due to the absence of an explicitly computable likelihood function. Finally, GANs are vulnerable to training data memorization (Nagarajan al. (2018)) problem that would be exacerbated in the temporal setting where memorizing only a partial segment of a medical time-series may suffice to reveal a patient's 's identify, which defeats the original purpose of using synthetic data in the first place.
Here, we propose an alternative explicit likelihood approach for generating time-series data based on a novel class of normalizing flows which we call Fourier flows. Our proposed flow-based model operates on time-series data in the frequency-doman rather than the time-domain-it converts variable-length time-series with varying sampling rates across different features to fixed-size spectral representation using the discrete Fourier transform (DFT), and then learns the distribution of the data in the frequency domain by applying a chain of data-dependent spectral filters to Beqqeency-transfore
time-series.
Using the convolution property of DFT (Oppenheim (1999)), we show that spectral filtering of time- series in the reqquency-domain an operation that mathematically resembles affine transformations used in existing flows (Dinh et al. (2016))-i is equivalent to a convolutional transformation in the time- domain. This enhancement in the richness of distributions learned by our flow comes at no extra computational cost: using Fast Fourier Transform (FFT) algorithms, we show that the entire steps of our flow run in ( log T) time, compared to the polynomial complexity of O(?²) for a direct, time-domain convolutional transformation. We also show that, because the DFT is linear transform with a Vandermonde transformation matrix, computation of its Jacobian determinant is trivial. The zero- sero-padding and interpolation properties of DFT enables natural handling of variable-length and inconsistently-San sampled time-series. Unlike existing explicit- likelioo models for time-series data, such as deep state-space models (Krishnan et al., 2017; Alaa & van der Schaar, 2019), our model can be optimized and assessed through the exact likelihood rather than variational lower bound.
2 PROBLEM SETUP
We consider general temporal data setup where each instance of (discrete) time-series comprises sequence of vectors x = [ -1l æ, E X, 0 S t < T 1, covering a period of time steps. We assume that each dimension in the feature vector œ; is sampled with different rate, i.e., at each time step I, the observed feature vector is 2t = xt, |........ [t.][r]]], where ra E N,, is the sampling period of feature dimension d E (1...., D). That is, for given sampling period ra we observe a value of 21,d every Fd time steps, and observe a missing value (denoted as +) otherwise, i.c.
Fr.d (ra] Fi,d: mod ra 0, *; mod ra + O.
The data setup described above is primarily motivated by medical time-series modeling problems, wherein a patient's 's clinical measurements and bio-markers are collected over time different rates (Johnson et al. (2016); Jagannatha & Yu (2016)). Despite our focus on medical data, our proposed generative modeling approach applies more generally to other applications, such as speech synthesis (Prenger et al. (2019)) and financial data generation (Wiese et al. (2020)).
Each realization of the time-series drawn from a probability distribution capture variable-length time-series (common in medical problems), the length T of each sequence is also assumed to be random variable- for notational convenience, we absorb the distribution of T into p. One possible way represent the joint distribution p(x) is through the factorization."
p(z). In order to
T-1
p(z) p( 3T-1, = p(T) II p(x, ao...., (.....,.._1,T))
We assume that the sampling period r'a for each feature d is fixed for all realizations of σ. The feature space X' is assumed to accommodate a mix of continuous and discrete variables on its D dimensions.
Our proposed method not restricted any specific factorization of p(x).
2

---

Published as a conference paper at ICLR 2021
Key objective. Using a training data set D fa comprising time- goal is to (I) estimate density function p(x) that best approximates p(z), and (2) sample synthetic realizations of the time-series æ from the estimated density p(x)) When dealing with data sets with variable lengths for the time-series, we model the distribution p(T) independently following the factorization in (2).
We model p(T) using binomial distribution. Throughout the paper, we focus developing flow-based model for the conditional distribution p(Zo,.... (2))
3 PRELIMINARIES
Let RR be random variable with known and tractable probability density function p(=), and let g: DRD + RDD be an invertible and differentiable mapping with an inverse mapping f = g-_ Let (() be a transformed random variable the probability density function p(æ) can be obtained using the change of variable rule as p(æ) p(z) det J [g]]]= -pf]---==(())))) p(f(=)) det J [f]], where J [ʃ]
and J[g] are the Jacobian matrices of functions f and g. respectively (Durrett (2019)).
Normalizing flows are compositions of M mappings that transform random draws from predefined distribution z ~ p(z) to a desired distribution p(z). Formally, a flow comprises a chain of bijective maps g = g() o g o ... o g(M) with an inverse mapping f = f()) o f(2) o ... o f(Al) Using the change of variables formula described above, and applying the chain rule to the Jacobian of the composition, the log-likelihood of x can be written as (Rezende & Mohamed (2015)):
M
log p(x) x) = log p()) + log det J[mm]--
Σ
(3)
m=1
Existing approaches to generative modeling with normalizing flows construct composite mappings g with structural assumptions that render the computation of the Jacobian determinant in (3) viable. Ex- amples of such structurally-constaall corresponding to perturbed diagonal matrix (Rezende & Mohamed (2015)), 1 x 1 convolutions for cross channel mixing, which exhibit a block diagonal Jacobian (Kingma & Dhariwal 2018)), and affine coupling layers that correspond to triangular Jacobian matrices (Dinh et al. (2016)).
mappings include: Sylvester transformations, with a Jacobian
3.1 FOURIER TRANSFORM
The Fourier transform is a mathematical operation that converts a finite-length, regularly-sampled time domain signal æ to its frequency domain representation X (Bracewell & Bracewell (1986)). A T-point discrete Fourier transform (DFT), denoted as X = Fr )z), transforms a (complex-valune valuedd time-stamped sequence x (So....,Ty) 1T- into length-T sequence of (complex-valued) frequency components, X è
XT- through the following operation (Oppenheim (1999)):
x Xx Σ
25j 7, 255
(4)
1=0
where j corresponds to the imaginary unit of split-complex number. Using Fuler's formula, the com- plex exponential terms in (4) can be expressed as e -2n + = cos (2ir #) - -in (2ir
kt 7, Thus,
the Fourier transform decomposes any time-series into a linear combination of sinusoidal signals of varying frequencies tthe resulting sequence of frequency components, X, corresponds to the coef- ficients assigned to the different frequencies of the sinusoidal signals constituting the time domain signal æ. The DFT is a key computational and conceptual tool in many practical applications involving digital signal processing and communications (Oppenheim (1999)).
3.2 FOURIER TRANSFORM PROPERTIES
We will rely in developing our model on various key properties of the DFT. These properties describe various operations on the time-domain data and their dual (equivalent) operations in the frequency domain. The DFT properties relevant to the development of our model are listed as follows:
Convolution: 2; @ 22 + X1 O X²
Symmetry: If is real-valued + Xx Xi-m Vm E N.
Even/Odd Transforms: F(Even(x)) Re x), F(Odd(z)) Im(
3

---

Published as a conference paper at ICLR 2021
where @ denotes element-wist multiplication, @ denotes circular convolution, Re(.) and [m).) denote the real and imaginary components, Even(z) = (x + x= x-)/2, Odd(z) = (x x x-)/2, where x- signifies the reflection of æ with respect to the = 0 axis, and æ Even(x) (x) + Odd(x). Another property that is relevant to our model is the interpolation property, which posits that zero-padding of in the time domain corresponds to up-sampled version of X in the frequency domain.
Table 1: The two main layers in an N-point Fourier flow, their inverses, and the corresponding log-determinant- Jacobian. The input to the flow, æ. is a D x T matrix comprising a set of D time-series each of length T, and the output, Y, is 2 D x N/2 tensor comprising the filtered spectral representation of æ. Here, we show the application of the Fourier transform layer to given feature dimension d' the same operation is applied independently to all feature dimensions. The vector X -- is the reversed conjugate of X and H [ When cascading multiple Fourier flows, we alternate between feeding either of the real and imaginary channels of X, lm(X (X) and Re( Re(X),
) to the BiRNN network in the different flows within the cascade.
Layer
Function
Inverse function
log |detJ]]
Fourier transform
ad x² U ON-T,
X² x
Fd +- 0, V VL /mod ra & O,
x" F N
)
log det W| 0.
X FN(
Ltd + & t mod ra ra=00 #0,
X [Xa,ds.... Xo.1N/2)
x² _ 1-
Spectral filtering
(log H, H,H) 14) = BiRNN (x)).
Y1, Y2 = split(Y), (log H BiRNN((Y)) (Y2),
Yi H Re(X) M,
Y2 = Im X): )
Re( = (Y, - -M) H,
Y concat (Y1, Y2).
X Re(X X) Im(X).
4 FOURIER FLOWS
We propose a new flow for time-series data, the Fourier Flow, which hinges on the frequency domain view of time-series data (explained in Section 3.,,, and capitalizes on the Fourier transform properties discussed in Section 3.2. In nutshell, Fourier Flow comprises two steps: (a) frequency transform layer (Section 4.1), followed by (b) a data-dependent spectral filtering layer (Section 4.2). The steps involved in Fourier Flow are summarized in Table 1 and Figure 1.
4.1 FOURIER TRANSFORM LAYER
In the first step of the proposed flow, we transform the time-series æ =[xo. xr-1] into its spectral representation via Fourier transform-we do so by applying the DFT operation (described in Section 3.1) to each feature dimension independently. Let xx = zo, [ra], ....l..................... ................................ arraa be the time-series associated with feature dimension d: the Fourier transform layer computes the N -oint DFT of xd for all d E (I....,)) D} through the following three steps:
Temporal zero padding:
ad d U ON-T, Fraalra O, 1I, mod ra + 0,
Fourier Transform:
X = FN(
Spectral cropping: X
(5)
Here, ON-T denotes a of N TT zeros, and the union denotes the padding operation that appends the zeros 0y- -r to the time-series xd. The temporal zero-padding step capitalizes on the frequency interpolation and sampling properties of DFT (Section 3.2) to ensure that the padded time-series a and its frequency spectrum X have fixed (predetermined) length of N, irrespective of the interval length 7' and the sampling period r.a Because the DFT coefficients are complex-valued X is tensor with dimensions 2 x 1 X N, and the collection of Fourier transforms for the D feature dimensions, X. is a tensor with dimensions 2 x D N. That is, the DFT layer converts each time-series x into a two-channel, "image-like like" D x N matrices Re(X) and Im shown in Figure 1. A flow with an N N-point DFT will be referred to as an N N-point Fourier flow in the rest of the paper. To guarantee a lossless recovery of the time-series via inverse DFT, we ensure that N > T.
4

---

Published as a conference paper at ICLR 2021
Finally, the spectral cropping step in (5) discards the N/2++1 1 to N7th frequency components in both Re(X) and [m) This is because a is real-valued, hence Re(X) and Im(X (X) are symmetric and anti-symmetric, respectively (See Section 3.2), which renders the discarded frequency components redundant. The final output of this layer, X, is a 2 x x N/2 tensor.
DFT computation. Since the DFT operation in (4) is linear, we can represent the frequeney transform step in (5) through a linear transformation and apply DFT via matrix multiplication as follows:
w
w
N-1
X Wa
W
w
2(N
and w = e 2-2jj/N (6)
VN
w (2 w( W 1)
When N is set to be power of 2, the DFT operation in (6) can be implemented using any variant of the Fast Fourier Transform (FFT) methods (Nusshaumer (1981); Van Loan (1992)), such the Cooley- Tukey FFT algorithm (Cooley & Tukey (1965)), with computational complexity of O(N log N).
Determinant of the DFT Jacobian. The DFT is a natural and intuitive transformation for temporal data, but how does introducing the DFT mapping affect the complexity of the Jacobian determinant of the flow? To answer this question, we note that the DFT matrix W in (6) is a (square) Vandermonde matrix, thus we can evaluate the DFT Jacobian determinant in closed-form as follows:
(et(J[
(a) | det(
(6)
II IenemgN
(w - wn)
1,
(7)
where (a) follows from the fact that the Jacobian of a linear transformation is equal to the transforma- tion matrix W, and (b) follows from the scalar multiplication property of determinants, which posits that det (aVVVV)) = a. (VN for scalar a and matrix V, combined with the formula for the determinants of Vandermonde matrices (Björck & Pereyra (1970)). (c) follows from the fact that the magnitude of the complex exponential is w 1, and the polynomial product in (7) comprises a total of N2 terms, each of which is of the form """"" = 1. The result in (7) shows that the DFT mapping does not incur extra computational costs when evaluating the flow likelihood in (3).
4.2 SPECTRAL FILTERING LAYER
The second layer of the Fourier flow is an affine coupling layer, similar to that originally introduced in (Dinh et al. (2014; 2016)), but applied to the time-series æ in the frequency domain as follows:
(log H, M) = RNN(Tm X)), Y H ④ Re(X) M, Y2 Tm (X): Y concat (Y, Y2),
where H and μ are D x N matrices, BiRNN denotes a bi-directional recurrent neural network (Schuster & Paliwal (1997)), and denotes the Hadamard (element-wise) product. Here, we split the real and imaginary channels in X rather than splitting the feature dimensions as in (Dinh et al. (2016)). The detailed steps of the flow and its inversion given in Table 1.
The affine transformation in the frequency domain can be thought of as a spectral filtering operation whereby the frequency transform of the even part of the time-series, Re(X), is applied to filter with transfer function H (Recall the even-odd decomposition properties of DFT in Section 3.2). The transfer function itself is data-dependent: depends on Im(X), or equivalently, the frequency trans- form of the odd part of the time-serie series The mapping from Im(X) to the transfer function H is implemented through an RNN that shares parameters across all different frequency components, since N can grow very large for meress with large T. We use bi-directional RNN since all frequency components are available at our disposal at the same time.
4.3 FOURIER FLOWSS IN THE TIME-DOMAIN
How does the Fourier flow mapping look like in the time domain? Using the convolution property of the DFT (Section 3.2), the time-domain view of the Fourier flow can be expressed as follows
y1 h()) & Even(x) + n, y/2 Odd( (),, y concat (y1, y2),
(8)
5

---

Published as a conference paper at ICLR 2021
(a) Fourier Transform Layer
(b) Spectral Filtering Layer
Re( X)
Het X
H
Y1
Y2
algorithm
Im( X)
DFT
FNf-1
Feature timension
Spectral fiter
Frequency index
BIRNN
Feature dimension
X x Tensor)
Imf X)
Y2 D N/2 Temor)
Time index
Figure 1: Pictorial depiction of the two layers underlying Fourier flow. Here, consider exemplary in- which we apply a 10-point Fourier flow (N 10). all
stance of ser-ss with displays. the r-axis corresponds to either the time or frequency indexes, the y-axis corresponds to the feature dimension, and the red dots correspond to the value associated with a given time (frequency) index in given feature dimension. (Darker shades of red correspond higher values.) (a) DFT layer. In this example, assume the sampling periods be 71 and na AA # 1. In the first step of the flow, values the under-sampled feature dimension to 0, and padd N T = 4 zeros to all features. (Padded zeros are depicted as white dotted circles), which results in padded time-series æ in the form of D x N matrix. Next, the DFT operation is applied to each row in resulting in a 2 D x N tensor X comprising the real and imaginary components of the frequency transform. (b) Spectral filtering layer. The imaginary component of the (cropped) spectrum is passed to bi-directional RNN, which views the input as an N/2 N/2-length sequence of D-dimensional features. The RNN outputs a N/2 x D sequence that corresponds to the spectral filter H, which then multiplied by Re(X). The final output concatenates the filtered spectrum, Y1, with Y2 = Im(X).
and 7'
all missing
where y, = F i E (1. (M). and = spectral filter H. That is, the Fourier flow in the time domain corresponds to a circular convolution between the even part
(H) (H) the impulse response of the
and the inverse DFT of the filter H. which depends on the odd part of
The time-domain view of a Fourier flow elucidates its multifaceted modeling advantages. First, there a representational advantage of having a more expressive convolutional transformation in (8) com- pared to the element-wise affine transformations in the coupling layers of existing methods (Dinh et al. (2014; 2016); Kingma & Dhariwal (2018)). Moreover, a Fourier Flow does not require splitting feature dimensions disjointly, but rather decomposes each feature to its even and odd components. Second, this richer representation comes with no extra computational cost: while time-domain convo- lution would run in (N'2 D2) time (Hunt (1971)), affine operation in the frequency domain, and the DFT runs in O(DN log N) time. Computation of the Jacobian determinant is achieved by simply adding all elements of H (Table 1) is the case in time-domain affine coupling methods. Finally, the DFT operation enables handling of variable-length time-series and inconsistent sampling periods across features without any extra modeling efforts.
spectral filter applies an O(N) element- dese
Related work. Existing explicit-likelihodd models for sequential data are based predominantly on state-space modeling approaches, which assume that hidden state dynamics control the sequence of ob- served data. Examples of such models include hidden Markov model (Beal et al. (2002)), deep Markov models (Krishnan al. (2017)), and attentive state-space models (Alaa van der Schaar (2019)). The key advantage of our model compared to these is that ours is optimized and assessed using the exact likelihood rather than variational lower bound.
Existing models based on normalizing flows have been primarily focused on static (non-temporal) data (Ziegler & Rush (2019))examppls include NICE (Dinh et al., 2014), ReaINVP (Dinh et al., 2016), and GLOW (Kingma & Dhariwal, 2018; Prenger et al., 2019). These flows assume highly-structured Jacobian determinants, e.g., diagonal, block diagonal and trian-
transformation with easy-to-comppe gular Jacobian matrices. Fourier flows assume structured transformation in the frequency domain, which corresponds to a richer and more complex representation than existing flows in time domain. In particular, Fourier flow can be thought of as the frequency- dual of the affine transformation

---

Published as a conference paper at ICLR 2021
(a) Distributions of the sinusoidal frequencies recovered by the different baselines.
Reall
TimeGAN True
True
True
Hitogram
Histgram
Histogram
3200
A350
1250
031
056
CMa
AB
Frequency
Frequency
Frequency
(b) Performance comparison between three baselines.
Res TimeGAN
Bea/VVP
FF
1200
Number of Epochs
Frequency
Time step
Figure 2: Performance of FF, RealNVP and TimeGAN in learning to generate random sinusoidal time-series with random frequencies. (a) Here, we estimate dominant frequency of the sinusoidal signals generated by each baseline and plot the histogram for estimated frequencies across that of the ground-truth distribution of the true frequency f- We plot kernel density estimate of the histogram in each plot. Note that the ranges of the x- and y-axes in each plot change according to the estimated distributions. (b) Assessment of the time-series data generated by the different baselines in terms of exact data likelihood, spectral properties and predictive utility.
of Re, which translates to a more complex convolutional mapping in the time domain. The closest work to ours is the invertible convolutional flows developed independently in (Karami et al. (2019)). This work does not deal with time-series data, but introduces a convolutional transformation on the feature dimension using Toeplitz matrices, and implements the transformation using DFT.
EXPERIMENTS
5.1 ILLUSTRATIVE RESULTS ON SYNTHETIC DATA
To examine the utility of our explicit-likelibe frequency-domain model, start off by comparing Fourier flows with their time-domain analogues, RealNVP flows (Dinh et al. (2016)), and a state-of- the-art implicit-likeliiee likelihood method, TimeGAN (Yoon et al. (2019)), in stylized experimental setup. In particular, we consider the following time-series data generation process:
x = sin(ft + Φ), & NO, 1), f Beta( = 2, 3 = 2), t E (O....,T-11) 1},
where Beta denotes Beta distribution with shape parameters a and B, and T 100. That is, our time-series data is set of sinusoidal sequences with random frequency and phase, where the phase is drawn from a normal distribution and the frequency is drawn from a Beta distribution. We generate total of 1,000 time-series based on (9) and use these train all baselines. We compare Fourier flow (FF) model with a composition of 10 flows and a (single-layer) BiRNN with 200 hidden units with an equivalent RealNVP model with the same number of flows and an Bi RNN for generating the coefficients of the affine transformation. For the TimeGAN model, we use the spper-parameter configuration recommended in (Yoon et al. (2019)). We train all models with 1,000 iterations and batch size of 128 we then generate 1,000 synthetic time-series from each trained model.
In Figure 2(a), we plot the histogram for the frequencies of the time-series generated by each model and compare it with the ground-truth Beta distribution of f. We estimate the frequency of each gener- ated time-series (which are generally not perfectly sinusoidal) by finding the dominant frequency com- ponent f in the DFT of each generated time-series. As we can see, the distribution of recovered by FF bears the strongest resemblance the true Beta distribution. Moreover, likelihood approach in TimeGAN concentrates the probability mass on few frequencies, indicating that it memorized some of the sinusoidals in training data, which is problematic in applications where
note that the implicit
7

---

Published as a conference paper at ICLR 2021
would want to keep the training samples private-thh problem does not arise in FF where the learned distribution is smooth and does not concentrate around any specifie frequencies. On the other hand, RealNVP learned distribution with much wider support than that of the original data.
The richness of the crlquency-domainn FF transformation compared to the (one-domain) ReaINVP is evident in the lower negative log- lngllikeiihodd achieved by our model (Figure 2(b), left). In Figure 2(b) (middle), we compare all baselines with respect to the power spectral densiry (PSD) of the 1,000 time- series sampled from each model; the PSD is a spectral representation of stochastic process given by PSD = F[ x x= }}. Figure 2(b) plots the cumulative mean absolute error (MAE) between the PSD of the original data and that of the samples generated by each model within all frequency bands [0, fl. Vf E [0, 1]. FF provides competitive accuracy in the recovered PSD compared to TimeGAN (MAE: 2.83 for FP, 3.44 for TimeGAN), and significantly outperforms RealNVP (MAE: 5.1 This implies that the distribution of frequencies generated by FF accurately resembles the original data.
Finally, we evaluated the accuracy of the sampled data in the time-domain by as- sessing their predictive usefulness fol- lows: we trained a vanilla RNN model using each of the three synthetically gen- erated samples to sequentially predict the next value in each time-series in the original data. Figure 2(c) (right) shows the MAE of the three RNN models across all time step as we can see, the RNN trained on FF-generated data con- sistently outperforms the baselines.
Table 2: Performance scores for all baselines.
(F-score, MAE)
Method
Stocks
Energy
Lung cancer
TimeGAN
(0.938, 0.173)
(0.479, 0.056)
(0.732, 3.248)
WGAN
(0,989, 0.010)
(0.985, 0.049)
(0.517, 4.824)
ARIMA
(0.648, 0.016)
(0.667. 0.037)
(0.659, 4.299)
WaveNet
(0.441, 0.123)
(0.013, 0.213)
RealNVP
(0,979, 0.020)
(0.965, 0.053)
(0,866, 5,762)
FF
(0.984, 0.009)
(0.949, 0.039)
(0.904, 3.207)
5.2 PERFORMANCE EVALUATION ON REAL DATA
We replicate the experimental setup (Yoon et al. (2019)) to test the performance of our model on mul- tiple time-series data sets with respect to its predictive utility, i.e., the extent to which models trained on the synthetic samples can perform well on prediction tasks in the original sample. We measure perfor- mance with respect to this criterion via a predictive score that corresponds to the accuracy of an RNN trained on the synthetic data to predict next step in real data. In addition, we measure the quality of the synthetic data generated by each baseline using the precision and recall metrics (Sajjadi et al. (2018) ()) averaged over all time steps, and then we combine both scores into a single F-score. We conduct experiments on: Google stocks data, UCI Energy data set, and longitudinal follow-up clinical data set for lung cancer patients. To highlight the richness of the learned FF distributions, we also visualize the t-SNE plots for the FF samples compared to their time-domain analogues generated by RealNVP.
We compare Fourier Flows with the following baselines: TimeGAN (Yoon al. (2019)), Wasserstein GAN (WGAN) trained on time steps as distinct feature dimensions, a Au- Dovinn-Average (ARIMA) model, WaveNet (Oord et al. (2016)) and a naive RealNVP bench- mark with time steps treated as feature dimensions. Per- formance results are provided in Table 2-the results indi- cate that FF consistently generates high-quality synthetic time-series that are useful for predictive model training (as quantified by the predictive score). FF outperforms all meth- ods with respect the predictive score on Stocks and Lung cancer data sets, and its learned distribution more closely resemble the real distribution compared to the time-domain method (Figure 3 shows the tSNE plots on Google's 's stock data). It also worth mentioning that FF performs competi- tively with respect to the domain- and task-agnostic precision and recall with respect to all data sets, which means that the FF- -e-geeerrteed synthetic samples could reliably replace origi- nal ones in a wide variety of modeling tasks.
(a) t-SNE plot for Fourier flows.
toregressive Integrated
X-+ SNE
(b) t-SNE plot for RealNVP.
VE-SEE
SNE
Figure 3: t-SNE plots for Stock.
8

---

Published as a conference paper at ICLR 2021
6 DISCUSSION
In this paper, we introduced an explicit likelihood model based on novel class of normalizing flows that models the distributions stochastic time-series data in the frequency-doma rather than the time-domain. Capitalizing on the Fourier transform properties, the proposed flow naturally handles arbitrary sample rates and duration of time-series data, and enables learning richer representation compared with the existing methods at extra computational cost.
Fourier flows enjoy three modeling advantages that enable them to accurately model time-series distributions. First, the DFT layer compresses temporal information into low-dimensional spectral representation, enabling a more efficient distribution learning. Consider for instance the sinusoidal example in Section 5. In this example the data is drawn from stochastic process sin(St 6) where f is a random frequency. To model length-T time-series drawn from such process using conventional methods, would need to model a 7-dimensional random variable. However, using the DFT transform, we would be modeling the spectral representation X = ] [[8(v+ f) 8(v - f)] where å is the Dirac-delta function. Thus, the spectral representation can be fully described with one piece of information, which is the location of the frequency component f. i.e. a I-dimensional random variable. Second, Fourier flows enable construction of complex transformations at extra cost for Jacobean computation. That is, DFT layer followed by a simple transform (such as an affine transform) would amount for complex overall transform without any extra complications associated with the Jacobean transformation. Finally, the usage of an RNN model in the novel spectral filtering layer captures the sequential nature of the spectral data.
Another key advantage for Fourier flows is that they are explicit likelihood models with a tractable exact likelihood. This means that they can be trained and evaluated using the exact model likelihood instead of a variational lower bound as in most existing approaches. Moreover, explicit likelihood models are hypothesized be superior implicit likelihood models, such GANs, in terms of privacy preservation, as they are less likely to memorize training points. While the theoretical generalization properties of GANs have been previously investigated Nagarajan et (2018), less work has been done in studying the theoretical generalization performance of flows. Proving that normalizing flows do not memorize data is beyond the scope of this paper as would amount for broader and more general result that is not limited to the time series setup. Such theoretical analysis is an interesting subject for future work.
While Fourier transforms enjoy various computational advantages, they may fall short when modeling non-stationary time-series, where we know that the distribution each new observation is not just function of the process history, but also the time index. Thus. investigating generalizations of our model with more general spectral transforms, such as generalized Fourier transforms or discrete Wavelet transforms are promising directions for future work.
ACKNOWLEDGMENTS
We would like to thank the reviewers for their valuable feedback. The research presented in this paper was supported by the US Office Naval Research (ONR), and by the National Science Foundation (NSF) under grant numbers 1407712, 1462245, 1524417, 1533983, and 1722516.
REFERENCES
Ahmed M Alaa and Mihaela van der Schaar. Attentive state-space modeling of disease progression.
In Advances in Neural Information Processing Systems, pp. 11338-1133 1338-1348,, 2019.
Matthew J Beal, Zoubin Ghahramani, and Carl E Rasmussen. The infinite hidden markov model. In Advances in neural information processing systems, pp. 577-584, 2002.
Rianne van den Berg, Leonard Hasenclever, Jakub M Tomezak, and Max Welling. Sylvester normalizing flows for variational inference. arXiv preprint arXiv:1803.05644 2018.
ke Björck and Victor Pereyra. Solution of vandermonde systems of equations. Mathematics of computation, 24(1 9233 1970.

---

Published as a conference paper at ICLR 2021
Ronald Newbold Bracewell and Ronald N Bracewell. The Fourier transform and its applications, volume 31999. McGraw-Hin New York, 1986.
Anna L Buczak, Steven Babin, and Linda Moniz. Data-driven approach for creating synthetic electronic medical records. BMC medical informatics and decision making, 10(1):59, 2010.
Edward Choi, Andy Schuetz, Walter F Stewart, and Jimeng Sun. Using recurrent neural network models for early detection heart failure onset. Journal of the American Medical Informatics Association, 24(2):361-377 2017.
James W Cooley and John W Tukey. An algorithm for the machine calculation of complex fourier series. Mathematics of computation, 297300) 1965.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.
Laurent Dinh, Jascha Dickstic,,,,,,, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605.08800 2016.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems. pp. 7511-7522 2019.
Rick Durrett. Probability: theory and examples, volume 49. Cambridge university press, 2019.
Cristóbal Esteban, Stephanie I. Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans. arXiv preprint arXiv: 1706:02033 2017.
Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint 130.v:130.008.0 2013.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. arXiv preprint 000.0050 2019.
Emiel Hoogeboom, Rianne van den Berg, and Max Welling, Emerging convolutions for generative 2019.
normalizing flows. arXiv preprint
Chin-Wer Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. International Conference Machine Learning (ICMI), 2018.
Hunt. A matrix theory proof the discrete convolution theorem. IEEE Transactions on Audio and Electroaconstics, 19(4):285-288 1971.
Abhyuday N Jagannatha and Hong Yu. Bidirectional rnn for medical event detection in electronic health records. In Proceedings of the conference. Association for Computational Linguistics. North American Chapter. Meeting, volume 2016, pp. 473. NIH Public Access, 2016.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3:160035, 2016.
Mahdi Karami, Dale Schuurmans, Jascha Gohl-Dicksteem, Laurent Dinh, and Daniel Duckworth. Invertible convolutional flow. In Advances in Neural Information Processing Systems, pp. 5635- 5645, 2019.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions, In Advances in neural information processing systems, pp. 10215-10224, 2018.
Ivan Kobyzev, Simon Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Rahul G Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear state space models. AAAI, 2017.
10

---

Published as a conference paper at ICLR 2021
Alex M Lamb, Anirudh Goyal Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. In Advances in neural information processing systems, pp. 4601-4609, 2016.
Vaishnavh Nagarajan, Colin Raffel, and Ian Goodfellow. Theoretical insights into memorization in gans. In Neural Information Processing Systems Workshop, 2018.
Henri J Nussbaumer. The fast fourier transform. In Fast Fourier Transform and Convolution Algorithms, pp. 80-111. Springer, 1981.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499) 2016.
Alan V Oppenheim. Discrete-time signal processing. Pearson Education India, 1999.
George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation. Advances in Neural Information Processing Systems, pp. 2338-2347, 2017.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 3617-3621. IEEE, 2019.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. International Conference Machine Learning (ICML), 2015.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 5234-5243, 2018.
Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(1 (((11))2673-6683 1997.
Benjamin Shickel, Patrick James Tighe, Azra Bihorac, and Parisa Rashidi. Deep ehr: survey of recent advances in deep learning techniques for electromic health record (chr) analysis. IEEE journal of biomedical and health informatics, 22(5):159-1164 2017.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Advances in Neural Information Processing Systems, pp. 3308-3318, 2017.
Charles Van Loan. Computational frameworks frameewrr for the fast Fourier transform. SIAM, 1992.
Jason Walonoski, Mark Kramer, Joseph Nichols, Andre Quina, Chris Moesel, Dylan Hall, Carlton Duffett, Kudakwashe Dube, Thomas Gallagher, and Scott McLachlan. Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record. Journal of the American Medical Informatics Association, 25(3):230-253 2018.
Magnus Wiese, Robert Knobloch, Ralf Korn, and Peter Kretschmer. Quant gans: Deep generation of financial time series. Quantitative Finance, pp. 1-22, 2020.
Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar. Time-series generative adversarial networks. In Advances in Neural Information Processing Systems, pp. 5508-5518 2019.
Jinsung Yoon, Lydia N Drumright, and Mihaela Van Der Schaar. Anonymization through data synthesis using generative ad adversarial networks (ads-gan). IEEE Journal of Biomedical and Health Informatics, 2020.
Zachary M Ziegler and Alexander Rush. Latent normalizing flows for discrete sequences. arXiv preprint ar Xw1::191111054400 2019.
11


Reviews:
[{"review_id": "k_LoXt_XItu", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a flow generative model for time-series in the Fourier domain. The time-series data are first converted to the Fourier domain. Instead of the affine coupling layer previously presented in literature, the authors have designed a frequency domain version of the same.\n\nPros:\n1. The paper is well-written and is largely easy to follow. \n2. Operating in the Fourier domain as well the affine coupling layer proposed in the paper are interesting novel contributions in the context of designing flow models for time-series.\n3. Based on the experiments presented, Fourier Flow is surprisingly effective when compared to RealNVP, and also slightly outperforms other methods like TimeGAN. \n\nCons:\n1. The authors should try to explain what it is about operating in the Fourier domain causes the improvements in prediction error. As the Fourier transform is invertible, I would imagine that the novel affine coupling layer has a larger role to play in this. This is also what the authors seem to suggest in Section 4.3. It would be great if the authors could add a baseline experiment operating in the time domain with the proposed affine layer (Equation (8)). \n2. In terms of the explanations in the paper, a lot of space has been devoted to explain the Fourier transform, its properties and how to compute it efficiently. While it is done well, I feel these are very well known facts. For the determinant of the Fourier matrix, we can just use the fact that it is unitary and we immediately have |det(W)| = 1. Instead, additional experiments analysing the experiments could be more useful, such as understanding the role of network depth etc. for this particular method, or additional t-SNE plots for the remaining datasets. \n3. As in all papers such as this one, it is not easy to see how the baseline approaches have been trained and how well the hyperparameters have been tuned. \n\nBased on the author response, I am willing to increase my rating for this paper. \n\nUPDATE AFTER AUTHOR RESPONSE:\n\nThe authors have addressed all my queries and made the changes that I requested. I am increasing my rating reflecting this. The paper has novelty, good experiments and improved performance. I would like to see the paper accepted. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "elrATjkRWSx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper introduces a new convolutional flow architecture that uses the DFT to convert the generated time series to the frequency domain. Convolutions are performed by multiplication in the frequency domain through a spectral affine layer that transform the even or odd part of the signal using a data dependent filter. The resulting time-domain convolution has input dependent weights, an interesting and original approach clearly different from other convolutional flow such as [1]. \n\nRelevance:\nTime series generative modeling has a wide range of crucial applications in fields ranging from medicine to finance. The new method shows very promising performance and it has the potential to become a state-of-the-art method for time series generation.\n\nOriginality:\nThe use of the DFT and spectral affine layers is original in the context of normalizing flows. Importantly, the DFT is very suitable for flows since it is an isometry and has therefore a trivial Jacobin. The use of input dependent convolutions is very interesting even in the context of regular ConvNet architectures. \n\nScientific quality:\nThe method is clearly presented and well motivated. The experiment section cover a decently large set of experiments and performance are compared with a very large number of state-of-the-art baselines.\n\nPros:\n- Original new architecture well motivated for time series applications\n- Rigorous experiments with multiple relevant baselines\n- Very promising experimental results\n\nCons:\n- It would have been useful to see a wider range of real world applications\n\nMinor points:\n- I would like to have a figure with the time domain sampled generated by the FF and the other baselines. It is strange to see a generative modeling paper without a figure showing the generated samples.\n\nReferences:\n[1] Karami, Mahdi, et al. \"Invertible convolutional flow.\" Advances in Neural Information Processing Systems. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "aSyqt4z7ms9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n+ This paper proposed a novel generative model named Fourier flow for modeling time series data. The model incorporates Fourier transformation into normalizing flows and considers the time series on the frequency domain, rather than the time domain. Such a combination looks interesting and novel. I think this paper will have some impact on the field of normalizing flows and time series analysis.\n+ The writing of this paper looks good, which makes it easy to follow.\n\nCons:\n- The empirical study is somewhat weak. The experimental results are not very impressive in the paper. The improvement of the performance seems to be marginal. Only one real-world dataset is considered. Please give a short description of the dataset. As the proposed method is for the general time series, the authors are suggested to evaluate their method on more datasets, especially from various fields. Besides, please give a definition of the metric predictive score. \n- The paper mentions that the computational cost is no larger than some SOTAmethods. It would be better to give a discussion on the complexities of your method and the SOTAs. It would also be more convincing if the authors can show the running time in the experiment.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PRQMT_SktCF", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThe paper presents Fourier Flows (FF), which is a time series generative model in the frequency domain. It shows that the Jacobian of the DFT is equal to 1, which means that DFT does not add too much overhead. The results on the real-world datasets are encouraging and expected because the predictive results mainly rely on the overall trend of the time series. By analysis in the frequency domain, we usually can capture the main trend accurately. The main concerns for the paper are the computational overhead on the proposed algorithm on non-periodic, long, and variable-length time series.\n\n\n\n### Feedback\n\n* The strongest point of the paper is when the authors show that the Jacobian of the DFT matrix is 1. Thus, taking DFT does not make the generative model much more complex.\n* The other advantage of FF can be in handling missing values. Unfortunately, the authors do not expand on this aspect.\n* The main issue with the FF is its computational complexity. The authors write: \"To guarantee a lossless recovery of the time-series x via inverse DFT, we ensure that $N\\geq T$.\" This means FF is impractical for long time series and quite inefficient in handling variable-length time series.\n* Moreover, Fourier transforms are mainly intended for periodic signals and do not provide concise representations of non-stationary signals. This inefficiency is why Wavelets and DWT have been invented. The synthetic data generation model gives an unfair advantage to FF because it uses periodic signals.\n* The authors' description for Eq. (8) is misleading. $\\mathbf{H}$ and $\\mathbf{h}$ are not fixed parameters, they are functions of the input. Thus, the model is a form of self-attention. I hope that the authors have done the training correctly. \n*The authors criticize GANs for memorization but never show that FFs do not memorize data points.\n* The choice of generating H using a BiRNN on only the $Im(X)$ is puzzling. Why didn't the authors use complex-valued RNNs to operate on both real and imaginary parts and not lose the phase information?\n* On page 2, the authors discuss a parametric model for T, but they never elaborate further on it.\n* \"where $x_-$ signifies the reversed version of $x$\": the authors should clarify that this is the reflection with respect to $x=0$ axis. Otherwise, we should add a time-shift operation too.\n\n----\n### Post-Response Update\nI don't think the authors have answered my second set of questions. While there are some doubts remaining in the paper, the idea looks fine. Although, I think a new paper with DWT will outperform this approach soon. I do not change my vote at this time.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Nice ideas with practical advantages.  
Decision: Accept (Poster)