GT-GAN: General Purpose Time Series Synthesis with Generative Adversarial Networks
Authors: Jinsung Jeon
JEONGHAK KIM
Haryong Song
Seunghyeon Cho
Noseong Park
Venue: NeurIPS 2022 Accept (2022)

Abstract: Time series synthesis is an important research topic in the field of deep learning, which can be used for data augmentation. Time series data types can be broadly classified into regular or irregular. However, there are no existing generative models that show good performance for both types without any model changes. Therefore, we present a general purpose model capable of synthesizing regular and irregular time series data. To our knowledge, we are the first designing a general purpose time series synthesis model, which is one of the most challenging settings for time series synthesis. To this end, we design a generative adversarial network-based method, where many related techniques are carefully integrated into a single framework, ranging from neural ordinary/controlled differential equations to continuous time-flow processes. Our method outperforms all existing methods.

GT-GAN: General Purpose Time Series Synthesis with Generative Adversarial Networks
Jinsung Jeon Yonsei University jjsjjs0902@ynnei..c.
Jeonghak Kim
Haryong Song Linger Studio Corp.
Kakao Corp.
haggie, pro@kakaccorp.... com
darong@lingercorrg
Seunghyeon Cho Yonsei University
Noseong Park
Yonsei University noseong@yonsei. ac.
seunglyyenchoo@yossii..ac ac.
Abstract
Time series synthesis is an important research topic in the field of deep learning, which can be used for data augmentation. Time series data types can broadly classified into regular or irregular. However, there are no existing generative models that show good performance for both types without any model changes. Therefore, we present general purpose model capable of synthesizing regular and irregular time series data. To our knowledge, we are the first designing a general purpose time series synthesis model, which is one of the most challenging settings for time series synthesis. To this end, we design generative adversarial network-basee method, where many related techniques are carefully integrated into a single framework, ranging from neural ordinary/controlled differential equations to continuous time-flow processes. Our method outperforms all existing methods.
Introduction
Time series data occurs frequently in real- world applications Reinsel 2003 Fu 2011 Li et al. 2018
Yu et al. 2018 Wu et a 2019 Guo et al. 2019 Bai et al. (2099 [ong et al. 2020 Huang et al. 2020a Ren al.
2021 Tekin al. 2021]. Among many tasks related time series, synthesizing time series data is one of the most important tasks because real-world time series data is frequently imbalanced and/or insufficient. Since regular and irregular time series data have different characteristics, however, different model designs had been adopted for them. Therefore, existing time series synthesis work focuses on either regular or irregular time series synthesis (Yoon et al. 2019 Alaa et al. 202T]. To knowledge, there
existing methods that work well for both types.
Regular time series means regularly sampled observations without any missing ones, and inregular times series means that some observations are missing from time to time. Irregular time series is much harder to process than regular time series. For instance, better after transforming time series data into its frequency domain, i.e., the Fourier transform, and some time series generative models use this approach [Alaa al. [02T]. However, it not easy to observe pre-determined frequencies from highly irregular time series Kidger et al. 2019). However, continuous-time models [Chen et al. 2018 Kidger et al. 2020 Brouwer et al., 2019] show good performance in processing both regular and irregular time series. By resorting them, we propose general purpose model that can synthesize both time series types without any model changes.
known that neural networks perform
To achieve the goal, we design sophisticated model which utilizes diverse set of technologies, ranging from generative adversarial networks (GANs [Goodfellow et al. 2014)), and autoencoders (AEs) to neural ordinary differential equations (NODEs [Chen et al. 2018 neural controlled differ-
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

GAN
Log-density
Synthesis
{(t;, )) L. (uke))
{(4,, xf
Encoder
Decoder
{tt;, (uue)
Wiener process
real
[tti, 2:))
Generator
Disc.
fake
Figure 1: The overall design of our method, GT-GAN, Each color means separate workflow. The autoencoder and the GAN share the workload synthesize regular and irregular time series.
ential equations (NCDEs (Kidger et al. 2020)), ), and continuous time-flow processes (CTFPs [Deng et al. 2020), which reflects the difficulty of the problem.
Fig. shows the overall design of our proposed method. One of the key points in our model design is that we combine the adversarial training of GANs and the exact maximun likelihood training of CTFPs into single framework. However, the exact maximum likelihood training is applicable only to invertible mapping functions whose input and output sizes are the same. Therefore, we design an invertible generator, and adopt an autoencoder, on whose hidden space our GAN performs the adversarial training.
In other words, i) the hidden vector size of the encoder is the same as the noisy vector of the generator, ii) the generator produces a set of fake hid- den vectors, iii) the decoder converts the set into a fake continuous path (cf. Fig. Fig. and iv) the discriminator provides feedback after reading the sampled fake sample. We note that in the third step. a fake continuous path is created by the decoder. There- fore, we can sample any arbitrary reg- ular/irregular time series sample from the fake path, which shows the flexibility in our method.
((t, x fake
Di
{(4, hy fake 1
Decoder (GRU-ODE)
{(tz, X]
fake
Figure 2: An example of sampling two irregular time se- ries from fake continuous path represented by an ordinary differential equation (ODE). In other words, we solve ODE problems to sample regular/irregular time series.
We conduct experiments with 4 datasets and 7 baselines. Since our method is able to support both the regular and irregular time series synthesis, we test for both of them. Our method outperforms other baselines in both environments. Our contributions can be summarized as follows:
1. We design model based on various state-of-the-ee deep learning technologies. Our method is able to process any types of time series data, ranging from regular irregular, without any model changes.
2. Our experimental results and visualization prove the efficacy of the proposed model.
3. Since our task is one of the most challenging tasks for time series synthesis, the proposed model architecture carefully designed. Our ablation studies show that our proposed model does not work well if any part is missing.
2 Related work and preliminaries
GANs are one of the most representative generative technology. Ever since the first introduction in its seminal research paper, GANs have been adopted to main different domains. Recently, researchers focused on their synthesis for time series data. Therefore, there have been proposed several GANs for time series synthesis. C-RNN-GAN [Mogren| 2016] has regular GAN framework that can be applied to sequential data by using LSTM in its generator and discriminator. Recurrent Conditional GAN (RCGAN |Esteban et al. 2017) took similar approach except that its generator and discriminator take conditional input for better synthesis. WaveNet [van den Oord et af.) (006)) also generates time series data from the conditional probability of previous data by using the dilated casual convolution.

---

GAN
-Log-densily
Syntheeie
Enonder
[(%, hy b
Deooder
{(4) 2 DE
((t, x Mis
Neural CDE
GRU ODE
{(4, foale
)
Wiener process
Wi
W. (10, u)
Invaribia Generator
{(t; 2 1i.
CTFP-1
real
((t by Ma Pr([(t, b. (44, 1
Dise
GRU-ODE
[(t,, x))
CTFP
Figure 3: The detailed architecture of our proposed method. Neural CDEs (or NCDEs) a recent breakthrough for processing time series. GRU-ODEs are a continuous interpretation of gated recurrent units (GRUs) based on NODEs. CTFPs are a flow-based concept convert an input time series process into target process. CTFPs are not a GAN- based concept but we integrate them into our framework, considering the challenging nature of the general purpose time series synthesis.
WaveGAN [Donahue al. 2019 has similar approach with DCGAN [Radford et al., 2016], where its generator is based on WaveNet. We can modify the teacher-forcing (T-Foreing [Graves] 2014]) and professor-forcing (P-Foreing [Lamb al. 2016)) models generate time series data from noise vectors, although they are not GAN models, by using the forecasting characteristic of those models.
TimeGAN Yoon et al. 2019 is yet another model for time series synthesis. This model aims mainly at synthesizing fake regular time series samples. They proposed a framework where the adversarial training of GANs and the supervised training of predicting X;+1 from X;, where and X,41 mean two multivariate time series values at time i; and f,41, respectively.
3 Proposed method
In this section, we describe our design. Since our general purpose time series synthesis is a challenging task, the proposed design is much more complicated than other baselines.
3.1 Overall workflow
We first describe the overall workflow in our model design, which consists of several different data paths (and several different training methods based on the data paths) as follows:
1. Autoencoder path: Given time series sample {(4,, ) the encoder produces a set of hidden vectors (hreal) Ji=u The decoder recovers a continuous path real which enhances the flexibility of our proposed method. From the path Xreal we sam- ple {(4 ))r >) We train the encoder and the decoder using the standard autoencoder (AE) loss to match x.eal and greal for all i.
2. Adversarial path: Given a set of noisy vectors {z, NN our generator produces a set of fake hidden vectors [b/ake J.N The decoder recovers a fake continuous path fake from [h.ukenN Ti=0- We sample {(t,, xfake) ) from Å and feed it into the discriminator. For irregular time series synthesis, decoder, and the discriminator using the standard adversarial loss.
sample lú in [0, 7). We train the generator, the
3. Log-density path: Given a set of hidden vectors {hreal) for an time series sample {(li, (41,xfea)) xxeaa) W the inverse path of the generator reproduces a set of noisy vectors {2.].N We feed No into its forward path again to reproduce [h.eal) Ji=o- where hreat h. for all i. During the forward pass, we calculate the negative log probability of - log p (h. for all with the change of variable theorem and minimize it for training, being inspired by Grover et al. 2018 and Deng al. 2020).
In particular, we note that the dimensionality of the hidden space in the autoencoder is the same as that of the latent input space of the generator, i.e., dim(h) dim(z), This is needed for the exact likelihood training in the generator the change of variable theorem requires that the input and
3

---

output sizes are the same to estimate the exact likelihood. In addition to it, we let the autoencoder and the generator share the workload synthesize fake time series by combining them into single framework, i.e., the generator synthesizes fake hidden vectors and the decoder reproduces hemanereadable fake time series from them.
3.2 Autoencoder
Encoder General NCDEs, which are considered as continuous analogue to recurrent neural networks (RNNs), are defined as follows:
beil1 f(b(t):8))dx((t
h((++1) = h((t)) b
(I)
=h(t,) + L.me f(b(t):0))
(t dt dt,
where X (t) is continuous path created from a raw discrete time series sample {(t,, ((tx))foa))) by an interpolation algorithm points the interpolation algorithm fills out values. Note that NCDEs keep reading the time-derivative of (1), denoted X (1)
note that X ((t)) (t,, xçeal) for all and for other non-observed time-
def dx(() In our case, collect (hreat)N j== follows:
de
( hrea f
f(h(t);0)) (X dt dt,
where hgeat = FCdim(x) (xfeal) and FCinput size- toze--output_sizz is fully-connected layer with specific input and output sizes. We refer to Appendix for the ODE function f definition.
Therefore, the input time series ((t,, > represented by a set of hidden vectors {(t, ) Because NCDEs are continuous analogue to RNNs, it shows the best fit to processing irregular time series [Kidger al. 2020].
Decoder Our decoder, which reproduces a time series from its hidden representations, is based on GRU-ODES (Brouwer et al. 2019 and is defined as follows:
a(4+1) = d(4) + / g(d()),
d(tx+1) CRU(h:+1 a(++1))),
(4)
(le+1, &(41) (le+1, ddm()( (d(4++1))
where d(lo) = FCd (ho) and h, means either the i-th real or fake hidden vector, i.e., Ireal or h/ recall that in Fig. the decoder is involved in both the autoencoder and the synthesis processes. x means a reproduced copy of GRU-ODEs uses the technology called neural ordinary differential equations (NODEs) to continuously interpret GRUs and refer to Appendix B for the ODE function definition.
In particular, the gated recurrent unit (GRU) at Eq. 4 is called as jump which is known to be effective processing time series with NODEs BBrowwer et al. 2019 Jia and Benson 2019]. We train the
encoder-decoder using the standard reconstruction loss between x and x.eat for all i in all training time series samples.
3.3 Generative adversarial network
Generator Whereas generators typically read noisy vector to generate fake sample in standard GANs, our generator reads a continuous path (or time series) sampled from a Wiener process to generate fake time series sample this generation concept is known as continuous time flow processes (CTFPs [Degg et al.) 2020). Appendix. [8.3shows an example of our generation process. The input
generation process is random path sampled from Wiener process, which is represented by a time series of latent vectors {44, 2,) ) in the path, and the output is a path of hidden vectors which is also represented by a time series of hidden vectors {(t,, h J

---

We use the mean squared reconstruction loss, i.c., the mean of | real encoder-decoder architecture. Then, the standard GAN loss train the generator and the discriminator. In our preliminary experiments, we found that the original GAN loss is suitable for our the Therefore, our generator can be written as follows:
h fake wi(1) w. + S. (r.rf(((()))) (())()),,(()):::))
(6)
where w; Z₁ a4( t.. Here, means a virtual time variable of the integral problem, and t; real physical time contained in time series sample {(t,, xx ) We note that this design corresponds to NODE model augmented with (t). We refer to Appendix B for the ODE function r definition.
Owing to the invertible nature of NODES, we can calculate the exact log-density of hreal, i.e., probability that hrea generated by the generator, using the change variable theorem and the Hutchinson's ' stochastic trace estimator as follows [Grathwohl et al. 2019 Deng et al. 2020
w(0) = heeal + J.
log Pr Pr((heal log Pr(w
I tr (T)) dr,
(8)
@w(T)
where w(0) means 2p00l in Fig. 3. hgoal means reproduced copy of he by our generator. Eq. corresponds to "CTFP 1n and Eqs. (6) and 8 to "CTFP" in Fig. We note that in Eq. 77 the
integral time reversed to solve the reverse-mode integral problem.
Therefore, minimize the negative log-density, denoted llog Pr(hheel), for each and our generator is trained by the two different training paradigms: i) the adversarial training against the discriminator, and ii) the maximum likelihood estimator (MLE) training with the log-density.
Discriminator
We design our discriminator based on the GRU-ODE technology as follows:
c(t(41) = c(ti) + 1441
q(c((),
(9)
c(t(+1) ( c(tr+1)),
(10)
where c(4o) FCaim (dmmii)))) (xo), and x; means the i-th time series value, i.e., xreal or x
fake
The ODE function q has the same architecture as but with its own parameters 0g. After that, we calculate the real or fake classification y ((FCdim(()==((((x))))) where is a softmax activation.
The role of each part of our proposed model is in Appendix M
3.4 Training method
- for all i, to train the
task. Instead of other variations, such WGAN-GP Gulrajani et al. 2017 therefore, standard GAN loss. We train our model in the following sequence:
1. We pre-train the encoder-decoder networks the reconstruction loss for kAE iterations.
2. After the above pre-training step, start jointly train all networks in the following sequence for KJOINT iterations: i) training the encoder-decoder networks with the re- construction loss, ii) training the discriminator-geeeraai training the decoder to improve the discriminator's classification output with the discrimina- tor loss and iv) the generator with the MLE loss every PMIE iteration. We found that too frequent MLE training incurs mode-collapse so we use every PMLLE iteration.
networks with the GAN loss, iii)
In particular, the 2-ii step to train the decoder to help the discriminator out is one additional point where the autoencoder and the GAN are integrated into a single framework. In other words, the generator should deceive both the decoder and the discriminator. Our training algorithm refer to Appendix
5

---

The well poll-esssd of NCDEs and GRU-ODEs was already proved in Lyons et al. 20007 Theorem 1.3] and Brouwer al. 2019] under the mild condition of the Lipschitz continuity. We show that our NCDE layers are also weol-p problems. Almost all activations, such as ReL.U, Leaky ReLUU, SoftPlus, Tanh, Sigmoid, ArcTTan, and Softsign, have a Lipschitz constant of Other common neural network layers, such as dropout, batch normalization and other pooling methods, have explicit Lipschitz constant values. Therefore, the Lipschitz continuity of ODE/CDE functions can be fulfilled in our case. In other words, it is well-posed training problem. As a result, our training algorithm solves a well-posed problem so its training process is stable in practice.
4 Experimental evaluations
Our software and hardware environments are as follows: UBUNTU 18.04 LTS, PYTHON 3.8.10, PYTORCH 188.1 TENSORFLOW 2.5.0, CUDA 11.2, and NVIDIA Driver 417.22, i9 CPU, and NVIDIA RTX 3090. The mean and variance of 10 runs are reported for model evaluation.
4.1 Experimental environments
Datasets We conduct experiments with 2 simulated and 2 real-world datasets. Sines has 5 features where each feature is created with different frequencies and phases independently. For each feature, i E {1,...,5), 5), (t) = sin f.t + 0;), where ~ [0 1] and 6; ~ 4[- T.. MuJoCo is multivariate physics simulation time series data with 14 features. Stocks is the Google stock price data from 2004 to 2019. Each observation represents one day and has 6 features. Energy is a UCI appliance energy prediction dataset with 28 values. To create the challenging irregular environments, 30, 50, 70% of observations from each time series sample {(t,,,;=]) xx==)) )) is randomly dropped in other words, N decreases to 0.7.N, O.5.N, 0.3.V. Dropping random values has been mainly used to create irregular time series env environments in the literature [Kidger et al.] [019 Xu and Xie 2020
Huang et al. 2020b Tang et al. 2020 Zhang et al., 2021 Uhin et al. 2021 Deng et al. 2021 Therefore, we conduct experiments with both the regular and the irregular environments.
Baselines We consider the following baselines for the regular time series experiments: TimeGAN, RCGAN, C-RNN-GAN, WaveGAN, WaveNet, T-Forcing, and P-Foreing. For the irregular experi- ments, we exclude WaveGAN and WaveNet, which cannot handle irregular time series, and redesign other baselines by replacing their GRU with GRU-Lt and GRU-Decay (GRU-D) [Che et al. 2018). GRU-At and GRU-D effective models for processing irregular time series data. GRU-Lt addi tionally uses the time difference between observations as input. GRU-D is a modification of GRU-Át to learnt exponential decays between observations. TimeGAN-AAt RCGAN-AL, C-RNN-GAN-AL. T-Forcing-/At, and P-Forcing-AAt (resp. TimeGAN-D, RCGAN-D, C-RNN-GAN-D, T-Forcing-DD, and (eoocingg-eccy)) are modified with GRU-At (resp. GRU-D) and can handle irregular data.
Our ablation studies also involve many advanced methods, based on NODEs, VAEs, flow models, and so forth. We intentionally leave these advanced methods for our ablation studies since our proposed method internally has them as sub-parts.
Evaluation metrics For quantitative evaluation of synthesized data, is evaluated with the discrim- inative score and the predictive score used in TimeGAN [Yoon al. 2019]. The discriminative score measures the similarity between the original data and the synthesized data. After learning a model that classifies the original data and the synthesized data using neural network, it is tested whether the original data and the synthesized data are classified well. The discriminative score is |Accuracy-O55 and if the score is low, classification difficult, so the original data and the synthesized data are decided to be similar. The predictive score measures the effectiveness of the synthesized data using (TSTR) method. After training model that predicts the next step using the synthesized data, the mean absolute error (MAE) is calculated between the predicted values and the ground-trute values in test data. If the MAE is small, the model trained using the synthesized data is decoded to be similar to the original data. For qualitative evaluation, the synthetic data is visualized with the original data. There are two methods for visualization. One to project original
the wwwthhssi--nnd-tss--eee-e
A well- posed problem means i) its solution uniquely exists, and ii) its solution continuously changes as input data changes.

---

Table 2: Irregular time series (30% dropped)
Method
Sines
Stocks
Energy
MuloCo
Table 1: Regular time series
GT-GAN
.363.1063
.251 .99
.33.11000 1,063 1.951.035
TimeGAN-AAt
4944.0012
4635
448:5.027
A16
Method
Sines
Stocks
Energy
MuJoCo
Disciiniative Soore
RCGAN-A
499 000 436 064 .500+:000 000
500+ 000
Discrinniabe Seore
GT-AAN
0124..044
07711.031
2.08
245
022
C-RNN-GAN At
55001.000
.500::000
-500-1.000
5001-000
TimeGAN
00111.008
022 2361.012 4099 028
T-Fording At
.3951.063
.305 1...00002220000 .47.1.011
.348.1044
ROGAN
02244.08
1964.027
3364.017
436=0012
P-Forcing-AA
.3444.127 :341::035
-50-1.000
4914-010
C-RNN-GAN
.399 1.28
499.1.001
.112:.095
.9611.008
.411
47911.000
463.1.025
T.Forcing
49541.001
2226:.033
48311.004
499=-000
ROGAN-D
.5001.000 500-1.000
500,1.000
500-1.000
P-Forcing
.30-4.227
2574:026 412::.06
5000=000
C-RNN-GAN-D
500+2000 500 000
500++.000
500+ 000
WaveNet
158:.011
.232::008
.39.2.010 .385
I-Forcing-D
.08:2.07 4099
.3474.046
494:
WaveGAN
277.4.013 221:4.022
P-Forcing-
.00.1.00
0.000.0000..0. .9114.020
GT-GAN 0974.000
040 000 312:1.002 055=_000
GT-GAN
0992.004
021
0666
001
.0482.001
TimeGAN
.9331.019 038+..001 001 .2734.004 062=0006
TimoGAN-AA
.145 087 001
3355..011
1188
Prodictive Score
ROGAN
097:1.001 040:1.001 292.1.005
-081=0003
RCGAN- At
144.1.028
18141.144
35111.056
.4331.021
Prodictive Score
CANN.GAN
11271.004 008.1.08.00 1.000 48811.005
055510044
C-RNN GAN
.7541.000
0911 000 -00-1.000
447.1.000
T-Forcing
.150-1.022 03%
.355+.005
142=:014
T-Forcing- At
.1162.002
U70:..013
2511
005-..001
P. P.Foniing
.116,1.004 043.1.001
.3031.006
10.
P.Forcing At
.102.1.02 0.3.1.018
1.00
18911011
WaveNet
.1174.008 042:1.001
331
333 004
TimeGANAD
19921.082
105:1.053
248.0.04
0931
WaveGAN
134-2.013 041
307
324=0006
RCGAN-D
.3882. 113 .523+.000
409
020
361
Original
094:2.001 036:4.001
250
001
C-RNN-GAN-D
.6444.001
.345tt0022
440-1.000
457
T-Forcing D
.1001.002 027.1.000 002
00000.0000000
10011.00
P.Forcing-D
.144 079:1 14744.001
.17:::002
Original
071.00 OTT
045
041 002
Original
Original
10
GT GAN
TimeGAN
10
10
-10
Original
-10
Original
GA
TimeGAN
-10
-10
-10
0
10
-10
10
-10
0
10
-10
10
Dendty Extimate
Original GT-GAN
Data Derctty Estimate
A
Original TimeGANN
Data Dereity Estimate
12
Original GAN
Data Estimate
12
Original TimeGAN
16
tV
>
N >.
0.0
0.5
1.0
0.0
1.0
0.2
0.4
0.6
0.2
0.4
0.6
Data Value
Data Value
Data Value
Data Value
(a) Stocks
(b) Energy
Figure 4: Visualizations and distributions of the regular time series synthesized by GT-GAN and TimeGAN
and synthetic data in a two dimensional space using t-SNE (an der Maaten and Hinton 2008 The other one the kernel density estimation to draw data distributions.
4.2 Experimental results
Regular time series synthesis In Table [ list the results of the regular time series synthesis. GT-GAN shows better performance on most cases than TimeGAN, the previous state-of- ofate-oftther -he- model. As shown in the 1st row in Fig. 4 GT-GAN covers original data areas better than TimeGAN. In addition, the 2nd row in Fig. 4 is the distributions of the fake data generated by GT-GAN and TimeGAN. The synthesized data's s distributions from GT-GAN are more similar to those of the original data than TimeGAN, which shows the efficacy of the explicit likelihood training of GT-GAN against the implicit likelihood training of TimeGAN.
Irregular time series synthesis In Tables and we list the results of the irregular time series synthesis. GT-GAN shows better discriminative and predictive scores than other baselines in cases. In Table where we drop random 30% observations from each time series sample, GT-GAN shows the best outcomes, outperforming TimeGAN by large margins. Baselines modified with GRU-AM and those with GRU-Decay show comparable results and is hard to say one better than the other in this table.
7

---

Table 3: Irregular time series (50% dropped)
Table 4: Irregular time series (70% dropped)
Method
Sines
Stocks
Energy
MuloCo
Method
Sines
Stocks
Energy
MuloCo
GT-GAN
.372.1.128
.655 1.033 .37 1110 .270..0016
GT-GAN
.278 .278 230 1. .033 .25 .047 .25 1. .223
TimeGAN-ZSt
496
019
479
025
.8234.022
TImeGAN-AAt
.0000000 488 0099 4966 008 4949
Discriminatve Score
ROGAN-AY
.06
165
479
000=-000 .500=20000
Dischimintiin Scare
RCGAN-A
433=.142
381 086 .500+.000
550++.000
C-RNN-GAN- At
500+1.000
500:0000
000
-500:1.000
C-RNN-GAN-AS
10000.00::0000
000 .500-.000 5500-10000
T-Forcingg At
.0811.137
308
100.1.010..00 .48.1
.48.1.011
.01
.486.1.005
T-Fording At
.374_,087
.3651.027 .468.1.00
1.281.022
P-Forcing-
4284.004
328.2
498
005
.491:.012
P-Forcing-AA
288=
.317::019 550++.000
498.4.003
TimeGAN D
.5001.000
477
443 0015
1500
D
498_ 006 .4851.0022 100100000
..99
RCGAN-D
.500+.000
5500+:000
.500= 000
.500:1000
RCGANDD
.500=0000
.5001-0000 .500
55001.000
C-RNN-GAN-D
500++000
5000=0000
550==0000
000
.500==000
C-RNN-GAN-D
500=0000
500 000 550++.000 550++.000
T-Forcing-D
.3002.101
.407
.376=
044
.498:1.001
I-Forcing-D
436=:067
.444
333:2.032 .493+.005
P-Forcing-D D
.99:1.000
550::0000..0. .500: 000
.500:1.000
P-Forcing-
550 000
449:..10
.4941.011
.49911000
GT-GAN
.1012.010 .018
0644.001
001 .0561.003
GT-GAN
088
.20 005
.76:1.001
.051
TimeGAN-24t
.23+4.00
058
008
4.021
TimoGAN-A1t
734=:000
072 000
.906
.4424.000
Prefictive Saure
RCGAN-A1
144::.005
094
.391 2:14
.277::061
Predietiv Soore
RCGAN- At
318
.98:1.000
222.1.041
C-RNN At
.7411.026
08911.001 .500..000
000
.448.1.001
C-RNN GAN At
.751
.04
084...00...
1.00000200000000 5000
.48.1.001
T-Forcing- At
.3994.029
075
=251=0000
.069-1002
T-Forcing- At
.113=000
077-+.022
.251:.000 .05
Forcing At
.12011.05
067.1.014... 2263 .003
.189.1. 1.26
P-Forcing At
.13.1.00
.550000000000 .002
1..6
117.1.034
TimeGAN-D
109:1.074 244:
.3392003
022
.375:.01T
TimetAN-D
752=2001
228 0010 44511.000
.72
RCGAN-D
5194.046
333
250
010
.314:t0023
RCGAN-D
404=
.41 2.45 045 .349
027 .20-+.056
C-RNN-GAN-D
.754:.000
-73==000 438
.479:1.000
C-RNN-GAN-D
6322
.336-.000
.479:10001
F-Forcing D
.0041.001
038
1. 003
090
000
.113:1.001
T-Foiiing D
.02
001 0031
1.002.002 .0911.99100000
11141.003
P-Forsing-D
1904.0022
089
198 005
.207:1.08
P.Forcing D
278
107:1.009 196-4.006 191:1.005
Original
071
011
045
001
0002
Original
071 004
0002
005
002
Original 1-arcingg
Original T-
0.0
Torcing
Forcing
Original GT-GAN
-5.0
Original
-7.5
-10
GT GAN
-10.0
-10
10
-5
0
10
0
5
30
Dota Densty Extimate
Original GT-GAN
Deta Density Estimate
0
Original Forcing at
Dalua Dersity Extimate
12
Original GT-GAN
Data Sensiity Extimate
12
Original T-Furcing-di
4
Frorcing-D
9-
Forcing D
6
0.0
0.5
1.0
0.0
0.5 Data Value
1.0
0.0
0.3
0.6
0.9
1.2
1.5
0.0
0.3
0.6
0.9
1.2
1.5
Cata Value
Data Value
Data Value
(a) Stocks
(b) Energy
Figure 5: Visualizations and distributions of the irregular time series (70% dropped) by GT-GAN,
T-Forcing-A1 At and T-Forcing-D
In Table (50% dropped), many baselines do not show reasonable synthesis quality, e.g. TimeGAN- D. TimeGAN-At, RCGAN-D, C-RNN-GAN-D, and C-RNN-GAN-AL have a discriminative score 0.5. Surprisingly, T-Forcing-D, T-Forcing-At, P-Forcing-D, DD and P-Forcing-At work well in this case. However, our model clearly shows the best performance in all datasets. Baselines modified with GRU-At show slightly better than them modified with GRU-Decay in this case.
(70% dropped) shows the results of the most chal- lenging experiments in our paper. All baselines do not work well because of the high dropping rate. T-Forcing- D. T-Forcing--At P-Forcing-D, and P-Forcing-Ast, which showed reason- able performance with dropping rate no larger than 50%, do not work well in this case. This shows that they are vulnerable to highly irregular time series data. Other GAN-based baselines are vulnerable as well. Our method greatly outperforms all existing methods, e.g., discriminative score of 0.278 by GT-GAN vs. 0.436 by T-Forcing-D vs. 0.288 by P-Forcing- At for Sines, and a predictive score of 0.051 by GT-GAN vs. 0. 1.114 by T-Forcing-D vs. 0.053 by T-Forcing- At for MuJoCo. Fig 5 visually compare our method and the best performing baseline for
Finally, Table 4
Table 5; Ablation study for training options. Refer to Ap- pendix E for other ablation studies with irregular time series.
Method (Regular)
Sines
Stocks
Energy
MuJoCo
GT-GAN
.012
.077
.221
.245
w/o Eq. g
.033
.159
.356
.278
w/o pre- pro-trasning
.446
.175
.322
.200
GT-GAN
097
040
312
.055
w/o Eq. B
.077
.043
315
.057
w/o pre-traiming
.966
.038
.299
.052
8

---

Table 6: Ablation study for model architecture in MuJoCo.
Fnergyy
GT-GAN (w.o AB)
GT-GAN (Flow only)
GT-GAN only)
GT-GAN (Full model)
Metric
Disc.
Pred.
Disc.
Pred.
Disc.
Pred.
Disc.
Pred.
30% dropped
.67
.455
.249
50% dropped
.500
064
457
111
.45
.622
.270
.056
70% dropped
.500
066
A55
107
.96
.146
.275
.051
the dropping rate of 70% - figures for other dropping rates and data are in Appendix land similar patterns are observed in them.
Ablation sensitivity analyses GT-GAN is characterized by the MLE training with the negative log-density in Eq. (8), and the pre-training step of the encoder and decoder. J shows the results of various GT-GAN modifications with some training mechanism removed. The model using the negative log-density training shows better performance than the model not using it. That is, the MLE training makes the synthetic data more like the real data. When the pre-trained autoencoder is not used, the predictive score is better than GT-GAN. However, the discriminative score is the worst.
In T we alter the architecture for our model. We modify our proposed GT-GAN model by re- moving its sub-parts to create simpler ablation models: i) In the first ablation model, we remove the autoencoder and perform the adversarial train- ing only with our generator and discriminator, denoted "GT-GAN (w.o. AE)". In other words, our generation directly outputs raw observations (instead of hidden vectors), which will be fed into our GRU-ODE-based discriminator. ii) The second ablation model, denoted "GT-GAN (Flow only)". has only our CTFP-based generator and we train it with the maximum likelihood training - we note that this construction is the same training flow-based models. This model is equivalent to the original CTFP model [Deng et al. 2020). iii) The third ablation model has only the autoencoder, denoted "GT-GAN (AE only)". However, we convert it to variational autoencoder (VAE) model. In the full GT-GAN model, the encoder produces a set of hidden vectors {(4,, h ) In this ablation model, however, this is changed to (1:, (V: real })) i= where, where. N(hreal, 1) means the unit Gaussian centered at hreal, The decoder is the same as its full model. We use the variational training for this model. Among the ablation models, GT-GAN (Flow only) outperforms the discriminator score in most cases. However, our full model is clearly the best in all cases. Our study shows that the ablation models of GT-GAN do not perform well as its full model if any parts missing, as shown in Fig 6 Refer to Appendix for the ablation studies with other datasets.
Original
DAN
(low only
02.0
Cata Valle
Figure 6: Visualization and distribution of Mu- JoCo (70% dropped) by GT-GAN and Flow only
N
The hyperparameters that significantly affect model performance are the absolute tolerance (atol), the relative tolerance (rtol), and the period of the MLE training (PMLE) for the generator. The atol and rtol determine the error control performed by the ODE solvers in CTFPs. We test with various options of the hyperparameters in Appendix F We found that there is an appropriate error tolerance (atol, rtol) depending on the data input size. For example, the datasets with small input sizes (i.e., Sines, Stocks) have good discriminator scores with (le-2, 1e-3), and the datasets with large input sizes (i.e., Energy, MuJoCo) show good results with (le-3, le-2).
5 Conclusions
Time series synthesis is an important research topic in deep learning and had been separately studied for regular or irregular time series synthesis. However, there are still no existing generative models that can handle both regular and irregular time series without model changes. Our proposed method, GT-GANN is based on various advanced deep learning technologies, ranging from GANs to NODEs, and NCDEs, and is able to process all passible types of time series without any changes in its model architecture and parameters. Our experiments, which incorporate various synthetic and real-world datasets, prove the efficacy of the proposed method. In our ablation studies, only our full method without any missing parts shows reasonable synthesis capabilities. The limitations and societal impacts of our proposed model are in Appendix O

---

Acknowledgments and Disclosure of Funding
Noseong Park is the corresponding author. This work was partly supported by the Yonsei University Research Fund of 2022 (10%), the Institute of Information & Communications Technology Planning Evaluation (ITTP) grant funded by the Korean government (MSIT) (No. 2020-0-01361, Artificial Intelligence Graduate School Program Yonsei University, 10%, and No. 2022-0-001 13, Developing a Sustainable Collaborative Multi-modal Lifelong Learning Framework, 70%), and the LG Display research fund (C2022000673, 10%)
References
Ahmed Alaa, Alex James Chan, and Mihaela van der Schaar. Generative time-series modeling with fourier flows. ICLR, 2021.
Lei Bai, Lina Yao, Salil S. Kanhere, Xianzhi Wang, and Quan Z. Sheng. Stg2seq: Spatial-temporal graph sequence model for multi-step passenger demand forecasting. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 1981- - 1987, 7 2019. doi: 10.24963/jcai.201199 2019/274.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically time series. In NeurlPS, 2019.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances neural information processing systems, 31, 2018.
Ruizhi Deng, Bo Chang, Marcus A. Brubaker, Greg Mori, and Andreas M. Lehrmann. Modeling continuous stochastic processes with dynamic normalizing flows. In NeurIPS, 2020.
Ruizhi Deng, Mareus A. Brubaker, Greg Mori, and Andreas M. Lehrmann. Continuous latent process flows. In NeurIPS, 2021.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis, 2019.
Cristóbal Esteban, L.
Stephanie Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans, 2017.
Tak-chung Fu. A review on time series data mining. Engineering Applicction of Artificial Intelligence, 1811 2011.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. In ICLR. 2019.
Alex Graves. Generating sequences with recurrent neural networks, 2014.
Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-gan: Combining maximum likelihood and adversarial learning in generative models. In AAAI, 2018.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans, arXiv preprint arXiv: 1704-00000, 2017.
Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. Attention based spatial- temporal graph convolutional networks for traffic flow forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):922-929, Jul. 2019. doi: /3300/ 3301922.
Rongzhou Huang, Chuyin Huang, Yubao Liu, Genan Dai, and Weiyang Kong. Lsgen: Long short- term traffic prediction with graph convolutional networks. In IJCAI. pages 2355-2361, 2020a.
10

---

Zijie Huang, Yizhou Sun, and Wei Wang. Learning continuous system dynamics from irregularly- sampled partial observations. In NeurIPS, 2020b.
Sheo Yon Jhin, Heejoo Shin, Seoyoung Hong, Minju Jo. Solhee Park, Noseong Park, Seungbeom Lee. Hwiyoung Maeng, and Seungmin Jeon. Attentive neural controlled differential equations for time-series classification and forecasting. In ICDM, 2021.
Junteng Jia and Austin R Benson. Neural jump stochastic differential equations. In NeurlPS, 2019.
Patrick Kidger, Patric Bonnier, Imanol Perez Arribas, Cristopher Salvi, and Terry J. Lyons. Deep signature transforms. In NeurlPS, 2019.
Patrick Kidger, James Morrill, James Foster, and Terry J. Lyons. Neural controlled differential equations for irregular time series. In NeurlPS, 2020.
Alex Lamb, Anirudh Goyal, Ying Zhang, Saizheng Zhang, Aaron Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks, 2016.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network: Data-driven traffic forecasting, In International Conference on Learning Representations (ICLR '18), 2018.
Terry J Lyons, Michael Caruana, and Thierry Lévy. Differential equations driven by rough paths. Springer, 2007.
Olof Mogren. C-rnn-gan: Continuous recurrent neural networks with adversarial training, 2016.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, 2016.
Gregory C Reinsel. Elements of multivariate time series analysis. Springer Science & Business Media, 2003.
Xiaoli Ren, Xiaoyong Li, Kaijun Ren, Junqiang Song, Zichen Xu, Kefeng Deng, and Xiang Wang. Deep learning-based weather prediction: A survey. Big Data Research, 23, 2021.
Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. Spatial-temporal synchronous graph convolutional networks: A framework for spatial-temporal network data forecasting. Pro- ceedings of the AAAI Conference on Artificial Intelligence, 34(01):914-921, Apr. 2020. doi:
Xianfeng Tang, Huaxiu Yao, Yiwei Sun, Charu Aggarwal, Prasenjit Mitra, and Suhang Wang. Joint modeling of local and global temporal dynamics for multivariate time series forecasting with missing values. In AAAI, 2020.
Selim Furkan Tekin, Oguzhan Karaahmetoglu, Fatih Ilhan, Ismail Balaban, and Suleyman Serdar Kozat. Spatio-temporal weather forecasting and attention mechanism on convolutional Istms. arXiv preprint, 2021.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio, 2016.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using I-sne. Journal of machine learning research, 9(11), 2008
Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. Graph wavenet for deep spatial-temporal graph modeling. In Proceedings of the Twenty- Eighth International Joint Conference Artificial Intelligence, IJCAI-19, pages 1907-1913, 7 2019.
Chen Xu and Yao Xie. Conformal prediction for dynamic time-series. In ICML, 2020.
Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar. Time-series generative adversarial networks. In NewrIPS, 2019.

---

Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In Proceedings of the Ewenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 3634-3640, 7 2018. 8. doi: 10.24963 2018/.20
Xiang Zhang. Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for inreuularly sampled multivariate time series. In JCLR, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We did.
(b) Did you describe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] See our supplemental material.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they chosen)? [Yes] In Appendix G specify all the training details.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] In Section 4 We report the results of 10 experiments by calculating the mean and variance.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Appendix C
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [Yes]
We release our model. See our supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "pGVZ-u7oW7i", "reviewer": "Reviewer_Besg", "summary": "This paper proposes GT-GAN, a framework for synthesizing regular and irregular time series with various techniques such as generative adversarial networks, auto-encoders, neural ordinary differential equations, neural controlled differential equations, and continuous time-flow processes. Experiments show that this method can outperform baselines in regular and irregular time series synthesis. Ablation studies show that each part of the model is critical to the synthesis performance.", "questions": "What are the difficulties of synthesizing regular and irregular time series in a single framework? How does the proposed method solve the difficulties? In other words, what parts of the method enable it to synthesize both regular and irregular time series? Since the author regards the ability to synthesize both regular and irregular time series as the core contribution of this work, it is better to clarify it more clearly.\n\nIn Table 5, GT-GAN without pre-training performs better than GT-GAN in terms of the predictive score, which contradicts the discriminative score. Can the authors explain this issue?", "limitations": "The authors have discussed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The studied problem is valuable for the field of machine learning.\n+ The proposed method outperforms all baselines across regular and irregular time series synthesis qualitatively and quantitatively according to the conducted experiments.\n+ The experiments are abundant, and the experimental settings are clearly stated.\n\nWeaknesses\n- The technical contributions are limited to this conference. The method proposed appears to be a combination of existing models. And some combinations of these models have been studied, e.g., Flow-GAN [Grover et al., 2017] combines maximum likelihood and GAN training. \n- The proposed method lacks motivation. It is unclear to me that why we need so many technologies together to achieve the goal.\n- The description of the training method is not clear to me. It would be better to give an algorithm pseudocode.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "hPONiaAGhg3", "reviewer": "Reviewer_BsWG", "summary": "The authors present a GAN-based architecture, called GT-GAN, for generating synthetic samples for regular and irregular time series. It comprises of two main components: (1) Autoencoder(AE) - encoder (Neural CDE) encodes time series into a latent space and decoder (GRU-ODE) recovers a continuous path from which synthetic time series is sampled. It is pre-trained using standard reconstruction loss. Decoder is trained additionally with GAN's discriminator loss. (2) Generator(CTFP) and Discriminator(GRU-ODE) networks which are trained using adversarial loss and maximum likelihood training with the log-density. Generator synthesize a time series latent vectors which is used by AE decoder to generate the time-series. This fake time series is then passed to discriminator along with true samples. The authors evaluate their method on 2 simulated and 2 real-world datasets and compare it against 7 related methods. While GT-GAN doesn’t always have a better performance for regular time series, however, it outperforms all other methods for irregular time series. The authors also present an ablation study to understand the role of each loss term and show that each loss component is required for the superior performance of the model.", "questions": "Good work. I’m just asking this out of curiosity. Since GT-GAN learns the temporal dynamics of irregular time series well, do you think if the model can be modified to do forecasting (which is history conditioned time series generation) for irregular time series?", "limitations": "As the authors mentioned in the limitations, the proposed method has many hyperparameters and it often becomes difficult to tune with standard methods like Grid Search. One way to deal with it is tying AutoML libraries like SyneTune which provides several state-of-the-art distributed hyperparameter optimizers (HPO).", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. First of all, the paper is very well-written and provide good amount of details to understand the motivation of the work, and the proposed method.\n2. The papers presents a through comparison with several related methods. The authors made extra efforts to modify competing methods for making compatible with irregular time series, and therefore, ensuring a fair comparison.\n3. The evaluation shows the superiority of the method both quantitatively and qualitatively, especially for irregular time series.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "etov3fijwyU", "reviewer": "Reviewer_xbxz", "summary": "The paper presents a time series GAN model that uses neural ODEs and continues time-flow processes to generate synthetic time series from regular and irregular data (irregular means time series data having missing observations). Specifically, the authors use neural ordinary differential equations, neural controlled differential equations, and continuous time-flow processes. The authors evaluate their approach against standard baselines and datasets and conduct an ablation study of different components. In particular, their method outperforms baselines in the irregular data setting with missing observations. ", "questions": "1. ODEs (and their variants) are a natural choice for modeling data with missing observations and this paper provides further evidence for this. However, the overall architecture is very similar to TimeGAN and therefore it is hard to judge how much gain is coming from the architecture itself or by replacing the standard RNN units with continuous counterparts. In particular, I one ablation that would be interesting is to replace the continuous units with GRU-$\\Delta$-t or GRU-D. How much performance does the model loose when using these units in GT-GAN? Likewise, how would the baselines perform (in particular TimeGAN) if the corresponding ODE units are used instead of the GRU-$\\Delta$-t or GRU-D? This would help to judge how much performance actually stems from the use of continuous units. \n\n2. The overall architecture is very similar to TimeGAN in the sense that it uses an autoencoder path, a GAN path and a way to link the two (in this case, the inversion/recovery and log density of the hidden representation from the autoencoder). However, this component is not analyzed in the ablation study. How does the model perform when removing this log-density path? Could this path be replaced with a supervised loss between the hidden (fake) representation of the CTFP and the autoencoder? Since this component is the biggest differentiator between the proposed architecture and TimeGAN (on top of the continuous units, of course) it would be interesting to measure this effect. \n\n3. The authors do not provide details on how the hyperparameters of GT-GAN are tuned. I would kindly ask the authors to discuss their tuning method in the paper. ", "limitations": "I appreciate that the authors include a limitations statement and discusses that the model contains many hyperparameters and is difficult to train because of that. While this might be the case it is a fairly general statement. It would be more interesting for the reader to understand what specific limitations arise from using continuous units (if there are any). Also, the authors discuss the training time in the Appendix and I think this could be moved to the main text. ", "rating": 7, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main contribution in this paper is the use of ODE/continuous time-flow processes into a GAN framework for time series synthesis. ODEs and CTFPs are a natural choice when learning from data with missing observations. The overall architecture is close to TimeGAN and uses an autoencoder path (with reconstruction loss) and an adversarial path with a discriminator that is trained on a standard adversarial loss (the individual components are continuous units, though). An interesting addition is the use of the invertible CTFP to use the hidden representation from the autoencoder to train the \"log-density\" path by converting the hidden representation into noise vectors and then again into a hidden representation by the CTFP and computing its log probability.\n\nThe overall architecture of the model is reasonable and the evaluation is sound overall. However, since the main contribution of this paper is using continuous layers for irregular time series synthesis, I would like to propose additional experiments to strengthen the paper. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept