Time-Transformer AAE: Connecting Temporal Convolutional Networks and Transformer for Time Series Generation
Authors: Yuansan Liu
Sudanthi Wijewickrema
Ang Li
James Bailey
Venue: Submitted to ICLR 2023 (2023)
Keywords: Time Series Generation
Adversarial Autoencoder
Temporal Convolutional Networks
Transformer

Abstract: Generating time series data is a challenging task due to the complex temporal properties of this type of data. Such temporal properties typically include local correlations as well as global dependencies. Most existing generative models have failed to effectively learn both the local and global properties of time series data. To address this open problem, we propose a novel time series generative model consisting of an adversarial autoencoder (AAE) and a newly designed architecture named `Time-Transformer' within the decoder. We call this generative model `Time-Transformer AAE'. The Time-Transformer first simultaneously learns local and global features in a layer-wise parallel design, combining the abilities of Temporal Convolutional Networks (TCN) and Transformer in extracting local features and global dependencies respectively. Second, a bidirectional cross attention is proposed to provide complementary guidance across the two branches and achieve proper fusion between local and global features. Experimental results demonstrate that our model can outperform existing state-of-the-art models in most cases, especially when the data contains both global and local properties. We also show our model's ability to perform a downstream task: data augmentation to support the solution of imbalanced classification problems.

Under review as a conference paper at ICLR 2023
TIME-TRANSFORMER AAE: CONNECTING TEMPORAL CONVOLUTIONAL NETWORKS AND TRANSFORMER FOR TIME SERIES GENERATION
Anonymous authors Paper under double-bling blind review
ABSTRACT
Generating time series data is a challenging task due to the complex temporal properties of this type of data. Such temporal properties typically include local correlations as well as global dependencies. Most existing generative models have failed effectively learn both the local and global properties of time series data. To address this open problem, we propose a novel time series generative model consisting of adversarial autoencoder (AAE) and newly designed architec- ture named "Time- Transformer' within the decoder. We call this generative model AAE'. The Transfrranf first simultaneously learns lo- cal and global features in layer-wise parallel design, combining the abilities of Temporal Convolutional Networks (TCNs) and Transformer in extracting local features and global dependencies respectively. Second, bidirectional cross at- tention is proposed to provide complementary guidance across the two branches and achieve proper fusion between local and global features. Experimental re- sults demonstrate that our model can outperform existing state-of-the in most cases, especially when the data contains both global and local properties. We also show our model's ability to perform a downstream task: data augmenta- tion to support the solution of imbalanced classification problems.
models
INTRODUCTION
Automatically generating realistic synthetic data assists solving real-world problems when there is limited access to real data and manual generation is cumbersome and/or impractical. Deep gen- crative models have shown considcrable success in domains such as computer vision and natural language processing in the last decade. Numerous models have been introduced to produce syn- thetic images or text to address downstream tasks such as image in in-painting (Pathak et al.) 2016), text to image translation (Zhang et al. 2016)), and automated captioning (Guo et al. 2017).
Although data generation is similarly important in the time series domain, there exist relatively few works that address this problem. This is due the fact that the generated data is required share a similar global distribution with the original time series data and also preserve its unique temporal properties. As such, generative models for time series data, especially those universally applicable to different types of time series data are relatively rare. Many existing works utilize Generative Ad- versarial Networks (GANs) (Goodiellow al. 2014) for time series generation and most of these address the temporal challenges using Recurrent Neural Networks (RNNs) such as Long Short Term Memory (LSTM) (Esteban al.) 2017 Yoon al. 2019; et al., (02I). There are also approaches that use Variational Autoencoder (VAE) (Kingma & Welling 2013) as the basic framework to gen- erate time series data (Fabius & van Amersfoort) [014] Desai et al. 2021 However, none of these works have succeeded in efficiently learning both local correlation and global interaction, which is crucial for time series processing.
Recently, Transformer based models have been successful in learning global features for different types of data including time series (Raffel et al., 2019 Dosovitskiy et al. 2020 Zerveas et al. 2021 Chen et al. 2021 2022). On the other hand, models based on Convolutional Neural Net- works (CNNs) have been shown to be better at extracting local patterns with their filters (Howard) al. 2017
Yamashita al. 2018 Liu et al. 2019. Temporal Convolutional Networks (TCNs),

---

Under review as a conference paper at ICLR 2023
consisting of dilated convolutional layers Oord et al. (2016a). preserve the original local processing capability of CNNs, but also have an enhanced ability to learn temporal dependencies in sequential data (Lea et al.) 2016 Bai et al.] 2018). This makes them appropriate for time series modeling. Therefore, it is natural to combine Transformer and TCN together to learn better time series fea- tures. For example, some previous works use sequential combinations for time series tasks (Lin et al., 2019 Cao et al., 2021). However, such sequential designs do not consider the interaction between local and global features inherent in these datasets.
Motivated by the above observations and analysis, we propose a novel time series generative model named 'Time-Trame Transformer AAE'. Specifically, we first select the Adversarial Autoencoder (AAE) Makhzani et al. (2013) as the generative framework due to its success at learning different types of tasks. The Time- Trae-Tranfformer both local and global features. In each Time- Transformer block, the temporal properties are learnt by both TCN layer and a Transformer block. They are then connected through a bidirectional cross attention block to fuse local and global features. This layer-wise parallel structure along with bidirectional interaction, combines the advantages the TCN and Transformer models: the ability of the TCN to efficiency extract local features, well as the Transformer's ability in building global dependency. We evaluate our proposed Time- Trme-Tranffomer AAE on different types of time series data including artificial and real-world datasets. Experiments show that the proposed model surpasses the existing state-of-the-ar (SOTA) models in addressing the time series generation task. Furthermore, we also show our model's effectiveness on downstream task imbalanced classification using several real-world datasets. To summarize, our contributions are as follows:
designed as part of the decoder to effectively learn and integrate
We propose a new time series generative model called Time-Trame Time-Tranffommr AAE, which effectively combines the advantages of TCN and Transformer in extracting local and global patterns respectively.
We introduce the Tranforranfrrmer module that simultaneously learns local and global features in a layer-wise parallel design and facilitates interaction between these two types of features by performing feature fusion bidirectional manner.
We show empirically that the proposed Time-Transformer AAE can generate better syn- thetic time series data, with respect to different benchmarks, compared to SOTA methods.
RELATED WORKS
2.1
TIME SERIES GENERATION
Deep generative models (DGMs) have gained increasing attention since their introduction. Kingma & Welling (2013) propose a Variational autoencoder (VAE) that uses Bayesian method to learn latent representations and turn the classic autoencoder into generative model. Goodrellow et al. (2014) introduce an adversarial approach to shape the output distribution and propose the Generative ad- versarial networks (GANs). Makhzani et al. (2015) combine the previous two models together in Adversarial autoencoders (AAE). They use the adversarial training procedure to perform variational inference in the VAF. Numerous models have been designed based these basic generative frame-
works and shown superior performance in image and text processing (Oord et al. 2016c b Pathak
al. 2016
Zhang al. 2016 Karras et al. 2017: Arjovsky al. 2017] et 2017 Kadurin
et al.
2017
He et al. 2019 Ahamad 2019).
Successes in the fields of graphs and text have led the application of DGMs in the time series domain. Most of them are derived from the GAN framework with additional modifications to incor- porate temporal properties. The first of these, called C-RNN-GAN (Mogren) 2016). directly uses the GAN structure with LSTM to generate music data. Esteban et al. (2017) propose a Recurrent Conditional GAN (RCGAN) which uses basic RNN as generator and discriminator and auxiliary label information as condition to generate medical time series. Since then, a number of works have utilized similar designs to generate time series data in various fields including finance, medicine and the internet (Zhou et al. 2018 Hartmann al. 2018 Chen & Jiang 2018, Koochali et al. 2019 Wiese et al.) 2020: Smith & Smith! 2020 Lin et al. 2020). TimeGAN Yoon et al. (2019) introduces embedding function and supervised loss to the original GAN framework generate uni-
versal time series. Pei et al. (202T) proposes RTSGAN based on WGAN (Arjovsky et al.) 2017)
and autoencoder. It focuses real-world data generation and achieves good performance.
Jeha
2

---

Under review as a conference paper at ICLR 2023
et al. (2022) utilizes the progressively growing architecture introduced by Karras al. (0017 to generate long time series. In addition, VAE has also been used for time series generation. Fabius & van Amersfoort (2014) designed recurrent VAE synthesize time series data. The recently in- troduced l'imeVAt Desai et al. (2021) implements an intepretable temporal structure together with VAE, and achieves state-of-the-ar results universal time series generation. Parrett al.) (2021) uses contrastive learning framework in stead of traditional DGMs to generate universal time series and achieves good results.
In contrast to sampling from learnt distribution, another line of works focus on step-wise gener- ation. They borrow the idea from the variational inference of VAE Kingma & Welling (2013) and build generative models to model the sequential data at each time step so that the future steps can be inferred based on the previous ones and the ground truth data. Some early works modify the basic RNN using the variational inference and get good results with regard to log-likeling ang-iikliihodd Chung et al. [013); Fraccaro et al. (20T6). Some recent works combine this idea with TCN to produce the latest state of the art models this area Lai et al. (2018) Aksan & Hilliges 2019). Though different to our work, these works provide an interesting direction for future time series generation.
2.2 TEMPORAL CONVOLUTIONAL NETWORKS AND TRANSFORMER
Temporal Convolutional Networks (TCNs) Lea et al. 2016 Bai et al., 2018) use dilated causal
convolutions in WaveNet Oord et al. 2016a) encode temporal patterns and avoid any information leakage from future to past. TCN based models have since been used for sequential data such as time series and show the ability successfully extract local features (Sen al. 2019 Zhao al. 2019 Hewage et al.) 2020; Deldari et al. 202T). Transformer Vaswani et al. (2017) implements a self-attention mechanism on the entire data to learn global interactions between each point, which enhances the models' ability to learn long range dependence and improves its performance on ma- chine translation tasks. Its variants also achieve great success on language processing, computer vision and time series tasks (Raffel et al. [2199] (Zhang et al. 2019 Dosovitskiy et al. 2020) Zerveas jet al., 2021 Chen et al., 2022). Some works combine these two types of models to take advantage of both their capabilities Lin et al. (2019); Cao et al. (202T). They link the two models sequentially, which assumes dependencies exist between the models. However, the models are required to learn different levels of features separately to preserve their independence, sequential combination is no longer suitable and instead, a parallel structure like our work is needed.
3 METHODS
In this section, we first describe the problem. Then, we introduce the proposed Time-Tranfformr AAE architecture. Afterwards, we discuss details of the model using one Time-Transformer block.
3.1 PROBLEM FORMULATION
Generally, multivariate time series generation via deep generative models like GANs involves train- ing model learn how map arbitrary prior distribution p to the real data distribution Pa, so that the model can produce realistic synthetic data x' ~ Pa based on any samples s drawn from p. For time series data, assume each time series
,,,, \T E R R where {t, (t,,,,..... t; = (t. (O)) is the C observations at time i (C is also called 'channel' in the rest of the paper), contains both local features and global dependencies. Thus, the map from p to pa here has to represent both types of features in order to generate realistic data. The goal of our work is to design model to learn such map that can represent both local processing and global interaction, in order to generate realistic and useful synthetic time series data that can be used in downstream machine learning problems.
3.2
TIME-TRANSFORME AAE
3.2.1 OVERVIEW
We choose Adversarial Autoencoder (AAE) as our generation framework due to the potential of extending it to supervised and semi-supervised learning settings (details of AAE can be found in
3

---

Under review as a conference paper at ICLR 2023
Appendix Ap. To modify the original AAE for time series generation, first Convolutional
Neural Networks (CNNs) as the basis of the AAE, and then insert the Time- Transformer (see Figure
lb into the decoder of the AAE. The overall structure is shown in Figure la Here, choose
s~N(1,1)
Diseriminator
Time- Trunsformer
CNN Block
DeCINN Block
Input
Flatten
Dense
Latent
Output
(a) Overview of Time- Transforrnrrme AAE
Layer
Anaddi
Connal
Dense
Outputs
Time- Trannsomoerr
Block
Bidirectional
Cruss Attention
(b) Tine-Transformer Structure
Figure 1:
AAE
Gaussian the prior distribution p(=) (N 1) and insert the Time-Transformee after De- Convolutional block. With this design, we expect the De-Convolutional block to first reconstruct a prototype of the time series, and then the Time- Transforroo to learn the detailed local/global features and generate realistic data.
Within the lutional Networks (TCNs) and the Transformer. The leamt prototype from the De-Convolutional block passed into a TCN layer and a Transformer block simultaneously. The TCN learns the local features of the time series, and the Transformer finds the global patterns of the data. The learnt results then go through cross-attention block to fuse with each other bidirectionally. At the end of this parallel structure we concatenate the outputs from both sides, and use full-connected layer to map them into the expected dimension (L x where and C' are length and number of channels of the time series respectively) and reshape into the original time series dimensions.
we use layer-wise parallelization to combine the Temporal Convo-
3.2.2 TIME-TRANSFORMET BLOCK
The Time-Tranffomme consists of several Time- Transfornnffrrr blocks, which have two key differences the standard TCN layers and Transformer blocks: (1) the layer-wise parallel design to combine local-window self-attention and depth-wise convolution and (2) the bidirectional cross attention over the two branches. Figure shows details of the Time-Transformer block.
Layer-wise Parallelization: In Mobile-Former (Chen et al.) 2021), the authors show the advantage of using a parallel structure. They combine MobileNets light-weight CNN) and Transformer in parallel, and achieve better performance than sequentially combined models image classification and object detection tasks. Inspired by this design, we also combine TCN and Transformer in a parallel manner for time series generation. However, instead of simply combining the entire TCN and Transformer, we only use one layer from the TCN and one block from the Transformer each Time-Transformes block.

---

Under review as a conference paper at ICLR 2023
Dense
TCN-in' Xi+1
Trans in Yest
Concat
Bi Cross-All
softmax
TCN-oout
Trans-oat
scale
Tranffrmet blook
AddiLayyr--B
TON layer
Layer-Norm
Feed Forward
Wo
Wx
We
AddiLaaee- Norm
Dilated-Conv
Multibead Self-Att
TUN
Trans
x;
Thons In X2
Bidiivctional Crass-åttention (Multi Head)
Figure 2: Detailed design of the Trae-Tranfformer block
As shown in Figure each Time-Transfooe Transformer block has two streams parallel to each other, the TCN stream and the Transformer stream. Inside the TCN stream we have a dilated convolutional layer followed by post layer normalization, which make up hidden layer of the TCN model. The inputs and outputs of each block (TCN-in and TCN-in' respectively in Figure 2 are time series data containing local features including those learnt directly from the last/current TCN layer (TCN-out in Figure 2} and the fusion of the local and global features. On the other side, the Transformer block has the default design accordance with (Vaswani al. 2017): Multi-head Self Attention layer and a Feed-forward Network, both with post layer normalization. Similarly, the inputs/outputs (Trans-in and Trans-in' respectively in Figure 2 of this side are global features from last/current Transformer block together with feature fusion.
The bidirectional cross attention block aims build interaction be- tween the two parallel branches, and thus fuse local and global features. As illustrated on the right hand side of Figure the output of the TCN layer T, E LxC and the output of the Transformer block E R LCC interact in mutual manner within the cross attention block, where it bidirection- ally fuses the local feature map I and the global feature map ÿ. Specifically, the TCN features are updated through residual connection with the attention matrix to obtain 24+1:
Bidirectional Cross Attention:
21+1 Σ Ag. Wev
where Wes is the learnable parameter for the value embedding layer, and AØ.+2; is the affinity ma- trix from Trans to TCN which can be calculated with matrix multiplication and a softmax function:
F. Wey eg
(T.
Ag soflmaa
VC
where We, and Was are learnable parameters of two linear projection layers. Similarly, the updated Trans feature map 3/+11 defined as:
Di+1 - + A5, W.Waa
T. J.Wag (H. (Fx
Az. -. =sosmmar
VC
where Wag W W and do are learnable parameters of three linear projection layers. As,- is the affinity matrix from TCN to Trans.

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 DATASETS
We use six datasets to evaluate our model. The first three datasets have been used in several pre- vious works (Yoon et al., 2019. Pei et al. [202]] Desai et al. 2021). Hence, they are referred to as preliminary datasets:
Sine Sim: A dataset containing 5000 samples of multivariate sinusoidal sequences with different frequencies f, amplitudes a, and phases In each dimension E {1.....5p the time series x7{4) generated using zy xr(d) asin(2n ft + (p) where ~ U[1, 2/ 0.15] and 4p 2/10, 2.
Stock: 3686 samples of stock-price sequences from historical Google daily stocks. Each sample has 6 features correlated with each other, including volume and high, low, opening, closing, and adjusted closing prices. Hence, this time series data is 6-dimensional.
Energy: The UCI Appliances energy prediction dataset contains time series data with high dimensionality (28) and correlated features. There totally 19735 continuous measure- ments in this dataset.
The next three datasets contain time series with both local and global patterns. They are used to evaluate the model's ability to learn both types of features, and are termed local-global datasets:
Sine. Cpx: Another 5000 samples of 5-dimensional sinusoidal sequences, which are more complex than the previous ones. They are simulated using sum of standard sinusoidal
waves (d) = aniinnnnnnnnmm (2m + 91) + a:ysin + (22) + agsin(2rfst fst + 43) where as; ~ 111 3), f; 0.11, ~ L[00, 2m 1, and a; * atj, Ji * JJ+ 44: F yjs when i * j. Thus, in each dimension, the sequence is sine wave globally, and also contains local patterns within each wave.
Music: waveform audio file (WAV) of the classical music 'Canon D' (converted from an MP3 file). The file is sampled at 8000H: frequency and double channels, which results in a 2845466-step, 2-dimensional time series. The music data also contains local patterns and seasonality globally. We select 10000-step segment of the original time series, which corresponds to the original music from 2'05" to 2'06".
ECochG: A medical dataset of historical patient data from cochlear implant surgeries. Each instance contains univariate time series that represents a patient's s inner ear response during the surgery. In general, each time series stochastic trend globally, with some local drops. The detailed background of this dataset can be found in Appendix.
4.2 BASELINE MODELS
We select four previous models to compare with. The inclusion criteria are: relevance of the work, accessibility of the code, executability of the code, and performance of the model. Tim and RTSGAN (Desai et al. 2021 Pei et al. 2021) are the two most recent models that have achieved state-of-the-ar (SOTA) performance. TimeGAN and RCGAN (Yoon et al. 2019 Esteban et al.) 2017) are two earlier methods that have gained much attention, both having achieved SOTA per- formance up to their introduction and have been used as a basis of comparison for many methods. All these works provide public access their code, together with detailed instructions. We the settings of the original designs as they have already been optimised by their authors. Details of our model's settings thyper-parameters and data pre-processing) can be found in Appendix C For reproducibility, we provide source code and corresponding instructions in supplementary files.
4.3 BENCHMARKS
To evaluate these generative models, we use a range of metrics and consider several desiderata for the generated time series: (1) They should be distributed close to the original data and different from each other, (2) They should be indistinguishable from the original data, and (3) They should preserve the temporal properties of the original data that they can be used for downstream tasks.

---

Under review as a conference paper at ICLR 2023
Visualisation is indicator of (1) above as it shows the distributions the original and generated data. We flatten the temporal dimension and use t-SNE (van der Maaten & Hinton 2008) plots to embed them into 2-dimensional space for viualisation.
Fréchet Inception Distance (FID) evaluates the quality of generated data for GAN type models. Following the ideas from (Hartmann al., 2018 |mmihh & Smith 2020 Jeha al.)
2022), we replace the Inception model of the original FID with a recently proposed state- of-the-art time series representation learning method called TS2 (Yue et al. (02T). We pre-train the TS2 on each dataset and use it to get the FID score of the model. The FID score provides a quantitative assessment of (1).
Discriminative Score quantitatively measures the indistinquishabiiiyy of the generated data. We follow the protocol from Yoon et al. (2019) when defining this and the next benchmarks. We first train a post-hoe sequential classifier (a 2-layer LSTM) guish original and generated data. Then we get the test accuracy accie from a held-out test set. Good generated time series data should be indistinguisaable to the real data (aceu- racy close to 0.5). Thus, we use Jaccre 0.5| as the Discriminative Score to measure the indistinguishability (2).
distin-
Predictive Score reflects how well the generated data inherit the predictive properties of the original data. We train a post-hoc sequential predictor (a 2-layer LSTM) to do one-more- step prediction, and calculate the mean absolute error (MAE). Here, we use the 'Train on Synthetic, Test on Real' (TSTR) technique introduced by Esteban et al. (2017). Good synthetic data should preserve the temporal properties of the original data, and thus the predictor should be able to learn from the generation and return good predictions of the real data. The Predictive Score provides a quantitative assessment of (3).
4.4 EXPERIMENTAL. RESULTS
In this section, we first report the performance of the models with respect to the aforementioned benchmarks. For the three scores, we execute the experiment several times to get the averages and confidence intervals (95% confidence). Afterwards, conduct some further studies using several real-world datasets.
Time- Tame-TTansfrmmer
TimeVAE
RTSGAN
TimeGAN
ROGAN
Syathetie
Music
Real
Sipe_Cpa
ECochG
Figure 3: t-SNE visualisations of models (columns) on 3 datasets (rows). The blue dots represent original data and the red dots represent synthetic data. Heavier overlaps represent better synthesis
Visualisation: Figure 3 shows visualization results of three locablglobal datasets (results of pre- liminary datasets can be found in Appendix F. Each row contains plots of one dataset and each column represents model. The blue and red dots represent original data points and generations re- spectively. It is obvious that proposed Time- Transformer AAE (denoted Time- Transformer hereafter) consistently produces synthetic data closely distributed to the original data. Time VAE and RTSGAN also have good distributions for most datasets.
FID Score: Table T shows the FID scores of each model on each dataset. These scores are mostly consistent with the observations from Figure 3 For example, proposed Time- Transforme ob- tains the best scores for all the datasets except 'Stock' where RTSGAN is slightly better. TimeVAE also generates well distributed data for this dataset, but as shown in the figure, the data points tend

---

Under review as a conference paper at ICLR 2023
Table 1: FID score: lower scores are better
Models
Time- Transformer
VAE
RTSGAN
TimeGAN
RCGAN
Sine. Sim
0.283=0.023
0.766=0.027
3.175+0. 142
4.572+0.15 153
9.705=0.417
Stock
0.273=0.033
0.303=0.037
0.254:0.031
0.645#0.081
22.673+0.975
Energy
0.983=000039
1.598=0.067
6.292+0.265
1.821=0.117
20.475#0.732
Music
0.395=0.013
13.415=0.937
0,957+0.095
1.89440.12 121
3.541
Sine-Cpx
1.502200.022
7.696+0.251
3,486+0.17 170
14.255=+0.49
18.922#0.736
ECochG
0.348=0.024
5.197=0.237
0.527+0.062
0.67420.098
12.65240.617
to fall on/around a line. This causes its FID score to be worse than the former two models. For those datasets where the models show similar visualisation results (e.g. Trme-Tranffommer and TAm on 'Sine. Sim') ) the FID scores provide way to distinguish the quality of the generation.
Discriminative Score: As shown in Table our model performs best all datasets except 'Stock' where RTSGAN takes the leading position. The Time-Transformn Transformer dominates the performances within the local-global datasets where the generations from other models are highly distinguishable while those from our model show much better indistinguishabiliyy
Table 2: Discriminative score: |a0C&e 0.5), lower scores are better
Models
Transfrranfeomer
TAE
RTSGAN
TimeGAN
RCGAN
Sine Sim
0.131+0.021
0.21740.015
0.489#0.007
0,485=0.008
0.500#0.000
Stock
0.463=0.023
0.476+0.050
0.374+0.022
0,486#0.030
0.500#0.000
Energy
0.496+0.006
0.499+0.000
0.499+0.00
0.500=0.000
0.500+0.000
Music
0.160#0.05%
0.495+0.008
0.489 0009
0,494=0.006
0.497+0.000
Sine_Cpx
0.168#0.041
0.47140.016
0.489#0.031
0.500#0.000
0.499 0011
ECochG
0.103#0.012
0.47440.016
0.405#0.004
0,424=0.015
0.496+0.003
To avoid long decimal numbers while highlighting significant differences, we report ten times MAE in Table 3 additional column 'Oracle" shows results from the 'Train on Real, Test on Real', which represent the theoretical best performances. As shown in the table, our model performs best in all datasets except 'Stock' and 'Energy' where RTSGAN and VAE show better performance. Note that our model's ss scores in local-global datasets are close to the original ones, which indicates that it has learnt most the predictive properties of these data.
Predictive Score:
Table 3: Predictive score: 10xMAE, lower scores are better
Model
Oracle
Time- Transformer
VAE
RTSGAN
TimeGAN
RCGAN
Sime
0.038=0.012
0.051=0.015
0.108+0.031
0.78920.063
0.465+0.055
1.135=0.324
Stock
0.015200.03
0.048+0.004
0.080=0.003
0.03620.003
0.094+0.007
1.310=0.051
Energy
0.04420.004
0.077+0.00
0.07240.006
0.228=0.022
0.197+0.007
0.733=0.021
Music
0.014=0.000
0.017+0.009
0.07740.051
0.021+0.009
0.027+0.007
0.179=0.016
Sine_Cpx
0.029=00.088
0.032+0.006
0.055+0.015
0.070+0.006
0.527+0.025
1.785=0.075
ECochG
0.01 100.009
0.013+0.006
0.08740.008
0.176#0.021
0,04240.006
0,40240. 170
Using the 'Music' dataset, we investigate how each component of the model con- tributes to the final result. Specifically, we (1) use only a De-convolutional block, (2) add only TCN block after (1), (3) add only Transformer block after (1), (4) add sequential combination of TCN and Transformer after (1), and (5) use our proposed Time- Transformer. The results of this study are shown in Table 4 (results of the other two gooal-global datasets can found the Appendix (E.2). From the FID column we can see clear improvement when adding either TCN Transformer. The sequential combination also slightly improves the performance. However, with our proposed parallel structure, the Time- Transformer achieves approximat the sequential one. The other two columns also reflect similar situations. All these phenomena indicate the strong ability of the proposed Time-Transforee Transformer AAE to generate time series.
Ablation Study:
approximate 40% improvement compared to
8

---

Under review as a conference paper at ICLR 2023
Table 4: Ablation study
Components
FID
Discriminator
Predictor
De-Convolution TCN
2.298#0.22 284
0.490#0.006
0.037#0.000
0.672+0.097
0.471
0.026+0.000
Transformer
0.815=0.039
0.479=0.009
0.027=0.006
TCN+Trans (Sequential)
0.627#0.031
0.455+0.021
0.022+0.008
Tran-Trranfrrmer AAE
0.395+0.013
0.160=0.054
0.017#0.009
Model Application: We briefly evaluate the utility of our model on imbalanced classification using the ECochG dataset and three real-world datasets from the UCR archive (Dau al. 2018): Wafer,
Herring, and SwedishLeaf, The details of these datasets can be found in Appendix F Previous
studies argued normal oversampling approaches (augmenting data via replication or simple modifi-
cations like jittering) can cause overfitting create out- oot-of--istte -ff--iidittrtuttion synthesis (Yap et al. 2014
Yan al. 2019). Our proposed model is expected address these issues. Table 5iists the classifi- cation results on the ECochG dataset using an aff-the-sheed Multi-layer Perceptron (MLP) with (1) no augmentations, (2) augmentations from repeating minority class (3) augmentations from jittering method which augments data by adding noises, (4) augmentation from RTSGAN, (5) augmentation from Time VAE, and (6) augmentation from our model (We do not list TimeGAN and RCGAN be- cause training data here are mostly not enough to train their models). The results the other three datasets can be found in Appendix
We can observe from the table that one crucial problem
Table 5: Data augmentation evaluation (higher scores are better)
Datasets
Components
Accuracy
Recall
Precision
AUC_ROC
AUC_PRC
No Aug
0.9751
0.7062
0.8706
0.9824
0.8521
Replication
0,9719
0.9552
0.7072
0.9824
0,8636
Jittering
0.9757
0.7074
0.8864
0.9866
0,8870
ECochG
RTSGAN
0.9702
0.6791
0.8235
0.9767
0.7352
Time
0,9738
0.7388
0.8319
0.9807
0.7906
Tran--Tranformer
0,9802
0.8178
0.9163
0.9945
0.9345
of imbalanced classification is low recall rates, which is caused by an under-represented positive class in the training set. The model tends to predict data as negative since has seen much more negatives than positives. Repeating the minority class can improve the recall at the cost of precision, which reflects the aforementioned overfitting problem. However, augmenting the training set with synthetic data from our model shows better results with respect to all metrics suggesting that it can aid solving imbalanced classification problems.
Additional Experiments: We further investigate our model's performance with respect to: different training sizes, local & global feature extraction, different encoder types, longer time series data, and different evaluation model. Due to space limitations, we provide the details of these experiments in Appendix E.4 to E.8 respectivel..
CONCLUSION
In this paper we introduce a novel time series generative model called Time- Transformer AAE, which contains adversarial autoencoder (AAE) and key component named Time-Trrasformer the decoder. Via a layer-wise parallelization and a bidirectional cross attention, Time-Transformer exploits the learning ability of Temporal Convolutional Networks and Transformer in extracting lo- cal features and building global interaction respectively. Through multiple experiments, we show the effectiveness of our proposed Transforaform AAE on time series generation tasks. Additionally, by training different sizes of data, we show the proposed model can achieve competitive per- formance ot-te-rf methods even with less training data. Furthermore, we see the model's utility in augmenting data for imbalanced classification problems. Possible future directions of our work can be: (1) altering the model setting with conditional generation in order to produce user- defined synthetic data and (2) extending the model to work with partial time series.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Afroz Ahamad. Generating text through adversarial training using skip-thought vectors. In Pro- ceedings of the 2019 Conference of the North. Association for Computational Linguisties, 2019. doi: 10.18653/v 1//19-300
Emre Aksan and Otmar Hilliges. Sten: Stochastic temporal convolutional networks. arXiv preprint arXiv:1902.006568, 2019.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan, 2017.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, 2018.
Luke Campbell, Arielle Kaicer, David Sly. Claire Iseli, Benjamin Wei, Robert Briggs, and Stephen O'Leary. Intraoperative real-time cochlear response telemetry predicts hearing preservation in cochlear implantation. Otology & Neurotology. 37(4)3332-338, 2016.
Yudong Cao, Yifei Ding, Minping Jia, and Rushuai Tian. A novel temporal convolutional network with residual self-attention mechanism for remaining useful life prediction of rolling bearings. Reliability Engineering & System Safery, 215:107813, 2021. ISSN 0951-8320. doi: 10.1016/j. ress.2021.107813
Weihuang Chen, Fangfang Wang, and Hongbin Sun. S2tnet: Spatio-tempont transformer networks for trajectory prediction in autonomous driving, 2022.
Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer, 2021.
Zhenghua Chen and Chaoyang Jiang. Building occupancy modeling using generative adversarial network. Energy and Buildings, 174, 07 2018. doi: 10.101//..cbbull.2118..0.000
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. recurrent latent variable model for sequential data. Advances neural information pro- cessing systems, 28, 2015.
Adrian Dalbert, Alexander Huber, Dorothe Veraguth, Christof Roosli, and Flurin Pfiffner. Assess- ment of cochlear trauma during cochlear implantation using electrocochleography and cone beam computed tomography. Otology & Neurotology, 37(5):446-433, 2016.
Hoang Anh Dau, Eamonn Keogh, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, Yanping, Bing Hu, Nurjahan Begum, Anthony Bag- nall, Abdullah Mueen, Gustavo Batista, and HeX The ucr time series classification archive, October 2018. URL htttps://wwr cs ucr
data 2018/]
Shohrch Deldari, Daniel V. Smith, Hao Xue, and Flora D. Salim. Time series change point detection with self-supervised contrastive predictive coding. In Proceedings of the Web Conference 2021, WWW '2,, 2021. doi: 10. 10.1145/3442881.344999
Abhyuday Desai, Cynthia Freeman, Zuhui Wang, and Ian Beaver. Timevae: A variational auto- encoder for multivariate time series generation, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition at scale, 2020.
Cristóbal Esteban, Stephanie L. Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans, 2017.
Otto Fabius and Joost R. van Amersfoort. Variational recurrent auto-encoders, 2014.
Marco Fraccaro, Søren Kaae Sønderby, Urrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. Advances in neural information processing systems, 29, 2016.
10

---

Under review as a conference paper at ICLR 2023
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information, 2017.
Kay Gregor Hartmann, Robin Tibor Schirmmesster, and Tonio Ball. Eeg-gan: Generative adversarial networks for electroenceephalggrhii (eeg) brain signals, 2018.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. Lagging inference net- works and posterior collapse in variational autoencoders. In International Conference on Learning Representations, 2019. URL Bolnhttss:: httpp/pppppeeeeeeeeeeeeeeeeeeeep net wtt.//orummmm/ het.ee/eooommmdd-yyeeeeeeee
Pradeep Hewage, Ardhendu Behera, Marcello Trovati, Ella Pereira, Morteza Ghahremani, Francesco Palmieri, and Yonghuai Liu. Temporal convolutional neural (ten) network for an effec- tive weather forecasting using time-series data from the local weather station. Soft Comput., 24
(21):16453-16482 2020. ISSN 1432-7643.
doi: 00.10077/005000000--9994444
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications, 2017.
Daniel Jarrett, Ioana Bica, and Mihaela van der Schaar. Time-series generation by contrastive imi- tation. Advances in Neural Information Processing Systems, 34:28968-22 28982, 2021.
Paul Jeha, Michael Bohlke- SchhkerSehneider, Pedro Mercado, Shubham Kapoor, Rajbir Singh Nirwan, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. PSA-GAN: Progressive self attention GANs for synthetic time series. In International Conference on Learning Representations, 2022.
net./ htt./forumm?coo URL hhttps://cpenreeeee httpr//reeeeeeeeeeeeoooorraaeaee 2irrrmmm?ddIIx___h2223x
Artur Kadurin, Sergey Nikolenko, Kuzma Khrabrov, Alex Aliper, and Alex Zhavoronkov. drugan: an advanced generative adversarial autoencoder model for de novo generation of new molecules with desired molecular properties in silico. Molecular pharmaceutics, 14(9):3098-311 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation, 2017.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 2013.
Alireza Koochali, Peter Schichtel, Andreas Dengel, and Sheraz Ahmed. Probabilistic forecasting of sensory data with generative adversarial networks ForGAN. IEEE Access, 7:63868-63888 2019. doi: 10. /ttp://ccee.211.99955555.
Guokun Lai, Wei- Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. Modeling long- and short-term temporal patterns with deep neural networks, 2017.
Guokun Lai, Bohan Li, Guoqing Zheng, and Yiming Yang. Stochastic wavenet: A generative latent variable model for sequential data, 2018
Colin Lea, Michael D. Flynn, Rene Vidal, Austin Reiter, and Gregory D. Hager. Temporal convolu- tional networks for action segmentation and detection, 2016.
Lei Lin, Beilei Xu, Wencheng Wu, Trevor W. Richardson, and Edgar A. Bernal. Medical time series classification with hierarchical attention-based temporal convolutional networks: A case study of myotonic dystrophy diagnosis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
Zinan Lin, Alankar Jain, Chen Wang, Giulia Fanti, and Vyas Sekar. Using GANs for sharing net- worked time series data. In Proceedings of the ACM Internet Measurement Conference. ACM, oct 2020. doi: 10. 145//441999434223633.
Bin Liu, Ruiming Tang, Yingzhi Chen, Jinkai Yu, Huifeng Guo, and Yuzhou Zhang. Feature gener- ation by convolutional neural network for thick-through rate prediction. In The World Wide Web Conference, wwW 19, pp. 1119-1129, 2019. doi: 10.1 0114/33085588333339977
11

---

Under review as a conference paper at ICLR 2023
Alireza Makhzani. Jonathon Shlens, Navdeep Jaitly, lan Goodfellow, and Brendan Frey. Adversarial autoencoders, 2015.
Olof Mogren. C-rnn-gan: Continuous recurrent neural networks with adversarial training, 2016.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio, 2016a.
Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks, 2016b.
Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelcnn decoders, 2016c.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting, 2016.
Hengzhi Pei, Kan Ren, Yuqing Yang, Chang Liu, Tao Qin, and Dongsheng Li. Towards generating real-world time series data, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019.
Rajat Sen, Hsiang-Fu Yu, and Inderjit S Dhillon. Think globally, act locally: A deep neural net- work approach to high-dimensional time series forecasting, In Advances in Neural Information Processing Systems, volume 32, 2019.
Kaleb E Smith and Anthony O Smith. Conditional gan for timeseries generation, 2020.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- chine Learning Research, 986))2557-2665, 2008. URL /ttp::t////// jmlr. www..rrrrgpppppp//////
html
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
Magnus Wiese, Robert Knobloch, Ralf Korn, and Peter Kretschmer. Quant GANs: deep generation of financial time series. Quantitative Finance, 20(9):1419-1440 2020. doi: 10. 18800146997888. 173046..
Sudanthi Wijewickrema, Christofer Bester, Jean-Mare Gerard, Aaron Collins, and Stephen O'Leary. Automatic analysis of cochlear response using electrocochleography signals during cochlear im- plant surgery. Plos one, 17(7), 2022.
Rikiya Yamashita, Mizuho Nishio, Richard K. G. Do, and Kaori Togashi. Convolutional neural networks: an overview and application radiology. Insights into Imaging, pp. 611-629, 2018. doi: 10. 10.10077/13244408---666
Yuguang Yan, Mingkui Tan, Yanwu Xu, Jiezhang Cao, Michael Ng, Huaqing Min, and Qingyao Wu. Oversampling for imbalanced data via optimal transport. In Proceedings of the Thirty-Thin Third AAAI Conference on Artificial Intelligence and Thirty-Firss Innovative Applications of Artificial Intelli- gence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, 2019. ISBN 978-1-57735 1.9-1.
doi: 10.1609/aaii.v333330 v33i...33......... 530015605.
Bee Wah Yap. Khatijahhusna Abd Rani, Hezlin Aryani Abd Rahman, Simon Fong, Zuraida Khairudin, and Nik Nik Abdullah. An application of oversampling, undersampling, bagging and boosting in handling imbalanced datasets. In Proceedings of the First International Conference on Advanced Data and Information Engineering, pp. 13 32-22. Springer Singapore, 2014.
Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar. Time-series generative adversarial net- works. In Advances in Neural Information Processing Systems, volume 32, 2019.
12

---

Under review as a conference paper at ICLR 2023
Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. Ts2vec: Towards universal representation time series, 2021.
George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. A transformer-based framework for multivariate time series representation learning. Proceed- ings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD '21, pp. 2114 2124, 2021. doi: 10. 0..11553447548.346444..
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks, 2016.
Han Zhang, lan Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative ad- versarial networks. In Proceedings the 36th International Conference on Machine Learning, pp. 7354-7363, 2019. URL UR//https::/prrceeddngs http://ceeeiiingmmmm.......e.e wttp...........///.... zhangl 9d. html
Wentian Zhao, Yanyun Gao, Tingxiang Ji, Xili Wan, Feng Ye, and Guangwei Bai. Deep temporal convolutional networks for short-term traffic flow forecasting. IEEE Access, 2019. doi: 10. N0.109//ACEESS2219999555
Xingyu Zhou, Zhisong Pan, Guyu Hu, Siqi Tang, and Cheng Zhao. Stock market prediction high-frequency data using generative adversarial nets. Mathematical Problems in Engineering, 2018:1-11, 04 2018. doi:
13

---

Under review as a conference paper at ICLR 2023
A ADVERSARIAL AUTOENCODER
The data generation in our model is based on a framework called Adversarial Autoencoder (AAE) MMahhzani al. 2015) which is shown in Figure _ It uses the adversarial training procedure from GANs to perform variational inference, and hence turns the decoder into a deep generative model.
Let us assume is the imput data and is the corresponding latent code produced by deep encoder B()). Let p(=) be an arbitrary prior distribution, g(=[ær) be the encoding distribution, and pa(#) be the data distribution. Then, can define an aggregated posterior distribution g(z) the latent code vector:
<(z) ( ((22)))p((x)d
The AAE attaches an adversarial network (a discriminator that distinguishes positive samples s ~ p(2) from negative samples ~ ((=)) on top of the latent code vector which guides the aggregated posterior q(2) to match the prior distribution p(z)
(see Figure
s~p(x)
D(-)
Figure 4: Adversarial Autoencoder
The objective function of the AAE consists of two parts: reconstruction loss and adversarial loss. The reconstruction loss Lree ensures that generator (decoder) G(-) can reconstruct the original data from the latent codes produced by the encoder E(-). Normally, it is measured using similarity sim(.) between the original and reconstructed data:
Lrec = (x- sim (z, G(z))
(5)
The adversarial loss ladv provides a solution for min-max adversarial game between generative model G(-) and a discriminative model ((:):
Lade = min max (x) [log D(x)] + Ez-p(=) log(1 D(G(2))))
(6)
G D
B ECOCHG DATASET
Intra-operative electrocochleography (ECochG) is used to monitor the response of the cochlea to sound during Cochlear Implant surgery. This surgery is cost-effective solution for hearing im- pairment which, however, has been received by only a few patients due to the high probability of losing their natural hearing due to trauma caused during the surgery. Previous studies show that the changes of some ECochG components can reflect such trauma (Campbell et al. 2016 Dalbert al. 2016). Specifically, CCambell al. 2016) demonstrate that 30% drop in amplitude of the "Cochlear Microphonic' (CM - one component the ECochG signal) leads to poorer natural hear- ing preservation, and thus can be used to predict trauma. This has motivated researchers to develop machine learning models to automatically detect these drops and assist the surgeon in preventing trauma during the surgery (Wijewickrema al. 2022).
C EXPERIMENTAL SETUP
C.I DATA PRE-PROCESSING
For the preliminary datasets, we follow the settings in previous work (Yoon et al. 2019 Desai et al.
2021 We use sliding window of 24 window size to sample the data resulting in time series of
14

---

Under review as a conference paper at ICLR 2023
length 24, and re- re-scale all data to [0, 1] using the min-max normalization formula:
min az)
x'
mox 22) min
where æ is the original data, min(x), maz are minimum and maximum of the data respectively.
We choose a longer length (128) for time series in local-global datasets, to preserve more local and global patterss, and evaluate the model's ability process longer data. We use min-max normalization to re-scale all the data to [0, 1] as mentioned previously.
C.2 MODEL SETTINGS
We search the hyper-parameter of the autoencoder based on the average reconstruction perfor- mances on the preliminary datasets. We divide the datasets into training and validation sets, and search the hyper-parameters including layer number, filter number, kernel size, stride number, head number, head size, and dilation rate using the training set. The value range of each parameter is shown in Table As result, the encoder is 3-layer 1-dimensional CNN with 64, 128, and 256 fil-
Table 6: Value range of Hyper-parameten
Hlyper-parameten
Value range
layer number
2,3
filter number
64, 24,,,22556,51
kernel size
4,6,8
stride number
1,2
head number
2,3,4
head size
64,128,256
dilation rate
1,2,4,16
ters and 'relu' activation. All filters have kernels of size 4, and move 2 unit-step each time (stride of 2). After this convolutional process, the results are flattened and go through fully connected layer which maps them into the latent code with pre-defined dimensions. Here we use 8-dimensional latent code for preliminary datasets (original: 24-step) and 16-dimensional vectors for local-global datasets (original: 128-step).
In the decoder, the de-convolutional part has 2 transposed convolutional layers with 128 and 64 filters. All filters have kernels of size 4 and strides of units. Then, the outputs are reshaped and mapped into the original data dimensions (length L and channels C*) via reshape and fully connected layers, which results in prototypes of the time series. The following Time- Transformer has two blocks, which represents TCN that has hidden dilated convolutional layers. The hidden layers have C' filters with kernel size of 4 and dilation rate of and 4 respectively. The Transformer blocks combined with each dilated convolutional layer consist of 3-head self-attention layer with head size of 64 and a feed-forward convolutional layer. The cross attention is also a 3-head attention module with head size of 64. It takes inputs from different sides which makes it different from the self-attentio module ahead. The discriminator, that accomplishes the adversarial process, is 2-layer fully connected layer with 32 hidden units in both layers and 'relu' activation.
We follow the original AAE training procedure to train the model: We first update the encoder and decoder using the reconstruction loss (Mean Squared Error). Then, the encoder and discriminator are updated with respect to the adversarial loss consisting of discriminative loss and generative loss (both are cross-entropy). We use the Adam optimiser to update all the losses. The learning rate for reconstruction loss is 0.005 initially, and reduces to 0.0001 via polynomial decay function (we directly implemented it using the tensorflow platform, the power of the polynomial is 0.5). Both the discriminative loss and generative loss have initial learning rates of 0.001, which also reduces to 0.0001 via a polynomial decay function (same as previous). The default training epoch is set to 1000.
15

---

Under review as a conference paper at ICLR 2023
D GENERATION EXAMPLES
We show some generated samples (together with the original data) of local-global datasets in Figure
Apul wpttr Man Maythhpp
* AnyW M Mhy Maap
Dow www.hmnnmmoa Papt
M
3 yANAt Now /
Avhyy "opp
A (v MAAA
Orignal
Tew-harabere
ToNGAN
SCCAS
Figure 5: Generation examples
16

---

Under review as a conference paper at ICLR 2023
E ADDITIONAL EXPERIMENTAL RESULTS
We provide more results from the experiments on visualization, training size, ablation study and model application as mentioned the paper.
E.1
VISUALIZATION
t-SNE plots of the preliminary datasets are shown Figure
Time- Transformer
TAmeVAE
RTSGAN
TimeGAN
Systhetic
Sine
Redi
Stock
Energy
Figure 6: t-SNE visualizations on preliminary datasets
E.2 ADDITIONAL ABLATION STUDY
Table 7 shows the results of the ablation study with respect to 'Sine_Cpx' and 'ECochG" datasets. Both show performance improvements when we add each part.
Table 7: Extra Ablation Study
Dataset
Components
FID
Discriminator
Predictor
De-Conv
6.97640. 157
0.490#0.021
1.832+0.526
TCN
2.570=0.007
0.417+0.270
0.529#0.057
Sine-Cpx
Transformer
2.472+00079
0.397+0.253
0.484+0.025
TCN+ Trans (Sequential)
1.72140.055
0.193+0.188 181
0.037+0.013
Time- Transtormer
1.50240.062
0.168+0.041
0.032+0.0G6 0006
De-Couv
0.78140.099
0.461#0.048
0.092+0.011
TCN
0.502+0.041
0.288+0.192
0.027+0.013
ECochG
Transformer
0.51340.045
0.256*0.166 164
0.02740.007
TCN+Trans (Sequential)
0.402+0.033
0.162+0.059
0.01740.006
Tranf
0.348+0.024
0.10420.012
0.013+0.006
E.3 IMBALANCED CLASSIFICATION
Table 8 lists imbalanced classification results on UCR datasets: Wafer, Herring, and SwedishLeaf..
E.4 TRAINING SIZE COMPARISON
We study how Time-Transforme performs with different sizes of training data. We train the model with 100%, 50%, and 20% of the data. The results are shown Figure including the mean scores (red dots) and corresponding intervals. Additionally, the best results from competitors are shown via a blue area, with the mean scores represented by the lines in the middle. Note that these results from the models trained full size datasets (100%). Generally, smaller training size leads to worse performance both in terms of quality and stability. The fact that our model is competitive with other models even when trained 50% of the data highlights its accuracy and stability.
17

---

Under review as a conference paper at ICLR 2023
Table Data Augmentation Evaluation on UCR datasets
Datasets
Components
Accuracy
Recall
Precision
AUC
AUC
No Aug
0.9919
0,9639
0,9610
0.9981
0.9826
Replication
0,9924
0.9835
0.9478
0.9979
0.9855
Jittering 0,9933
0,9774
0,9643
0.9985
0,9892
Wafer
RTSGAN
0.9940
0.9759
0.9686
0.9973
0.9847
Time VAE
0.9932
0,9789
0,9587
0.9978
0.9868
Time-Transe Transformer
0.9945
0.9849
0,9618
0.9992
0,9928
No Aug
0.5469
0.3461
0,5625
0.6549
0.5762
Replication
0.6563
0.4615
0,6000
0.7287
0.6429
Jittering 0.6250
0.3846
0.5556
0.6781
0.5455
Herring
RTSGAN
0.6563
0.4231
0.6111
0.7146
0.6307
Time VAE
0.6406
0,3846
0,5882
0.6817
0.5891
Trae
0.6875
0.4615
0.6667
0.7712
0.6455
No Aug
0.9264
0,0000
0,0000
0.9383
0,4296
Replication
0.8848
0.8043
0.3700
0.9487
0.5337
Jittering 0,9279
0.3043
0,5185
0,9426
0,5230
SwedishLeaf
RTSGAN
0.9232
0.1956
0,4500
0.9418
0.4656
Time VAE
0.9360
0.6304
0,5577
0.9439
0.5433
Time-Transiee Transformer
0,9408
0.8478
0.5652
0.9536
0.6341
0005
8.000
6003
6.00
Traning See
Taining Ste
Tariig Ste
Music
33
0.07
0.35
0.08
25
0.05
0.44
1.5
0.03
(b) Sine_Cpx
0.05
0.64
0.03
0.0
Teming SUN
Tean Sto
0.0
Tamin Sax
(c) ECochG
Figure 7: Results of training size experiments on 'Music', 'Sine Cpx', and 'ECochG' datasets
E.5 CASE STUDY FOR LOCAL & GLORAL FEATURE LEARNING
We first define the local and global features: Assume we have time series x = (t......fd t and function F that maps segment of into some feature space. Global features are defined by applying the function to the whole time series: i.c. F(ft)...., Local features are defined with reference to sub-part of the whole series: i.e F((li.... li+m)) where, <
18

---

Under review as a conference paper at ICLR 2023
Then, three simple datasets to show different performance of the models when encountering time series containing both global and local features. Figure g shows the underlying idea. The green series and the orange series each represent local and global features respectively. They are generated with high (50Hz) and low (5Hz) frequencies with respect to the Fast Fourier Transform. By adding them together we get the blue series which, hence, contains both global and local features, it 'mixture' (the global features would be the total trend of the low frequency wave and this needs be learnt using the entire series, and local features would be the time span and the amplitude of single oscillation of the high frequency wave, which would be learnt via sub- par--patt of the time series). To be specific, the formulae generate local, global and mixture time series are:
name
Local: asin (2x50t)
Global: asin
Mixture: (25m(2 asin(2n501)
where a ~ 3[.,3]. For each of the three cases, we generate 2500 time series, each consisting of 128-step time steps (we use 128 time steps ensure global features are non-trivial). We thus obtain three different datasets, each containing 2500 time series.
global
Figure 8: Global local feature example
We train the different generation models on these datasets and compare their performance. First, we show some generated examples in Figure 9 Comparing these generated examples, we can see: when the time series contain only global features or only local features, all the models can generate comparatively good time series (see Figure 9a and 9b). However, when comes to the mixture of the two, baseline models fail to generate as effectively. RTSGAN fails to extract local properties. TimeGAN and RCGAN seem to be able to learn the local patterns some extent, but they fail to capture the global features. TAm performs better than others but is less effective than our model (see Figure 9c).
capture the global trends but
Table 9 lists the quantitative evaluations results of each model, which also shows the effectiveness of our model in extracting both global and local features simultaneously, as it gets the best scores with respect to all metries for the mixture dataset.
E.6 CHOICE OF ENCODER
Table 10 lists different performance outcomes from our model with two different encoders, namely, CNN and Transforaformerr.. The first is the simple design which we used our proposed model, and the second is the one with a Time- Transformer module inserted in the encoder. We test their per- formance on Sine Sim dataset to briefly investigate how much improvement a complex encoder can bring to the model. As shown in the table, using Time-TTansforre Transformer encoder does improve the results a little, but it requires much more time to train. This indicates that a simple encoder can generate relatively good synthetic data with high efficiency.
E.7 LONGER TIME SERIES DATA
Most existing related works Yoon et al. (019); Desai et al. (201): Pei et al. (202)): Jarrett et al. 2021 for synthetic time series generation have used lenghh time series as their experimental
19

---

Under review as a conference paper at ICLR 2023
Orginal
The www.eee
Toewin
KOGGAM
ICANN
MWW
VVWVV
MWW
JVVVVV
VVVVV
WWMV
(a) Local feature series
(b) Global feature series
orgiaal
wwww
NAMM
WM
www.
VVW
MMM
www.
VMM
mwM
www.
VVV
WMM
MMM
www.
(c) Mixture series
Figure 9: Generation examples
Table 9: Evaluation results for global & local features learning (lower scores are better)
Model
Benchmark
Local
Global
Mixture
RCGAN
8.879=0.461
4.81720.592
10.217=0.762
TimeGAN
4.736+0.224
1.597+0.299
5.15..0.316
TAE
FID
0.271+0.074
0.241+0.037
1.081#0.068
RTSGAN
0.299#0.06 00.8
0.163+0.055
1.788+0.09
Hime-TTrnsformer
0.317#0.071
0.229
0.474+0.014
RCGAN
0.455+0.026
0.395+0.011
0,497+0.005
TimeGAN
0.178#0.066
0,048+0.00 0.00
0.485+0.011
TimeVAE
Discriminative
0.015+0.026
0.018+0.009
0.177+0.073
RTSGAN
0.022+0.013
0.012+0.004
0.294+0.078
Time-Traas Transformer
0.029+0.027
0.017+0.015
0.061+0.060
Oracle
0.009#0.002
0.014+0.008
0.024+0.000
RCGAN
0.196#0.052
0.285+0.057
0.21640.079
TimeGAN
0.077+0.018
0.052+0.013
0.103+0.043
Predictive
VAE
0.01320.006
0031+0.005
0.052+0.011
RTSGAN
0.01320.006
0.027=0.012
0.073+0.015
Time-Tre Transformer
0.014=0.005
0,030+0.009
0.03140.007
datasets. A recent work (PSA-GAN Jeha al. (022 has claimed to generate long time series and the longest time series used in their work are 256- length.. They apply a sampling method from L.STet Lai al. (2017) to the electricity datasets, to produce these 256-1ength time series. Hence,
20

---

Under review as a conference paper at ICLR 2023
Table 10: Comparison between different encoders (lower results better)
Encoder CNN
Training Time (s)
FID
Discriminative 0.131#0.021 0.120#0.011
Predictive 0.051=0.015 0.036#0.010
530.24 1651.33
0.283=+0002) 0.256*0.0014
Trne-Tranffommrr
use this dataset, with the same sampling method as Lai (0017. to further evaluate our model. The results are shown in Table [1 below. We can see our model still performs well with this longer (256-length) time series data, as has the best performance against the baselines (we only use RTSGAN and TimeVAE) VAE).
Table 11: Long time series evaluation (lower is better for all metrics)
Model
FID
Discriminative
Predictive
Oracle Time
N/A
0.010#0.004
1.655=0.621
0,415=0.037
0.034+0.005
RTSGAN
0.896=0. 147
0.397+0.007
0.021+0.007
Time-Trrnsformer
0.338+0.058
0.057=0.027
0.014+0.006
E.8 EVALUATION WITH TRANSFORMER
We replace the LSTM with Transformer and perform the quantitative experiments again, in order to investigate a stronger sequential model can obtain different results. We re-do the experiments on the local-global datasets and only the two most recent models (RTSGAN, VAE) We can see the transformer is more effective at iden-
baselines. The results are shown in Table I2
Table 12: Two scores with Transformer based model (lower numbers better)
Model
Benchmark
Music
Sine-Cpx
ECochG
Time VAE
0.500=0.000
0.487
0,495#0.003
RTSGAN
Diseriminative
0.498+0.000
0.499#0.001
0.472+0.014
Foreerormer
0.384::.026
0.425+0.034
0.36620.041
Oracle
0.030+0.001
0.040#0.001
0.007
Time VAE RTSGAN
0.135=0.019
0.051
0.033+0.008
Predictive
0.046:0.0002
0.079
0.016#0.006
TimesFormer
0.041#0.002
0.042+0.001
0.009+0.002
tifying differences between real data and synthetic ones, as the discriminative scores become worse
than those from the LSTM model. However, our model still out-performs the baselines. Regarding the predictive score, our model still has the most effective forecasting performance (closest to using the original data).
F DETAILS OF IMBALANCED DATASETS
The original ECochG dataset contains data from 77 patients. When sampled and pre-processed as previously mentioned, this results in 13982 time series including two classes. However, this dataset extremely imbalanced: only 874 instances positive, which makes it hard to train machine learning model. One possible solution is augmenting the minority class to make it balanced. As the UCR datasets, the first two imbalanced binary datasets while the last one contains 15 classes, we create an imbalanced situation by assigning class I as the positive class and all remaining classes negative (labeled as O). Here, provide statistical details of the datasets used in Model Application of Section 4.4 Table [3 shows this information including: number of training data (#Train). number of testing data (#Test), length of each time series (Len), positive of training set (Pos Rate, equals to "Positive/HTrraie and number of training data for generative model training (#Generative Training).
21

---

Under review as a conference paper at ICLR 2023
Table 13: Details of Imbalanced Datasets
Dataset
Train
# Test
Len
Pos Rate
# Generative Training
ECochG
9787
4195
128
0.063
874
Wafer
1000
6164
152
0.097
97
Herring
64
64
512
0.391
25
SwedishLeaf
500
625
128
0.058
29
22


Reviews:
[{"review_id": "ZRJug1F1F3", "reviewer": "Reviewer_Kd3s", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces Time-Transformer AAE. The proposed method is an adversarial autoencoder approach, and the Time-Transformer is a component of the decoder. Each Time-Transformer block is comprised of (1) two parallel modules: a TCN block and a transformer block, and (2) a bi-directional cross-attention. The experimental results show that the proposed method could significantly outperform SOTA methods.", "summary_of_the_review": "In general, this paper introduces a novel method for time series generation, and the results show that it can significantly outperform the SOTA methods. However, the motivations for some choices and designs are not quite clear (please see weaknesses). In my opinion, the strengths outweigh the weaknesses. ", "strength_and_weaknesses": "Strengths:\n1. The proposed Time-Transformer is a novel architecture for time series generation.\n2. Experimental results show that the proposed method could significantly improve the results.\n3. Writing is clear and easy to follow.\n\n\nWeaknesses and questions:\n1. How will your model perform if you use GAN rather than the adversarial auto-encoder framework?\n2. The encoder is much simpler than the decoder, what's the motivation behind your choice?\n3. How does Time-Transformer perform if you use two parallel transformer blocks rather than a TCN and a transformer?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "L2USMmRgcE", "reviewer": "Reviewer_xKc6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a decoder architecture for time-series generation. The proposed model, namely the Time-Transformer, uses both a temporal convolutional network (TCN) and a Transformer to process information at different timescales. While the TCN focuses on learning local features, the Transformer block aims to capture global features, which are then fused via a cross-attention operation. The decoder network stacks several of these Time-Transformer layers. The proposed decoder is trained via the adversarial autoencoder framework to learn a latent space. The model is evaluated on a variety of time series benchmarks (Sine, Stock, and Energy datasets). It demonstrates better performance compared to the baselines.", "summary_of_the_review": "It is an interesting line of work, and the paper presents strong performance in the evaluations. However, considering the originality of the architecture, I think the paper requires a revision to provide more insights. The related work discussion also misses relevant time-series models. I believe providing more comprehensive evidence for the contribution of the proposed model would be very helpful for the community. The following questions can be a starting point. Is the Time-Transformer also effective in other frameworks (i.e., seq2seq VAE or autoregressive setting instead of the adversarial autoencoders)? How would other decoder classes perform in the benchmarks?\n\nI am leaning towards a rejection for the current version of the submission as it would require a major revision. However, I am willing to change my score upon clarification and new evidence.\n\n", "strength_and_weaknesses": "Strengths:\n\n1- The paper leverages the adversarial autoencoder in the temporal domain.\n\n2- The proposed Time-Transformer block controls access to local and global information via a convolution and a self-attention layer, respectively. This design addresses the main concern of the paper: learning local and global features explicitly. It also seems to perform well in the evaluations. \n\n---\n\nWeaknesses:\n\n1- Generative modeling of time-series data is a vast literature involving a number of areas, such as deterministic autoregressive approaches [1*, 2*], state-space models [3*, 4*], and contrastive learning [7*]. Furthermore, [5*, 6*] augment TCNs with a hierarchy of latent variables to capture short-and long-term dependencies. The discussion in the related work section and the baselines in the experiments ignore this line of research. I can suggest another set of ablations. To show the contribution of the proposed Time-Transformer architecture, other model classes could be used in the decoder (i.e., after the De-Convolutional block).\n\n2- The evaluations are performed on \"toy\" datasets. Sequences of 24 or 128 steps are rather short and hence easily explainable by a fixed-dimensional latent variable. I think this is a limitation as the entire sequence is encoded via a single, global, fixed-dimensional latent space. It is hard to tell how the model would perform when the task requires modeling longer sequences like sequential image generation.\n\n3- Finally, the performance of the baselines, TimeGAN and RCGAN, looks different from the reported performance in the TimeGAN paper. In the t-SNE plots on the preliminary \"Sine_Sim\" and the \"Stock\" datasets (see Appendix E), the baselines perform poorly. However, in the TimeGAN paper, (see Fig. 3 a and b in https://proceedings.neurips.cc/paper/2019/file/c9efe5f26cd17ba6216bbe2a7d26d490-Paper.pdf), the generated and the real data are not distinguishable. Could it be possible that the baseline models have not converged? Could the authors explain the source of the difference?\n\n---\n\nAdditional References\n\n[1*] Van den Oord, Aaron, et al. \"Conditional image generation with pixelcnn decoders.\" Advances in neural information processing systems 29 (2016).\n\n[2*] Van Den Oord, Aäron, Nal Kalchbrenner, and Koray Kavukcuoglu. \"Pixel recurrent neural networks.\" International conference on machine learning. PMLR, 2016.\n\n[3*] Chung, Junyoung, et al. \"A recurrent latent variable model for sequential data.\" Advances in neural information processing systems 28 (2015).\n\n[4*] Fraccaro, Marco, et al. \"Sequential neural models with stochastic layers.\" Advances in neural information processing systems 29 (2016).\n\n[5*] Lai, Guokun, et al. \"Stochastic wavenet: A generative latent variable model for sequential data.\" arXiv preprint arXiv:1806.06116 (2018).\n\n[6*] Aksan, Emre, and Otmar Hilliges. \"STCN: Stochastic temporal convolutional networks.\" arXiv preprint arXiv:1902.06568 (2019).\n\n[7*] Jarrett, Daniel, Ioana Bica, and Mihaela van der Schaar. \"Time-series generation by contrastive imitation.\" Advances in Neural Information Processing Systems 34 (2021): 28968-28982.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KAhkWkWUL4", "reviewer": "Reviewer_Cnjw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce Time-Transformer AAE, a generative model for time series that incorporates a new \"time-transformer\" architecture into the adversarial autoencoder (AAE) generative framework. The time-transformer consists of parallel temporal convolutional and transformer blocks interleaved with bidirectional cross-attention to share information. Empirical studies on a variety of datasets show that this approach achieves good coverage and captures the predictive information in the original data, while ablations show that the specific architecture of the time-transformer block outperforms reasonable alternatives.", "summary_of_the_review": "The authors propose an architecture and framework for generative time series modeling that borrows significantly from previous work, yet is shown to perform well relative to state-of-the-art competitors in a broad range of experiments. Moreover, the quality of these results seems to depend specifically on the novel way in which these familiar components are combined. Other issues in the presentation would be straightforward to address and could improve my overall evaluation of the work.", "strength_and_weaknesses": "Strengths:\n- The paper is well written and the main ideas are clearly presented. The figures and supplementary details are relevant and informative.\n- The model is studied through a broad set of experiments that show it consistently outperforms recent, relevant competitor methods in terms of some standard quality metrics for generative modes: predictive value of the synthetic data for a real test set, difficulty of discriminating between real and generated sequences, and overlap of the empirical and generative distributions.\n- The ablation study (Table 4) provides convincing evidence that the specifically parallel structure of the time-transformer block is key to achieving the results above. This goes some way towards alleviating weakness #1 below.\n\nWeaknesses:\n- The novelty of the proposal is modest overall. The specific innovation here is the parallel arrangement of two well-known architectures as part of a decoder block in a well-known generative framework. \n- Throughout the paper, the authors repeatedly claim that simultaneous learning of local and global structure in time series is a major challenge that is specifically overcome by their approach. Despite this, this problem is not defined at any level of technical detail, the property is not established to obtain with any level of rigor in the provided data, and there is no evidence provided the superior performance of the proposed method is due to solving this specific problem.\n- There are no training details whatsoever in the main paper. The supplement describes the learning rate schedule but not the algorithmic details of training. The algorithmic details of training need to be included somewhere in the paper or supplement.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "44Wn12pb1-", "reviewer": "Reviewer_GBz2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduced a new time series generative model called `Time-Transformer AAE` that consists of an adversarial autoencoder and a newly designed architecture called `Time-Transformer`, where temporal properties are learnt by both a TCN layer and a Transformer block. This model was proposed to better learn local correlations as well as global dependencies. Empirical results demonstrated that this model can achieve state-of-the-art performance on datasets, especially datasets with long term dependencies.\n\nThe contributions of this paper comes from the creation of this Transformer based adversarial auto-encoder model, proposing the layer-wise parallel design & interaction to simultaneously learn local & global features, and demonstrating its performance via empirical experiments.\n", "summary_of_the_review": "Overall, this paper proposed an interesting idea of combining TCN & Transformer in a Time-Transformer block for adversarial autoencoder time series generation. The presentation of model architecture and performance are clear. On the other hand, the motivation of such a proposed model is a bit unclear, and the application of such a model in new areas seems limited. It also appears unclear to me whether the performance gain comes from combining different layers of previous state-of-the-art model layers or it's a genuine change. Therefore, I recommend this paper to be rejected.", "strength_and_weaknesses": "Generally this paper does well in presenting its idea and performance, but struggles to provide a coherent story to help us understand how the idea was developed and how the claim of better capturing local correlation & global pattern is supported.\n\nStrengths of this paper:\n1. The figures are very informative in demonstrating the model architecture & bidirectional cross attention.\n2. Extensive experiments are conducted over multiple aspects to understand the performance difference between the proposed model and multiple baselines.\n3. The experiment results look very promising, huge improvement over previous state-of-the-art.\n4. Ablation study demonstrated the proposed bidirectional cross attention mechanism is very useful.\n\nWeaknesses and how to improve:\n1. The datasets are a bit limited and on a small scale. Can you try a larger dataset like [eletricity](https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014) or [traffic](https://archive.ics.uci.edu/ml/datasets/PEMS-SF) and see if there's any difference in the results?\n2. While the paper claimed to better capture local correlation and global dependency, there's no clear analysis to support this claim. Case study or explanation with characteristics of the dataset and performance could mitigate this gap.\n3. The discriminative score and predictive score are generated with a weak baseline (2 layer LSTM), in real applications people use more sophisticated methods like a Transformer based model to do classification & prediction. Would you please try better classifiers or predictors and see if the result changes? This is because these new methods do better in capturing subtle long term dependency and local patterns.\n4. The motivation of using the proposed architectures is not well explained. There're more choices in how to capture local / long term dependency. Would you please add some motivations to help me understand the necessities in using the proposed architecture?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject