SDformer: Similarity-driven Discrete Transformer For Time Series Generation
Authors: Chen Zhicheng
FENG SHIBO
Zhong Zhang
Xi Xiao
Xingyu Gao
Peilin Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: time series generation; generative model; discrete token modeling;
Primary area: other

Abstract: The superior generation capabilities of Denoised Diffusion Probabilistic Models (DDPMs) have been effectively showcased across a multitude of domains. Recently, the application of DDPMs has extended to time series generation tasks, where they have significantly outperformed other deep generative models, often by a substantial margin. However, we have discovered two main challenges with these methods: 1) the inference time is excessively long; 2) there is potential for improvement in the quality of the generated time series. In this paper, we propose a method based on discrete token modeling technique called Similarity-driven Discrete Transformer (SDformer). Specifically, SDformer utilizes a similarity-driven vector quantization method for learning high-quality discrete token representations of time series, followed by a discrete Transformer for data distribution modeling at the token level. Comprehensive experiments show that our method significantly outperforms competing approaches in terms of the generated time series quality while also ensuring a short inference time. Furthermore, without requiring retraining, SDformer can be directly applied to predictive tasks and still achieve commendable results.

SDformer: Similarity-driver Discrete Transformer For Time Series Generation
Zhicheng Chen 1.2,, Shibo Feng", Zhong Zhang", Xi Xiao Xingyu Gao", Peilin Zhao?
Shenzhen International Graduate School, Tsinghua University
Tencent AI Lab
School of Computer Science and Engineering, Nanyang Technological University
KKey Laboratory of Data Protection and Intelligent Management (Sichuan University), Ministry of Education Instittte of Microelectronics, Chinese Academy of Sciences
fczc220mails, xiaoxesz). ts tsinghua. edu. cm, shibo001 @ntu. @nt.tnduuuuu.....emmm edu
gaoxingyudime. ac. cn, {todzhang, masonzhao masonzhao)@tencen. @teenent....... com
Abstract
The superior generation capabilities of Denoised Diffusion Probabilistic Models (DDPMs) have been effectively showcased across multitude of domains. Re- cently, the application of DDPMs has extended to time series generation tasks, where they have significantly outperformed other deep generative models, often by a substantial margin. However, we have discovered two main challenges with these methods: 1) the inference time is excessively long: 2) there is potential for improvement in the quality of the generated time series. In this paper. we propose method based discrete token modeling technique called Similarity-driver Dis- crete Transformer (SDformer). Specifically, SDformer utilizes a similarity-din -riven vector quantization method for learning high-quality discrete token representations of time series, followed by discrete Transformer for data distribution modeling at the token level. Comprehensive experiments show that our method significantly outperforms competing approaches in terms of the generated time series quality while also ensuring short inference time. Furthermore, without requiring training. SDformer can be directly applied to predictive tasks and still achieve commendable results.
Introduction
Time series data is prevalent across wide array of real-world applications. spanning fields such as finance [22 9 38, 36), healthcare (27), energy [28] 22 [3], retail (2) 45). and climate science [35] Despite significance, the limited availability of dynamic data can pose a significant barrier to the development of machine learning solutions, particularly in scenarios where data sharing could lead to privacy violations [2). The generation of synthetic yet realistic time series data has emerged as promising alternative, garnering increased interest due to recent advancements in deep learning techniques.
Existing works on time series generation (TSG) is mainly based on common deep generative models, such methods based on generative adversarial networks (GAN) [2 39
37 26 [6] [7] and
methods based on Variational Autoencoders ( (VAE) 7 25). Currently holding state-of-the-arr results DDPMs-based methods [19] 6 42 which have the capability to generate high-quality, realistic time series. However, they are not without their challenges. Firstly, these methods often require lengthy inference times due to the substantial number of denoising steps involved. Secondly, despite
This work done when Zhicheng Chen works as an intern in Tencent Al Lab. "Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

these methods achieving significant advancements in generation quality compared to other deep generative models, we observe that the quality of the time series they generate still has potential for further enhancement.
Currently, large hansformer-based language models, often known LMs LLMs, have become the standard choice for natural language generation tasks [ 33. As time has progressed. these LMs have evolved to produce content across a wide range of modalities, such as images (29 40 5 and
videos 4 33). using what is referred to as Discrete Token Modeling (DTM) technique. In general, these approaches function by learning a discrete representations of images (or videos, etc.), treating them as if they were natural language, and harnessing the power of existing language models for the generation process. Inspired by their success, we aim to explore the application of these techniques in the domain of multivariate time series generation, potentially unlocking new possibilities and advancements in this area.
To address the aforementioned challenges, we propose a novel two-stage method for time series generation, called Similarity-driven Discrete Transformer (SDformer). The primary objective of the first stage is to employ Vector Quantized Variational Autoencoders (VQ-VAE) [31] for learning high-quality discrete representations of time series. To enhance this process, we introduce similarity- driven vector quantization approach, which identifies the most suitable code from the codebook by maximizing similarity. The experiments in 5.4 further substantiate the superiority of our method over distance-driven vector quantization. Moreover, to prevent code collapse a phenomenon where only a small portion of the codes are updated during training, thereby hindering the performance of the VQ-VAF-we incorporate two standard recipes (44 during training: Exponential Moving Average (EMA) for codebook updates and Resetting inactivated codes during the training process (Code Reset). In the second stage, we implement two Discrete Token Modeling (DTM) techniques: Masked Token Modeling (MTM) and Autoregressive Token Modeling (ARTM), underpinning the SDformer-ar and SDformer-m variants, respectively. SDformer-ar adopts autoregressive approach for both training and inference, mitigating the inconsistency between these two phases through random replacement 43 SDformer-m utilizes random masking for training and iterative decoding Our findings reveal that SDformer, particularly SDformer-ar, surpasses existing models in time series generation. Moreover, SDformer demonstrates robust predictive performance without requiring retraining.
for inference g 5
In summary, our contributions include:
We propose an efficient time series generation model SDformer. which successfully in- troduces DTM technique into time series generation and demonstrates its feasibility and efficiency.
We introduce novel memilarrty-driveer vector quantization approach that outperforms the traditional distance-driven method in learning discrete representations of time series. This innovative approach offers straightforward yet powerful technique for applying discrete token modeling in various fields.
Our experimental results confirm that the SDformer's performance time series generation notably surpasses that of the current sate-off-the-rr arr models, exemplified by an average enhancement of 80 in Discriminative Score and 86.5% in Context-FID Score across multiple datasets.
2 Related work
2.1 Discrete token modeling
Discrete token modeling, a staple in natural language processing (NLP), has recently been adapted for lan-uaee modalities through vector quantization models like VQ-VAE (3I) and VQGAN [10 These models enable the encoding of diverse data types into discrete tokens, allowing the application of advanced language modeling techniques to generate content across various domains. This expansion significantly broadens the utility of NLP methodologies, extending their impact beyond traditional language tasks. Among these techniques, Autoregressive Token Modeling (ARTM) is a common approach that predicts the next token in a sequence, given the previous tokens, using categorical distribution. Models such as DALL-E [29 and Parti 40 employ ARTM to accomplish text-to-image generation tasks. Similarly, T2M-GPT
43 and MotionGPT
[8] utilize ARTM for

---

eext-to-moton generation, Another widely used technique is Masked Token Modeling (MTM), which trained using masked token objective [8] In this approach, some tokens in the sequence are randomly masked and need to be predicted based on the observed tokens. Models such as MaskGIT [5] and MUSE [4] leverage MTM for image generation tasks. Furthermore, MAGVIT 44 and Phenaki
3
employ MTM for video generation, showcasing the versatility of this technique.
2.2 Time series generation
Deep generative models demonstrate high-quality sample generation across various domains, does time series generation. At first, people mostly relied on GAN to complete time series generation
24
39
37 26 [6]. For example. TimeGAN (39 improves temporal dynamics capture by adding
an embedding function and supervised loss. COT-GAN 37 combines GAN and Causal Optimal Transfer (COT) principles
efficiently and stably generate low- and -igh-imensiooal time series data. Due to the challenges of training instability and mode collapse in GAN, researchers have started exploring alternative deep generative models for TSG. Tim employs an interpretable temporal structure and achieves promising results in time series synthesis using VAE. Moreover, several studies focus on addressing the generation of irregular time series, such as GT-GAN [7] and KoVAE (25].
With the emergence of Denoising Diffusion Probabilistic Models (DDPMs) 14] new class of generative models, impressive generative capabilities have been demonstrated across various domains. Recently, diffusion models have also been adapted for TSG. For instance, DiffWave applies DDPMs to waveform generation, while DiffTime [6] harnesses the latest advancements in score-based diffusion models for time series generation. Furthermore, Diffusion-TS 42 gener- ates time series samples by utilizing an encoder-decoder transformer with disentangled temporal representations, showcasing the versatility and potential of these alternative generative models.
[99 directly
3 Definitions and problem formulation
We define multivariate time series as X1:r (I1, ;.,, E IR where 7 and d are the number of time steps and variables respectively. Assuming that dataset containing n time series can be expressed as D = {Xi,r 12 the goal of unconditional generation is to use a model fo to generate time series with the same distribution as D,i.e.
X - fo()).
where Z is the input sampled from any known distribution, such as the Gaussian distribution.
Time series forecasting is common conditional time series generation. We denote historical values as X1.1 E R where < I < r is the number of historical time steps. Therefore, the goal of conditional generation is model fo predict future values, i.e.,
ć = (X....)
(2)
In this paper, objective is to develop effective approach that not only accomplishes uncondi- tional generation tasks efficiently but also adapts to conditional generation tasks without retraining, while maintaining high accuracy.
4 Methods
In this section, we illustrate proposed innovative model SDformer for time series generation. Specif- ically, SDformer is two-stage method, the framework of which is illustrated in Figure [ In the first stage, a pre-trained time series tokenizer utilizes similarity-driver vector quantization to obtain high-quality discrete token representations. Following this, discrete Transformer is employed to learn the distribution of time series data at the discrete token level, with the two generative ways (Masked and Autoregressive strategies).
4.1 Time series tokenizer
To represent time series in discrete tokens, we pre-train a multivariate time series tokenizer based on the VQ-VAE architecture (31). Our time series tokenizer consists of encoder E and decoder
3

---

Stage 1: Similarity-drivee driven Encoder/Decoder Training
Stage Masked//Autoregressive Token Generation
24
1B
Masked/ Maskedd/Autorrgrree
Similariy-ddriee VQ
Encoder
Decoder
Transformeer
24
19
Original
Reconstruction
Masked/Shifted Tokens
Predicted Tokens
Figure The workflow SDformer. In stage I, we pre-train a time series tokenizer which uses similarity-driven vector quantization to obtain high-quality discrete token representations. In stage2, two optional techniques are introduced for time series modeling at discrete token level: MTM and ARTM. For MTM, the input tokens are randomly masked and fed into the Masked Transformer, an encoder-only model, to predict the masked tokens. Conversely, for ARTM, the input tokens are shifted back by step with the [BOS] token added at the starting position, and then processed by the Autoregressive Transformer, a decoder-only model, to predict subsequent tokens for all input tokens.
D. The encoder is responsible for the generation of discrete time series tokens, while the decoder is capable of reconstructing these tokens back into their original time series form. This methodology allows us to represent time series akin to a language, therehy enabling the application of multitude of efficient language models to address various time series-related tasks.
Specifically, the encoder & initially applies ID convolutions to time series features X along the temporal dimension, resulting in latent vectors H1G = (hi,.... h2) IR Lxd, where L T/r, signifies the temporal downsampling rate and de is hidden dimension. Subsequently, we employ the codebook to discretely quantize h, to obtain discrete token. The learnable codebook C' {cx K-] 12d comprises K latent embedding vectors, each with dimension de. The process of similarity-drive vector quantization Q(-) involves identifying the index of the vector in the codebook that exhibits the highest similarity h4, which can be expressed as:
h2
Ck
34 Qh+,) arg max
[b,]]]
||
k= K-
where y, = 0, K -,, denotes the inner product, and represents the modulo operation. For simplicity, introduce normalization step in the final output layer of the encoder resulting in unit modulus length for ht Furthermore, we ensure that the code in the codebook always has a unit modulus length for cg. The similarity-driver quantization process can be re-simplified as:
arg max 14 Ck-
k= K-1
Following quantization, the dequantization process (-) reverts back to the latent embedding vector, denoted as:
(5) Ultimately, the decoder D restores to raw time series space, i.e., X D.....). To train this time series tokenizer, we utilize two distinct loss functions for training and optimizing the parameters of E and DD:
h, = - Cy.
L = 6= - X
(1-h-- sggh))) L
where the first loss is the reconstruction loss, the second loss is embedding loss, sg(.) represents the stop gradient, and À is hyperparameter used to adjust the weights different parts. For the codebook, use Exponential Moving Average and Codebook Reset techniques (34) update.
When the time series tokenizer training is completed, the codebook and parameters will be frozen. By employing this time series tokenizer, multivariate time series X1.r gx can be mapped to

---

sequence of time series tokens YI:L E {0..... (0,...,K-11) - learn the distribution of time series data at the discrete token level. For the choice of DTM, we can opt for methods such as ARTM or MTM. We will introduce these two methods in Sections 4. and respectively.
Therefore, we can use DTM technique to
4.2 Autoregressive token modeling time series generation
In this part, we utilize ARTM technique to learn the distribution of time series data at the discrete token level, based the time series tokenizer. We refer this approach SDformer-ar. During training. we take shifted tokens YLL = ([BOS) 1/1... ... -/--1) as input and real tokens 1I.L as target for training, where [BOS] represents Beginning of Sentence token. In particular, use index K as the [BOS] token, which is distinet from the codebook's 's index range {O,... K- 13. The training objective is to minimize the negative log-likelihood of all tokens:
Car =
Σ log P()))T
Concretely, we input V into a Decoder-only Transformer to predict the probabilities P(y (Y for each token, where the negative log-likee likelihoodd is computed as the cross-entrope between the ground-truth one-hot token and predicted token. For inference, we start from the [BOS] token and generate next token in autoregressive fashion. The detailed training and inference algorithm of SDformerarr are respectively shown in AlgorithmmZann and Appendix E Note that we are able to generate diverse time series by sampling from the predicted distributions given by the transformer.
Random replacement. Autoregression known to exhibit inconsistency between the training and inference phases. Specifically, during training, the first i ground-truth tokens used predict the i-th token. However, during inference, there's s no guarantee that all the preceding tokens used as conditions are correct. To alleviate this issue, we implement random replacement strategy form of data augmentation during training. In this approach, each token is processed individually. A random number is compared probability threshold If it meets the threshold, the token is replaced randomly; otherwise, it remains unchanged. The random replacement can be expressed as:
(andint(O, K), if Uniform(O. /) S
V.
yi,
otherwise
where Randint(O, K) is a random integer sampled uniformly from the range to K 1, and Uniform(0, I) is random number sampled from uniform distribution in the range 0 to 1. Therefore, during training, we use y) instead of ys in Vin to achieve data augmentation.
4.3 Masked token modeling on time series generation
In this part, we utilize MTM technique to learn the distribution of time series data at the discrete token level, based the time series tokenizer. We refer this approach as SDformer-m. During training, we sample a probability from the uniform distribution (O, 1) as the mask probability. We then replace tokens in the original token sequence with the [MASK] token according to the mask probability In particular, we use index K as the [MASK] token, which is distinct from the codebook's 's index range (0.... K- 1}. In other words, when the token y: K a certain position, it indicates that the position has been masked. Denote Y 1: as the result after applying random mask to YEL.- The training objective is to minimize the negative log-likelihood of the masked tokens:
Lmass -E
Σ B.=K
log P(y: 1...)
Concretely, feed the masked V into multi-layer bi-directional transformer to predict the probabilities P(y Y i..)) ) for each masked token, where the negative log-likelihood is computed as the cross-entropy between the arounnd-rutth one-hot token and predicted token.
During inference, we generate a new token sequence using iterative decoding, as proposed in [5]. Initially, we set an iteration number N and mask schedule of length N. Here, S(I) represents number of masks needed after the completion of step L. It is required that S < N. S - 1] 0. and S(D) strictly decreases with an increase in The detailed training and inference algorithm of SDformer-n respectively shown in Algorithm 3 and 5 in Appendix
5

---

SDformer-
SDromer-n
Difiusion TS
(a) I-SNE: Energy
(b) I-SNE: ETTh
(c) KDE: Energy
(d) KDE: ETTh
Figure 2: Visualizations of the time series synthesized by SDformer and Diffusion-TS.
Table 1: Descriptions of datasets.
Dataset
Sines
Stocks
ETTh
MuJoCo
Energy
fMRI
# of Samples
10000
3773
17420
10000
19711
10000
dim
5
14
28
50
5 Experiments
In this section, commence by assessing proposed methods through comparative analysis with several state-of-the-art baseline methods on unconditional time series generation tasks. Subsequently. delve deeper into the analysis of methods' versatility and high performance in conditional generation tasks. Lastly, through ablation experiments, we confirm the superior effectiveness of similarity-driven vector quantization and discrete token modeling.
5.1 Experimental setups
Datasets To evaluate the performance of SDformer, conduct experiments on 4 real-world datasets (Stocks, ETTh, Energy and fMRI) and 2 simulated datasets (Sines and MuJoCo). Table I provides partial description of each dataset. For more detailed information, please refer to Appendix A
Metrics For quantitative evaluation of synthesized data, employ the discriminative score and predictive score as described in [39], along with the Context-FID score proposed by [16], For detailed descriptions, please refer Appendix A
5.2 Unconditional time series generation
Table provides summary of the performance for each of the compared algorithms all the datascts. From these results, we can make several observations. Firstly, our proposed methods based on DTM outperform other methods in most cases, demonstrating the fcasibility and effectiveness of DTM for time series generation tasks. Secondly, SDformer-rr exhibits a significantly better performance than SDformer-m, which contrasts with the findings in the visual domain. This can be attributed to the fact that autoregressive token modeling is better suited to capture temporal correlations compared to masked token modeling.
To further investigate the capability our proposed methods in handling longer sequences, we compare the generative abilities of different methods on longer time series, as shown in Table 3 Based on the results, is evident that many methods exhibit significant distortions when dealing with longer time series, particularly when the discriminative score approaches 0.5, prominently

---

Table 2: Results of all methods all datasets
Metries
Methods
Sines
Stocks
ETTh
MuJoCo
Energy
IMRI
SDformer-rr
0.006=.004
0.010=.006
0.003=.001
0.008
0.0062.004
0,017=.007
SDforme-mm
0.008+.004
0.020=.011 011
0.022+.001
0.0250+ 007
0.062+.006
0.043+.000 006
Diffusion-TS
0.0062.007
0.067=.015
0.061#0099
0.008+0002
0.122
0.167+0023
Discriminative
TimeGAN
0.01 0000
0. 0.102
0.1 14...50055
0.238=.068
0.2362.012
0.484:t042
Score
TAm
0.041#0044
0. 0.145s.120
0.209+.058
0.230=
0.499 000
0.476*.044
Diffwave
0.017+0008
0.232+.061
0, 0.19020008
0.203=0096
0.493+.004
0,402#.029
DiffTime
0.01 34.066
0.097*.016
0. 100000077
0. 045
0.445
0.245+.051
Cot-GAN
0.254s.137
0.230s.016
0.325s.099
0.426s.022
0.498s.002
0.492+.018
SDforer-ar
0.0932.000
0.037+.000
0.118:.002
0.007=2001
0.249:.000
0.091
SDformer-m
0.093#0000
0.037+ 000
11:1 002
0.007=.001
0.250+000 000
0.091+.001
Diffusion-TS
0.093#.000
0.03662.000
0.1 192.002
0.007=.000
0.250#0000
0.099#0000
Predictive Score
TimeGAN
0.093+.019
0.038+.001
0. 0.124+.000
0.025#.003
0.273+.004
0. 126+.002
VAE
0.093#.000
0.039=0000
0.126:t00*
0.012=000
0.292:.000
0.1 13+.003
Diffwave
0.093#.000
0.047+.000
0. 0.1
0.0133+000
0.251 0.00
0. 0.101+0000
DiffTime
0.093+.000
0.038=0001
0.121#.004
0.010= 001
0.252+0000 000
0.100#
Cot-GAN
0.10000000
0.047=.001
0. 0.0
0.068=0009
0.259+.000
0. 0.185..003
Original
0.0942.001
0.036+.001
0,121 005
0.007=0001
0.250+0003
0.090 001
SDforer-ar
0.001=.000
0.002#.000
0.008a.001
0.005+.001
0.0031.000
0.015+.001
SDformer-m
0.010:.002
0.034:.008
0.019#.003
0.0
0.041#.005
0.035#.003
Diffusion-TD TS
0.006a.000
0.147s.025
0.1 16:
0.013+.001
0.089
0.1054.006
Context-FID Scorel,
TimeGAN
0.101#:014
0.13
0.300+.013
0,563=052
103
1.292+.218
TimeVAE
0.307+.000
0.215
0.805 186
0,251=.015
1.631
14.449:.966
Diffwave
0.014+000
0.232+.032
0.873+.061
0.393+.041
1.031+.111
0.244
DiffTime
0.006+:001
0.2362.074
0.299#.044
0, 02888=008
0.279:.045
0.340:
Cot-GAN
1.3374.068
0.408+.086
0.980+.071
1.094a.079
1.039+0028
748134.550
in the Energy dataset. Despite these challenges, both SDformer-ar and SDformer-m continue to demonstrate exceptional performance.
To visualize the performance time series generation, we adopt two visualization methods: projecting original and synthetic data in 2-dimensional space using t-SNE [32], and drawing data distributions using Kernel Density Estimation (KDE). Figure 2 illustrates the visualization of our methods in comparison with Diffusion- TS on the Energy and ETTh datasets, revealing that the data generated by SDformer-ar ar more closely resembles the real data, followed by SDformer-m.
5.3
Conditional time series generation
Pred
Pred 16
Pred 24
Pred
0.45
0.50
0.50
0.40
0.43
0.40
0.40
0.40
(5% uu
32
0.35
16
0.35
40
Stamps
Time
Stamps
Pred
Pred 16
Pred
Pred 32
0.6
0.6
0.6
Valuuc
0.5
0.5
0.5
0.5
Origral S0hermener
Ori St
8
16
32 Time Stamps
24
40
40
16
24
32
40
16
24 32 40
Time Stamps
Time Stamps
Time Stamps
Figure 3: Examples of time series forecasting for Energy (1st row) and fMRI (2 row) datasets. Green and gray colors correspond to SDformer-ar and Diffusion-TS, respectively.
7

---

Table 3: Results of long-term time series generation
ETTh
Energy
Metrics
Methods
64
128
256
64
128
256
SDfomec-arr
0.0182.007
0.0132.005
0.0082.006
0.010=2007
0.013+.007
0.0172.003
SDfomer--m
0.034+0017
0.038+.008
0.0414:024
0.053+.018
0.0694.014
0.035+
Diffusion-Te TS
0.106+.048
0.144+.060
0.060+.030
0.078+.021
0.143+.075
0.290+.123
Discriminative Score|
TimeGAN
0.227+.078
0.188+.074
0.4424.056
0.4982.001
0.49920001
0.499=000
VAE
0.171#.144 142
0.154+.087
0.178+.076
0.4992.000
0.4992.000
0 0.499 000
Diffwave
0.254+.074
0.274s.047
0.3044.068
0.497:.004
0.499+.001
0.499=0000
DiffTime
0.150=t003
0.176+.015
0.243+0005
0.328s.031
0.396a.024
0 0.437=0095
CoI-GAN
0.296+.348
0.451+.080
0.4614:010
0.499#.001
0.499+.001
0.498 004
SDfomer-ar
0.1162.006
0.1 1000000
0.095+.003
0.2474.001
0.244s.000
0.243+0002
SDformer-m
0. 0.1202.004
0,1072.004
0.110+:007
0 0.2484.001
0,245+.000
0.244=.003
Diffusion-TS
0.1162.000
0.110+.003
0.1094.013
0.249:.000
0.2474.001
0.245
Prodictive
TimeGAN
0.1324.008
0.153+.014
0.220+.008
0.291+.003
0.303+.002
0.341
Score|
Time VAE
0. 0.0
0.1 0.113+0005
0.1
0.302:.001
0.318+.000
0,353#0003
Diffwave
0.1334.008
0.1 0.129+.003
0.132+:001
0.2524.001
0.252+.000
0,251
DittTime
0.1 0.118+.004
0.1 0.120+0008
0.118+.003
0.252+.000
0.251.2:000
0,251=000
Cot-GAN
0. 0.135+.003
0.126#.001
0.129:0000
0.2624.002
0.2692.002
0,275=0004
Original
0.1142.006
0.10840005
0.106t.010
0.245+.002
0.243+.000
0,243
SDfommer-ar
0.0182.003
0,024:.001
0.021
0.031+.002
0.0362.002
0,041=0003
SDformer-m
0.0862.008
0.094+.007
0.078+.006
0.160:.025
0.151#.011
0.136 014
Diffusion- TS
0.631
0.7874.062
0.423+.038
0.135+.017
0.087+.019
0.126e 024
Context-FID Score|
TimeGAN
1.130t. 102
1.553 169
5.8724.208
1.230s.070
2.535s.372
5.032+.831
Time VAE
0.827#.146
1.0624. 134
0.826*.093
2.662:.087
3.125#. 106
3,768 998
Diffwave
155432.153
2.354
2.899+.289
2.6972.418
5.552+.528
5.572+
DiffTime
1.2792.083
2.554
3.524
0.7624.157
1.344s.131
4.735=.729
Cot-GAN
3.008+.277
2.639+.427
4.0754.894
1.8244. 144
1.8224271
2.5334.467
Apart from unconditional generation, we also explore the performance of our proposed methods in conditional generation tasks. Our objective to evaluate the model's versatility in handling both conditional and unconditional tasks. More specifically, we aim to train single model that can effectively manage both unconditional and conditional tasks under different settings. Referring (42) we set = 48, = 8, 16, 24, 32 in Equation (2). and then directly use the model trained under the unconditional generation task to complete the forecasting tasks under these different settings. Figure 3 disiplays several examples of forecasting tasks. The median values of forecasting are represented as the dotted line, and 5% and 95% quantiles are depicted as the shade areas (Green: SDformer-ar, Gray: Diffusion-TS). This demonstrates that SDformer-ar provides more reasonable forecasts with higher confidence compared to Diffusion-TS. Furthermore, more detailed results are illustrated in Figure Based on these findings, the methods employing discrete token modeling demonstrates adaptability to both unconditional and conditional generation tasks of varying lengths without the need for retraining, while maintaining good performance.
5.4 Ablation study
To understand the contribution of each component proposed methods, conduct ablation experi- ments for two aspects, 1) The impact of vector quantization methods based on different measurements 2) The advantages of discrete representations in time series generation. More experimental results refer to Appendix C due to limited space.
Effect of similarity-driven vector quantization in Equation For discrete token modeling method, the quality of discrete representations learning from continuous data determines the perfor- mance potential of the entire method, with vector quantization playing crucial role. Therefore, we compare the impact of our proposed similarity-driver vector quantization with the commonly used distance-driven vector quantization on the overall method performance.

---

Energy
fMRI
1.5
sitamern
steameeee
1.33
MSE or DS
MSE or D5
1.0
2.35
0.160
0.5
1.33
116
5/6 6.27
6.3
511615
321
Pred
Pred
Pred 24
Pred 32 Uncondition
Pred 8
Pred Pred 24
Pred 32
Uncondition
Figure Performance for time series forecasting and generation under different setting. All forecasting tasks utilize Mean Square Error (MSE) performance metric, while unconditional generation tasks employ Discriminative Scores (DS). Note: The data in this figure has been scaled by factor of 100 for the forecasting tasks and 10 for the unconditional generation tasks to streamline the presentation.
Table Results of ablation study.
Metries
Methods
Sines
Stocks
ETTh
MuJoCo
Energy
IMRI
Stommer-ar
0.006s.004
0.010:.006 (066
0.003+.001
0,0084.005
0.006:.004
0.017s.007
w/o similarity
0.006e.004
0.011
0.010+.005
0.0134.000
0.018+.005
0.024+.003
Discriminative Score.
continuous
0.047+.012
0.065s.012
0.145+.020
0.0554.013
0.3224.012
0,243+.214
continuous, w/ first
0.012+.004
0.021s.015
0.0064.000
0.020#.006
0.2774.007
0.0744.006
SDoomer-m
0.0084.004
0.020s.011
0.022s.001
0,0254.007
0.0624.006
0.0431.006
w/o similarity
0.015s.007
0.0811.010
0.055a.004
0.0701.005
0.0681.005
0.0551.009
SDforer--r
0.093e.000
0.037s.000
0.118 000
0,007:.001
0.249*.000
0.09 +.02
w/o similarity
0.093t.000
0.037s.000
0.122 002
0.008+.001
0.249:t000
0.091+0002
Predictive
continuous
0.093e.000
0.038+.000
0.124 0003
0.009+.001
0.255+.000
0. 0.105.0000
Score]
continuous, w/ first
0.093e.000
0.037s.000
0.122 003
0.0074.001
0.255 0.251++000
0.087+.003
SDoomer-m
0.093e.000
0.037s.000
0.119
0,0074.001
0.250+.000
0.091#0001
w/o similarity
0.093e.000
0.037s.000
0.123+.00
0.008+.001
0.250+0000
0.093+.000
SDfomee-ar
0.001=:000
0.0022.000
0.008+.001
0,005:.001
0.003++000
0.015+.001
w/o similarity
0.002=0000
0.012+:001
0.013+.001
0.005+.001
0.004+0000
0.011+0000
Context-FID Score]
continuous
0.056=004
0.101#.022
0.433#.049
0.065#.008 008
0.213+.022
5.512+.390
continuous, w/ first
0.004=0000
0.015
0.002+.000
0.006+.001
0.021#.003
0.003+.000
SDfomer-m
0.010s.0012
0.0342.008
0.0192.003
0.0301.003
0.0411.005
0.035+.003
w/o similarity
0.0442.006
0.1231.009
0.1061.012
0.0981.009
0.0624.014
0.0384.002
The term "w/o similarity" in Table denotes the variant of the corresponding method that employs distance-driven vector quantization instead of similarity-drive vector quantization. As per the comprehensive results, similarity driven vector quantization significantly outperforms distance-driven vector quantization. For instance, the discriminative score witnessed an increase 35.6% for SDformer-ar and 46.2% for SDformer-m.
Effect of discrete token in Equation 4 and (5). To explore the impact of discrete versus continuous tokens, introduce variant replacing the original discrete token with continuous one within similar framework. Initially, we substitute VQ-VAI VAE with VAE as per [4] to encode continuous latent space. In the second stage, we reconfigure the Transformer to accommodate continuous inputs, altering its initial input strategy due to the inapplicability of a fixed token like [BOS]. Thus, the first token of each time series is not used as the prediction target during training. For inference, we devised two methods: one involves sampling from a multivariate Gaussian, calculated from all initial training tokens, for generating the first token; the second uses the actual initial token directly. Although the latter does not lend itself to a fair comparison with the original model, is included for more comprehensive evaluation reference.
Table the term "continuous" represents the first variant, which involves using continuous tokens in place of discrete ones. Meanwhile, the term "continuous, w/ first" denotes the second variant, which builds upon the first by providing the actual first token during inference. worth noting that, as SDformer-m necessitates the utilization of category sampling during inference for achieving

---

iterative sampling, we abstain from conducting ablation experiments on discrete tokens specifically for this model. Based on the results, the model's performance experiences significant decline upon the removal of discrete tokens. Even when incorporating the condition information of the first token, often fails to surpass the original method most scenarios.
6 Conclusions
In this paper, present discrete token modeling for the time series generation tasks and propose a innovative two-stage model. Specifically, it is built upon an efficient time series tokenizer, which attains high-quality discrete token representations through similarity- vector quantization. Leveraging this foundation, we employ autoregressive token modeling and masked token modeling techniques to learn the distribution of time series data at the discrete token level. Experimental results showcase the efficacy and adaptability of our approach in various time series generation tasks. Owing to the flexibility of our method in the second stage, future work could explore referencing more efficient language models to design increasingly effective time series generation strategies.
Acknowledgements
We would like thank Tencent AI Lab for supporting Zhicheng Chen as student researcher during his internship. The study was partially supported by the Key Laboratory of Data Protection and Intelligent Management, Ministry of Education, Sichuan University and also the Fundamental Research Funds for the Central Universities under Grant SCU2023D008.
References
[I] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.00874,, 2023.
[2] Ahmed Alaa, Alex James Chan, and Mihaela van der Schaar. Generative time-series modeling with fourier flows. International Conference on Learning Representations, 2020.
[3] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
[4] Huiwen Chang. Han Zhang, Jarred Barber, Aaron Maschinot, José Lezama, Lu Jiang, Ming- Hsuan Yang, Kevin Patrick Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. In International Conference on Machine Learning, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA. [5] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
[6] Andrea Coletta, Sriram Gopalakrishnan, Daniel Borrajo, and Svitlana Vyetrenko. On the constrained time-series generation problem. Advances in Neural Information Processing Systems, 36, 2024.
[7] Abhyuday Desai, Cynthia Freeman, Zuhui Wang, and Ian Beaver. Timevae: A variational auto-encoder for multivariate time series generation. arXiv preprint arXiv:2111.08091 2021.
[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLI 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Pupers), 2019.
[9] Qianggang Ding, Sifan Wu, Hao Sun, Jiadong Guo, and Jian Guo. Hierarchical multi-scale gaussian transformer for stock movement prediction. In IJCAI, pages 4640-4644 2020.
10

---

[10] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-1222 12883, 2021.
[11]
Cristóbal Esteban, Stephanie L Hyland, and Gunnar Rätsch. Real-valued (medical) time series generation with recurrent conditional gans. arXiv preprint arXiv:1706.02633, 2017.
[12] Shibo Feng. Chunyan Miao, Ke Xu, Jiaxiang Wu, Pengcheng Wu, Yang Zhang, and Peilin Zhao. Multi-scale attention flow for probabilistic time series forecasting. IEEE Transactions on Knowledge and Data Engineering, 2023.
[13] Shibo Feng, Chunyan Miao, Zhong Zhang, and Peilin Zhao. Latent diffusion transformer for probabilistic time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11979-11987, 2024.
[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[15] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 988):1735-1788 1997.
[16] Paul Jeha, Michael Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Psa-gan: Progressive self attention gans for synthetic time series. In The Tenth International Conference on Learning Representations, 2022.
Pedro Mercado, Shubham Kapoor, Rajbir Singh Nirwan,
[17] Jinsung Jeon, Jeonghak Kim, Haryong Song. Seunghyeon Cho, and Noseong Park. Gt-gan: General purpose time series synthesis with generative adversarial networks. Advances in Neural Information Processing Systems, 35:36999-37011 37010, 2022.
[18] Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion forcign language. Advances in Neural Information Processing Systems, 36, 2024.
[19] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A
versatile diffusion model for audio synthesis. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021.
[20] Bryan Lim, Sercan Ö Ank, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, (((4):774-1764 2021.
[21] Bryan Lim and Stefan Zohren. Time-series forecasting with deep learning: survey. Philosoph- ical Transactions of the Royal Society A, 379(2194):2020022))) 2021.
[22] Chenghao Liu, Steven CH Hoi, Peilin Zhao, and Jianling Sun. Online arima algorithms for time series prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[23] I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
[24] Olof Mogren. C-mn-gan: Continuous recurrent neural networks with adversarial training. arXiv preprint 1XX1vv6611.09000 2016.
[25] Ilan Naiman, N. Benjamin Erichson, Pu Ren, Michael W. Mahoney, and Omri Azencot. Gen- erative modeling of regular and irregular time series data via koopman VAEs. In The Twelfth International Conference on Learning Representations, 2024.
[26] Hengzhi Pei, Kan Ren, Yuqing Yang, Chang Liu, Tao Qin, and Dongsheng Li. Towards generating real-world time series data. In 2021 IEEE International Conference Data Mining (ICDM), pages 469 478. IEEE, 2021.
[27] Robert B Penfold and Fang Zhang. Use of interrupted time series analysis evaluating health care quality improvements. Academic pediatrics, 13(6):S3-SAA 2013.
[28] Minghui Qiu, Peilin Zhao, Ke Zhang. Jun Huang, Xing Shi, Xiaoguang Wang, and Wei Chu. A short-term rainfall prediction model using multi-task convolutional neural networks. 2017
IEEE international conference on data mining (ICDM), pages 395-404. IEEE, 2017.
[29] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821-8833 Pmlr, 2021.

---

[30] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks, IEEE transactions on Signal Processing, 45(11):2673-268/ 1997.
[31] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[32] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11). 2008.
[33] Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Herman Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2022.
[34] Will Williams, Sam Ringer, Tom Ash, David Macleod, Jamie Dougherty, and John Hughes. Hicrarchical quantized autoencoders, Advances in Neural Information Processing Systems, 33:4524 4535, 2020.
[35] Sifan Wu, Xi Xiao, Qianggang Ding, Peilin Zhao, Ying Wei, and Junzhou Huang. Adversarial sparse transformer for time series forecasting. Advances in neural information processing systems, 33:17105-171155 2020.
[36] Ke Xu, Yifan Zhang, Deheng Ye, Peilin Zhao, and Mingkui Tan. Relation-awar transformer for portfolio policy learning. In Proceedings of the twenty-ninth international conference on international joint conferences artificial intelligence, pages 4647-4653, 2021.
[37] Tianlin Xu, Li Kevin Wenliang, Michael Munn, and Beatrice Acciaio. Cot-gan: Generating sequential data via causal optimal transport. Advances neural information processing systems, 0188798-8009 2020.
[38] Jaemin Yoo, Yejun Soun, Yong-chan Park, and U Kang. Accurate multivariate stock movement prediction via data-axis transformer with multi-level contexts. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2037-2045, 2021.
[39] Jinsung Yoon, Daniel Jarrett, and Mihaela Van der Schaar. Time-series generative adversarial networks. Advances in neural information processing systems, 32, 2019.
[40] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. Trans. Mach. Learn. Res., 2022.
[41] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10459-10469, 2023.
[42] Xinyu Yuan and Yan Qiao. Diffusion-ts: Interpretable diffusion for general time series genera- tion. arXiv preprini arXiv:2403.01744 2024.
[43] Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. arXiv preprint arXiv:2301.066
2023.
[44] Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling for regularized loss minimization. In international conference on machine learning, pages 1-9. PMIR, 2015.
[45] Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. One fits all: Power general time series analysis by pretrained Im. Advances in neural information processing systems, 36:43322-43355, 2023.
12

---

Supplementary Materials for SDformer
In the supplementary, we provide more implementation details, more experimental results, and visualization of test samples of our SDformer. We organize our supplementary follows
In Section A
give the detailed description of used datasets and the metrics.
In Section
we provide the experiment settings.
In Section
C
show more experimental results verify the effectiveness and efficiency
SDformer.
In Section
D
we provide the details operations of the architecture in stage I and 2.
In Section
we provide the algorithms training and inference in SDformer.
In Section
we specify the limitations of our method.
In Section
showcase visualization results and test samples six TSG datasets.
A Dataset and metric details
The Stocks dataset consists of Google's stock price data between 2004 and 2019, with each observation representing a day and containing features. The ETTh dataset includes data obtained from electrical transformers, encompassing load and oil temperature measurements taken every 15 minutes from July 2016 to July 2018. The Energy dataset, a UCI appliance energy prediction dataset, comprises 28 features. The fMRI dataset serves benchmark for causal discovery and features (BOLD) time series; we chose a simulation
Dataset..
realistic simulations of blood-ox wead--ooyee--lee--eeeeeeee with 50 features from the original dataset referring to 42 The Sines dataset contains features, each generated independently with varying frequencies and phases. The MuJoCo dataset is a multivariate physics simulation time series dataset with 14 features.
Metric. The Discriminative Score quantifies the similarity between original and synthesized data. Initially, classification model trained using both the original and synthesized data. Subsequently, the model's capability to classify these data types assessed, and the discriminative score is computed |Aceuracy-O2. 0.5]. The Predictive Score evaluates the usefulness of the synthesized data by training a post-hoc sequence model to predict next-step temporal vectors using the train-synthesis-an -aadssss-andd--t-rrelllll (TSTR) method. The Context-FID Score quantifies the quality of the synthetic time series samples by computing the difference between representations of time series that fit into the local context.
B Experimental settings
Table 5: Detailed hyperparameters of SDformer.
Stage
Stage
Dataser
dim
Ene/Dee Layers
de
Hidden
Transformer Layers
Sines
512
2
1024
512
0.5
1024
2
0.3
6
15.4.3221100
Stocks
2
512
256
2.0
1024
0.3
[5,3,0]
ETTh
(512, 1024)
2
512
512
0.5
1024
6
0.3
6
15.4.3.211.01
MuloCo
2
512
512
0.5
1024
(2.6)
0.1
[5.4332.11.0]
Energy
512
2
512
512
0.01
4
1024
2
0.1
6
15.4.3,2,1.0
IMRI
(1,2)
0.01
[2.4]
[5,4.3.2.1.0]
In this part, we introduce our main experimental settings. For the unconditional generation task, we conduct five evaluations to obtain the experimental results. Similarly, for the conditional generation task, we run the process five times when computing the performance metrics, and escalate it to 1000 runs when calculating the median, along with the 5% and 95% quantiles of the predicted samples. Furthermore, we summarize the detailed hyperparameters values in { *) are the hyperparameters of SDformerar and SDformer-m respectively. Our primary experiments are executed on an Nvidia V-100 GPU with the Adam can use importance sampling [4] to further accelerate it.
SDformer, shown as Table The two
23 optimizer. In the future,
13

---

Spommer
Bitudeei's
Inference Time (s)
80
60
40
36.75
20
3.04
2.94
3.03
2300
2.93
2.88
Sines
Stocks
ETTh
MujoCo
Energy
fMRI
Figure 5: Inference time of SDformer and Diffusion-TH
Table 6: Results of additional ablation study.
Metries
Methods
Sines
Stocks
MuJoCo
Energy
[MRI
Stormer-ar
0.006+.004
0.010# 006
0.0034.001
0.008+:000 005
0.006:2.004
0.017+.007
Discriminative
w/o Transformer
0.007+0007
0.008+.055
0.007+0002
0.0062.003
0.009+0003
0.024+.000 009
Soorel
SDformer-m
0,008+.004
0.0201.011
0.022+.00) 001
0.025:.007
0.0624.006
0.0434.006
w/o Transformer
0.012+0007
0.033+0010
0.025+0007
0.027+0008
0.063+.009
0.060#
SDformer-ar
0.093+.000
0.037+.000
0.118:.002
0.007#.001
0.249::.000
0.091+.002
Predictive
w/o Transformer
0.093:.000
0.037=0000
0.1 0.119
0.008+0001
0.249:.001
0.092
Score,)
SDformerm
0.093+.000
0.037s.000
0.1 092.020002
0.0074.001
0.250:.000
0.091+.001
w/o Transfomee
0.0932.000
0.037:.000
0.1172.003
0.008+.001
0.250::.0M0
0.093+.001
SDformer-ar
0.0012.000
0.002::000
0.008:.001
0.005+.001
0.003++.000
0.015+.001
Context-FID
Transformer
0.003+.000
0.003+.000
0.009+.001
0.006+.000
0.004+.000
0.018+.001
Score
SDormer-m
0.010:.002
0.034a.008
0.0192.003
0.030:.003
0.041#.005
0.035
w/o Transformer
0.013+.001
0.016:.002
0.045+.005
0.031#.005
0.038:.008
0.044+.002
C Additional experimental results
Comparison inference time. To validate the superiority of our method in terms of inference time over DDPMs-based approaches, we conduct comparison between the inference times of SDformer and Diffusion-TS across datasets, as illustrated in Figure 5 By integrating Figure 5 and Table 2] it can be demonstrated that our approach possesses higher generation capability compared to the current DDPMs-based methods, while not necessitating the extensive inference time they require.
Effect of Transformer architecture. To investigate the contribution of the Transformer architecture SDformer, designed a variant that replaces the original Transformer architecture with LSTM architecture. Specifically, for SDformer-ar, we employ a unidirectional LSTM [15] to replace the Decoder-only Transformer. Conversely, for SDoormer-m, we utilize a bidirectional LSTM 30 to replace the Encoder-only Transformer, ensuring equivalent functionality. The term "w/o Transformer" Table denotes the variant of the corresponding method that employs LSTM architecture instead of Transformer architecture. Based on the results, the SDformer employing the LSTM architecture is somewhat less effective than one based on the Transformer overall, but the difference is generally not substantial. We believe this is because the time series tokenizer, which is based on similarity- driven vector quantization the first stage, has learned high-quality discrete token representations that are shorter and simpler than the original time series. As result, even when using simpler model the second stage, it can still learn the data distribution of the time series quite effectively.
Effect of model size. To examine the impact of model size on SDformer, Table 7 illustrates the performance of SDformer across various model sizes. is evident that with increasing model size, there is a noticeable enhancement in performance. Furthermore, investigate whether our SDformer remains competitive with other methods when the model size is low, we reduce model size by adjusting parameters such as the hidden dimension and code dimension, resulting in smaller version of the SDformer (SDformer-s) that has a model size comparable the current state-of-the-arr baseline, Diffusion-TT TS. We then compare the performance of SDformer-s with Diffusion- -SS, as demonstrated
14

---

Table Performance discrepancy of SDformer across different model sizes on the Energy Dataset.
Model Size (M)
1.4
3.0
11.9
44.9
Discriminative Score +
0.1492.007
0.084+.009
0.01
0.006+.004
Context-FID Score
0.033*.003
0.022+.002
0.0041.000
0.003+.000
Table 8: The comparison results of small version SDformer (SDformer-s) against the baseline Diffusion-TS on Sines, Stocks and ETTh datasets.
Metrics
Methods
Sines
Stocks
ETTh
Discriminative
SDformer-s
0.003+.003
0.019
0.0232.001
Score.
Diffusion-TS
0.0062.007
0.067*.015
0,061+.009
Context-FID Score4
SDformer-s
0.0062.000
0.015#.002
0.071+.001
Diffusion-TS
0.0062.000
0.1474.025
0.1162.010
Inference Time (s)
SDfommer-
2.68
2.59
2.67
Diffusion-TV
27.12
30.01
32.97
Model Size (M)
SDformer-s
0.17
0.16
0.28
Diffusion-TSS
0.24
0.29
0.35
in Table g We can observe that SDformer maintains competitive performance even with reduced model parameters and showcases notably faster inference times compared to Diffusion-TS.
D Model details
In this section, we present the detailed network architecture of the SDformer. The time series tokenizer in stage 1 primarily consists of ID Convolution and 1D ResNet networks, illustrated in Tables and [0] In stage 2, the Autoregressive Transformer and Masked Transformer are implemented as standard decoder-only and encoder-only Transformer, respectively. The specifics of their blocks depicted in Tables and 12
Table 9: The detailed architecture of the time serie tokenizer's encoder.
Layer
Function
Descriptions
Convolution
input channelzd, output channel= kemel size=3, stride=1, padding=1
ReLU
nn.ReLUO
Convolution
input channel=D, output channel=D, kernel size=1, stride=2, padding=! input channel=D), depth=3, dilation growth rate=3 un.ReLUO
ResNet
5
ReLU
Convolution
input channel=D, outpur channel=D) kernel sizc=4, strid==2, padding=1 inpul channel=D, depth=3, dilation growth rate=3 in.RcLUO
ResNet
ReLU
Convolution
input channel= de, output channel=H, kernel size=3, stride=1, padding=1
E
Algorithms
In this section, we detail the training and inference algorithms for SDformer. Specifically, Algorithm outlines the training procedure for the time series tokenizer the first stage, while Algorithm and elucidate the training processes for the Autoregressive Transformer and Masked Transformer the second stage, respectively. I Lastly, Algorithm 4 and represent the inference procedures for 5
SDformer-ar and SDformer-m. respectively.
15

---

Table 10: The detailed architecture of the time serie tokenizer's decoder.
Layer
Function
Descriptions
Convolution
input channel=de, oulput channel=D, kernel size=3, stride=1, padding=1
ReLU
un.Rel.UO)
ResNet
input channel=D, depth=3, dilation growth rate=3
ReLU
nn. un.ReLUU
5
Upsample
nn. Unsamppeel
Convolation
input channel-D. output channel=D, kernel size=3, stride=1, padding=1
ResNet
inpul channel=D, depth=3, dilation growth rate=3 un.ReLUO
ReLU
Upsample
un.Upsample))
10
Convolation
input channel=D, output channcl=D, kernel size=3, stride=1, padding=l
ReL.U
nn.Rel.U()
Convolution
input channel=D, output channel=D. kemell size=3, stride=l,, padding=1
13
ReLU
nn.ReLUO)
14
Convolution
input channel=D, output channel=d, kernel size=3, stride=1, padding=1
Table 11: The detailed architecture of the Autoregressive Transformer block.
Layer
Function
Descriptions
1
Layernorm
mo.LayerNomm)
2
Casual-attention
CasualAttenniion(qxxx, k=x, v=x)
3
Layernorm
m.LaaerNormm)
MLP
nn.ean)
5
ReLU
bn.ReLUO
6
MLP
an.Line
Table 12: The detailed architecture of the Masked Transformer block.
Layer
Function
Descriptions
1
Layernorm
nn.LayerNormo)
2
Self-attention
Attention(q=x. k=x, v=x) nn.L.ayerNorm) nn. nn.Line
3
Layernorm
4
MLP
5
ReLU
nn. ReLUO
6
MLP
nn an.car)
Algorithm 1 Training of time series tokenizer.
Input: Time series dataset D {Xirr H
Output:
Encoder E, Decoder ID and codebook C.
1: for +- to K do
2: Get the X1.r D;
3: Feed the X1.r to Encoder and get the HIL:;
4: Get the YI.L by Equation
5: Get the H1s by Equation (5)
6: Feed the HI.L to Decoder D and get the X
7: Compute the training loss L by Equation 66:
8: Complete backpropagation process based L and update the parameters of & and D;
9: Use HI.L to update the codebook through exponential moving average;
end for
11: Return trained E, D, and C.
16

---

Algorithm 2 Training of Autoregressive Transformer.
Input:
Time series dataset D = [Xi=> ippiizzee
optimized time series tokenizer.
Output: Autoregressive Transformer Par.
1: for < to K do
2: Get the discrete tokens YI.1 by & and Equation
3: Get the shifted tokens Y by shifting YI.L and appending [BOS] token;
4: Update I.f. by applying random replacement via Equation (8):
5: Feed the 1EL Autoregressive Transformer Gari
6: Compute the training loss Lar by Equation 7
Complete backpropagation process based Car and update the parameters of Gari
8: end for
9: Return trained Gar.
Algorithm 3 Training of Masked Transformer.
Input: Time series dataset D {x M optimized time series tokenizer.
Output: Masked Transformer 9m.-
1: for k 4- 1 to K do
2: Get the discrete tokens YI.. by and Equation 4
Get the masked tokens YI.Z by randomly masking YI.L:
4: Feed the Y I. to Masked Transformer 9m:
5: Compute the training loss Lmask by Equation P
6: Complete backpropagation process based Lmask and update the parameters of Gmi
7:
end for
8: Return trained Gm.
F Limitations
the goal is achieve superior generation capabilities, the time series tokenizer must possess relatively large codebook and a higher number of parameters, However, this will result in increased memory pressure.
G Additional visualizations
To visualize the performance unconditional time series generation, we adopt three visualization methods: 1) projecting original and synthetic data in 2-dimensional space using t-SNE [32]; 2) projecting original and synthetic data in 2-dimensional space using Principal Component Analysis (PCA); 3) drawing data distributions using Kernel Density Estimation (KDE). Figure 3 along with Figures thtrough display the visual outcomes of unconditional generation. evident that the
Algorithm 4 Inference process of SDformer-ar.
Require: The token sequence length E Σ, codebook size K E (0) [K]
[BOS] token as starting input.
2: for E (0,1, L 1) do
3: Liagits + Gar (X );
> Logitts E R (++1)x Categorical sampling, Iampled (o,... -1
4: + softmax |
> p e R
5: Isampled Sample(p);
6:
r((+1)
( Isampteal;
D Concatenate the newly sampled token Dequantization in Equation
7: end for
8: H1.L Q 1-1 :]);
(5), H E RLdd
9: XI.- D(HH::):
& X1.r E Bexd
10: return X1.T-
> Return final output.
17

---

Algorithm 5 Inference process of SDformer-m.
Require:
The token sequence length L E &, codebook size K E E, iteration number N E Z, and mask schedule S Z.N
y K. 1
1 an all-ones vector
2: for t (0,1,...., -W-1
1 do Ligits t 9m (F()))
Logits E RR LLLK
3:
4:
p p+ softmax (Llosta
p E R
5:
Teampled* + Sample(p): Ps - slaampptedl
b Categorical sampling, Isampled E (0. ... K 13
6:
b Ps E IR indicates the probabilityy of the sampled tokens
7:
Ps - where(P + K. *.11L.Ph ps):
Assign probability of to unmasked tokens Find the (S[t) 1)-th smallest value from p. > Mask low-probability tokens
8:
Tihreshold sorted (p))St +1;
9:
Lammpted + where Pa < Tihreshold, K 1x, sampled));
Y ((++)) - where(Y (1) # K, k,Y(,) lsamptea);
10:
Update masked tokens
end for
H.L = Q Q-1(x((M):
Dequantization in Equation (5), H R2ddd X lor Brxd
X 1:r I(H
return X 1:r-
Return final output
SDiormer-m SDfomer-aa
Diffusion-TT TS
(a) Sines
(b) Stocks
(c) MuJoCo
(d) IMRI
Figure 6: t-SNE visualizations of the time series synthesized by SDformer and Diffusion-TT on the Sines, Stocks, MuJoCo and IMRI datasets.
data generated by SDformer more closely aligns with the actual data compared to that produced by Diffusion-TS.
Figures 9 through 12 show the prediction samples of SDformer in different datasets and different prediction lengths. The median values of forecasting are represented as the dotted line, and 5% and 95% quantiles are depicted as the shade areas. per the results, SDformer generates precise predictions in the majority of examples, signifying its strong performance in conditional generation tasks. Additionally, SDformer eliminates the need for retraining when undertaking these prediction tasks, emphasizing its remarkable versatility.
18

---

SDformer ar
SDforme--m
Diffusion- IS
(a) Sines
(b) Stocks
(c) MuJoCo
(d) fMRI
Figure 7: Kernel density estimation visualizations of the time series synthesized by SDformer and Diffusion- TS on the Sines, Stocks, MuloCo and fMRI datasets..
SDrormer-m SDformer-al
Diffusion- Is
(a) Sines
(b) Stocks
ETTh
(d) MuJoCo
Energy
fMRI
Figure 8: PCA visualizations of the time series synthesized by SDformer and Diffusion- TS across all datasets.
19

---

0.54
0.50
0.44
0.46
0.48
0.42
0.46
0.48
0.38
0.46
0.44
0.42
0.36
0.42
0.40
16
24
32
40
16
24
32
40
8
16
24
32
40
8
16
24
32
40
0.52
0.54
0.44
0.45
0.50
0.52
0.48
0.40
0.46
0.48
0.38
0.35
0.46
0.36
0.42
0.54
0.52
0.40
0.50
0.32
0.38
0.48
0.30
0.46
0.28
32
40
16
24
32
40
16
24
32
16
40
0.54
0.46
0.50
0.375
0.44
0.350
0.50
0.42
0.46
0.325
0.44
0.48
0.42
0.38
0.40
16
24
32
16
0.38
0.48
0.550
0.46
0.525
0.44
0.500
0.32
0.42
0.450
16
24
32
40
16
24
32
40
8
16
24
32
40
8
16
32
40
0.44
0.46
0.54
0.42
0.44
0.40
0.48
0.50
0.42
0.38
0.46
0.40
0.36
0.44
0.38
32
40
16
24
32
16
24
32
Figure 9: Examples of time series forecasting for the Energy dataset with a prediction length of 24. Green colors correspond to Predictions of SDformer.
20

---

0.46
0.52
0.42
0.48
0.40
0.46
0.44
0.42
0.36
16
24
32
40
16
24
32
40
16
24
32
40
16
24
32
40
0.42
0.48
0.50
0.35
0.46
0.30
0.36
0.46
0.25
0.44
0.52
0.44
0.50
0.46
0.44
0.42
0.44
0.40
24
32
40
16
24
32
40
8
16
24
32
40
16
24
32
40
0.425
0.42
0.550
0.52
0.525
0.400
0.375
0.48
0.38
0.350
0.450
0.325
0.44
0.34
0.425
24
32
16
0.52
0.50
0.52
0.48
0.50
0.46
0.48
0.44
0.46
0.42
16
24
32 40
16
24
32
40
8
16
24
32
40
8
16
24
32
40
0.62
0.60
0.44
0.46
0.40
0.38
0.42
0.56
0.44
0.36
0.40
0.54
0.42
0.34
0.52
0.38
32
40
16
24
32
40
16
24
32
Figure 10: Examples of time series forecasting for the Energy dataset with prediction length of 32. Green colors correspond to Predictions of SDformer.
21

---

0.625
0.625
0.60
0.600
0.58
0.56
0.575
0.575
0.55
0.54
0.550
0.550
0.52
0.525
0.525
0.50
0.500
0.500
0.60
0.600
0.55
0.54
0.550
0.50
0.52
0.50
0.50
8
16
24
32
40
16
24
32
40
16
24
32
40
8
16
24
32
40
3.600
0.500
0.625
0.60
0.680
0.625
0.575
0.575
0.575
3.550
0.550
0.525
0.50
0.525
0.500
0.500
0.500
16
24
32
40
3.600
0.625
0.600
Am
0.575
0.600
0.575
0.55
0.525
0.525
0.525
0,500
0.50
0.500
0.500
16
24
32
40
16
24
32
16
24
32
40
0.65
3.600
0.60
0.58
0.55
0.56
3.525
0.52
0.500
0.45
0.50
16
24
32
40
40
16
24
16
24
32
40
0.600
0.60
0.60
3.575
0.58
5.550
0.56
0.55
0.55
0.525
0.50
0.500
0.52
0.50
0.45
32
40
16
16
24
32
40
Figure 11: Examples of time series forecasting for the fMRI dataset with prediction length of 24. Green colors correspond to Predictions of SDformer.
22

---

0.600
0.60
0.600
0.575
0.55
0.550
0.550
0.525
0.50
0.525
0.500
0.500
16
24
32
40
16
24
32
40
8
16
24
32
40
8
16
24
32
40
0.625
0.58
0.575
0.55
0.550
0.56
0.550
0.54
0.50
0.525
0.525
0.500
0.500
0.50
0.60
0.625
0.60
WWW
0.600
0.60
0.58
0.55
0.575
0.55
0.550
0.54
0.50
0.50
0.525
0.52
0.500
16
24
32
40
16
24
32
40
16
24
32
40
8
16
24
32
40
0.600
0.625
0.58
0.600
0.56
0.575
0.550
0.54
0.550
0.50
0.52
0.500
0.50
16
3.625
0.600
0.60
0.575
0.575
0.550
0.550
0.55
0.525
0.50
0.50
0.500
0.500
16
24
32
40
16
40
8
16
24
32
40
8
16
24
32
40
0.60
0.60
1
WW
0.60
0.55
0.50
0.54
0.52
32
40
16
16
24
32
40
Figure 12: Examples of time series forecasting for the fMRI dataset with prediction length of 32. Green colors correspond to Predictions of SDformer.
23

---

1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have added the contributions and scope in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have added the limitations in Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
24

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
The experimental results all can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [NA]
Justification:
We will release all the experimental related codes upon acceptance.
25

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips.c http:///ggiiieee///oeuuuun/eeeee
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We have specified the details of experiment in Experimental section and Appendix.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
All the experimental results are reported with mean and variances.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided the required compute resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https htt///neurppppp http:/pp//pppppp....o////aaaaaaa
Answer: [Yes]
Justification:
The research is conform to the NeurIPS Code of Ehics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The societal impact of our work lies primarily in its potential to broaden the applicability of machine learning models in real-world scenarios. For example, in critical sectors such as transportation, energy, finance, and healthcare, the integration of advanced forecasting methods can provide decision-makes with substantial support, thereby facilitating the realization of intelligent decision decision-making ideals.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original paper that produced the code package and dataset.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "Zw9FdpuHeo", "reviewer": "Reviewer_h9mP", "summary": "This paper proposed a MAE type attention model for time series generation. One major advantage is its efficiency, which is much faster than the diffusion based method. Various experiments are conducted and the proposed model outperform all baselines.", "questions": "Please see the weaknesses part.", "limitations": "Please see the weaknesses part.\n\nAt the current stage, this paper proposed an interesting model in time series generation and I tend to recommend acceptance. However, I am open to reconsidering this assessment based on the further discussions.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Efficiency: The paper introduces SDformer, a novel approach to time series generation that addresses key limitations of Denoised Diffusion Probabilistic Models (DDPMs). The proposed model demonstrates superior computational efficiency compared to diffusion-type models, which is a significant advancement in the field. \n\n2. Methodology: SDformer's two-step process, combining similarity-driven vector quantization and discrete Transformer modeling, presents an innovative solution to the challenges of long inference times and quality improvement in time series generation. \n\n3. Empirical Results: The numerical results presented in the paper are compelling, showing significant improvements over competing approaches in terms of generated time series quality and inference time. This empirical evidence strongly supports the effectiveness of the proposed method.", "weaknesses": "1. Reproducibility: One concern of the current submission is the lack of sample code provided to reviewers for verifying the reported experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive reviews and suggestions. In the following, we will answer your question.\n\n**Q1.** One concern of the current submission is the lack of sample code provided to reviewers for verifying the reported experimental results.\n\n**A1.** Thank you for your suggestion. We have provided the anonymous link containing the sample code to the AC. This will allow you to directly verify our experimental results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LTyT0Epcgi", "reviewer": "Reviewer_t6Rv", "summary": "This paper introduces SDformer, a method for time series generation, which addresses challenges in inference time and quality improvement. It utilizes a similarity-driven vector quantization technique to learn high-quality discrete token representations of time series. It employs a discrete Transformer for data distribution modeling at the token level. Experimental results demonstrate that SDformer outperforms competing approaches in generating high-quality time series while maintaining short inference times.", "questions": "As I stated above in the weakness, what's the exact limitation of the proposed model?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of similarity-driven vector quantization approach is novel and makes sense.\n\n2. The experiment conducted is adequate and the results are impressive. The analysis of the experimental results is detailed.\n\n3. The paper is well-written and the presentation is good.", "weaknesses": "1. The description of the proposed method is not very detailed.\n\n2. The limitations of the method have not been well explained. It should be compared with other methods, rather than just stating that achieving better generation results will lead to an increase in the number of parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive reviews and suggestions. In the following, we will answer your questions one by one.\n\n**Q1.** The description of the proposed method is not very detailed.\n\n**A1.** Thank you for your feedback. SDformer is a two-stage method. \n* **First Stage:** In the first stage, we develop a variant of VQ-VAE [1], introducing an alignment method based on maximizing the similarity between discrete and continuous representations, abandoning the distance-minimization alignment approach used in VQ-VAE, as shown in Eq. 5. To further enhance the precision and stability of the training process, we apply Exponential Moving Average (EMA) for codebook updates and reset inactive codes (Code Reset) during training, which is included in line 48-50 of introduction. Also, the specific structures of the encoder and decoder in the first stage are presented in Tables 8 and 9 in Appendix D, and the detailed training process for the first stage is described in Algorithm 1.\n* **Second Stage:** In the second stage, we use the trained encoder and tokenizer in the first stage to obtain the discrete representation of the target time series, as shown on the right side of Figure 1. In this stage, we employ two different strategies: autoregressive (SDformer-ar) and non-autoregressive (SDformer-m) using a Transformer structure to accomplish the task of generating discrete indexes. The Transformer structures for the two different strategies in the second stage are presented in Tables 10 and 11 in Appendix D. The detailed processes for training and inference are described in Algorithms 2-5. \n\nTo further address your concern, we would appreciate it if you could specify the aspects of the method description that you found unclear or insufficient. This will allow us to better understand your concerns and enhance the clarity and detail of our method description accordingly.\n\n**Q2.** The limitations of the method have not been well explained. It should be compared with other methods, rather than just stating that achieving better generation results will lead to an increase in the number of parameters.\n\n**A2.** Thank you for your suggestion. The table below shows the Context-FID Score of SDformer and Diffusion-TS at varying parameter quantities. While SDformer demonstrates competitive performance with smaller parameters, it’s evident that there is significant potential for improvement as the parameters increase. This suggests that achieving higher performance may require managing increased memory demands. Conversely, using smaller parameters could result in a notable drop in performance. \n\nOn the other hand, Diffusion-TS shows a gradual performance improvement initially, but with a large enough parameter quantity, its performance may become distorted. This indicates that one can opt for smaller parameters in Diffusion-TS without a substantial loss in performance. We will incorporate these findings into our paper to better address our limitations. Your insights are highly appreciated and play a crucial role in improving our manuscript.\n\n| Model Size (M) | [1.1-1.5]  | [3.0-3.1]  | [11.9-12.1] | [44.9-48.0] | \n|----------------|------------|------------|-------------|-------------|\n| SDformer       | 0.033±.003 | 0.022±.002 | 0.004±.000  | 0.003±.000  |\n| Diffusion-TS   | 0.089±.003 | 0.071±.023 | 0.060±.015  | 2.002±.339  |\n(e.g. [1.1-1.5]:The model size is between 1.1M and 1.5M.)\n\n**Q3.** What's the exact limitation of the proposed model?\n\n**A3.** The SDformer has a slightly larger parameters to ensure high accuracy, which results in relatively higher memory requirements. In future work, we will explore how to maintain the model's accuracy while reducing the parameters.\n\n[1]. Van Den Oord, Aaron, and Oriol Vinyals. \"Neural discrete representation learning.\" Advances in neural information processing systems 30 (2017).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MKaiiYpces", "reviewer": "Reviewer_2bQV", "summary": "In this paper, the authors propose a time series generation model named SDformer. The model is based on the discrete token modeling techniques and demonstrate empirically its feasibility for the time-series generation task. The method presented in the paper surpasses the current state-of-the-art models on multiple metrics and datasets.", "questions": "- How did the authors choose the size of the codebook K?\n- Code collapse is mentioned in the intro and not discussed further. What does it mean?\n- Adding more precise descriptions for implementing the random replacement would be helpful for reproduction. For instance, how do you choose the value of $\\pi$? Is it a hyperparameter?\n- When zero-shot forecasting, it can be helpful to compare with some baseline. Is the diffusion-TS trained specifically in the conditional case? or is it compared to the same setting where you train for unconditional generation and perform zero-shot forecasting? The qualitative examples are nice but do not give us a thorough understanding of the model performance in that case.\n- Why the unconditional results in Fig. 4 (bottom) have value > 0.5?\n- For the long-range benchmark, I think it would be beneficial if the authors compare their results against [1]. The results of TimeVAE and TimeGAN can degrade as the length of the sequence grows merely due to the fact that they rely on LSTM\\GRU at their backbone, which can be replaced easily for an encoder handling long-sequences.\n- The authors do not compare the number of parameters for their original results. For a fair comparison, the authors should refer to Table 7 in the main text. In addition, can you add the results of the reduced size model on Energy and MuJoCo to Table 7 and compare the full model number parameters?\n- Some time-series generation-related work and comparisons are missing. For instance, [2,3]. \n- I think the authors should also compare with [3]. Its performance is stronger than Diffusion-TS for some datasets and is much smaller regarding parameters and inference time. \n\n\n\n[1] Zhou, Linqi, et al. \"Deep latent state space models for time-series generation.\" \n\n[2] Jeon, Jinsung, et al. \"GT-GAN: General purpose time series synthesis with generative adversarial networks.\"\n\n[3] Naiman, Ilan, et al. \"Generative modeling of regular and irregular time series data via koopman VAEs.\"", "limitations": "See the Weaknesses and Questions sections.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- In general, the paper presents interesting results that empirically show that the discretization of time series is very effective for the task of generation, which is a challenging problem.\n- The method presents strong empirical results over various number of metrics.", "weaknesses": "Overall, employing transformers for time series data generation tasks is challenging, and there are only a few works in this direction.\nThere are some details that are missing and make this paper's results hard to reproduce (see Questions section).\nMy main concern is the lack of details in the paper and the missing of recent related work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive reviews and suggestions. In the following, we will answer your questions one by one.\n\n**Q1.** How did the authors choose the size of the codebook K. \n\n**A1.** K is a hyperparameter that requires experimentation to determine the optimal value. Specifically, we choose from {128, 256, 512, 1024}. The table below presents the Discriminative Score of SDformer under different K. Our final choice can be found in Appendix B (Experimental Settings).\n\n| K      | 128        | 256         | 512         | 1024       |\n|--------|------------|-------------|-------------|------------|\n| Stocks  | 0.012±.004 | 0.010±.006  | 0.010±0.007 | 0.010±.007 |\n| Energy | 0.102±.012 | 0.017±.006  | 0.006±.004  | 0.005±.004 |\n\n**Q2.** Code collapse is mentioned in the intro and not discussed further.\n\n**A2.** Code collapse occurs when only a small part of the codebook gets updated during training, which degrades the performance of the VQ-VAE. The table below shows how code collapse happens on the Energy dataset due to the lack of EMA and Code Reset. I'll include it in the introduction.\n\n|                        | Codebook Usage (%) | Reconstruction loss (MSE) |\n|------------------------|--------------------|---------------------------|\n| w/ EMA and Code Reset  | 100                | 0.0001                    |\n| w/o EMA and Code Reset | 3.3                | 0.0061                    |\n\n**Q3.** Adding more precise descriptions for implementing the random replacement would be helpful for reproduction.\n\n**A3.** I'll revise the original text as follows: \"In this approach, each token is processed individually. A random number is compared to a probability threshold, π. If it meets the threshold, the token is replaced randomly; otherwise, it remains unchanged.\" Pseudocode is provided below for clarification.\n```plaintext\nif Uniform(0,I) <= π:\n    y_i <- randint(0,K)\nelse:\n    y_i <- y_i\n```\nIn this pseudocode, Uniform(0,I) denotes a number randomly generated between 0 and 1, randint(0,K) signifies a token index randomly chosen from the codebook, and y_i represents the i-th token in the original sequence.\n\n\n**Q4.1.** When zero-shot forecasting, it can be helpful to compare with some baselines.\n\n**A4.1.** In our task setting, we focus on training for unconditional generation and performing zero-shot forecasting. As a result, it would be unjust to directly compare our approach with specialized forecasting models, which are typically trained exclusively on conditional generation tasks. However, if you have any suggested baselines, we would be more than happy to conduct a comparison with them.\n\n**Q4.2.** Is the diffusion-TS trained specifically in the conditional case? or is it compared to the same setting where you train for unconditional generation and perform zero-shot forecasting? \n\n**A4.2.** For the prediction task, our settings are the same, both training for unconditional generation and performing zero-shot forecasting. \n\n**Q4.3.** The qualitative examples are nice but do not give us a thorough understanding of the model performance in that case.\n\n**A4.3.** The qualitative examples in Fig. 3 and 9-12 are randomly selected instances from the prediction task, with their corresponding quantitative metrics shown in Fig. 4.\n\n**Q5.** Why the unconditional results in Fig. 4 (bottom) have value > 0.5?\n\n**A5.** Because the evaluation metrics for the prediction task and the unconditional generation task differ. To display them on the same bar chart, we have scaled the results accordingly, which is indicated in the caption of Fig. 4. \n\n**Q6.** For the long-range benchmark, comparing with LS4 would be beneficial. \n\n**A6.** Regarding LS4, we have attempted to reproduce the results but faced unforeseen environmental issues and time constraints. We are continuing to address these challenges and will update you promptly with any progress.\n\n**Q7.1.** The authors do not compare the number of parameters for their original results.\n\n**A7.1.** The parameter counts for our models are detailed in the following table, which we'll update in the main text.\n\n| Dataset        | Sines  | Stocks | ETTh   | MuJoCo | Energy | fMRI   |\n|----------------|--------|--------|--------|--------|--------|--------|\n| Model Size (M) | 45.9   | 44.4   | 95.3   | 44.9   | 44.9   | 36.9   |\n\n\n**Q7.2.** Can you add the results of the reduced size model on Energy and MuJoCo？\n\n**A7.2.** The table below presents the results of the reduced-size model on the Energy and MuJoCo datasets. It can be observed that the smaller SDformer model still maintains a performance advantage.\n\n|                      | Context-FID Score↓ | Model Size (M) |\n|----------------------|--------------------|----------------|\n| Energy: SDformer-s   | 0.033±.003         | 1.41           |\n| Energy: Diffusion-TS | 0.089±.003         | 1.14           |\n| MuJoCo: SDformer-s   | 0.008±.001         | 0.34           |\n| MuJoCo: Diffusion-TS | 0.013±.001         | 0.36           |\n\n**Q8.** Some time-series generation-related work and comparisons are missing, such as KoVAE and GT-GAN. \n\n**A8.** The table below shows the results of my reproduction of KoVAE and GT-GAN in the same environment. Some results differ from those in the original paper. We are in contact with the corresponding authors and will provide updates as they come.\n\n|               | Sines      | Stocks     | Energy     | MuJoCo     |\n|---------------|------------|------------|------------|------------|\n| DS: SDformer  | 0.006±.004 | 0.010±.006 | 0.006±.004 | 0.008±.005 |\n| DS: KoVAE     | 0.005±.004 | 0.027±.021 | 0.256±.025 | 0.052±.006 |\n| DS: GT-GAN    | 0.044±.025 | 0.012±.059 | 0.476±.006 | 0.246±.022 |\n| FID: SDformer | 0.001±.000 | 0.002±.006 | 0.003±.000 | 0.005±.005 |\n| FID: KoVAE    | 0.011±.001 | 0.074±.018 | 0.192±.035 | 0.034±.005 |\n| FID: GT-GAN   | 0.121±.029 | 0.171±.041 | 0.542±.062 | 0.547±.033 |\n(DS: Discriminative Score, FID: Context-FID Score)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yt2Hbw41oe", "reviewer": "Reviewer_DHCH", "summary": "This paper investigates the use of a vector-quantization transformer for time series generation. The authors propose a similarity-driven vector quantization method for time series modeling. After training the encoder and decoder structures for generating embeddings, the authors propose two ways to generate time series: auto-regressive and masking. They empirically prove their idea across various settings, including different setups and datasets.", "questions": "• First of all, the idea is novel and distinguishable from previous time series generation models. I wonder about the motivation behind this work.\n\n• For conditional time series generation, not for imputation or forecasting where other time series values are conditioned, generating the time series for other conditions such as trend or global minimum. Can the authors suggest an idea to use their method for these kinds of generation tasks? In my opinion, VQ-based methods can be harder to control conditionally than diffusion frameworks. Please correct me if I have any misunderstandings.\n\n• In terms of mask selection for masked token modeling, time series models often use special masking, such as geometric masking [1]. Randomly selected masks cannot fully learn the properties of the time series domain.\n\n• In the proposed vector-quantized samples, is there any relationship between the distance of the embedding and the generated data samples, i.e., do small distances in vectors really exhibit similar time series data after decoding?\n\n• Can the authors suggest more ablation study about model size and code embedding size?\n\n[1] On the Constrained Time-Series Generation Problem, NeurIPS 2023\n\n[2] A Transformer-based Framework for Multivariate Time Series Representation Learning, KDD 2021", "limitations": "The authors clarify the limitations in the Appendix.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "• The paper investigates the use of vector-quantization transformers, similar to MUSE or DALLE. This method is popular in LLMs; however, existing time series generation frameworks largely follow the image domain, such as diffusion models.\n\n• The authors propose a similarity-driven vector quantization method, which is much more effective for time series than existing methods.\n\n• The paper conducts experiments on various datasets, including well-known ones, demonstrating the effectiveness of the vector-quantization methods.\n\n• The authors establish a time series forecasting task as a conditional time series generation and show the effectiveness of their methods.", "weaknesses": "Please refer to the Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive reviews and suggestions. In the following, we will answer your questions one by one.\n\n**Q1.** I wonder about the motivation behind this work.\n\n**A1.** Our motivations include the following three points:\n* **Inference efficiency.** The inference time of previous SOTA models based on DDPMs is lengthy. To reduce this, we use vector quantization to compress the length of time series in latent space, significantly speeding up the second-stage Transformer. Additionally, non-autoregressive masked token modeling further decreases inference time.\n* **Generation quality.**  There is room to improve the quality of generated time series. To address this, we use similarity-driven vector quantization to capture correlations within time series, resulting in higher-quality discrete token representations. Additionally, we apply NLP techniques like autoregressive and masked token modeling to better learn time series distributions, effectively enhancing the quality of the generated time series.\n* **Compatibility with LLMs.** In recent years, LLM-related technologies have advanced rapidly. By aligning the format of time series data with natural language data (i.e., learning discrete time series tokens), we can directly leverage these advancements to tackle time series tasks. Our work confirms the feasibility of this approach and sets the stage for future research on integrating time series with LLM technology.\n\n**Q2.** Can the authors suggest an idea to use their method for other conditional time series generation, such as trend or global minimum?\n\n**A2.** You are correct that, compared to diffusion frameworks, VQ-based methods face greater challenges in handling conditional information, making this a valuable research direction. One approach could be to replace the [BOS] token in autoregressive token modeling with conditional input information from a conditional encoder. For instance, during training, we could input the trend or global minimum into the encoder and integrate it into the Transformer for joint training.\n\n**Q3.** In terms of mask selection for masked token modeling, time series models often use special masking, such as geometric masking. Randomly selected masks cannot fully learn the properties of the time series domain.\n\n**A3.** Thank you for your insight. Geometric masking is indeed more appropriate than random masking for capturing the properties of the time series domain. In time-domain datasets, a single missing point can often be accurately predicted using surrounding values, such as through interpolation, without requiring complex neural network modeling. Thus, such masking instances do not contribute to learning the time series data distribution. However, for SDformer-m, the processed data is a token sequence, where each token represents $\\( r \\times d \\)$ data points from the original data space, with  $r$ being the temporal downsampling factor and $d$ the number of time variables. In this case, accurate prediction using adjacent tokens via interpolation or other methods is not feasible, necessitating neural network modeling. Additionally, we compare the impact of the two masking strategies on the Context-FID Score, as shown in the table below. The results indicate that random masking is more effective for token sequence modeling.\n\n| Method | randomly masking | geometric masking |\n|--------|------------------|-------------------|\n| Stocks | 0.034±.008       | 0.107±.006        |\n| Energy | 0.041±.005       | 0.065±.005        |\n\n\n**Q4.** In the proposed vector-quantized samples, is there any relationship between the distance of the embedding and the generated data samples, i.e., do small distances in vectors really exhibit similar time series data after decoding?\n\n**A4.** Yes, typically, the closer the latent vectors are, the more similar their decoded time series data will be. To validate this, we construct positive and negative sample pairs. Positive samples represent data pairs that belong to the same token after quantization (i.e., the latent vectors are similar), while negative samples represent data pairs that do not belong to the same token after quantization (i.e., the latent vectors are dissimilar). Subsequently, we compute the average time series distances of the positive and negative sample pairs after decoding, as shown in the table below. It is evident that the average distance of the positive sample pairs is significantly smaller than that of the negative sample pairs, indicating that the closer the latent vectors are, the more similar their decoded time series data will be.\n\n| Data             | Positive Sample Pairs | Negative Sample Pairs | Mixed Samples Pairs |\n|------------------|-----------------------|-----------------------|---------------------|\n| Average Distance | 0.005938              | 0.072874              | 0.072730            |\n\n\n**Q5.** Can the authors suggest more ablation study about model size and code embedding size?\n\n**A5.** Thank you for your suggestion. We will incorporate these ablation studies into the main text. The following two tables show the Context-FID Score of SDformer on the **Energy** dataset with varying model sizes and code embedding sizes, respectively. It can be observed that as the model size increases, the performance improves significantly. Moreover, a larger code embedding size also contributes to performance enhancement, albeit to a limited extent.\n\n| Model Size (M)     | 1.4        | 3.0        | 11.9       | 44.9       |\n|--------------------|------------|------------|------------|------------|\n| Context-FID Score↓ | 0.033±.003 | 0.022±.002 | 0.004±.000 | 0.003±.000 |\n\n| Code Embedding Size | 32           | 64           | 128          | 256          | 512          |\n|---------------------|--------------|--------------|--------------|--------------|--------------|\n| Context-FID Score↓  | 0.0039±.0001 | 0.0035±.0004 | 0.0029±.0004 | 0.0031±.0002 | 0.0026±.0002 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:

- The proposed method, SDformer, addresses key limitations of existing DDPM-based models for time series generation - it improves inference time and generation quality.
- SDformer utilizes a novel similarity-driven vector quantization approach to learn high-quality discrete token representations of time series, followed by a discrete Transformer for data distribution modeling. This is a novel and promising approach.
- The paper is generally well-written and the presentation is good.

Weaknesses:
- The description of the proposed method is not very detailed, and some aspects could be explained better for reproducibility.
- The limitations of the method, such as the relationship between model size/parameters and performance, have not been fully explored.
- There are some discrepancies between the results reported for baseline methods (KoVAE, GT-GAN) and the original papers.
- The comparison to specialized time series forecasting models is limited, as the paper focuses on unconditional generation.

Based on the overall positive reviews and the novel contributions of the work, I would recommend accepting this paper. The authors have provided satisfactory responses to the reviewers' questions and concerns, and have agreed to incorporate more details and analysis into the final version.
Decision: Accept (poster)