Utilizing Image Transforms and Diffusion Models for  Generative Modeling of Short and Long Time Series

Authors: ilan  naiman
nimrod  berman
itai  pemper
idan  arbiv
gal  fadlon
omri  azencot
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 20:04:30

Abstract: Lately, there has been a surge in interest surrounding generative modeling of time series data. Most existing approaches are designed either to process short sequences or to handle long-range sequences. This dichotomy can be attributed to gradient issues with recurrent networks, computational costs associated with transformers, and limited expressiveness of state space models. Towards a unified generative model for varying-length time series, we propose in this work to transform sequences into images. By employing invertible transforms such as the delay embedding and the short-time Fourier transform, we unlock three main advantages: i) We can exploit advanced diffusion vision models; ii) We can remarkably process short- and long-range inputs within the same framework; and iii) We can harness recent and established tools proposed in the time series to image literature. We validate the effectiveness of our method through a comprehensive evaluation across multiple tasks, including unconditional generation, interpolation, and extrapolation. We show that our approach achieves consistently state-of-the-art results against strong baselines. In the unconditional generation tasks, we show remarkable mean improvements of $58.17$% over previous diffusion models in the short discriminative score and $132.61$% in the (ultra-)long classification scores. Code is at https://github.com/azencot-group/ImagenTime.

Main paper: 
 introduction: 
 Generative modeling of real-world information such as images [72] , texts [13] , and other types of data [99, 55, 8] has drawn increased attention recently. In this work, we focus on the setting of generative modeling (GM) of general time series information. There are several factors that govern the complexity required from sequential data generators including the sequence length, its number of features, the appearance of transient vs. long-range effects, and more. Existing generative models for time series are typically designed either for multivariate short-term sequences [44, 19] or univariate long-range data [103] , often resulting in separate and completely different neural network frameworks. However, a natural question arises: Can one develop a unified framework equipped to handle both high-dimensional short sequences and low-dimensional long time series? Earlier approaches for processing time series based on recurrent neural networks (RNNs) handled short sequences well [62, 3, 43, 76] , however, modeling long-range dependencies turned out to be significantly more challenging. Particularly, RNNs suffer from the well-known vanishing and exploding gradient problem [9, 70] that prevents them from learning complex patterns and long-range dependencies. To address long-context modeling and memory retention, extensive research is devoted to approaches such as long short-term memory (LSTM) models [42] , unitary evolution RNNs [5] transforms within diffusion models, including integration with advanced latent diffusion spaces [58] . Furthermore, combining time series images and Wasserstein GANs [12, 39] have been considered for generative modeling. Yet, representing general time series as images within diffusion models for tasks such as unconditional generation, interpolation, and extrapolation, remains largely underexplored. The goal of this work is to make a step toward bridging this gap.Diffusion models. Both denoising diffusion probabilistic models (DDPM) [81, 40] and score-based generative models [84, 85] have demonstrated their effectiveness across diverse domains including images [74, 41] , audio [15, 52] , and graphs [69, 97, 10] . Song et al. [86] showed that DDPM and score-based models can be both interpreted as stochastic differential equations (SDE). Further works focused on improving generation quality by using latent diffusion processes in autoencoder architectures [74] . Another research direction deals with lowering the number of neural function evaluations (NFEs), which originally ranged from hundreds to thousands NFEs. For instance, Karras et al. [45] obtain a low Fréchet inception distance (FID) with only 35 evaluations, whereas the recent consistency models [82] achieved comparable results with only a single function evaluation.Generative modeling of time series. Generative adversarial networks (GANs) [31] have shown remarkable success in generating realistic data across various domains. Specifically, their application to time series information by joint optimization of supervised and adversarial objectives via TimeGAN captured the inherent dynamics of real-world signals [99] . Similarly, GT-GAN [44] utilizes diverse tools including ordinary differential equations (DE) [16] , neural controlled DE [51] , and continuous time-flow processes to model both regularly-and irregularly-sampled data [21] . Nevertheless, GANs suffer from challenges, primarily due to unstable training dynamics and mode collapse [59] . Beyond GANs, variational autoencoders (VAEs) have been also considered for generative modeling of sequential data [22, 54, 73] , where the work [66] achieved strong results using Koopman-based approaches [7, 6, 65, 11] . To process long-range dependencies and stiff dynamics [79] , Zhou et al. [103] introduced LS4, a latent generative model based on linear state space equations. Following the success of diffusion models in other domains, there is a growing desire to adapt them for time series data. However, this adaption is not straightforward and entails the design of a suitable backbone [88, 57, 19, 100, 67] . Other approaches focused on regression problems [49] , based on manifold learning tools [47, 48] . Instead, we propose a new framework for generative modeling of time series by transforming such data to images and using existing strong diffusion vision models. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The conclusion section discusses the shortcomings of our framework and future improvements.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
background: 
 In what follows, we state the problem, we mention two effective time series to image transformations, and we briefly discuss the essentials of diffusion-based generative modeling.Problem statement. We address the problem of generating time series (TS), sampled from a learned distribution p(x) that is similar to an unknown distribution p(x), for which we have a set of observed TS data. The given observations include data samples x ∈ R L×K , where L represents the sequence length and K denotes the number of features. Formally, the generative modeling task is often termed "unconditional generation" [37] , and it entails learning a model M capable of sampling unseen time series x from p(x). Additionally, our work addresses a secondary problem known as "conditional generation". In this setting, given an additional signal c, we learn the unknown (conditional) distribution p(x|c). For example, the signal c can be an observed part from the TS. This conditional modeling proves useful for tasks such as time series interpolation and extrapolation. Time series to image transforms. We focus in our study on two invertible time series to image transformations: 1) the delay embedding; and 2) the short time Fourier transform. We provide below a brief overview of these transforms and their inverse. We consider additional transforms and we discuss more details in App. A. Fig. 3 illustrates a time series signal and its related images.Delay embeddings [87] transform a univariate time series x 1:L ∈ R L to an image by arranging the information of the series in columns and pad if needed. Let m, n be two user parameters representing the skip value and the column dimension, respectively. We construct the following matrix X,EQUATIONwhere q = ⌈(L -n)/m⌉. The image x img is created by padding with zeros to fit the neural network input constraints. Given x img , the original time series x 1:L can be extracted in multiple ways. For instance, if m = 1, then x 1:L is formed by concatenating the first row and last column of x img . The delay embedding scales naturally to long sequences, e.g., setting m = n = 256 allows to encode 65k sequences with 256 × 256 images.Short Time Fourier Transform (STFT) [35] is a well-known transformation that maps a signal from its original domain into the frequency domain. To preserve the temporal structure, STFT applies a rolling window on the time axis, extracting time series segments for which the fast Fourier transform (FFT) is applied. Given an input signal x ∈ R L×K , STFT produces an image x img ∈ R 2K×H×W , where the channels are doubled to store the real and imaginary parts, and H, W are derivatives of user parameters. STFT requires a minimum window length, and thus, short sequences may require a linear interpolation to match length constraints. Remarkably, the short time Fourier transform is invertible via reverse STFT with a negligible loss of information. Importantly, in contrast to the common practice in audio processing, we do not further compute the spectrogram of STFT, avoiding non-trivial inverse transformations.Diffusion models. Diffusion processes gradually add noise to an image, following a predefined noise scheduling scheme. Generating new images is possible by learning a model that removes noise. The diffusion process {x(t)} T t=0 is the path of a stochastic differential equation (SDE) [86] , where an initial sample x(0) is drawn from the data distribution p 0 (x). The initial sample is modified to x(T ), sampled from a simple prior distribution such as a normal Gaussian N (0, I). Formally, the forward process is governed by an SDE of the form,EQUATIONwhere f (•, t) : R d → R d represents the drift coefficient, g ∈ R is the diffusion scalar, and w denotes a standard Wiener process. To facilitate sampling, we need to derive the reverse SDE. It is well-known that the reverse process [4] is given by,EQUATIONwhere t denotes reverse time and w is a reverse Wiener process. Given Eq. ( 3), one can derive a deterministic process, characterized by trajectories that share identical marginal probability densities. Formally, we obtain the following ordinary differential equation (ODE),EQUATIONDiffusion models compute an estimator s θ (x, t) to approximate the infeasible ∇ x log p t (x) viaEQUATIONwhere p 0t denotes the joint distribution of the initial data and noisy sample. In practice, minimizing Eq. ( 5) is done by learning the noise pattern of input images. For a more comprehensive discussion regarding score-based models, we refer to [86, 45] . We specify in Sec. 4 the particular diffusion model employed in this work, along with further additional details.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [NA] Justification: All datasets are public, and the code will be publicly available at the end of the double-blind process. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
method: 
 Our approach to generative modeling of time series information is based on the following simple observation and straightforward idea. We observe that diffusion models for vision have demonstrated remarkable progress and results recently [40, 74, 72] . Therefore, our idea is to transform sequences to images, allowing their processing using established diffusion vision models. Fortunately, there are several efficient time series to image maps with effective inverse transforms [87, 35, 95, 56] . Our computational pipeline is composed of three main building blocks: 1 only learnable parameter-based part of our neural network. We illustrate our generative modeling framework for time series information in Fig. 1 , depicting the above building blocks with proper notations for inputs and outputs. Formally, given an input time series x ∈ R L×K with L the sequence length and K the number of features, we transform it to an image x img ∈ R C×H×W . Noise is added to the latter image yielding the tensor x img (t) which is processed with our diffusion model, whose output s(x img , t) ∈ R C×H×W represents the cleaned image. During inference, noise x img (T ) is sampled from N (0, I), iterated backward to x img (0) and transformed to a time series x ∈ R L×K .There are several options to choose from regarding the time series to image (ts2img) transform, T : R L×K → R C×H×W . While all transforms are applicable in our framework, we opt for ts2img maps that are efficient to compute, provide informative images, scalable across short and long sequences, and have a closed-form inverse. For instance, line graphs [56] are efficient with a closedform inverse, however, they produce images that are mostly non-informative as they contain blank pixels. Similarly, the Gramian angular field transform [95] essentially stores the sequence in its main diagonal, and thus, it is not straightforward to apply it to long-range data. In this work, we focus on using the delay embedding and short time Fourier transforms. Both ts2img maps satisfy all requirements above. Moreover, our empirical ablation analysis in Sec. 5.5 highlights that these transformations attain the best results on average. The inverse transforms T -1 for delay embedding and STFT are parameter-less, deterministic, and highly efficient. See Sec. 3 and App. A.At the heart of our ImagenTime framework lies the generative diffusion model backbone. Diffusion models for vision data have enjoyed increased attention over the past few years, with strong techniques appearing at an unprecedented rate [81, 84, 40, 85, 86, 74] . One limitation, shared among all diffusion models, is the requirement to iteratively denoise the image during inference, resulting in costly neural function evaluations (NFEs). While multiple works focused on alleviating this issue [77, 83, 28, 82] , the work by Karras et al. [45] offers an enhanced score-based model with a good balance between rapid sampling and high-quality generations. Specifically, they presented a clearer design space for the factors that determine the performance of diffusion models, and they suggested EDM that employs a second-order ODE for the reverse process, yielding low FID images in 35 NFEs. Thus, we utilize in this work the EDM diffusion model as our generative backbone.We conclude by briefly discussing the training and inference procedures. During training, we process batches of time series data X, for which we apply T using either delay embedding or STFT to obtain a batch of images, i.e., X img = T (X). Subsequently, we employ the training procedure of EDM to learn the score function s θ (X img , t). For inference, we use the trained EDM model and we compute the reverse ODE in Eq. ( 4) for sampling new data points. In practice, we follow the same inference procedure specified in [45] . Finally, given a batch of sampled images, Ximg , we apply the inverse transform T -1 to achieve a batch of generated time series samples, i.e., X = T -1 ( Ximg ). 
experiments: 
 We use standard unconditional and conditional quantitative and qualitative benchmarks to extensively validate our framework's ability to generate high-quality time series samples. First, we test our framework on short-term and long-term standard time series unconditional generation benchmarks (Sec. 5.1, Sec. 5.2). Then, we introduce a novel benchmark for ultra-long sequences (above 10k steps) and evaluate our method in comparison to strong baselines (Sec. 5.3). Then, we consider interpolation and extrapolation benchmarks, similar to [78] , to test our model on conditional generation tasks (Sec. 5.4). Further, we extended these benchmarks with additional short-and ultra-long setups, which test the framework's robustness to lengths. Finally, we conclude with an extensive ablation of our framework (Sec. 5.5). More details on the experimental settings can be found in App. B.Data, baselines, and metrics. We employ our framework on the unconditional generation benchmark reported in [19] . The benchmark includes four synthetic and real-world datasets with a fixed length of 24. The first dataset, Stocks, consists of daily historical Google stock data from 2004 to 2019, comprising six channels: high, low, opening, closing, and adjusted closing prices, as well as volume. This data lacks periodicity and is dominated by random walks. The second dataset, Energy, is a multivariate appliance energy prediction dataset [14] , featuring 28 channels with correlated features, and it exhibits noisy periodicity and continuous-valued measurements. The third dataset, MuJoCo (Multi-Joint dynamics with Contact), serves as a versatile physics generator for simulating TS data with 14 channels [89] . We report results on the simple synthetic Sine dataset of sine functions in App. C.1. Our framework is compared with state-of-the-art short-term time series generative models. KoVAE [66] , DiffTime [19] , GT-GAN [44] , TimeGan [99] , RCGAN [25] , C-RNN-GAN [64] , T-Forcing [33] , P-forcing [32] , WaveNet [91] , WaveGAN [23] , and LS4 [103] , which is the state-of-the-art generative model for modeling long sequences. The benchmark employs two metrics:1) The Predictive (pred) metric assesses the utility of the generated data.2) The Discriminative (disc) metrics gauge the similarity of distributions using a proxy discriminator. For all experiments, we used the delay embedding transform with an embedding of n = 8 and a delay of m = 3, yielding a 8 × 8 image. We use 18 sampling steps with the EDM model [45] as the diffusion generative backbone.Quantitative and qualitative results. The results for the short-term unconditional benchmark are shown in Tab. 1. Our framework achieves state-of-the-art results on all datasets and metrics. Particularly, we note MuJoCo, where we improved the second-best method by 88% and 21% in the discriminative and predictive scores. In general, the second-best approach is DiffTime. Importantly, while LS4 performs well on long sequences, our results indicate that it struggles with short sequences.In comparison, we will show below that in addition to obtaining SOTA results on short-term time series, we also achieve strong results in the long-term case (Sec. 5.2). We also evaluate our method using two common qualitative tests [99] . First, we compute a two-dimensional t-SNE [92] embedding for real and synthetic data. The desired outcome is that both datasets span similar regions and shapes in 2D. We plot the embeddings of the real data, ours and GT-GAN in Fig. 2A , highlighting that our generated point clouds are closer to the real data in comparison GT-GAN. Tab. 11 reports the Wasserstein distances between the t-SNE embeddings of the generated data and the real data, showing that our approach is superior to GT-GAN. Second, we estimate the probability density functions in Fig. 2D . Our approach generates densities similar to the real densities, whereas GT-GAN introduces noticeable errors. The rest of the short-term datasets' qualitative analysis appears in App. C.3. Data, baselines, and metrics. We utilize the long-term time series benchmark presented in [103] .It includes three long-term real-world time series datasets obtained from the Monash Time Series Forecasting Repository [29] : FRED-MD, NN5 Daily, and Temperature Rain. We omit Solar Weekly as it is short-term. These datasets were chosen based on their average 1-lag autocorrelation metric, measuring their correlation over time. Their 1-lag values range from 0.38 to 0.98, highlighting a diverse range of temporal dynamics that present challenges for generative learning tasks. Each dataset contains approximately 750 time steps. We compare our method with state-of-the-art long-term generative methods: LS4 [103] , SaShiMi [30] , SDEGAN [50] , TimeGAN [99] , Latent ODE [75] , ODE 2 VAE [98] , GP-VAE [27] and RNN-VAE [18] . Three different metrics are used to evaluate the generative performance: Marginal (marg), Classification (class), and Prediction (pred). Marginal scores measure the absolute difference between the empirical probability density functions of two distributions. Classification scores use a sequence model to classify samples as real or generated; high scores indicate less distinguishable samples. Prediction scores utilize a train-on-synthetic-test-on-real sequence-to-sequence model to predict future steps; lower scores indicate higher predictability. We used the STFT transform in all our experiments, creating a 32 × 32 size image. The number of sampling steps is 18 and we use the EDM model [45] as the diffusion generative backbone.Quantitative and qualitative results. We present the results in Tab. 2. LS4 performs well across most datasets and metrics. In comparison, our method outperforms LS4 and the other techniques in almost all cases. On NN5 Daily pred and on Temp Rain marg we achieve inferior results. Notably, Zhou et al. [103] discuss the challenge of measuring the marginal score for the Temp Rain dataset due to frequent zero values. We highlight that our approach substantially improves the classification and prediction scores for the Temp Rain dataset. Additionally, our framework achieves strong results in the classification scores for the FRED-MD and NN5 Daily datasets. We report our results with standard deviations in App. C.4; the results emphasize the statistical significance improvement our framework achieves. Our qualitative results are shown in Fig. 2(B, D ) and in App. C.3.Data, baselines, and metrics. We conclude our unconditional generation evaluation by considering the challenging setting of ultra-long sequences. As far as we know, this setup is underexplored in the literature, and moreover, considering short, long, and ultra-long time series for a single framework is novel to our work. Specifically, we use the following real-world datasets from the Monash Time Series Forecasting Repository [29] : San Francisco Traffic (Traffic) [53] and KDD-Cup 2018 (KDD-Cup) [61] . The datasets' lengths are 17544 and 10920, respectively. In addition to the unconditional generation benchmark we consider above, we also evaluate our approach on conditional generation tasks. We focus on the imputation (interpolation) and forecasting (extrapolation) tasks, following the experimental setup in [75, 78] . Our approach can be adapted to solve these tasks via a simple modification. For instance, in the interpolation task, the goal is to generate the missing values. Thus, we apply our diffusion model only in the missing locations using a corresponding mask. The rest of the values are left unchanged. A similar mechanism can be applied to extrapolation. Generally, this approach is similar to image inpainting techniques [60] . In the interpolation task, we randomly mask 50% of the sequence values, whereas in the extrapolation challenge, we split the sequence in half, where the second half represents the target values. In this benchmark, we consider short, long, and ultra-long sequences. Our comparison focuses on generative methods that can handle long-range dependencies including ODE-RNN [75] , Latent ODE [75] , CRU [78] , and LS4 [103] . Further details about the experiments can be found in App. B.Datasets. In the short-term setting, we use ETT* datasets [102] , that contain electricity loads of various resolutions (ETTh1, ETTh2, and ETTm1, ETTm2) from two electricity stations. The sequence length is 96. For the long-term case, we utilize an established benchmark [75, 78, 103] , including the Physionet and USHCN datasets. The Physionet dataset [80] improve by two-and one-orders of magnitude on Physionet and USHCN, respectively. Finally, we also highlight our ultra-long interpolation results which are ≈ 4 times better than ODE-RNN. We conclude that our approach shows robustness to varying sequence lengths, presenting extremely strong results across several datasets and in comparison to state-of-the-art generative models.We conclude our empirical section by thoroughly inspecting different components of our framework. Specifically, we show that our approach is robust to different image resolutions (App. C.8). We also experiment with a range of hyper-parameters (App. C.9), demonstrating the stability of our approach.Our performance evaluation highlights that our method is comparable to LS4 in terms of training and inference time (App. C.10). Below, we ablate the effect of various image transforms on the performance in the unconditional test. We evaluate our model using four different transforms: folding, Gramian angular field (GAF), delay embedding (DE) and STFT, and we detail the results in Tab. 5.While DE and STFT are slightly better on short and long sequences, respectively, we emphasize that all other transforms perform reasonably well across the various datasets and metrics. GAF does not scale to long sequences as it produces huge images, and thus, it is omitted from the long-term test. We conclude that our framework is robust to the choice of image transformation.  
conclusion: 
 While new generative models for general time series data appear rapidly, the majority of existing frameworks are specifically designed to process either short or long sequences. The lack of a unified framework for varying lengths time series can be justified by the shortcomings of current available tools: gradient issues of recurrent networks, temporal computational costs of transformers, and limited expressiveness of state space models. In this work, we address this problem by introducing a novel generative model for time series based on signal-to-image invertible transforms and a vision diffusion backbone. The benefit of our approach is threefold: we exploit advanced diffusion models for vision, we seamlessly process short-to-ultra-long sequences, and we can utilize tools from the signal-toimage literature. We extensively evaluate our framework in the unconditional and conditional settings using short, long, and ultra-long sequences, considering multiple datasets, and in comparison to state-of-the-art models. Our experiments show the superiority of our framework, setting new SOTA results. Further, we demonstrate the robustness of our method through several ablation studies. Our approach requires slightly higher computational resources, which we leave for further consideration and future work. Finally, we believe that the proposed framework has the potential to be applicable in additional tasks including classification, anomaly detection, few-shot learning, and more generally, serve as a foundation model.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification:Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We give a detailed description for each domain transformation and its corresponding forward and inverse processes. We present in Fig. 3 a visual example of a time series and its various different image transformations.Folding is a simple naïve transformation. Given a time series x, we fold it into an image x img by starting from the first row on the left and continuing to the right, jumping to a new row whenever reaching the end of a row. Finally, if needed, we pad with zeroes at the end of the image. The inverse transformation back to the time series is simply taking the in-padded area of the image and unfolding it back to the time series. Although it is simple, this transformation can scale to very long sequences.Folding can be viewed as a particular example of delay embedding, as we detail below.Gramian Angular Field is introduced in [95] for subsequent imputation and classification tasks. It depicts a time series within a polar coordinate system rather than the usual Cartesian coordinates.In the Gramian matrix, each element corresponds to the cosine of the sum of angles. The inverse action, from the image to the time series, is to take the main diagonal. While being a very informative transformation, a major constraint of this transformation is that the height and the width are linear with the size of the time series, preventing it from scaling to long sequences.Delay Embedding [87] transforms a univariate time series x 1:L ∈ R L into an image by organizing the series' information into columns and padding as necessary. The hyperparameters for this transformation are m and n, where m represents the skip value and n is the column dimension. For a given arbitrary channel of a time series, the transformation constructs the matrix X as follows:X =    x 1 x m+1 . . . x L-n . . . . . . . . . . . . x n x n+m+1 . . . x L    ∈ R n×q ,where q = ⌈(L -n)/m⌉. The image x img is created by padding with zeros to meet the neural network input requirements. The example presented here is for a single channel; scaling to multiple dimensions is straightforward by concatenating each matrix X along another channel. Given an input signal x ∈ R L×K , the transformation produces an output x img ∈ R K×n×q . We pad the image with zeroes right after to create an x img ∈ R K×n×n .The original time series x 1:L can be reconstructed from x img by taking each marginal progression from the columns of the matrix X. There are various methods to reverse the transformation. For instance, if m = 1, x 1:L is formed by concatenating the first row and the last column of x img . The delay embedding naturally scales to long sequences; for example, setting m = n = 256 allows encoding 65k sequences with 256 × 256 images.Short Time Fourier Transform (STFT) [35] is a widely used transformation that converts a signal from its original time domain into the frequency domain. The process of computing STFT involves dividing a time-domain signal into shorter, fixed-length segments and then applying the Fourier transform to each segment individually. Given an input signal x ∈ R L×K , the STFT produces an output x img ∈ R 2K×H×W . In this output, the number of channels is doubled to store both the real and imaginary parts of the transformed signal, and H and W are determined by user-defined parameters. These parameters include n_fft, which specifies the size of the Fourier transform, and hop_length, which defines the distance between successive sliding window frames. Unlike typical audio processing practices, we do not compute the magnitude spectrogram from the STFT output. Instead, we retain both the real and imaginary components within the image, thereby avoiding the need for additional complex spectrogram estimation. This approach maintains the integrity of the full spectral information. After obtaining the STFT images, we normalize them to the range [-1, 1], ensuring that the data is scaled appropriately for subsequent processing.We use the same architectural backbone for all experiments: EDM [45] . We use the AdamW optimizer and train for 1000 epochs, although in practice, all models converged in the range 300-500 epochs. For each task, we elaborate on specific variations in settings and hyperparameters and provide additional information on the training and evaluation protocol.B.1 Short-term unconditional generation.Data. For the short-term unconditional generation (Sec. 5.1), we utilize four synthetic and realworld datasets with a fixed length of 24: Stocks, consisting of daily historical Google stock data from 2004 to 2019, comprising six channels: high, low, opening, closing, and adjusted closing prices, as well as volume. This data lacks periodicity and is dominated by random walks. The second dataset, Energy, is a multivariate appliance energy prediction dataset [14] , featuring 28 channels with correlated features, and it exhibits noisy periodicity and continuous-valued measurements. The third dataset, MuJoCo (Multi-Joint dynamics with Contact), serves as a versatile physics generator for simulating TS data with 14 channels [89] . The last dataset, Sine, is a multivariate simulated dataset, where each sample x i t (j) is defined as sin(2πηt + θ), where η is sampled from a uniform distribution [0, 1] and θ is sampled from a uniform distribution [-π, π], with five channels for j.Hyperparameters. We describe below in Tab. 6 the different hyperparameters used in our framework. For all datasets, we used the same default sampler of EDM [45] , and we mention the hyperparameters of their U-net model that we tune in our work. Please see [45] for further details about the U-net model hyperparameters. Evaluation. We utilize the benchmark proposed in [99] to evaluate short-term unconditional generation and adhere to its evaluation protocol. This protocol comprises two scores: a predictive score and a discriminative score. The predictive score assesses the utility of the generated data by training an independent prediction model on the generated data; superior generations result in better prediction scores for this model. The discriminative score evaluates the similarity of distributions using a proxy discriminator trained to distinguish between generated and original samples; higher scores indicate that the generative model has accurately captured the underlying distribution of the data. For more details about the evaluation protocol, please refer to [19] or [99] .B.2 Long-term unconditional generation.Data. In our exploration of long-term unconditional generation, we employ the benchmark for longterm time series data as presented in [103] . This benchmark encompasses three extensive real-world time series datasets from the Monash Time Series Forecasting Repository [29] : FRED-MD, NN5 Daily, and Temperature Rain. These datasets were meticulously selected based on their average 1-lag autocorrelation metric, which quantifies the 1-step correlation over time. The 1-lag values, ranging from 0.38 to 0.98, exemplify a broad spectrum of temporal dynamics, thereby presenting significant challenges for generative learning models. To ensure uniformity in the NN5 Daily and FRED-MD datasets, each sequence within these datasets is normalized such that each trajectory is centered at its mean and adheres to a normal distribution. This normalization approach is advantageous for datasets like NN5 Daily, where the minimum and maximum values can vary substantially across different data points. For the Temperature Rain dataset, sequences are scaled to the [0, 1] range, considering the data's consistently positive values and its tendency to cluster around the x-axis with occasional sharp spikes. Each dataset comprises approximately 750 time steps, providing a robust basis for evaluating long-term generative performance.Hyperparameters. We describe below in Tab. 7 the different hyperparameters used in our framework. For all datasets, we used the same default sampler of EDM [45] , and we mention the hyperparameters of the U-net model that we tune in our work, please see [45] for more details about these hyperparameters. In addition, For all long-term experiments, we use the AdamW optimizer with a weight decay of 10 -5 . Evaluation. To assess model performance, we follow the benchmark used in [103] . Our evaluation comprises classification and prediction models, each employing linear encoders and decoders with a single S4 layer having 16 hidden state dimensions. In the classification model, the encoder maps data dimensions to 16 hidden states. The S4 layer's output sequence is averaged before being passed to the decoder, which produces logits for binary classification using cross-entropy loss. Similarly, the prediction model's encoder maps input to a 16-dimensional hidden state, while the decoder maps it back to the original data dimension, predicting k = 10 future steps. Both models are trained using the AdamW optimizer, which has a learning rate of 0.01 over 100 epochs and a batch size of 128.The optimizer generates samples equal to testing data points to train the models together.Data. For the ultra-long-term unconditional task, we introduce a novel benchmark consisting of two datasets: San Francisco Traffic (Traffic) [53] and KDD-Cup 2018 (KDD-Cup) [61] . The datasets' lengths are 17544 and 10920, respectively. Traffic includes an hourly time series detailing the road occupancy rates on the San Francisco Bay Area freeways from 2015 to 2016. KDD-Cup represents the air quality level from 2017 to 2018 estimated by 59 stations across two cities, Beijing (35 stations) and London (24 stations), measured in an hourly rate. We follow the same normalization procedure applied to Fred-MD and NN5 daily, as described in B.2.Hyperparameters. In Tab. 8 below, we outline the various hyperparameters used in our framework. For all datasets, we employed the default sampler of EDM [45] , and we specified the U-net model hyperparameters that we tuned in our study. For more information on the U-net model hyperparameters, please refer to [45] . Evaluation. For the ultra-long-term unconditional generation task, we follow the same procedure outlined in B.2. We use the same classification and prediction models, as they effectively distinguish between low-and high-quality ultra-long-term generations.Data. For short-term interpolation and extrapolation benchmarks (Sec. 5.4), we use the ETT* datasets [102] , each with a fixed length of 96. The ETT datasets are crucial indicators for long-term electric power deployment, containing two years of data from two separate counties in China. The datasets are divided into ETTh1 and ETTh2 for 1-hour intervals, and ETTm1 and ETTm2 for 15-minute intervals. Each data point includes the target value "oil temperature" and six power load features.For long-term interpolation and extrapolation, we employ a well-established benchmark [75, 78, 103] , incorporating the Physionet and USHCN datasets. The data extraction for the USHCN dataset follows the procedure detailed by [20] . Notably, both datasets exhibit sparsity across many features and contain numerous zero values. The Physionet dataset [80] includes health measurements from 41 sensors collected from 8,000 ICU patients within the first 48 hours of admission. The United States Historical Climatology Network (USHCN) [63] provides daily measurements from 1,218 weather stations across the United States, covering precipitation, snowfall, snow depth, and minimum and maximum temperatures. For the conditional tasks, we strictly follow the training and evaluation procedures described by [78] , and we refer readers to this work for a comprehensive explanation of the evaluation protocol. Both datasets span approximately 1,000 to 2,000 time steps.For the ultra-long-term conditional generation task, we utilize the same data used in the ultra-longterm unconditional generation benchmark, and we refer to App. B.3 for more details about the datasets.Hyperparameters. We describe below in Tab. 9 the different hyperparameters used in our framework. The hyperparameters are similar for both tasks. Therefore, we present them in a unified table. For all datasets, we used the same default sampler of EDM [45] ; we also present the hyperparameters of the U-net model; for further details about them, please see [45] . Evaluation. For the short-term and ultra-long-term datasets (ETT*, Traffic, KDD-Cup), we follow the next procedure. In the interpolation task, we randomly mask 50% of the input data and train the models to predict the missing masked 50%. In the extrapolation task, we mask the second half of the sequence and train the models to predict this missing half. We measure the distance between the models' generated outcomes and the ground truth using MSE loss. Accurate generation will lead to a smaller distance between the prediction and the ground truth, thus indicating the models' interpolation and extrapolation capabilities. For the long-term datasets, we follow a similar procedure as above; however, since the data is sparse and irregularly sampled, the masking is slightly different. We adhere to the exact interpolation and extrapolation processes described in [78] and refer to that source for more details.Due to space constraints in the main paper, we report the rest of the benchmark here. In Tab. 10, we show our model performance on the simple toy Sine dataset.In Tab. 11, we present the Wasserstein distances calculated between our generated 2D point cloud and the actual data. A lower score indicates greater similarity between the clusters, meaning that a lower score is preferable. Our approach yields the best scores across all datasets in comparison to GT-GAN on short sequences and LS4 on long and ultra-long time series.We include the Stocks, Energy and the MuJoCo qualitative t-SNE evaluation (Fig. 4(A, B, C )) and density analysis (Fig . 4(D, E, F )). In addition, we show in Tab. 11 a quantitative evaluation of the t-SNE clusters Wasserstein distance. Both the visual results and the quantitative results demonstrate our framework's ability to learn the true distribution across multiple datasets. In Tab. 12, we present the results of our method on unconditional generation of long sequences with standard deviation. The results demonstrate our method's statistical significance compared to the state-of-the-art method LS4. We include the qualitative t-SNE evaluation for Temp Rain and NN5 Daily (Fig. 5(A, B )) and their density analysis (Fig. 5(D, E )). Additionally, we provide in Tab. 11 the quantitative evaluation of the t-SNE clusters using the Wasserstein distance. Our results indicate the superiority of our approach in comparison to other techqniques. We plot the 2D t-SNE embeddings of synthetic data generated with our method and SOTA tools vs. the real data (top). Then, we compare their probability density functions (bottom).In Tab. 13, we present the results of our method for unconditional generation of ultra-long sequences, including standard deviations. These results demonstrate the statistical significance of our method compared to the state-of-the-art competitive methods. We include the qualitative t-SNE evaluation for Traffic and KDD-Cup (Fig. 6(A, B )) and the density analysis (Fig. 6(D, E )). Additionally, we provide in Tab. 11 the quantitative evaluation of the t-SNE clusters using the Wasserstein distance. Our results highlight our method ability to handle very long sequences.Given the significant impact of image size on the computational resources required by our method, we investigate whether varying image size influences different transformations. We explore the effect of different image sizes on our framework, utilizing long-term datasets with short time Fourier transform (STFT) and short-term datasets with delay embedding. For short time series consisting of 24 time steps, we test image sizes of 8 and 16, as we observe that scaling to larger sizes may not yield benefits. For long time series of approximately 750 steps, we experiment with sizes of 32, 64, and 128. Notably, 32 is the minimum size required to contain enough pixels for representing the long sequence adequately. We present the results in Tab. 14. While most results demonstrate high In this section, we compare the computational resources required by our proposed method and the LS4 method, focusing on training and inference wall-clock runtime and model size in terms of parameters, and we analyze the FLOPs used per method. Although our method, which utilizes image transforms and diffusion models, has a larger model size in terms of parameters, it remains comparable to LS4 regarding training and inference time. Despite the larger model size, our method achieves similar training and inference efficiency, making it a viable and scalable solution for large-scale time series generation tasks as shown in Tab. 16 . Furthermore, the rapid advancements and growing research interest in faster sampling techniques for diffusion models [82] present an opportunity to further enhance our method's efficiency. Leveraging these developments, our approach can integrate even more optimized diffusion models, potentially reducing the computational time and resources required for training and inference, thus improving scalability for large-scale time series generation tasks. Finally, we analyze the FLOPs used in our method compared to LS4 and DiffTime on the Stock, nn5daily and KDD Cup datasets in Tab. 17. In this section, we investigate how the performance of our proposed method scales with the size of the underlying image diffusion model. Specifically, we evaluate the impact of increasing the model size from a few thousand parameters to several hundred million on various time-series datasets. Additionally, we compare this trend with other state-of-the-art time-series generative models, analyzing how their performance is affected by model size increments. Interestingly, merely increasing the model parameters does not improve their performance. It demonstrates that simply enlarging previous methods does not necessarily enhance their generation capabilities. Moreover, in the case of LS4 on the KDD Cup dataset, increasing the model's parameters to 100 million results in memory collapse, making it infeasible to run a batch size of one with the current resources used for training all models. We present the results for our method in Tab. 18, for LS4 in Tab. 19 and the results for DiffTime, in Tab. 20 and Tab. 21. Finally, we add a visualization of the results for the Stocks dataset in Fig. 7 .Our goal in this paper is to leverage recent advancements in computer vision to develop an elegant and robust solution for time-series data, addressing different sequence lengths and setting a baseline for handling short, long, and ultra-long sequences. We aim to take advantage of the fact that image architectures are more thoroughly explored. We hypothesize that the improvements we observe are largely due to the more advanced development of image architectures compared to time-series architectures. To further investigate this, we used NVAE [90] , and StyleGAN [46] , instead of the diffusion model. We observe the results in Tab. 22 . The results imply that using recently betterexplored architecture yields better results when using the same transformations. This understanding strengthens our hypothesis for the robustness and efficiency of diffusion models. Justification: The experiment section, related work and method section support the main claims.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This work presents a simple but effective approach for time-series modeling by transforming the timeseries into an image and then applying standard image diffusion modeling. While the novelty of the approach is relatively limited (and is similar to prior work applying diffusion to spectrograms), the authors demonstrate consistent empirical performance across standard timeseries benchmarks and show that their approach improves over prior work like LS4.

Reviews: review: paper_summary: paper_summary: The paper proposes using invertible transforms to map varying-length time series to images. Using this technique, generative modeling of time series can be done using diffusion vision models. The authors demonstrate state-of-the-art performance on unconditional generation, interpolation, and extrapolation on short and long time series benchmarks. An additional contribution of the paper is the introduction of a novel benchmark for ultra-long (>10k timesteps) time series.
strength_weakness: strength_weakness: - One crucial point is that the proposed method involves 1-2 orders of magnitude more parameters than e.g. LS4, a close competitor (Table 16). I think this point should be more clearly emphasized and investigated in the main paper.
  - The authors mention that in terms of wall-clock time, their method's training and inference efficiency is comparable despite the difference in size. However, this seems to be mostly a statement on how much work has been done improving efficiency of image diffusion models -- it may be possible to drastically improve efficiency of existing time series methods.
questions: questions: - How does the number of parameters compare between the proposed method and all the other methods you compare to, e.g. in Tables 1 and 2?
- Were any experiments done evaluating how the proposed method compares to e.g. a parameter-matched LS4 model?
- Similarly, were any scaling law experiments done to investigate how the performance of the proposed method improves as the size of the image diffusion model increases?
  - For example, it would be quite interesting to find that the scaling of the image diffusion model architecture is better than the scaling of e.g. LS4 or time series diffusion architectures.
limitations: limitations: The authors adequately address the limitations of their work.
paper_summary: paper_summary: The paper argues for the use of image generative modelling architectures for the time-series generative modelling task. Doing so involves converting a time-series to an image-shaped object, modelling it as an image, and then converting back.
strength_weakness: strength_weakness: Weaknesses:
- The authors mention that their approach "requires slightly higher computational resources". They go into more detail in Appendix C.10 but I would appreciate further detail. In particular, the cost is given in terms of hours/minutes, but I cannot see anything about e.g. GPUs used or FLOPs used. Are the number of GPUs matched between methods? A comparison in terms of GPU-hours for the same GPU type would be informative. To be really confident of the improvement, ideally there would be a comparison in which the methods were compared when given equal GPU-hours. Investigating how the performance of different methods scales as training FLOPs are increased would also be very interesting.
- It is a little unclear whether the advantage shown comes from the fact that image architectures are just better-explored than time-series architectures, or whether the inductive bias of transforming into an image shape is helpful. Can the authors comment on this? An interesting potential experiment would be to try training an image architecture with an older architecture and compare its performance.
- A more detailed description of the Predictive and Discriminative metrics used in Section 5.1 would be helpful.
questions: questions: See weaknesses
limitations: limitations: Adequately addressed
paper_summary: paper_summary: In this paper, authors propose to convert general time sequences into images by employing invertible transforms and incorporate advanced diffusion vision models to process short- and long-range time-series within the same framework. Through experiments, improvements have been made on multiple tasks, such as unconditional generation, interpolation and extrapolation.
strength_weakness: strength_weakness: 1.  The time-series to image (ts2img) transform methods and diffusion backbone used in the paper are all from other existing works, which compromise the novelty and significance of the paper. 
2. In Sec.5.1 and Sec.5.2, the ts2img transform methods used for short- and long-range generation are different. It is hard to tell whether the transform choice or the diffusion model plays a more important role. Taken together with Sec.5.5, the proposed method is only effective (i.e. outperforms others) with some specific transform method on different tasks, which conflicts with the authors’ claim that the proposed model can seamlessly process sequences with different lengths.
questions: questions: Please see the weaknesses part above.
limitations: limitations: Yes.

Comments: title: Response
Response
comment: Thank you for your thorough response to my concerns. Your comment that your aim is to "leverage recent advancements in computer vision to develop an elegant and robust solution for time-series data" does help to clarify the contribution of this paper for me. I have raised my score to a 6.
Thank you for your response and for reevaluating our work. We are happy that our response address all your concerns and questions, and we would be more than willing to address any additional ones you may have.
Thanks to the authors for the response! The new results (regarding matched parameter / FLOPs comparisons) are enlightening and compelling. I will raise my score accordingly.
Thank you for your response and for reconsidering the evaluation of our work.

We will clarify in the main paper that image transformation is selected as a hyperparameter and direct readers to Sections B.1, B.2, and B.3 of the appendix, where all relevant hyperparameter details, including image transformation, are described. We appreciate your feedback, which has improved the clarity of our work.

Regarding the automatic choice based on sequence length, we will emphasize in the final revision that some transformations, like Delay Embedding, are robust across any length, while others, such as STFT, are better suited for specific lengths, like long or ultra-long sequences.

We're happy to address any further questions you may have.
Thank you for the additional information. After reconsidering the contribution of this paper, I agree with your views on novelty. And the extra experiments help address my concerns. As you said, the type of time-series to image transform is set by a hyper-parameter, but I didn’t find the details about it. Instead of manual choice, it would be better if the transform type could be automatically chosen according to the time-series length. All things considered, I will increase my rating. Good luck to you.