充分分析下下面的论文内容，列出创新点、解决问题的场景和实现逻辑；使用ptorch 来实现这个timedif； TIMEDIT: GENERAL-PURPOSE DIFFUSION TRANSFORMERS FOR TIME SERIES FOUNDATION MODEL
A PREPRINT
Defu Cao∗, Wen Ye∗, Yizhou Zhang, Yan Liu Department of Computer Science University of Southern California
Los Angeles, CA 90007
{defucao, yewen, zhangyiz, <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
September 5, 2024
ABSTRACT
With recent advances in building foundation models for texts and video data, there is a surge of interest in foundation models for time series. A family of models have been developed, utilizing a temporal auto-regressive generative Transformer architecture, whose effectiveness has been proven in Large Language Models. While the empirical results are promising, almost all existing time series foundation models have only been tested on well-curated “benchmark” datasets very similar to texts. However, real-world time series exhibit unique challenges, such as variable channel sizes across domains, missing values, and varying signal sampling intervals due to the multi-resolution nature of real-world data. Additionally, the uni-directional nature of temporally auto-regressive decoding limits the incorporation of domain knowledge, such as physical laws expressed as partial differential equations (PDEs). To address these challenges, we introduce the Time Diffusion Transformer (TimeDiT), a general foundation model for time series that employs a denoising diffusion paradigm instead of temporal auto-regressive generation. TimeDiT leverages the Transformer architecture to capture temporal dependencies and employs diffusion processes to generate high-quality candidate samples without imposing stringent assumptions on the target distribution via novel masking schemes and a channel alignment strategy. Furthermore, we propose a finetuning-free model editing strategy that allows the seamless integration of external knowledge during the sampling process without updating any model parameters. Extensive experiments conducted on a varity of tasks such as forecasting, imputation, and anomaly detection, demonstrate the effectiveness of TimeDiT.
1 Introduction
Time series analysis is pivotal in a diverse set of AI applications, such as natural science [Cuomo et al., 2022, Wang et al., 2020], social science [Zhang et al., 2022, Sharma et al., 2021],sustainability [Krenn and Buffoni, 2023],health [Kaushik et al., 2020, Kamra et al., 2021], etc. These applications root in diverse domains [Li et al., 2018, Bi et al., 2023, Cao et al., 2023c, Zhang et al., 2021, Ye and Gao, 2022], leading to time series with various distributions [Wang et al., 2023] and a divers set of analysis tasks, such as forecasting [Nie et al., 2022, Jia et al., 2024], imputation [Tashiro et al., 2021], anomaly detection [Tuli et al., 2022], etc. Even though considerable progress has been made in developing specialized models optimized for specific scenarios and individual tasks, an open question remains: Can a single time series foundation model excel across domains? Recent initiatives have explored the possibility of universal time series models on zero-shot setting [Ansari et al., 2024, Liu et al., 2024, Gruver et al., 2024, Cao et al., 2023b], drawing inspiration from large pre-trained language models in natural language processing(NLP) and computer vision(CV), such as GPT[Radford et al., 2018], CLIP [Radford et al., 2021], which are known for their robust transfer learning capabilities. However, due to the fundamentally different semantics between text/images and time series data, the unique challenges of achieving a truly flexible and general-purpose time series model remain an open problem.
Recently, the emergence of LLMs like GPT-4 [OpenAI, 2023] and LLaMA [Touvron et al., 2023] suggests the potential for building time series foundation models enabling a general solution to handle multiple time series distributions. Previous attempts typically build upon the transformer backbone, which has achieved state-of-the-art performance on various time series tasks, particularly in modeling long-term dependencies. However, the tokenization of time series data for transformers is especially sensitive to variations in data sources and sampling rates. Previous tokenization approaches with different schemes including token patching [Das et al., 2023, Woo et al., 2024a]; discretization tokens [Talukder et al., 2024] and tokens based on time series features [Yue et al., 2022, Ansari et al., 2024, Rasul et al., 2023] have either fragmented the global information or have been constructed in a manner that inherently loses important information. Most, if not all, of them employ a channel independence strategy [Nie et al., 2023] or focus solely on univariate time series. Channel independence strategy, though beneficial in certain contexts, often overlooks the complex inter-temporal and cross-feature dependencies in practical applications and thus presents an opportunity for optimization [Zeng et al., 2023].
Moreover, compared with texts and images, time series exhibit unique characteristics such as missing values [Kollovieh et al., 2023], irregular sampling [Kidger et al., 2020, Cao et al., 2023a, Zhang et al., 2022], multi-resolution [Niu et al., 2023, 2022], etc. To address these challenges, a foundation model for time series must be capable of demonstrating flexibility across different scales to handle diverse inputs with varying distributions. However, these unique natures and challenges are not covered by the popular well-curated benchmark datasets[Li et al., 2018, Zhou et al., 2021, Alexandrov et al., 2020]. As a result, most existing works, which are developed and evaluated primarily on these datasets, may not fully address the complexities encountered in real-world time series applications. In addition, time series processes are often governed by underlying physical principles [Li et al., 2021, Meng et al., 2022, Xiao et al., 2023] and can be guided by domain-specific textual information [Jin et al., 2023, Sun et al., 2023]. However, integrating these diverse sources of information into a unified model poses further challenges, as the model must effectively leverage the relevant physics context while adapting to the unique characteristics and distributions of each domain. Addressing each of these issues requires innovative approaches in data preprocessing, model architecture, and training strategies to create models that can seamlessly handle the diverse and complex nature of time series data.
Current approaches to time series modeling lack a unified framework for handling the aforementioned diverse and imperfect data inputs, frequently prioritizing performance on well-curated benchmarks over addressing real-world challenges. Diffusion models, such as DDPM [Ho et al., 2020], offer a promising solution by framing data generation as a series of conditional transformations, effectively recasting density estimation as sequential reconstruction. Unlike autoregressive methods that generate future tokens sequentially, diffusion models can directly produce high-quality samples through a reverse denoising process. This process can be analogized to solving partial differential equations (PDEs), allowing for the natural incorporation of physics-based knowledge. This capability, combined with transformers’ strength in capturing temporal dependencies, presents an opportunity to develop a more versatile and robust time series foundation model. Such a hybrid approach could effectively address the complexities of real-world data while maintaining the flexibility to adapt to various forecasting tasks and data conditions.
In this work, we introduce TimeDiT—a diffusion transformer-based foundation model equipped with a standardized training pipeline for different shapes of input time series and tailored for diverse distributions and downstream tasks. TimeDiT leverages the Transformer architecture’s inherent ability to capture temporal dependencies through its attention mechanisms, while also benefiting from the scalability that allows for increased model capacity crucial for complex time series tasks. By adopting a diffusion model approach, TimeDiT treats time series holistically, avoiding the error accumulation issues common in autoregressive solutions. The model incorporates a novel comprehensive mask mechanism that enables a single, unified foundation model to handle multiple tasks without additional modules or parameters. This design naturally addresses real-world challenges such as multi-resolution data and missing values. During the sampling stage, TimeDiT introduces an innovative strategy to incorporate physics knowledge as an energy- based prior, supported by theoretical guarantees. This approach guides the reverse diffusion process using physics-based constraints, including partial differential equations, resulting in generated samples that adhere to known physical laws and domain-specific requirements, thereby enhancing sample quality and model applicability across various scientific and engineering contexts.
TimeDiT’s performance is rigorously evaluated through an extensive experimental setup encompassing over 20 diverse datasets from domains including traffic, weather, finance, etc. The model is benchmarked against more than 25 open-source baselines, ranging from linear-based models to diffusion-based models, transformer-based models, and other forecasting foundation models. These comprehensive experiments cover multiple challenging time series tasks, including in-domain and zero-shot probabilistic forecasting, imputation, anomaly detection, and synthetic data generation. TimeDiT demonstrated state-of-the-art or highly competitive results across these tasks, showcasing its effectiveness and efficiency as a foundation model for various time series applications. Notably, TimeDiT achieved new state-of-the-art CRPSsum scores on the Electricity and Traffic datasets for probabilistic forecasting. In addition, the results on zero-shot experiments show that our model can be used as a foundation model even without fine-tuning,although fine-tuning may be necessary in some cases. Furthermore, TimeDiT’s scalability and adaptability are evident in its ability to incorporate external knowledge, such as physical constraints, during the sampling stage. This feature allows for the generation of samples that better conform to known physical laws and domain-specific requirements. This combination of state-of-the-art performance, adaptability across diverse tasks, scalability, and the ability to incorporate domain-specific knowledge positions TimeDiT as a powerful and versatile foundation model, capable of addressing a wide spectrum of time series challenges and opening new avenues for advanced time series analysis across various fields.
In summary, our contributions can be summarized as three unfolds:
• We introduce TimeDiT, a novel diffusion transformer-based foundation model for time series analysis. By combining the strengths of diffusion models and transformers, our approach offers a flexible architecture adaptable to various downstream tasks. The model incorporates a comprehensive mask mechanism for reconstruction pretraining and task-specific fine-tuning, ensuring a standardized training pipeline capable of handling diverse input shapes and distributions.
• Unlike autoregressive approaches, TimeDiT addresses real-world challenges in time series data by directly processing multivariate inputs and employing a denoising process to generate cohesive target time series. This method effectively handles issues such as missing values and multi-resolution data. In addition, TimeDiT can generate time series that adhere to known physical laws and domain-specific requirements, enhancing its applicability in scientific and engineering contexts.
• Evaluated on over multiple datasets across different domains and tasks, TimeDiT achieves state-of-the-art or competitive results. It excels in probabilistic forecasting, imputation, anomaly detection, and data generation, showcasing its versatility as a foundation model in both in-domain and zero-shot settings.
2 Related Work
General Purpose Time Series Model In the past decades, researchers have excelled in designing sophisticated models for specific time series analysis tasks [Zhang et al., 2024b, Fan et al., 2024, Cao et al., 2020, 2022]. However, in recent years, the emergence of large language models has inspired the development of general-purpose time series models [Zerveas et al., 2021, Zhang et al., 2024a, Garza and Mergenthaler-Canseco, 2023] and the field of time series has seen tremendous exploration efforts towards foundation models. [Gruver et al., 2024] simply encoded time series as strings while [Jin et al., 2023] converted time series into language representations by alignment. [Cao et al., 2023b] and [Pan et al., 2024] further incorporated decomposition technique and prompt design and generalizes to unseen data and multimodal scenarios. [Rasul et al., 2023] worked towards foundation model from a probabilistic perspective but only considered univariate time series only which rarely appears in real-life. Additionally, many studies started to follow a two-stage training paradigm of pretraining and finetuning [Chang et al., 2023, Dong et al., 2024, Nie et al., 2022]. However, these works mainly focused on the forecasting task only [Woo et al., 2024a, Das et al., 2023]. [Zhou et al., 2023] first adapted GPT2 as a general-purpose time series analysis model and extended it to various time series tasks. [Talukder et al., 2024] leveraged VQVAE as a tokenizer for transformer to handle time series tasks and [Ansari et al., 2024] employed a scaling and quantization technique to embed time series. For more detailed literatures of the general-purpose time series model, please refer to recent surveys and position paper [Liang et al., 2024, Jin et al., 2024, Jiang et al., 2024]
Diffusion models for Time Series Despite the growing interest of diffusion models in various scenarios [Peebles and Xie, 2022, Li et al., 2022a, Lu et al., 2024, Sui et al., 2024a,b], the use of diffusions in time series analysis is less explored compared to pre-trained language models and transformers. Most existing studies also focused solely on forecasting and the choice of backbone model also varies among VAE[Li et al., 2022b], RNN[Rasul et al., 2021], and transformer. CSDI [Tashiro et al., 2021] utilized diffusion model for time series imputation. [Yuan and Qiao, 2024] incorporated decomposition into diffusion model to improve interoperability. Although [Kollovieh et al., 2023] build a diffusion pipeline for multiple tasks with refinement, they still train different models for each task. To the best of our knowledge, there has been no exploration of leveraging unified diffusion models for a comprehensive set of time series tasks yet. Please refer to [Yang et al., 2024] for a comprehensive literature review on diffusion models for time series analysis.Variance
Noise
Random Position Mask
Stride Mask
Block Mask
Inference
Fixed Position
Mask
Future Block Mask
Full Block
Mask
Cond Obs
Cond Mask
Noise
Target
Final Projection Layer
Time DiT Block N
Scale Feed Forward Scale, Shift Layer Norm
Scale
Multi-Head
Attention
Scale, Shift Layer Norm
Embed
Timestamp
Embedding Layer
Time Series Mask Unit
Imputation
Forecast
Generation
Anomaly Detection
Padded Data
Healthcare Transportation
Padding Mask
Sustainability Science
Embed
Other Cond
Pos Embed Noise
MLP
Pos Embed Cond Obs
Figure 1: TimeDiT Architecture. Left: TimeDiT framework with diverse multivariate time series from different domains with multi-resolution, missing values; Middle: Structure of TimeDiT block; Right top: Illustration of masks generated by Time Series Mask Unit, reconstruction mask is neglected as it’s an all-zero mask; Right bottom: Masks for downstream tasks that TimeDiT handles during inference.
3 Preliminaries 3.1 Diffusion Model
In recent years, diffusion models have emerged as a promising approach in generative modeling. A diffusion process is a Markov chain that incrementally adds Gaussian noise to data over a sequence of steps, effectively destroying the data structure in forward process and destroying the data structure in backward structure.
The forward process adds noise to the data x0 over a series of timesteps t according to a variance schedule βt, resulting in a set of noisy intermediate variables x1, x2, . . . , xT . Each subsequent xt is derived from the previous step by applying Gaussian noise:
q(xt | xt−1) = N (xt; p1 − βtxt−1, βtI) (1)
The reverse process aims to denoise the noisy variables step by step, sampling each xt−1 from the learned distribution pθ(xt−1 | xt). This distribution, modeled by a neural network parameterized by θ, approximates the Gaussian distribution:
pθ(xt−1 | xt) = N(xt−1;μθ(xt,t),Σθ(xt,t)) (2)
By iterating this reverse process from t = T down to t = 0, the model gradually reconstructs the original data from noise. The reverse process learns to predict the mean and covariance of each intermediate distribution, effectively approximating the original data distribution.
4 Methodology
In this section, we present our main contributions: the proposed foundation model, TimeDiT , a diffusion model with transformer backbone designed for multiple time series tasks, along with uniform masking strategies and incorporation of physics knowledge and textual information as an extension. We first outline the uniformed problem setting for multiple down-stream tasks and offer an in-depth examination of the model architecture. Subsequently, we delve into the training pipeline with mask strategies, which help to build the training scheme in self-supervised learning for time series. Next, we present how to incorporate external information to improve the model’s performance during both the training and inference stages. By doing so, TimeDiT can generate samples that better conform to real-world requirements and enhance its performance on various downstream tasks. These extensions showcase the flexibility and adaptability of our proposed model, making it a powerful tool for a wide range of time series applications.
4.1 Problem Definition
We denote a multivariate time series as X = {xi,j} ∈ RK×L, where K is the number of features and L is the
length of the time series. Each individual entry xi,j represents the j-th feature at time step l, for i ∈ {1, . . . , K} and
j ∈ {1, . . . , L}. We define an observation mask Mobs = {mi,j } ∈ {0, 1}K×L, where mi,j = 0 if xi,j is missing, and
m = 1 if x is observed. Let xobs ∈ Xobs denote the observed subsequence; xtar denote the target subsequence i,ji,j0 0
of xobs which could be forecast target or imputation target or the whole sequence depending on the task. Let xcon 00
denote the unmasked partial observations in xobs which acts like conditions for the masked area xtar. Let us use all 00
subscripts of x to denote diffusion timestamp, and a subscript of 0 means no noise has been applied to the original data.
Formally, the goal of our task is to approximate the true conditional data distribution given the conditional information
q (xta | xcon) with a model distribution p (xtar | xcon), which can be calculated by a diffusion model with conditional X00 θ00
information:
T
p xtar | xcon := pxtarYp xtar | xtar,xcon,xtar ∼ N(0,I),where θ0:T0 Tθt−1t0T
t=1
p xtar | xtar,xcon := N xtar ;μ xtar,t | xcon,σ xtar,t | xconI. (3)
θ t−1 t 0 t−1 θ t 0 θ t 0
The mask mechanism M plays a critical role in identifying the positions of xcon and xtar. By leveraging these positional
differences, our model adeptly adapts to various downstream tasks, including forecasting, imputation, anomaly detection, etc, within a unified framework.
4.2 Time Series Diffusion Transformer
Figure 1 shows the overall framework of TimeDiT. We first establish the M and xobs based on the given input from obs 0
different distributions with multivariate sequences, missing value and multi-resolution by injecting placeholders to
standardize the input shape across different time series, facilitating more efficient and consistent processing. Then,
the unified time series mask unit adapts to diverse time series scenarios and builds the xcon, M and xtar, with shape 00
RB×L×K, to help TimeDiT learn robust representations in a self-supervised manner by reconstructing the original sequence through denoising xtar. After that, the embedding layer directly treats xcon and xtar as without any patching, as
T00
the diffusion process is designed to handle multivariate input and operate in a continuous token space. By preserving the
integrity of the input time series, TimeDiT ensures that the model can effectively capture and utilize the rich information contained within the data. The TimeDiT block’s attention mechanism is designed to autonomously learn cross-channel and temporal correlations through end-to-end training.
Standardize Pipeline We introduce placeholders within the input sequences to standardize the input shape across
different time series, accounting for varying channel numbers K and sequence length L. Specifically, we define the
maximum channel number Kmax such that any input with channel k < Kmax is padded to have Kmax channels while
any input with more than Kmax channels are segmented into ⌈ k ⌉ blocks of inputs where each block has Kmax Kmax
channels and undergoes independent processing. This segmentation allows our model to manage high-dimensional data efficiently, reducing computational overhead and maintaining relative positional integrity of the data and consistency across inputs. Additionally, for any input with sequence length less than the designated maximum length Lmax, we pad the sequence in the front to achieve the desired length. This standardization is essential for establishing a uniform input structure that enhances processing efficiency and consistency.
Time Series Mask Unit We propose a unified time series mask mechanism that includes a variety of masks that seamlessly integrates with the model during self-supervised task agnostic pre-training and task specific fine-tuning to cater to diverse time series scenarios. The time series mask unit generate four types of masks: reconstruction mask, stride mask, block mask, and random mask. Firstly, the task-agnostic pre-training aims to improve the overall time series representation by encouraging the model to learn robust and generalizable features from the input data. Secondly, the task-specific training is designed specifically for the most common downstream tasks, including forecasting and imputation, enabling the model to adapt to the unique requirements of each task.
00
A PREPRINT - SEPTEMBER 5, 2024
As shown in Figure 1 right top, given x ∈ RK×L, the random mask MR can be generated by: R 1 zi,j > r,z ∈ RK×L,z ∼ Uniform(0,1)
M (x, r) = 0, otherwise, (4) where r is the mask ratio. In addition, for task-specific training and inference, we allow the user to supply customized imputation masks that could simulate the naturally missing data and multi-resolution cases.
Block mask MB can be generated via:
B 1 j<L−l,
M (x, l) = 0, otherwise, (5)
where l is the predicted length. We can randomly select l during pretraining and use the designated prediction length during the finetuning and inference stage for specific experiment settings.
Stride mask MS, a variant of MB, is placed intermittently within the series and is defined as follows:
1 jmod2=0
MS(x, nblocks) = b (6)
0 otherwise
where nblock is the number of blocks into which the sequence is divided; b = l L m is the length of each block; j is
A PREPRINT - SEPTEMBER 5, 2024
nblocks
the index of the sequence. MSis designed for task-agnostic pretraining further to enhance the model’s representation
ability by integrating information across non-contiguous parts of the series.
In addition, reconstruction mask MRec = 0 is used for tasks including synthetic data generation and anomaly detection, where we can directly generate synthetic data or obtain an anomaly score for each temporal position based on the difference between original and reconstructed series.
For the pretraining stage, we random select one conditional mask type from M = {MR, MS, MB, MRec} for each instance. TimeDiT’s goal is to reconstruct the xtar , defined as x0 × (J − M) where J is unit matrix. The target sequence is the masked portion of the original sequence. In the finetuning and inference stage, the choice of mask is tailored to align with the specific requirements of the user. This flexibility allows TimeDiT to apply the most appropriate masking strategy based on the context of the task and application.
Condition Injection Instead of following [Peebles and Xie, 2022] to integrate diffusion timestep and label information (texts in out case) through the layer normalization preceeding the attention block, we add the diffusion timestep and texts information to the target noise as these are universal information to the series.
Given that TimeDiT utilizes a transformer-based architecture, a straightforward and intuitive approach is to include
conditional information directly as part of the input sequence by concatenation as done in latent diffusion [Rombach
et al., 2022]. However, we empirically found out that controlling mean and variance through layer normalization is a
stronger form of conditional information injection. We incorporate the partial observations xc0on through adaptive layer
normalization to control the scale and shift of the xtar. This design choice is motivated by the face that the scale and 0
shift of the partial observations are more relevant to the target observations. This integration can be expressed as
AdaLN(h, c) = cscale LayerNorm(h) + cshif t (7)
where h is the hidden state and cscale and cshift are the scale and shift parameters derived from the partial observations. We perform temporal attention in the self-attention block to capture the temporal dependency within the input.
Inference We perform pre-training and finetuning for forecasting and imputation. We exclude the pre-training process from anomaly detection and synthetic generation because the two tasks are very dataset specific and do not necessarily benefit from learning distributions beyond the target dataset. Let n represent the number of samples generated for each prediction, which we set to n = 10 (n = 30 for forecasting tasks) in our experimental setup at inference time. We use the median of these n predictions as the final prediction, providing the added benefit of obtaining a confidence interval for TimeDiT ’s predictions. To prevent channel padding from affecting the generated samples, we mask out the invalid channels during sampling at each diffusion timestep so that TimeDiT does not falsely treat information in the non-valid channels as meaningful information. Padding is applied at the beginning of the temporal dimension to ensure that the most relevant information remains at the end, thereby mitigating the effect of padding.
4.3 Physics-Informed TimeDiT
Physics principles are fundamental in shaping the evolution of temporal signals observed in real-world phenomena, such as climate patterns and oceanographic data. Therefore, it is essential to integrate physical knowledge into foundational time series models. In this section, we propose a strategy to incorporate physics knowledge as an energy-based prior for Algorithm 1 Physics Informed TimeDiT through Energy-based Sampling
1: 2: 3:
4:
5: 6:
7:
8: 9:
xT ∼N(0,I)
for t = T,...,1 do
z∼N(0,I)ift>1,elsez=0 11−αt 
xt−1=√αt xt−√1−α ̄tεθ(xt,t) +σtz end for
A PREPRINT - SEPTEMBER 5, 2024
forj=0,1,..,k−1do
xtar = xtar +ε∇K(xtar;xobs)+αε∇logp(xtar|xobs)+
√
2εσ,σ ∼ N(0,1)
j+1jj j end for
return xtar k
TimeDiT during inference, which iteratively refines the reverse diffusion process. By guiding the denoising process during inference with gradients derived from physical laws represented by partial differential equations (PDEs), the integration of this knowledge can significantly enhance the quality of the generated samples.
A generic form of a physical law represented as a PDE that describes the evolution of a continuous temporal signal x(u, t) over a spatial coordinate u is given by:
∂x=F(t,x,u,∂x, ∂2x ,...) (8) ∂t ∂ui ∂ui∂uj
Based on this PDE representation of physical knowledge, the consistency between the predicted time series xtar and the physics knowledge can be quantified using the following squared residual function:
∂xtar ∂xtar ∂2xtar
K(xtar;F)=−|| ∂t −F(t,xtar,u, ∂u ,∂u∂u ,...)||2 (9)
iij
This function reaches its maximum when the predicted time series is perfectly consistent with the physical model, resulting in a residual of 0. Using this metric K, physics knowledge can be integrated into a probabilistic time series foundation model p(xtar|xcon) by solving the following optimization problem to obtain a refined model q(xtar|xcon):
q(xtar|xcon) = argmaxExtar∼qK(xtar;F)−αDKL(q(xtar|xcon)||p(xtar|xcon)) (10) q
where the first term represents the aforementioned physics knowledge metric, and the second term controls the divergence between q(xtar|xcon) and p(xtar|xcon). The following theorem provides a closed-form solution to the above optimization problem:
Theorem 4.1. The optimal q(xtar|xcon) in Eq.10 is the Boltzmann distribution defined on the following energy function: E (xtar ; xcon ) = K (xtar ; F ) + α log p(xtar |xcon ) (11)
in other words, the optimal q(xtar|xcon) is:
q(xtar|xcon) = Z1 exp(K(xtar;F)+αlogp(xtar|xcon)), (12)
where Z = R exp(K (xtar ; F ) + α log p(xtar |xcon ))dxtar is the partition function.
ThetheoremillustratesthatsamplingfromtheBoltzmanndistributiondefinedinEq. 11,isanalogoustoincorporating physics knowledge into model edition. In the context of diffusion models, this distribution can be effectively sampled using Langevin dynamics [Stoltz et al., 2010]:
xtar = xtar + ε∇ log q(xtar|xcon) + √2εσ, σ ∼ N (0, 1)
j+1 j √ (13)
= xtar + ε∇K(xtar; xcon) + αε∇ log p(xtar|xcon) + 2εσ, σ ∼ N (0, 1) jjj
In diffusion model, precisely calculate the likelihood log p(xtar|xcon) is intractable. To tackle this issue, following previous works [Kollovieh et al., 2023], we approximate likelihood with the objective to edit the pre-trained diffusion
model:
log p(xtar |xcon ) = −Eε,t [||εθ (xtar , t; xcon ) − ε||2 ] (14) The approximation presented above constitutes the optimizable component of the evidence lower bound(ELBO).
Algorithm 1 summarizes the comprehensive model editing process. 7 5 Experiments
To comprehensively assess our time series foundation model, we evaluate on a diverse set of tasks that reflect real-world challenges and applications. We begin by testing the model’s performance in practical scenarios and its ability to integrate domain knowledge. This includes handling missing data and multi-resolution forecasting on customized datasets, which allows us to evaluate the model’s robustness in situations that frequently occur in real-world applications, as well as physics-informed modeling crucial for scientific and engineering domains[Yuan and Qiao, 2024], which uses 6 practical partial differential equations (PDEs). We then assess the model’s capabilities in well-established benchmarking tasks across various fields such as finance, healthcare, and industrial monitoring. These tasks include forecasting on Solar, Electricity, Traffic, Taxi, and Exchange datasets[Tashiro et al., 2021, Rasul et al., 2021] to evaluate temporal dependency modeling, imputation on ETTh, ETTm, Weather and Electricity datasets[Zhou et al., 2021] to assess the handling of missing data, anomaly detection on MSL, SMAP, SWaT, SMD, and PSM datasets[Xu et al., 2021, Zhao et al., 2020] to gauge sensitivity to unusual patterns, and synthetic data generation on Stock, Air Quality, and Energy datasets[Yoon et al., 2019, Desai et al., 2021] to test understanding of underlying distributions. By evaluating these diverse tasks, we can demonstrate that our model truly serves as a foundation for various time series applications, potentially reducing the need for task-specific models.
A PREPRINT - SEPTEMBER 5, 2024
0.8 0.6 0.4 0.2 0.0
5 10 15 20 25 30 35 40 45 50 Missing Ratio (%)
CRPS sum
CSDI DiffTS TimeDiT
0.5 0.4 0.3 0.2 0.1 0.0
23456 Resolution
CRPS sum
CSDI DiffTS TimeDiT
(a) (b)
Figure 2: Visualization of miss value (a) and multi resolution (b) forecasting results on the Exchange dataset. Compared between our model TimeDiT and state-of-the-art diffusion-based methods. The x-axis number in (b) is the number of different sampling resolutions in the multivariate input.
5.1 Practical Scenarios: Missing Data and Multi-Resolution Forecasting
To simulate more realistic scenarios in time series tasks, we introduced two additional challenges: missing values and multi-resolution data. These conditions are common in real-world applications and test a model’s robustness and adaptability. For the missing value scenario, we created datasets with various missing ratios, simulating incomplete data often encountered in practice. In the multi-resolution setting, we sampled each individual time series within the multivariate dataset at different resolutions, reflecting the diverse sampling frequencies often present in real-world data collection.
Figure 2 illustrates TimeDiT’s performance in realistic scenarios, showcasing its effectiveness across different sampling frequencies on the Exchange dataset. In Figure 2 (a), we observe TimeDiT’s superior performance in handling missing data. As the missing ratio increases from 5% to 50%, TimeDiT maintains the lowest CRPSsum across all scenarios, indicating its robustness to data gaps. The performance gap between TimeDiT and other models widens as the missing ratio increases, highlighting its effectiveness in more challenging conditions. Figure 2 (b) demonstrates TimeDiT’s ability to manage multi-resolution data, where it maintains a clear performance advantage as the number of different sampling resolutions increases from 2 to 6. This demonstrates its ability to effectively integrate and forecast time series data sampled at varying frequencies. These findings underscore TimeDiT’s potential as a practical and versatile tool for time series forecasting in diverse, challenging scenarios that more closely resemble real-world applications.
5.2 Domain Knowledge Integration: Physics-Informed TimeDiT
A key advantage of our approach is the ability to directly incorporate physics knowledge into the pretrained foundation model without additional fine-tuning. This is possible because the diffusion model reconstructs the entire process, allowing for seamless integration of PDE-based constraints during inference. By encoding the known physical laws governing the system into the sampling process, we can guide the model towards more physically consistent and MSE
DDPM 0.011(0.000) DDIM 0.015(0.000) TSDiff 0.011(0.000) TimeDiT 0.010(0.000)
DDPM 0.016(0.001) DDIM 0.018(0.000) TSDiff 0.017(0.001) TimeDiT 0.011(0.001)
DDPM 0.309(0.004) DDIM 0.349(0.013) TSDiff 0.309(0.008) TimeDiT 0.284(0.005)
RMSE MAE
Advection 0.106(0.001) 0.084(0.001) 0.122(0.002) 0.096(0.001) 0.106(0.022) 0.085(0.001) 0.103(0.002) 0.082(0.001) Burgers 0.128(0.004) 0.101(0.003) 0.136(0.001) 0.116(0.001) 0.129(0.005) 0.102(0.004) 0.104(0.005) 0.083(0.003) Diffusion Sorption
CRPS
0.472(0.007) 0.559(0.009) 0.472(0.011) 0.464(0.008)
1.787(0.040) 1.858(0.015) 1.800(0.055) 1.395(0.053)
0.415(0.006) 0.470(0.013) 0.414(0.007) 0.423(0.007)
MSE RMSE MAE CRPS
Navier-Stokes
0.309(0.004) 0.556(0.004) 0.332(0.005) 0.415(0.006) 0.350(0.014) 0.591(0.011) 0.377(0.009) 0.470(0.013) 0.399(0.008) 0.556(0.007) 0.331(0.006) 0.414(0.007) 0.299(0.006) 0.546(0.006) 0.322(0.06) 0.403(0.007) Vorticity
1.917(0.020) 1.385(0.007) 0.851(0.009) 0.476(0.005) 1.567(0.031) 1.252(0.012) 0.754(0.012) 0.401(0.006) 1.966(0.073) 1.402(0.026) 0.866(0.010) 0.485(0.005) 1.524(0.523) 1.234(0.021) 0.772(0.009) 0.445(0.006) CFD
0.004(0.000) 0.065(0.001) 0.054(0.000) 0.082(0.000) 0.039(0.002) 0.194(0.006) 0.188(0.006) 0.313(0.012) ---- 0.004(0.000) 0.062(0.001) 0.051(0.001) 0.080(0.001)
A PREPRINT - SEPTEMBER 5, 2024
0.556(0.004) 0.591(0.011) 0.556(0.007) 0.533(0.005)
0.332(0.005) 0.377(0.009) 0.331(0.006) 0.327(0.005)
Table 1: Physics informed results: This table presents results of PDE forecasting for various sampling strategies, demonstrating the model’s ability to incorporate physics constraints without updating the foundation model. Metrics shown include point estimates metrics, which is MSE, RMSE and MAE; probabilistic metrics, CRPS, for uncertainty quantification. Lower values indicate better performance and closer adherence to physical laws.
accurate predictions. In this section, we evaluate how effectively our pre-trained foundation model can integrate physics- informed knowledge into time series forecasting without the need for fine-tuning. We study four 1D partial differential equations (PDEs) from [Takamoto et al., 2022]: general Navier-Stokes Equations, Kolmogorov Flow (a specific case of Navier-Stokes Equations), Advection Equations, Burgers Equations, Diffusion Soeption and Computational Fluid Dynamics (CFD). These equations are used to generate synthetic data with random initial conditions, and we apply diffusion models to forecast time series based on data from a historical window.
Table 1 presents the results, including both mean error and error bars. The table clearly demonstrates that our proposed model editing solution, which incorporates physics knowledge, significantly outperforms previous sampling strategies such as DDPM [Ho et al., 2020], DDIM [Song et al.], and TS Diffusion’s Self-Guidance [Kollovieh et al., 2023]. By leveraging domain-specific physical information, our approach achieves substantial performance improvements over these baselines, highlighting the effectiveness of integrating physics-informed priors into the diffusion model sampling process. This performance gain underscores the potential of combining pretrained foundation models with domain-specific knowledge. Our method offers a flexible framework for enhancing time series forecasting in scientific and engineering applications where the underlying physical laws are partially known. It demonstrates that by bridging the gap between data-driven approaches and physics-based modeling, we can achieve more accurate and physically consistent predictions without the computational overhead of retraining or fine-tuning the entire model. The ability to easily incorporate physics knowledge into a pretrained foundation model represents a significant advance in the field of scientific machine learning. It opens up new possibilities for rapid adaptation to specific physical systems and phenomena, potentially accelerating research and discovery in various scientific domains.
5.3 Forecasting on Full-shot Setting and Zero-shot Setting
In the forecasting task, we conduct two types of experiments. First, we compare our proposed TimeDiT with baselines in a full-shot setting, where models are trained and tested on separate datasets. This approach evaluates their performance on conventional time series forecasting tasks, ensuring that models can effectively learn and generalize from complete data. Second, we assess TimeDiT as a foundation model in a zero-shot setting, comparing it to previous transformer- based time series models. This setting is crucial as it tests the model’s ability to generalize and adapt to entirely new datasets without prior exposure, highlighting its robustness and versatility. Together, these experiments provide a holistic view of TimeDiT’s capabilities, addressing both specialized performance and broad applicability in time series forecasting.
Table 2 presents the full-shot forecasting results, comparing TimeDiT with state-of-the-art models in two categories: deterministic forecasting models, which are trained with the Student’s t-distribution head to support probabilistic results, and inherently probabilistic time series forecasting models, including diffusion-based models, such as CSDI and non-diffusion-based, such as GP-copula. Our model achieves the lowest CRPSsum on four datasets and the Solar 0.432(0.002) 0.457(0.019) 0.445(0.002) 0.467(0.002) 0.337(0.024) 0.301(0.014) 0.287(0.020) 0.298(0.004) 0.286(0.003)
Electricity 0.033(0.000) 0.037(0.002) 0.140(0.017) 0.033(0.001) 0.024(0.002) 0.021(0.011) 0.021(0.001) 0.017(0.000) 0.019(0.002) 0.017(0.000)
Taxi 0.177(0.000) 0.190(0.005) 0.181(0.006) 0.187(0.001) 0.208(0.183) 0.179(0.002) 0.114(0.020) 0.123(0.003) 0.303(0.004)
Exchange 0.011(0.001) 0.026(0.001) 0.013(0.001) 0.013(0.001) 0.007(0.000) 0.005(0.003) 0.006(0.001) 0.007(0.001) 0.009(0.001) 0.005(0.001)
Exchange 0.030(0.001) 0.007(0.000) 0.012(0.000) 0.011(0.000) 0.024(0.001)
A PREPRINT - SEPTEMBER 5, 2024
TimeDiT 0.278(0.001)
Table 2: Forecasting results on CRPSsum with full shot setting.
Zero-shot
TEMPO Moirai small Moirai base Moirai large LagLLama
Solar 0.581(0.002) 0.884(0.005) 0.948(0.002) 1.042(0.002) 0.690(0.005)
Electricity 0.081(0.003) 0.079(0.002) 0.072(0.002) 0.039(0.001) 0.065(0.005)
Traffic 0.147(0.000) 0.215(0.000) 0.191(0.001) 0.111(0.000) 0.275(0.001)
Taxi 0.400(0.001) 0.463(0.001) 0.428(0.000) 0.597(0.000) 0.620(0.003)
TimeDiT 0.457(0.002)
Table 3: Forecasting results on CRPSsum. Zero-shot implies that the model has not encountered any samples from the
above datasets during training.
second-best performance on the Taxi dataset. In the zero-shot setting, TimeDiT is compared with the open-sourced foundation models including TEMPO [Cao et al., 2023b], which is pre-trained with Student’s t-distribution head to support probabilistic results, Moirai [Woo et al., 2024b] and LagLLama [Rasul et al., 2023] in Table 3. TimeDiT’s ability to outperform other open-source foundation models in most cases is particularly noteworthy, as it suggests that TimeDiT can be effectively applied to a wide range of time series forecasting tasks across different domains with minimal adaptation.
0.026(0.001)
0.185(0.010)
0.398(0.001)
0.021(0.002)
Datasets
DLinear LightTS ETSformer FEDformer Autoformer PatchTST TimesNet GPT2(3)
ETTh1 ETTh2 ETTm1 ETTm2 Weather Electricity 1st Pl MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE Count
0.201 0.306 0.142 0.259 0.093 0.206 0.096 0.208 0.052 0.110 0.132 0.260 0 0.284 0.373 0.119 0.250 0.104 0.218 0.046 0.151 0.055 0.117 0.131 0.262 0 0.202 0.329 0.367 0.436 0.120 0.253 0.208 0.327 0.076 0.171 0.214 0.339 0 0.117 0.246 0.163 0.279 0.062 0.177 0.101 0.215 0.099 0.203 0.130 0.259 0 0.103 0.214 0.055 0.156 0.051 0.150 0.029 0.105 0.031 0.057 0.101 0.225 0 0.115 0.224 0.065 0.163 0.047 0.140 0.029 0.102 0.034 0.055 0.072 0.183 0 0.078 0.187 0.049 0.146 0.027 0.107 0.022 0.088 0.030 0.054 0.092 0.210 1 0.069 0.173 0.048 0.141 0.028 0.105 0.021 0.084 0.031 0.056 0.090 0.207 1
TimeDiT 0.042 0.135 0.042 0.139 0.023 0.098 0.024 0.083 0.031 0.036 0.069 0.174 10 Table 4: Imputation result on 96-length multivariate time series averaged over the four mask ratios. We calculate MSE
and MAE for each dataset. Bold indicates best result, Underline indicates the second best result. 5.4 Imputation Task
We conduct experiments on six benchmark time-series datasets: ETTh1, ETTh2, ETTm1, ETTm2, Electtricity, and Weather. We use random mask ratios {12.5%, 25%, 37.5%, 50%} following previous studies’ settings with sequence length set to 96. We finetune on model checkpoints pretrained on solar, traffic, exchange, taxi, Huawei cloud, air quality, and weather (different from the evaluation weather data). Table 4 shows the imputation result averaged over the four mask ratios. TimeDiT achieves the best performance on most dataset. obtaining 10 first places out of the 12 evaluations while the remaining baselines obtained 2 first place count in total. In particular, TimeDiT achived a 39% reduction in MSE and 22% reduction in MAE compared to the strongest baseline on ETTh1 dataset. For full result on each mask ratio, please refer to section F.
5.5 Anomaly Detection Task
We conduct experiments on five real-world datasets from industrial applications: MSL, SMAP, SWaT, SMD, and PSM. The diffusion model, renowned for its proficiency in distribution learning, may inadvertently overfit by reconstructing anomalies alongside normal data points. To counteract this, we opted to bypass pretraining and introduced spectral residue (SR) transformation at the preprocessing stage of TimeDiT . This transformation helps to conceal points most likely to be anomalies and their immediate neighbors. The number of neighbors affected is controlled by the hyperparameter nneighbor. The SR method utilizes Fourier Transformation to convert the original time series into a saliency map, thereby amplifying abnormal points, as detailed in [Ren et al., 2019, Zhao et al., 2020]. For additional information about this transformation, please see section G. Consistent with prior methodologies, we set the sequence length to be 100 identify anomalies using the 99th percentile of reconstruction errors. During evaluations, we apply standard anomaly adjustments as suggested by [Xu et al., 2018]. As demonstrated in table G, TimeDiT outperforms baseline models on four of the five datasets. In particular, TimeDiT 23.03 points of improvement in terms of F1 score on the SMAP dataset compared to the strongest baseline.
Methods TimeDiT GPT2(6) TimesNet PatchTS. ETS. FED. LightTS DLinear Auto. Anomaly.
MSL 89.33 82.45 81.84 SMAP 95.91 72.88 69.39 SWaT 96.46 94.23 93.02 SMD 83.28 86.89 84.61 PSM 97.57 97.13 97.34
78.70 85.03 78.57 78.95 68.82 69.50 70.76 69.21 85.72 84.91 93.19 93.33 84.62 83.13 85.08 82.53 96.08 91.76 97.23 97.15
84.88 79.05 69.26 71.12 87.52 92.74 77.10 85.11 93.55 93.29
83.31 71.18 83.10 85.49 79.40
A PREPRINT - SEPTEMBER 5, 2024
1stPlCount 4 1 0 0 0 0 0 0 0 0
We replace the joint criterion in Anomaly Transformer with reconstruction error for fair comparison.
Table 5: Anomaly Detection result on 100-length multivariate time series. We calculate F1 score as % for each dataset. ’.’ notation in model name stands for transformer. Bold indicates best result, Underline indicates the second best result.
Metric
Discriminative Score
Predictive Score
Methods
Sine
0.1217(0.039) 0.0489(0.0562)
Stocks
0.2038(0.057)
Air Quality Energy
0.3913(0.039) 0.4969 (0.000) 0.2869(0.053) 0.4993(0.001) 0.1227(0.006) 0.2301(0.006) 0.2109(0.003) 0.0053(0.002)
0.035(0.002) 0.3305(0.003) 0.0269(0.001) 0.2878(0.001) 0.022(0.002) 0.2506(0.000) 0.0217(0.000) 0.2489(0.000)
TimeGAN TimeVAE
0.1987(0.037) Diffusion-TS 0.0099(0.003) 0.1869(0.0159)
TimeDiT
TimeGAN TimeVAE Diffusion-TS TimeDiT
0.0086(0.004)
0.2797(0.015) 0.2285(0.000) 0.2262(0.000) 0.1915(0.000)
0.0087(0.006)
0.0481(0.002) 0.0485(0.000) 0.042(0.000) 0.0445(0.000)
Table 6: Synthetic Generation result on 24-length multivariate time series. We calculate discriminative and predictive score according to Yoon et al. [2019] and results are averaged over five runs. Bold indicates the best performance. 5.6 Synthetic Generation Task
A PREPRINT - SEPTEMBER 5, 2024
0.4 0.3 0.2 0.1 0.0 0.1 0.2 0.3
TimeDiT
1 x-pca
0.3
0.2
0.1
0.0
0.1
0.2
Diffusion-TS
1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0 2.5 x-pca
0.3
0.2
0.1
0.0
0.1
TimeGAN
1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0 2.5 x-pca
0.4 0.3 0.2 0.1 0.0 0.1
TimeVAE
1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0 2.5 x-pca
1
0
2
3
4
Original Synthetic
Original Synthetic
Original Synthetic
y_pca
y_pca
y_pca
y_pca
Original Synthetic
Figure 3: PCA Evaluation of Synthetic Time Series from TimeDiT and other baselines on stock dataset.
We conduct experiments to synthesize multivariate time series and evaluate performance using the discriminative score and predictive score metrics under a "train on synthetic test on real" experimental setup with sequence length set to 24 [Yuan and Qiao, 2024]. We finetune on model checkpoints pretrained on etth1, etth2, ettm1, ettm2, electricity, and weather. Table 5.5 shows the result on synthetic generation where TimeDiT, in general, consistently generates more realistic synthetic samples compared to baselines, even on high-dimensional energy datasets. This demonstrates TimeDiT’s strength in complex time series synthesis. We visualize synthesis performance using PCA and t-SNE in Appendix E. As shown in Figure 3, TimeDiT’s samples markedly overlap the original data distribution better than other methods. Qualitative and quantitative results confirm TimeDiT’s superior ability to model intricate characteristics for realistic time series synthesis, even on multidimensional, complex datasets.
6 Conclusion
In this paper, we introduced TimeDiT, a pioneering approach to creating a versatile and robust foundation model for various time series tasks under practical scenarios. By integrating transformer architecture with diffusion model, TimeDiT effectively captures temporal dependencies and addresses real world challenges unique to time series regarding multi-resolution and missing values as well as incorporating external knowledge. Our innovative masking strategies allow for a consistent training framework adaptable to diverse tasks such as forecasting, imputation, and anomaly detection and synthetic data generation. Extensive experiments demonstrated the strong performance of TimeDiT on both practical scenarios and standard benchmarks. However, we recognize some limitations. We primarily explored common sequence lengths and did not assess TimeDiT’s performance on very long sequences. While we have introduced randomness in prediction length and feature numbers up to a maximum, we aim to develop more scalable solutions for highly variable multivariate time series. Additionally, our understanding of how different types of external information contribute to performance is still developing. For future work, we envision several key directions: enhancing scalability to improve TimeDiT’s ability to handle practical time series with varying multivariate numbers; developing techniques for seamless multi-modal integration, allowing TimeDiT to leverage diverse data sources for improved performance across different tasks; and extending TimeDiT’s capabilities to effectively process and analyze very long time series sequences, addressing a critical need in many real-world applications.