TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model

## Abstract

Abstract With recent advances in building foundation models for texts and video data, there is a surge of interest in foundation models for time series. A family of models have been developed, utilizing a temporal auto-regressive generative Transformer architecture, whose effectiveness has been proven in Large Language Models. While the empirical results are promising, almost all existing time series foundation models have only been tested on well-curated “benchmark” datasets very similar to texts. However, real-world time series exhibit unique challenges, such as variable channel sizes across domains, missing values, and varying signal sampling intervals due to the multi-resolution nature of real-world data. Additionally, the uni-directional nature of temporally auto-regressive decoding limits the incorporation of domain knowledge, such as physical laws expressed as partial differential equations (PDEs).
To address these challenges, we introduce the Time Diffusion Transformer (TimeDiT), a general foundation model for time series that employs a denoising diffusion paradigm instead of temporal auto-regressive generation. TimeDiT leverages the Transformer architecture to capture temporal dependencies and employs diffusion processes to generate high-quality candidate samples without imposing stringent assumptions on the target distribution via novel masking schemes and a channel alignment strategy.
Furthermore, we propose a finetuning-free model editing strategy that allows the seamless integration of external knowledge during the sampling process without updating any model parameters.
Extensive experiments conducted on a varity of tasks such as forecasting, imputation, and anomaly detection, demonstrate the effectiveness of TimeDiT.

## 1 Introduction

Time series analysis is pivotal in a diverse set of AI applications, such as natural science (Cuomo et al., ; Wang et al., ), social science (Zhang et al., ; Sharma et al., ),sustainability (Krenn and Buffoni, ),health (Kaushik et al., ; Kamra et al., ), etc. These applications root in diverse domains  (Li et al., ; Bi et al., ; Cao et al., ; Zhang et al., ; Ye and Gao, ), leading to time series with various distributions (Wang et al., ) and a divers set of analysis tasks, such as forecasting (Nie et al., ; Jia et al., ), imputation (Tashiro et al., ), anomaly detection (Tuli et al., ), etc. Even though considerable progress has been made in developing specialized models optimized for specific scenarios and individual tasks, an open question remains: Can a single time series foundation model excel across domains? Recent initiatives have explored the possibility of universal time series models on zero-shot setting (Ansari et al., ; Liu et al., ; Gruver et al., ; Cao et al., ), drawing inspiration from large pre-trained language models in natural language processing(NLP) and computer vision(CV), such as GPT(Radford et al., ), CLIP (Radford et al., ), which are known for their robust transfer learning capabilities. However, due to the fundamentally different semantics between text/images and time series data, the unique challenges of achieving a truly flexible and general-purpose time series model remain an open problem.

Recently, the emergence of LLMs like GPT-4 (OpenAI, ) and LLaMA (Touvron et al., ) suggests the potential for building time series foundation models enabling a general solution to handle multiple time series distributions.
Previous attempts typically build upon the transformer backbone, which has achieved state-of-the-art performance on various time series tasks, particularly in modeling long-term dependencies.
However, the tokenization of time series data for transformers is especially sensitive to variations in data sources and sampling rates. Previous tokenization approaches with different schemes including token patching (Das et al., ; Woo et al., ); discretization tokens (Talukder et al., ) and tokens based on time series features (Yue et al., ; Ansari et al., ; Rasul et al., ) have either fragmented the global information or have been constructed in a manner that inherently loses important information. Most, if not all, of them employ a channel independence strategy (Nie et al., ) or focus solely on univariate time series. Channel independence strategy, though beneficial in certain contexts, often overlooks the complex inter-temporal and cross-feature dependencies in practical applications and thus presents an opportunity for optimization (Zeng et al., ).

Moreover, compared with texts and images, time series exhibit unique characteristics such as missing values (Kollovieh et al., ), irregular sampling (Kidger et al., ; Cao et al., ; Zhang et al., ), multi-resolution (Niu et al., , ), etc.
To address these challenges, a foundation model for time series must be capable of demonstrating flexibility across different scales to handle diverse inputs with varying distributions. However, these unique natures and challenges are not covered by the popular well-curated benchmark datasets(Li et al., ; Zhou et al., ; Alexandrov et al., ). As a result, most existing works, which are developed and evaluated primarily on these datasets, may not fully address the complexities encountered in real-world time series applications. In addition, time series processes are often governed by underlying physical principles (Li et al., ; Meng et al., ; Xiao et al., ) and can be guided by domain-specific textual information (Jin et al., ; Sun et al., ). However, integrating these diverse sources of information into a unified model poses further challenges, as the model must effectively leverage the relevant physics context while adapting to the unique characteristics and distributions of each domain. Addressing each of these issues requires innovative approaches in data preprocessing, model architecture, and training strategies to create models that can seamlessly handle the diverse and complex nature of time series data.

Current approaches to time series modeling lack a unified framework for handling the aforementioned diverse and imperfect data inputs, frequently prioritizing performance on well-curated benchmarks over addressing real-world challenges. Diffusion models, such as DDPM (Ho et al., ), offer a promising solution by framing data generation as a series of conditional transformations, effectively recasting density estimation as sequential reconstruction. Unlike autoregressive methods that generate future tokens sequentially, diffusion models can directly produce high-quality samples through a reverse denoising process. This process can be analogized to solving partial differential equations (PDEs), allowing for the natural incorporation of physics-based knowledge.
This capability, combined with transformers’ strength in capturing temporal dependencies, presents an opportunity to develop a more versatile and robust time series foundation model. Such a hybrid approach could effectively address the complexities of real-world data while maintaining the flexibility to adapt to various forecasting tasks and data conditions.

In this work, we introduce TimeDiT—a diffusion transformer-based foundation model equipped with a standardized training pipeline for different shapes of input time series and tailored for diverse distributions and downstream tasks. TimeDiT leverages the Transformer architecture’s inherent ability to capture temporal dependencies through its attention mechanisms, while also benefiting from the scalability that allows for increased model capacity crucial for complex time series tasks. By adopting a diffusion model approach, TimeDiT treats time series holistically, avoiding the error accumulation issues common in autoregressive solutions. The model incorporates a novel comprehensive mask mechanism that enables a single, unified foundation model to handle multiple tasks without additional modules or parameters. This design naturally addresses real-world challenges such as multi-resolution data and missing values. During the sampling stage, TimeDiT introduces an innovative strategy to incorporate physics knowledge as an energy-based prior, supported by theoretical guarantees. This approach guides the reverse diffusion process using physics-based constraints, including partial differential equations, resulting in generated samples that adhere to known physical laws and domain-specific requirements, thereby enhancing sample quality and model applicability across various scientific and engineering contexts.

TimeDiT’s performance is rigorously evaluated through an extensive experimental setup encompassing over 20 diverse datasets from domains including traffic, weather, finance, etc. The model is benchmarked against more than 25 open-source baselines, ranging from linear-based models to diffusion-based models, transformer-based models, and other forecasting foundation models. These comprehensive experiments cover multiple challenging time series tasks, including in-domain and zero-shot probabilistic forecasting, imputation, anomaly detection, and synthetic data generation. TimeDiT demonstrated state-of-the-art or highly competitive results across these tasks, showcasing its effectiveness and efficiency as a foundation model for various time series applications. Notably, TimeDiT achieved new state-of-the-art CRPSsum scores on the Electricity and Traffic datasets for probabilistic forecasting. In addition, the results on zero-shot experiments show that our model can be used as a foundation model even without fine-tuning, although fine-tuning may be necessary in some cases. Furthermore, TimeDiT’s scalability and adaptability are evident in its ability to incorporate external knowledge, such as physical constraints, during the sampling stage. This feature allows for the generation of samples that better conform to known physical laws and domain-specific requirements. This combination of state-of-the-art performance, adaptability across diverse tasks, scalability, and the ability to incorporate domain-specific knowledge positions TimeDiT as a powerful and versatile foundation model, capable of addressing a wide spectrum of time series challenges and opening new avenues for advanced time series analysis across various fields.

In summary, our contributions can be summarized as three unfolds:

- •
We introduce TimeDiT, a novel diffusion transformer-based foundation model for time series analysis. By combining the strengths of diffusion models and transformers, our approach offers a flexible architecture adaptable to various downstream tasks. The model incorporates a comprehensive mask mechanism for reconstruction pretraining and task-specific fine-tuning, ensuring a standardized training pipeline capable of handling diverse input shapes and distributions.
- •
Unlike autoregressive approaches, TimeDiT addresses real-world challenges in time series data by directly processing multivariate inputs and employing a denoising process to generate cohesive target time series. This method effectively handles issues such as missing values and multi-resolution data. In addition, TimeDiT can generate time series that adhere to known physical laws and domain-specific requirements, enhancing its applicability in scientific and engineering contexts.
- •
Evaluated on over multiple datasets across different domains and tasks, TimeDiT achieves state-of-the-art or competitive results. It excels in probabilistic forecasting, imputation, anomaly detection, and data generation, showcasing its versatility as a foundation model in both in-domain and zero-shot settings.

## 2 Related Work

#### General Purpose Time Series Model

In the past decades, researchers have excelled in designing sophisticated models for specific time series analysis tasks (Zhang et al., ; Fan et al., ; Cao et al., , ). However, in recent years, the emergence of large language models has inspired the development of general-purpose time series models (Zerveas et al., ; Zhang et al., ; Garza and Mergenthaler-Canseco, ) and the field of time series has seen tremendous exploration efforts towards foundation models. (Gruver et al., ) simply encoded time series as strings while (Jin et al., ) converted time series into language representations by alignment. (Cao et al., ) and (Pan et al., ) further incorporated decomposition technique and prompt design and generalizes to unseen data and multimodal scenarios. (Rasul et al., ) worked towards foundation model from a probabilistic perspective but only considered univariate time series only which rarely appears in real-life. Additionally, many studies started to follow a two-stage training paradigm of pretraining and finetuning (Chang et al., ; Dong et al., ; Nie et al., ). However, these works mainly focused on the forecasting task only (Woo et al., ; Das et al., ). (Zhou et al., ) first adapted GPT2 as a general-purpose time series analysis model and extended it to various time series tasks. (Talukder et al., ) leveraged VQVAE as a tokenizer for transformer to handle time series tasks and (Ansari et al., ) employed a scaling and quantization technique to embed time series. For more detailed literatures of the general-purpose time series model, please refer to recent surveys and position paper (Liang et al., ; Jin et al., ; Jiang et al., )

#### Diffusion models for Time Series

Despite the growing interest of diffusion models in various scenarios (Peebles and Xie, ; Li et al., ; Lu et al., ; Sui et al., , ), the use of diffusions in time series analysis is less explored compared to pre-trained language models and transformers. Most existing studies also focused solely on forecasting and the choice of backbone model also varies among VAE(Li et al., ), RNN(Rasul et al., ), and transformer. CSDI (Tashiro et al., ) utilized diffusion model for time series imputation. (Yuan and Qiao, ) incorporated decomposition into diffusion model to improve interoperability.
Although  (Kollovieh et al., ) build a diffusion pipeline for multiple tasks with refinement, they still train different models for each task. To the best of our knowledge, there has been no exploration of leveraging unified diffusion models for a comprehensive set of time series tasks yet. Please refer to  (Yang et al., ) for a comprehensive literature review on diffusion models for time series analysis.

## 3 Preliminaries

### 3.1 Diffusion Model

In recent years, diffusion models have emerged as a promising approach in generative modeling. A diffusion process is a Markov chain that incrementally adds Gaussian noise to data over a sequence of steps, effectively destroying the data structure in forward process and destroying the data structure in backward structure.

The forward process adds noise to the data $\mathbf{x}_{0}$ over a series of timesteps $t$ according to a variance schedule $\beta_{t}$, resulting in a set of noisy intermediate variables $\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{T}$. Each subsequent $\mathbf{x}_{t}$ is derived from the previous step by applying Gaussian noise:

$$ $q(\mathbf{x}_{t}\mid\mathbf{x}_{t-1})=\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta _{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I})$ (1) $$

The reverse process aims to denoise the noisy variables step by step, sampling each $\mathbf{x}_{t-1}$ from the learned distribution $p_{\theta}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t})$. This distribution, modeled by a neural network parameterized by $\theta$, approximates the Gaussian distribution:

$$ $p_{\theta}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{t-1}; \mu_{\theta}(\mathbf{x}_{t},t),\Sigma_{\theta}(\mathbf{x}_{t},t))$ (2) $$

By iterating this reverse process from $t=T$ down to $t=0$, the model gradually reconstructs the original data from noise. The reverse process learns to predict the mean and covariance of each intermediate distribution, effectively approximating the original data distribution.

## 4 Methodology

Figure: Figure 1: TimeDiT Architecture. Left: TimeDiT framework with diverse multivariate time series from different domains with multi-resolution, missing values; Middle: Structure of TimeDiT block; Right top: Illustration of masks generated by Time Series Mask Unit, reconstruction mask is neglected as it’s an all-zero mask; Right bottom: Masks for downstream tasks that TimeDiT handles during inference.
Refer to caption: x1.png

In this section, we present our main contributions: the proposed foundation model, TimeDiT , a diffusion model with transformer backbone designed for multiple time series tasks, along with uniform masking strategies and incorporation of physics knowledge and textual information as an extension. We first outline the uniformed problem setting for multiple down-stream tasks and offer an in-depth examination of the model architecture. Subsequently, we delve into the training pipeline with mask strategies, which help to build the training scheme in self-supervised learning for time series. Next, we present how to incorporate external information to improve the model’s performance during both the training and inference stages. By doing so, TimeDiT can generate samples that better conform to real-world requirements and enhance its performance on various downstream tasks. These extensions showcase the flexibility and adaptability of our proposed model, making it a powerful tool for a wide range of time series applications.

### 4.1 Problem Definition

We denote a multivariate time series as $\mathbf{X}=\{x_{i,j}\}\in\mathbb{R}^{K\times L}$, where $K$ is the number of features and $L$ is the length of the time series. Each individual entry $x_{i,j}$ represents the $j$-th feature at time step $l$, for $i\in\{1,\dots,K\}$ and $j\in\{1,\dots,L\}$.
We define an observation mask $\mathbf{M_{obs}}=\{m_{i,j}\}\in\{0,1\}^{K\times L}$, where $m_{i,j}=0$ if $x_{i,j}$ is missing, and $m_{i,j}=1$ if $x_{i,j}$ is observed.
Let $\mathbf{x}_{0}^{\text{obs}}\in X^{\text{obs}}$ denote the observed subsequence; $\mathbf{x}_{0}^{\text{tar}}$ denote the target subsequence of $\mathbf{x}_{0}^{obs}$ which could be forecast target or imputation target or the whole sequence depending on the task. Let $\mathbf{x}_{0}^{\text{con}}$ denote the unmasked partial observations in $\mathbf{x}_{0}^{obs}$ which acts like conditions for the masked area $\mathbf{x}_{0}^{tar}$. Let us use all subscripts of $x$ to denote diffusion timestamp, and a subscript of 0 means no noise has been applied to the original data. Formally, the goal of our task is to approximate the true conditional data distribution given the conditional information
$q_{\mathbf{X}}\left(\mathbf{x}_{0}^{\mathrm{ta}}\mid\mathbf{x}_{0}^{\mathrm{
con}}\right)$
with a model distribution $p_{\theta}(\mathbf{x}^{\text{tar}}_{0}\mid\mathbf{x}^{\text{con}}_{0})$, which can be calculated by a diffusion model with conditional information:

$$ $\displaystyle p_{\theta}\left(\mathbf{x}_{0:T}^{\mathrm{tar}}\mid\mathbf{x}_{0 }^{\mathrm{con}}\right):=p\left(\mathbf{x}_{T}^{\mathrm{tar}}\right)\prod_{t=1 }^{T}p_{\theta}\left(\mathbf{x}_{t-1}^{\mathrm{tar}}\mid\mathbf{x}_{t}^{ \mathrm{tar}},\mathbf{x}_{0}^{\mathrm{con}}\right),\mathbf{x}_{T}^{\mathrm{tar }}\sim\mathcal{N}(\mathbf{0},\mathbf{I}),\text{where}$ $\displaystyle p_{\theta}\left(\mathbf{x}_{t-1}^{\mathrm{tar}}\mid\mathbf{x}_{t }^{\mathrm{tar}},\mathbf{x}_{0}^{\mathrm{con}}\right):=\mathcal{N}\left( \mathbf{x}_{t-1}^{\mathrm{tar}};\bm{\mu}_{\theta}\left(\mathbf{x}_{t}^{\mathrm {tar}},t\mid\mathbf{x}_{0}^{\mathrm{con}}\right),\sigma_{\theta}\left(\mathbf{ x}_{t}^{\mathrm{tar}},t\mid\mathbf{x}_{0}^{\mathrm{con}}\right)\mathbf{I} \right).$ (3) $$

The mask mechanism $\mathbf{M}$ plays a critical role in identifying the positions of $\mathbf{x}_{0}^{\mathrm{con}}$ and $\mathbf{x}_{0}^{\mathrm{tar}}$. By leveraging these positional differences, our model adeptly adapts to various downstream tasks, including forecasting, imputation, anomaly detection, etc, within a unified framework.

### 4.2 Time Series Diffusion Transformer

Figure 1 shows the overall framework of TimeDiT. We first establish the $\mathbf{M_{obs}}$ and $\mathbf{x}_{0}^{\text{obs}}$ based on the given input from different distributions with multivariate sequences, missing value and multi-resolution by injecting placeholders to standardize the input shape across different time series, facilitating more efficient and consistent processing.
Then, the unified time series mask unit adapts to diverse time series scenarios and builds the $\mathbf{x}_{0}^{\text{con}}$, $\mathbf{M}$ and $\mathbf{x}_{0}^{\text{tar}}$, with shape $\mathbb{R}^{B\times L\times K}$,
to help TimeDiT learn robust representations in a self-supervised manner by reconstructing the original sequence through denoising $\mathbf{x}_{T}^{\text{tar}}$. After that, the embedding layer directly treats $\mathbf{x}_{0}^{\text{con}}$ and $\mathbf{x}_{0}^{\text{tar}}$ as without any patching, as the diffusion process is designed to handle multivariate input and operate in a continuous token space.
By preserving the integrity of the input time series, TimeDiT ensures that the model can effectively capture and utilize the rich information contained within the data. The TimeDiT block’s attention mechanism is designed to autonomously learn cross-channel and temporal correlations through end-to-end training.

#### Standardize Pipeline

We introduce placeholders within the input sequences to standardize the input shape across different time series, accounting for varying channel numbers $K$ and sequence length $L$. Specifically, we define the maximum channel number $K_{max}$ such that any input with channel $k<K_{max}$ is padded to have $K^{max}$ channels while any input with more than $K_{max}$ channels are segmented into $\lceil\frac{k}{K_{max}}\rceil$ blocks of inputs where each block has $K_{max}$ channels and undergoes independent processing. This segmentation allows our model to manage high-dimensional data efficiently, reducing computational overhead and maintaining relative positional integrity of the data and consistency across inputs. Additionally, for any input with sequence length less than the designated maximum length $L_{max}$, we pad the sequence in the front to achieve the desired length. This standardization is essential for establishing a uniform input structure that enhances processing efficiency and consistency.

#### Time Series Mask Unit

We propose a unified time series mask mechanism that includes a variety of masks that seamlessly integrates with the model during self-supervised task agnostic pre-training and task specific fine-tuning to cater to diverse time series scenarios. The time series mask unit generate four types of masks: reconstruction mask, stride mask, block mask, and random mask. Firstly, the task-agnostic pre-training aims to improve the overall time series representation by encouraging the model to learn robust and generalizable features from the input data. Secondly, the task-specific training is designed specifically for the most common downstream tasks, including forecasting and imputation, enabling the model to adapt to the unique requirements of each task.

As shown in Figure 1 right top, given $\mathbf{x}\in\mathbb{R}^{K\times L}$, the random mask $\text{M}^{\text{R}}$ can be generated by:

$$ $\mathbf{M}^{\text{R}}(x,r)=\begin{cases}1&z_{i,j}>r,z\in\mathbb{R}^{K\times L} ,z\sim Uniform(0,1)\\ 0,&otherwise,\end{cases}$ (4) $$

where $r$ is the mask ratio. In addition, for task-specific training and inference, we allow the user to supply customized imputation masks that could simulate the naturally missing data and multi-resolution cases.

Block mask $\mathbf{M}^{\text{B}}$ can be generated via:

$$ $\mathbf{M}^{\text{B}}(x,l)=\begin{cases}1&j<L-l,\\ 0,&otherwise,\end{cases}$ (5) $$

where $l$ is the predicted length. We can randomly select $l$ during pretraining and use the designated prediction length during the finetuning and inference stage for specific experiment settings.

Stride mask $\mathbf{M}^{\text{S}}$, a variant of $\mathbf{M}^{\text{B}}$, is placed intermittently within the series and is defined as follows:

$$ $\mathbf{M}^{\text{S}}(x,n_{\text{blocks}})=\begin{cases}1&\left\lfloor\frac{j} {b}\right\rfloor\bmod 2=0\\ 0&\text{otherwise}\end{cases}$ (6) $$

where $n_{block}$ is the number of blocks into which the sequence is divided; $b=\left\lceil\frac{L}{n_{\text{blocks}}}\right\rceil$ is the length of each block; $j$ is the index of the sequence. $\mathbf{M}^{\text{S}}$is designed for task-agnostic pretraining further to enhance the model’s representation ability by integrating information across non-contiguous parts of the series.

In addition, reconstruction mask $\mathbf{M}^{\text{Rec}}=0$ is used for tasks including synthetic data generation and anomaly detection, where we can directly generate synthetic data or obtain an anomaly score for each temporal position based on the difference between original and reconstructed series.

For the pretraining stage, we random select one conditional mask type from $\mathbf{M}=\{\mathbf{M}^{\text{R}},\mathbf{M}^{\text{S}},\mathbf{M}^{\text{B}}
,\mathbf{M}^{\text{Rec}}\}$ for each instance. TimeDiT’s goal is to reconstruct the $\mathbf{x}^{\text{tar}}$, defined as $\mathbf{x}_{0}\times(J-\mathbf{M})$ where $J$ is unit matrix. The target sequence is the masked portion of the original sequence. In the finetuning and inference stage, the choice of mask is tailored to align with the specific requirements of the user. This flexibility allows TimeDiT to apply the most appropriate masking strategy based on the context of the task and application.

#### Condition Injection

Instead of following  (Peebles and Xie, ) to integrate diffusion timestep and label information (texts in out case) through the layer normalization preceeding the attention block, we add the diffusion timestep and texts information to the target noise as these are universal information to the series.

Given that TimeDiT utilizes a transformer-based architecture, a straightforward and intuitive approach is to include conditional information directly as part of the input sequence by concatenation as done in latent diffusion  (Rombach et al., ). However, we empirically found out that controlling mean and variance through layer normalization is a stronger form of conditional information injection. We incorporate the partial observations $x_{0}^{c}on$ through adaptive layer normalization to control the scale and shift of the $x_{0}^{tar}$. This design choice is motivated by the face that the scale and shift of the partial observations are more relevant to the target observations. This integration can be expressed as

$$ $\text{AdaLN}(h,c)=c_{scale}\text{LayerNorm}(h)+c_{shift}$ (7) $$

where $h$ is the hidden state and $c_{\text{scale}}$ and $c_{\text{shift}}$ are the scale and shift parameters derived from the partial observations. We perform temporal attention in the self-attention block to capture the temporal dependency within the input.

#### Inference

We perform pre-training and finetuning for forecasting and imputation. We exclude the pre-training process from anomaly detection and synthetic generation because the two tasks are very dataset specific and do not necessarily benefit from learning distributions beyond the target dataset. Let $n$ represent the number of samples generated for each prediction, which we set to $n=10$ ($n=30$ for forecasting tasks) in our experimental setup at inference time. We use the median of these $n$ predictions as the final prediction, providing the added benefit of obtaining a confidence interval for TimeDiT ’s predictions. To prevent channel padding from affecting the generated samples, we mask out the invalid channels during sampling at each diffusion timestep so that TimeDiT does not falsely treat information in the non-valid channels as meaningful information. Padding is applied at the beginning of the temporal dimension to ensure that the most relevant information remains at the end, thereby mitigating the effect of padding.

### 4.3 Physics-Informed TimeDiT

Figure: Algorithm 1 Physics Informed TimeDiT through Energy-based Sampling

Physics principles are fundamental in shaping the evolution of temporal signals observed in real-world phenomena, such as climate patterns and oceanographic data. Therefore, it is essential to integrate physical knowledge into foundational time series models. In this section, we propose a strategy to incorporate physics knowledge as an energy-based prior for TimeDiT during inference, which iteratively refines the reverse diffusion process. By guiding the denoising process during inference with gradients derived from physical laws represented by partial differential equations (PDEs), the integration of this knowledge can significantly enhance the quality of the generated samples.

A generic form of a physical law represented as a PDE that describes the evolution of a continuous temporal signal $\mathbf{x}(\mathbf{u},t)$ over a spatial coordinate $\mathbf{u}$ is given by:

$$ $\frac{\partial\mathbf{x}}{\partial t}=F(t,\mathbf{x},\mathbf{u},\frac{\partial \mathbf{x}}{\partial\mathbf{u}_{i}},\frac{\partial^{2}\mathbf{x}}{\partial \mathbf{u}_{i}\partial\mathbf{u}_{j}},\dots)$ (8) $$

Based on this PDE representation of physical knowledge, the consistency between the predicted time series $\mathbf{x}^{\text{{tar}}}$ and the physics knowledge can be quantified using the following squared residual function:

$$ $K(\mathbf{x}^{\text{{tar}}};F)=-||\frac{\partial\mathbf{x}^{\text{{tar}}}}{ \partial t}-F(t,\mathbf{x}^{\text{{tar}}},\mathbf{u},\frac{\partial\mathbf{x}^ {\text{{tar}}}}{\partial\mathbf{u}_{i}},\frac{\partial^{2}\mathbf{x}^{\text{{ tar}}}}{\partial\mathbf{u}_{i}\partial\mathbf{u}_{j}},\dots)||^{2}_{2}$ (9) $$

This function reaches its maximum when the predicted time series is perfectly consistent with the physical model, resulting in a residual of 0. Using this metric $K$, physics knowledge can be integrated into a probabilistic time series foundation model $p(\mathbf{x}^{\text{{tar}}}|\mathbf{x}^{\text{{con}}})$ by solving the following optimization problem to obtain a refined model $q(\mathbf{x}^{\text{{tar}}}|\mathbf{x}^{\text{{con}}})$:

$$ $q(\mathbf{x}^{\text{{tar}}}|\mathbf{x}^{\text{{con}}})=\arg\max_{q}\mathbb{E}_ {\mathbf{x}^{\text{{tar}}}\sim q}K(\mathbf{x}^{\text{{tar}}};F)-\alpha D_{KL}( q(\mathbf{x}^{\text{{tar}}}|\mathbf{x}^{\text{{con}}})||p(\mathbf{x}^{\text{{ tar}}}|\mathbf{x}^{\text{{con}}}))$ (10) $$

where the first term represents the aforementioned physics knowledge metric, and the second term controls the divergence between $q(\mathbf{x}^{\text{{tar}}}|\mathbf{x}^{\text{{con}}})$ and $p(\mathbf{x}^{\text{{tar}}}|\mathbf{x}^{\text{{con}}})$.
The following theorem provides a closed-form solution to the above optimization problem:

###### Theorem 4.1 .

The optimal $q(\mathbf{x}^{\text{tar}}|\mathbf{x}^{\text{{con}}})$ in Eq.10 is the Boltzmann distribution defined on the following energy function:

$$ $E(\mathbf{x}^{tar};\mathbf{x}^{con})=K(\mathbf{x}^{tar};F)+\alpha\log p( \mathbf{x}^{tar}|\mathbf{x}^{con})$ (11) $$

in other words, the optimal $q(\mathbf{x}^{tar}|\mathbf{x}^{con})$ is:

$$ $q(\mathbf{x}^{tar}|\mathbf{x}^{con})=\frac{1}{Z}\exp(K(\mathbf{x}^{tar};F)+ \alpha\log p(\mathbf{x}^{tar}|\mathbf{x}^{con})),$ (12) $$

where $Z=\int\exp(K(\mathbf{x}^{tar};F)+\alpha\log p(\mathbf{x}^{tar}|\mathbf{x}^{con
}))d\mathbf{x}^{tar}$ is the partition function.

The theorem illustrates that sampling from the Boltzmann distribution defined in Eq.  11, is analogous to incorporating physics knowledge into model edition. In the context of diffusion models, this distribution can be effectively sampled using Langevin dynamics (Stoltz et al., ):

$$ $\displaystyle\mathbf{x}^{\text{{tar}}}_{j+1}$ $\displaystyle=\mathbf{x}^{\text{{tar}}}_{j}+\epsilon\nabla\log q(\mathbf{x}^{ \text{{tar}}}|\mathbf{x}^{\text{{con}}})+\sqrt{2\epsilon}\sigma,\sigma\sim \mathcal{N}(0,1)$ (13) $\displaystyle=\mathbf{x}^{\text{{tar}}}_{j}+\epsilon\nabla K(\mathbf{x}^{\text {{tar}}}_{j};\mathbf{x}^{\text{{con}}})+\alpha\epsilon\nabla\log p(\mathbf{x}^ {\text{{tar}}}_{j}|\mathbf{x}^{\text{{con}}})+\sqrt{2\epsilon}\sigma,\sigma \sim\mathcal{N}(0,1)$ $$

In diffusion model, precisely calculate the likelihood $\log p(\mathbf{x}^{\text{{tar}}}|\mathbf{x}^{\text{{con}}})$ is intractable. To tackle this issue, following previous works (Kollovieh et al., ), we approximate likelihood with the objective to edit the pre-trained diffusion model:

$$ $\log p(\mathbf{x}^{\text{{tar}}}|\mathbf{x}^{\text{{con}}})=-\mathbb{E}_{ \epsilon,t}[||\epsilon_{\theta}(\mathbf{x}^{\text{tar}},t;\mathbf{x}^{\text{{ con}}})-\epsilon||^{2}]$ (14) $$

The approximation presented above constitutes the optimizable component of the evidence lower bound(ELBO). Algorithm 1 summarizes the comprehensive model editing process.

###### Theorem 4.1 .

## 5 Experiments

To comprehensively assess our time series foundation model, we evaluate on a diverse set of tasks that reflect real-world challenges and applications. We begin by testing the model’s performance in practical scenarios and its ability to integrate domain knowledge. This includes handling missing data and multi-resolution forecasting on customized datasets, which allows us to evaluate the model’s robustness in situations that frequently occur in real-world applications, as well as physics-informed modeling crucial for scientific and engineering domains(Yuan and Qiao, ), which uses 6 practical partial differential equations (PDEs). We then assess the model’s capabilities in well-established benchmarking tasks across various fields such as finance, healthcare, and industrial monitoring. These tasks include forecasting on Solar, Electricity, Traffic, Taxi, and Exchange datasets(Tashiro et al., ; Rasul et al., ) to evaluate temporal dependency modeling, imputation on ETTh, ETTm, Weather and Electricity datasets(Zhou et al., ) to assess the handling of missing data,
anomaly detection on MSL, SMAP, SWaT, SMD, and PSM datasets(Xu et al., ; Zhao et al., ) to gauge sensitivity to unusual patterns,
and synthetic data generation on Stock, Air Quality, and Energy datasets(Yoon et al., ; Desai et al., ) to test understanding of underlying distributions. By evaluating these diverse tasks, we can demonstrate that our model truly serves as a foundation for various time series applications, potentially reducing the need for task-specific models.

Figure: Figure 2: Visualization of miss value (a) and multi resolution (b) forecasting results on the Exchange dataset. Compared between our model TimeDiT and state-of-the-art diffusion-based methods. The x-axis number in (b) is the number of different sampling resolutions in the multivariate input.
Refer to caption: x2.png

### 5.1 Practical Scenarios: Missing Data and Multi-Resolution Forecasting

To simulate more realistic scenarios in time series tasks, we introduced two additional challenges: missing values and multi-resolution data. These conditions are common in real-world applications and test a model’s robustness and adaptability.
For the missing value scenario, we created datasets with various missing ratios, simulating incomplete data often encountered in practice. In the multi-resolution setting, we sampled each individual time series within the multivariate dataset at different resolutions, reflecting the diverse sampling frequencies often present in real-world data collection.

Figure  2 illustrates TimeDiT’s performance in realistic scenarios, showcasing its effectiveness across different sampling frequencies on the Exchange dataset. In Figure  2 (a), we observe TimeDiT’s superior performance in handling missing data. As the missing ratio increases from 5% to 50%, TimeDiT maintains the lowest CRPSsum across all scenarios, indicating its robustness to data gaps. The performance gap between TimeDiT and other models widens as the missing ratio increases, highlighting its effectiveness in more challenging conditions.
Figure  2 (b) demonstrates TimeDiT’s ability to manage multi-resolution data, where it maintains a clear performance advantage as the number of different sampling resolutions increases from 2 to 6. This demonstrates its ability to effectively integrate and forecast time series data sampled at varying frequencies.
These findings underscore TimeDiT’s potential as a practical and versatile tool for time series forecasting in diverse, challenging scenarios that more closely resemble real-world applications.

**Table 1: Physics informed results: This table presents results of PDE forecasting for various sampling strategies, demonstrating the model’s ability to incorporate physics constraints without updating the foundation model. Metrics shown include point estimates metrics, which is MSE, RMSE and MAE; probabilistic metrics, CRPS, for uncertainty quantification. Lower values indicate better performance and closer adherence to physical laws.**
|  | MSE | RMSE | MAE | CRPS | MSE | RMSE | MAE | CRPS |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Advection | Navier-Stokes |  |  |  |  |  |  |
| DDPM | 0.011(0.000) | 0.106(0.001) | 0.084(0.001) | 0.472(0.007) | 0.309(0.004) | 0.556(0.004) | 0.332(0.005) | 0.415(0.006) |
| DDIM | 0.015(0.000) | 0.122(0.002) | 0.096(0.001) | 0.559(0.009) | 0.350(0.014) | 0.591(0.011) | 0.377(0.009) | 0.470(0.013) |
| TSDiff | 0.011(0.000) | 0.106(0.022) | 0.085(0.001) | 0.472(0.011) | 0.399(0.008) | 0.556(0.007) | 0.331(0.006) | 0.414(0.007) |
| TimeDiT | 0.010(0.000) | 0.103(0.002) | 0.082(0.001) | 0.464(0.008) | 0.299(0.006) | 0.546(0.006) | 0.322(0.06) | 0.403(0.007) |
|  | Burgers | Vorticity |  |  |  |  |  |  |
| DDPM | 0.016(0.001) | 0.128(0.004) | 0.101(0.003) | 1.787(0.040) | 1.917(0.020) | 1.385(0.007) | 0.851(0.009) | 0.476(0.005) |
| DDIM | 0.018(0.000) | 0.136(0.001) | 0.116(0.001) | 1.858(0.015) | 1.567(0.031) | 1.252(0.012) | 0.754(0.012) | 0.401(0.006) |
| TSDiff | 0.017(0.001) | 0.129(0.005) | 0.102(0.004) | 1.800(0.055) | 1.966(0.073) | 1.402(0.026) | 0.866(0.010) | 0.485(0.005) |
| TimeDiT | 0.011(0.001) | 0.104(0.005) | 0.083(0.003) | 1.395(0.053) | 1.524(0.523) | 1.234(0.021) | 0.772(0.009) | 0.445(0.006) |
|  | Diffusion Sorption | CFD |  |  |  |  |  |  |
| DDPM | 0.309(0.004) | 0.556(0.004) | 0.332(0.005) | 0.415(0.006) | 0.004(0.000) | 0.065(0.001) | 0.054(0.000) | 0.082(0.000) |
| DDIM | 0.349(0.013) | 0.591(0.011) | 0.377(0.009) | 0.470(0.013) | 0.039(0.002) | 0.194(0.006) | 0.188(0.006) | 0.313(0.012) |
| TSDiff | 0.309(0.008) | 0.556(0.007) | 0.331(0.006) | 0.414(0.007) | - | - | - | - |
| TimeDiT | 0.284(0.005) | 0.533(0.005) | 0.327(0.005) | 0.423(0.007) | 0.004(0.000) | 0.062(0.001) | 0.051(0.001) | 0.080(0.001) |

### 5.2 Domain Knowledge Integration: Physics-Informed TimeDiT

A key advantage of our approach is the ability to directly incorporate physics knowledge into the pretrained foundation model without additional fine-tuning. This is possible because the diffusion model reconstructs the entire process, allowing for seamless integration of PDE-based constraints during inference. By encoding the known physical laws governing the system into the sampling process, we can guide the model towards more physically consistent and accurate predictions.
In this section, we evaluate how effectively our pre-trained foundation model can integrate physics-informed knowledge into time series forecasting without the need for fine-tuning. We study four 1D partial differential equations (PDEs) from  (Takamoto et al., ): general Navier-Stokes Equations, Kolmogorov Flow (a specific case of Navier-Stokes Equations), Advection Equations, Burgers Equations, Diffusion Soeption and Computational Fluid Dynamics (CFD). These equations are used to generate synthetic data with random initial conditions, and we apply diffusion models to forecast time series based on data from a historical window.

Table 1 presents the results, including both mean error and error bars. The table clearly demonstrates that our proposed model editing solution, which incorporates physics knowledge, significantly outperforms previous sampling strategies such as DDPM (Ho et al., ), DDIM (), and TS Diffusion’s Self-Guidance (Kollovieh et al., ). By leveraging domain-specific physical information, our approach achieves substantial performance improvements over these baselines, highlighting the effectiveness of integrating physics-informed priors into the diffusion model sampling process.
This performance gain underscores the potential of combining pretrained foundation models with domain-specific knowledge. Our method offers a flexible framework for enhancing time series forecasting in scientific and engineering applications where the underlying physical laws are partially known. It demonstrates that by bridging the gap between data-driven approaches and physics-based modeling, we can achieve more accurate and physically consistent predictions without the computational overhead of retraining or fine-tuning the entire model. The ability to easily incorporate physics knowledge into a pretrained foundation model represents a significant advance in the field of scientific machine learning. It opens up new possibilities for rapid adaptation to specific physical systems and phenomena, potentially accelerating research and discovery in various scientific domains.

### 5.3 Forecasting on Full-shot Setting and Zero-shot Setting

In the forecasting task, we conduct two types of experiments. First, we compare our proposed TimeDiT with baselines in a full-shot setting, where models are trained and tested on separate datasets. This approach evaluates their performance on conventional time series forecasting tasks, ensuring that models can effectively learn and generalize from complete data. Second, we assess TimeDiT as a foundation model in a zero-shot setting, comparing it to previous transformer-based time series models. This setting is crucial as it tests the model’s ability to generalize and adapt to entirely new datasets without prior exposure, highlighting its robustness and versatility.
Together, these experiments provide a holistic view of TimeDiT’s capabilities, addressing both specialized performance and broad applicability in time series forecasting.

**Table 2: Forecasting results on CRPSsum with full shot setting.**
|  |  | Solar | Electricity | Traffic | Taxi | Exchange |
| --- | --- | --- | --- | --- | --- | --- |
| Deterministic | DLinear | 0.432(0.002) | 0.033(0.000) | 0.070(0.001) | 0.177(0.000) | 0.011(0.001) |
| PatchTST | 0.457(0.019) | 0.037(0.002) | 0.405(0.001) | 0.190(0.005) | 0.026(0.001) |  |
| Latent ODE | 0.445(0.002) | 0.140(0.017) | 0.095(0.004) | 0.181(0.006) | 0.013(0.001) |  |
| GPT2(6) | 0.467(0.002) | 0.033(0.001) | 0.069(0.001) | 0.187(0.001) | 0.013(0.001) |  |
| Probabilistic | GP-copula | 0.337(0.024) | 0.024(0.002) | 0.078(0.002) | 0.208(0.183) | 0.007(0.000) |
| TransMAF | 0.301(0.014) | 0.021(0.011) | 0.056(0.001) | 0.179(0.002) | 0.005(0.003) |  |
| TimeGrad | 0.287(0.020) | 0.021(0.001) | 0.044(0.006) | 0.114(0.020) | 0.006(0.001) |  |
| CSDI | 0.298(0.004) | 0.017(0.000) | 0.020(0.001) | 0.123(0.003) | 0.007(0.001) |  |
| Diffusion-TS | 0.286(0.003) | 0.019(0.002) | 0.097(0.001) | 0.303(0.004) | 0.009(0.001) |  |
| TimeDiT | 0.278(0.001) | 0.017(0.000) | 0.019(0.000) | 0.123(0.001) | 0.005(0.001) |  |

**Table 3: Forecasting results on CRPSsum. Zero-shot implies that the model has not encountered any samples from the above datasets during training.**
|  |  | Solar | Electricity | Traffic | Taxi | Exchange |
| --- | --- | --- | --- | --- | --- | --- |
| Zero-shot | TEMPO | 0.581(0.002) | 0.081(0.003) | 0.147(0.000) | 0.400(0.001) | 0.030(0.001) |
| Moirai small | 0.884(0.005) | 0.079(0.002) | 0.215(0.000) | 0.463(0.001) | 0.007(0.000) |  |
| Moirai base | 0.948(0.002) | 0.072(0.002) | 0.191(0.001) | 0.428(0.000) | 0.012(0.000) |  |
| Moirai large | 1.042(0.002) | 0.039(0.001) | 0.111(0.000) | 0.597(0.000) | 0.011(0.000) |  |
| LagLLama | 0.690(0.005) | 0.065(0.005) | 0.275(0.001) | 0.620(0.003) | 0.024(0.001) |  |
| TimeDiT | 0.457(0.002) | 0.026(0.001) | 0.185(0.010) | 0.398(0.001) | 0.021(0.002) |  |

Table  2 presents the full-shot forecasting results, comparing TimeDiT with state-of-the-art models in two categories: deterministic forecasting models, which are trained with the Student’s t-distribution head to support probabilistic results, and inherently probabilistic time series forecasting models, including diffusion-based models, such as CSDI and non-diffusion-based, such as GP-copula. Our model achieves the lowest CRPSsum on four datasets and the second-best performance on the Taxi dataset. In the zero-shot setting, TimeDiT is compared with the open-sourced foundation models including TEMPO (Cao et al., ), which is pre-trained with Student’s t-distribution head to support probabilistic results, Moirai (Woo et al., ) and LagLLama (Rasul et al., ) in Table 3. TimeDiT’s ability to outperform other open-source foundation models in most cases is particularly noteworthy, as it suggests that TimeDiT can be effectively applied to a wide range of time series forecasting tasks across different domains with minimal adaptation.

**Table 4: Imputation result on 96-length multivariate time series averaged over the four mask ratios. We calculate MSE and MAE for each dataset. Bold indicates best result, Underline indicates the second best result.**
| Datasets | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Weather | Electricity | 1st Pl |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | Count |
| DLinear | 0.201 | 0.306 | 0.142 | 0.259 | 0.093 | 0.206 | 0.096 | 0.208 | 0.052 | 0.110 | 0.132 | 0.260 | 0 |
| LightTS | 0.284 | 0.373 | 0.119 | 0.250 | 0.104 | 0.218 | 0.046 | 0.151 | 0.055 | 0.117 | 0.131 | 0.262 | 0 |
| ETSformer | 0.202 | 0.329 | 0.367 | 0.436 | 0.120 | 0.253 | 0.208 | 0.327 | 0.076 | 0.171 | 0.214 | 0.339 | 0 |
| FEDformer | 0.117 | 0.246 | 0.163 | 0.279 | 0.062 | 0.177 | 0.101 | 0.215 | 0.099 | 0.203 | 0.130 | 0.259 | 0 |
| Autoformer | 0.103 | 0.214 | 0.055 | 0.156 | 0.051 | 0.150 | 0.029 | 0.105 | 0.031 | 0.057 | 0.101 | 0.225 | 0 |
| PatchTST | 0.115 | 0.224 | 0.065 | 0.163 | 0.047 | 0.140 | 0.029 | 0.102 | 0.034 | 0.055 | 0.072 | 0.183 | 0 |
| TimesNet | 0.078 | 0.187 | 0.049 | 0.146 | 0.027 | 0.107 | 0.022 | 0.088 | 0.030 | 0.054 | 0.092 | 0.210 | 1 |
| GPT2(3) | 0.069 | 0.173 | 0.048 | 0.141 | 0.028 | 0.105 | 0.021 | 0.084 | 0.031 | 0.056 | 0.090 | 0.207 | 1 |
| TimeDiT | 0.042 | 0.135 | 0.042 | 0.139 | 0.023 | 0.098 | 0.024 | 0.083 | 0.031 | 0.036 | 0.069 | 0.174 | 10 |

### 5.4 Imputation Task

We conduct experiments on six benchmark time-series datasets: ETTh1, ETTh2, ETTm1, ETTm2, Electtricity, and Weather. We use random mask ratios $\{12.5\%,25\%,37.5\%,50\%\}$ following previous studies’ settings with sequence length set to 96. We finetune on model checkpoints pretrained on solar, traffic, exchange, taxi, Huawei cloud, air quality, and weather (different from the evaluation weather data). Table 4 shows the imputation result averaged over the four mask ratios. TimeDiT achieves the best performance on most dataset. obtaining 10 first places out of the 12 evaluations while the remaining baselines obtained 2 first place count in total. In particular, TimeDiT achived a 39% reduction in MSE and 22% reduction in MAE compared to the strongest baseline on ETTh1 dataset. For full result on each mask ratio, please refer to section F.

### 5.5 Anomaly Detection Task

We conduct experiments on five real-world datasets from industrial applications: MSL, SMAP, SWaT, SMD, and PSM. The diffusion model, renowned for its proficiency in distribution learning, may inadvertently overfit by reconstructing anomalies alongside normal data points. To counteract this, we opted to bypass pretraining and introduced spectral residue (SR) transformation at the preprocessing stage of TimeDiT . This transformation helps to conceal points most likely to be anomalies and their immediate neighbors. The number of neighbors affected is controlled by the hyperparameter $n_{neighbor}$. The SR method utilizes Fourier Transformation to convert the original time series into a saliency map, thereby amplifying abnormal points, as detailed in (Ren et al., ; Zhao et al., ). For additional information about this transformation, please see section G.
Consistent with prior methodologies, we set the sequence length to be 100 identify anomalies using the 99th percentile of reconstruction errors. During evaluations, we apply standard anomaly adjustments as suggested by (Xu et al., ). As demonstrated in table 11, TimeDiT outperforms baseline models on four of the five datasets. In particular, TimeDiT 23.03 points of improvement in terms of F1 score on the SMAP dataset compared to the strongest baseline.

**Table 5: Anomaly Detection result on 100-length multivariate time series. We calculate F1 score as % for each dataset. ’.’ notation in model name stands for transformer. Bold indicates best result, Underline indicates the second best result.**
| Methods | TimeDiT | GPT2(6) | TimesNet | PatchTS. | ETS. | FED. | LightTS | DLinear | Auto. | Anomaly.* |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MSL | 89.33 | 82.45 | 81.84 | 78.70 | 85.03 | 78.57 | 78.95 | 84.88 | 79.05 | 83.31 |
| SMAP | 95.91 | 72.88 | 69.39 | 68.82 | 69.50 | 70.76 | 69.21 | 69.26 | 71.12 | 71.18 |
| SWaT | 96.46 | 94.23 | 93.02 | 85.72 | 84.91 | 93.19 | 93.33 | 87.52 | 92.74 | 83.10 |
| SMD | 83.28 | 86.89 | 84.61 | 84.62 | 83.13 | 85.08 | 82.53 | 77.10 | 85.11 | 85.49 |
| PSM | 97.57 | 97.13 | 97.34 | 96.08 | 91.76 | 97.23 | 97.15 | 93.55 | 93.29 | 79.40 |
| 1st Pl Count | 4 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |

**Table 6: Synthetic Generation result on 24-length multivariate time series. We calculate discriminative and predictive score according to Yoon et al. () and results are averaged over five runs. Bold indicates the best performance.**
| Metric | Methods | Sine | Stocks | Air Quality | Energy |
| --- | --- | --- | --- | --- | --- |
| Discriminative Score | TimeGAN | 0.1217(0.039) | 0.2038(0.057) | 0.3913(0.039) | 0.4969 (0.000) |
| TimeVAE | 0.0489(0.0562) | 0.1987(0.037) | 0.2869(0.053) | 0.4993(0.001) |  |
| Diffusion-TS | 0.0099(0.003) | 0.1869(0.0159) | 0.1227(0.006) | 0.2301(0.006) |  |
| TimeDiT | 0.0086(0.004) | 0.0087(0.006) | 0.2109(0.003) | 0.0053(0.002) |  |
| Predictive Score | TimeGAN | 0.2797(0.015) | 0.0481(0.002) | 0.035(0.002) | 0.3305(0.003) |
| TimeVAE | 0.2285(0.000) | 0.0485(0.000) | 0.0269(0.001) | 0.2878(0.001) |  |
| Diffusion-TS | 0.2262(0.000) | 0.042(0.000) | 0.022(0.002) | 0.2506(0.000) |  |
| TimeDiT | 0.1915(0.000) | 0.0445(0.000) | 0.0217(0.000) | 0.2489(0.000) |  |

### 5.6 Synthetic Generation Task

Figure: Figure 3: PCA Evaluation of Synthetic Time Series from TimeDiT and other baselines on stock dataset.
Refer to caption: x4.png

We conduct experiments to synthesize multivariate time series and evaluate performance using the discriminative score and predictive score metrics under a "train on synthetic test on real" experimental setup with sequence length set to 24  (Yuan and Qiao, ). We finetune on model checkpoints pretrained on etth1, etth2, ettm1, ettm2, electricity, and weather. Table 6 shows the result on synthetic generation where TimeDiT, in general, consistently generates more realistic synthetic samples compared to baselines, even on high-dimensional energy datasets. This demonstrates TimeDiT’s strength in complex time series synthesis. We visualize synthesis performance using PCA and t-SNE in Appendix E. As shown in Figure 3, TimeDiT’s samples markedly overlap the original data distribution better than other methods. Qualitative and quantitative results confirm TimeDiT’s superior ability to model intricate characteristics for realistic time series synthesis, even on multidimensional, complex datasets.

## 6 Conclusion

In this paper, we introduced TimeDiT, a pioneering approach to creating a versatile and robust foundation model for various time series tasks under practical scenarios. By integrating transformer architecture with diffusion model, TimeDiT effectively captures temporal dependencies and addresses real world challenges unique to time series regarding multi-resolution and missing values as well as incorporating external knowledge. Our innovative masking strategies allow for a consistent training framework adaptable to diverse tasks such as forecasting, imputation, and anomaly detection and synthetic data generation. Extensive experiments demonstrated the strong performance of TimeDiT on both practical scenarios and standard benchmarks. However, we recognize some limitations. We primarily explored common sequence lengths and did not assess TimeDiT’s performance on very long sequences. While we have introduced randomness in prediction length and feature numbers up to a maximum, we aim to develop more scalable solutions for highly variable multivariate time series. Additionally, our understanding of how different types of external information contribute to performance is still developing. For future work, we envision several key directions: enhancing scalability to improve TimeDiT’s ability to handle practical time series with varying multivariate numbers; developing techniques for seamless multi-modal integration, allowing TimeDiT to leverage diverse data sources for improved performance across different tasks; and extending TimeDiT’s capabilities to effectively process and analyze very long time series sequences, addressing a critical need in many real-world applications.

## Appendix A Datasets

- 1.
The ETT datasets Zhou et al. ()(^1^11ETT: [https://github.com/zhouhaoyi/ETDataset](https://github.com/zhouhaoyi/ETDataset)) include electricity load data at various resolutions (ETTh & ETTm) from two different electricity stations.
- 2.
The Weather dataset comprises 21 meteorological indicators collected in Germany over the span of one year.
- 3.
The Electricity dataset provides information on electricity consumption.
- 4.
The SMD dataset Su et al. () includes multivariate time-series data collected from server machines in a data center. It typically contains metrics such as CPU usage, memory usage, and disk activity.
- 5.
The PSM dataset Abdulaal et al. ()is used for predictive maintenance and includes sensor data from industrial machines. It often contains readings such as temperature, pressure, and vibration over time.
- 6.
The MSL dataset Hundman et al. () comes from the Mars Science Laboratory mission, specifically the Curiosity rover. It includes telemetry data from the rover’s sensors and systems.
- 7.
The SWaT dataset Mathur and Tippenhauer () originates from a scaled-down water treatment testbed designed to reflect a real-world water treatment process. It includes sensor and actuator data collected over time.
- 8.
The SMAP dataset Hundman et al. ()comes from NASA’s Soil Moisture Active Passive (SMAP) mission, which measures soil moisture and freeze/thaw state. It includes time-series data from multiple sensors aboard the SMAP satellite.
- 9.
The Sine dataset Yoon et al. () is synthetically generated by sinusoidal waves.
- 10.
The Air Quality dataset (^2^22Air Quality: [https://archive.ics.uci.edu/dataset/360/air+quality](https://archive.ics.uci.edu/dataset/360/air+quality))contains hourly averaged readings from five metal oxide chemical sensors integrated into an Air Quality Chemical Multisensor Device. This device was positioned at road level in a highly polluted area of an Italian city. Data were collected from March 2004 to February 2005, making it the longest freely available record of on-field air quality chemical sensor responses.
- 11.
The Stock dataset (^3^33Stock: [https://finance.yahoo.com/quote/GOOG](https://finance.yahoo.com/quote/GOOG)) contains daily historical Google stocks data from 2004 to 2019.
- 12.
The UCI Appliances Energy prediction dataset (^4^44Energy: [https://archive.ics.uci.edu/ml/datasets](https://archive.ics.uci.edu/ml/datasets))consists of multivariate, continuous-valued measurements including numerous temporal features measured at close intervals.
- 13.
The Cloud dataset: The Huawei cloud datasets contain serverless traces (Joosen et al., ). Following (Rasul et al., ), we selected 8 time series containing metrics based on the minute-frequency occurrences of the top 10 functions over a period of 141 days. The metrics included in these series are: Function delay; Platform delay; CPU usage; Memory usage; CPU limit; Memory limit; Instances; Requests. The functions were chosen based on their median occurrences throughout the dataset.
- 14.
The Weather_2 dataset: The Weather_2 dataset comprises hourly climate time series data collected near Monash University, Clayton, Victoria, Australia, from January 2010 to May 2021. It includes series for temperature, dewpoint temperature, wind speed, mean sea level pressure, relative humidity, surface solar radiation, surface thermal radiation, and total cloud cover (Godahewa et al., ).

**Table 7: Dataset details**
| Dataset | Domain | Length | Dimension | Frequency |
| --- | --- | --- | --- | --- |
| ETTh | Energy | 17420 | 7 | 1 hour |
| ETTm | Energy | 69680 | 7 | 15 min |
| Weather | Nature | 52696 | 21 | 10 min |
| Electricity | Energy | 26304 | 321 | 1 hour |
| Air Quality | Nature | 9357 | 13 | 1 hour |
| Sine | Synthetic | 10000 | 5 | N/A |
| Stock | Finance | 3685 | 6 | 1 day |
| Energy | Energy | 19745 | 28 | 10 min |
| MSL | Space | 132046 | 55 | 1 min |
| PSM | Cloud | 220322 | 25 | 1 min |
| SMAP | Space | 562800 | 25 | 1 min |
| SMD | Cloud | 1416825 | 38 | 1 min |
| SWaT | Energy | 944920 | 51 | 1 second |
| Requests Minute | Cloud | 64800 | 10 | 1 min |
| Function Delay Minute | Cloud | 64800 | 10 | 1 min |
| Platform Delay Minute | Cloud | 64800 | 10 | 1 min |
| Memory Usage Minute | Cloud | 64800 | 10 | 1 min |
| CPU Limit Minute | Cloud | 64800 | 10 | 1 min |
| Memory Limit Minute | Cloud | 64800 | 10 | 1 min |
| Instances Minute | Cloud | 64800 | 10 | 1 min |
| Weather_2 | Climate | 3001 | 695 | 1 day |

## Appendix B Forecasting Experiment Setting

For the forecasting task, we utilized five widely-used open datasets to evaluate probabilistic time series forecasting performance. These datasets were collected in GluonTS  (Alexandrov et al., ) and have been previously employed in  (Tashiro et al., ; Salinas et al., ):

- •
Solar(^5^55Solar: [https://www.nrel.gov/grid/solar-power-data.html](https://www.nrel.gov/grid/solar-power-data.html)): Hourly solar power production records from 137 stations in Alabama State, as used in  (Lai et al., ).
- •
Electricity(^6^66Electricity: [https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014](https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014)): Hourly time series of electricity consumption for 370 customers, as used in  (Asuncion and Newman, ).
- •
Traffic(^7^77Traffic_nips: [https://archive.ics.uci.edu/dataset/204/pems_sf](https://archive.ics.uci.edu/dataset/204/pems_sf)): Hourly occupancy rates of 963 San Francisco freeway car lanes, with values between 0 and 1  (Asuncion and Newman, ).
- •
Taxi(^8^88Taxi: [https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data)): Half-hourly spatio-temporal time series of New York taxi rides taken at 1,214 locations, using data from January 2015 for training and January 2016 for testing, as proposed in  (Tlc, ).
- •
Exchange rate(^9^99Exchange: [https://github.com/laiguokun/multivariate-time-series-data](https://github.com/laiguokun/multivariate-time-series-data)): Daily exchange rates between 8 currencies, namely Australia,
the United Kingdom, Canada, Switzerland, China, Japan, New Zealand, and Singapore, as used in  (Lai et al., ).

Table  8 summarizes the characteristics of each dataset. The task for these datasets is to predict the future $L_{2}$ steps given the observed $L_{1}$ steps. We set $L_{1}$ and $L_{2}$ values based on previous studies  (Tashiro et al., ; Salinas et al., ). For training, we randomly selected $L_{1}+L_{2}$ consecutive time steps as a single time series and designated the last $L_{2}$ steps as forecasting targets. We adhered to the train/test splits used in previous studies and utilized the last five samples of the training data as validation data.

For the full-shot setting, we trained separate models on different datasets. Due to the large number of features in multivariate time series, we adopted subset sampling of features for training. For each input, we split them into subsets based on their order. If the last subset was smaller than the fixed shape, we applied padding to ensure equal input sizes across all subsets. In the multi-resolution setting, we used different resolutions identified by the resolution number, which corresponded to different sampling rates for the exchange features. It is worth to note that the aforementioned strategy was also employed for zero-shot training, as the input feature length varied across datasets.

## Appendix C Training details

The codebase for TimeDiT is modified from [https://github.com/facebookresearch/DiT](https://github.com/facebookresearch/DiT), where they provide different model sizes, including Small (S), Big (B), Large (L), Extra Large (XL) (Peebles and Xie, ).
In our training, we used Adam optimizer with a training rate of 0.0001 without weight decay. Batch size is set to 512. The maximum channel number $K_{max}$ is set to 40. All experiments are run on NVIDIA A100 GPUs. The zero-shot foundation model was trained on the ETT, weather, illness, air quality, and cloud datasets and used for different downstream tasks. We will include more available time series datasets to develop a more robust time series foundation model as the future work.

**Table 8: Evaluate time series dataset for forecasting tasks.**
|  | Features | Time Steps | History Length ($L_{1}$) | Prediction Horizon ($L_{2}$) | Rolling Windows | Frequency | Domain |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Solar | 137 | 7009 | 168 | 24 | 7 | 1 hour | $\mathbb{R}^{+}$ |
| Electricity | 370 | 5833 | 168 | 24 | 7 | 1 hour | $\mathbb{R}^{+}$ |
| Traffic | 963 | 4001 | 168 | 24 | 7 | 1 hour | (0, 1) |
| Taxi | 1214 | 1488 | 48 | 24 | 56 | 30 mins | $\mathbb{N}$ |
| Exchange | 8 | 6071 | 60 | 30 | 5 | 1 day | $\mathbb{R}^{+}$ |

**Table 9: Training Details. Imp stands for Imputation. SG stands for Syntheric Generation. AD stands for Anomaly Detection. FC stands for Forecasting**
| Dataset | Task | Loss | model size | hidden size | attention head | depth |
| --- | --- | --- | --- | --- | --- | --- |
| ETTh | Imp | $L_{simple}$ | S | 384 | 6 | 12 |
| ETTm | Imp | $L_{simple}$ | S | 384 | 6 | 12 |
| Weather | Imp | $L_{simple}$ | S | 384 | 6 | 12 |
| Electricity | Imp | $L_{simple}$ | S | 384 | 6 | 12 |
| Air Quality | Imp | $L$ | S | 384 | 6 | 12 |
| Sine | SG | $L$ | S | 384 | 6 | 12 |
| Stock | SG | $L$ | S | 384 | 6 | 12 |
| Energy | SG | $L$ | S | 384 | 6 | 12 |
| MSL | AD | $L_{simple}$ | S | 384 | 6 | 12 |
| PSM | AD | $L_{simple}$ | S | 384 | 6 | 12 |
| SMAP | AD | $L_{simple}$ | S | 384 | 6 | 12 |
| SMD | AD | $L_{simple}$ | S | 384 | 6 | 12 |
| SWaT | AD | $L_{simple}$ | S | 384 | 6 | 12 |
| Solar | FC | $L_{simple}$ | B | 768 | 12 | 12 |
| Taxi | FC | $L_{simple}$ | B | 768 | 12 | 12 |
| Traffic | FC | $L_{simple}$ | B | 768 | 12 | 12 |
| Exchange | FC | $L_{simple}$ | B | 768 | 12 | 12 |
| Electricity | FC | $L_{simple}$ | B | 768 | 12 | 12 |

## Appendix D Metrics

#### MAE

describes the mean absolute error that measures the absolute difference between ground truth and prediction.

$$ $\text{MAE}=\frac{1}{n}\sum_{i=1}^{n}|y_{i}-\hat{y}_{i}|$ (15) $$

#### MSE

describes the mean squared difference between ground truth and prediction.

$$ $\text{MSE}=\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}$ (16) $$

#### RMSE

is the sqaure root of MSE.

$$ $\text{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}}$ (17) $$

#### Discriminative score

Following TimeGAN, we train a post-hoc time-series classification model (by optimizing a 2-layer LSTM) to distinguish between sequences from the original and generated datasets. First, each original sequence is labeled real, and each generated sequence is labeled not real. Then, an off-the-shelf (RNN) classifier is trained to distinguish between the two classes as a standard supervised task. We then report the classification error on the held-out test set.

#### Predictive Score

Following TimeGAN, we train a post-hoc sequence-prediction model (by optimizing a 2-layer LSTM) to predict next-step temporal vectors over each input sequence. Then, we evaluate the trained model on the original dataset. Performance is measured in terms of the mean absolute error (MAE); for event-based data, the MAE is computed as the absolute value of 1 - estimated probability that the event occured.

#### Computations of CRPS

We explain the definition and calculation of the CRPS metric. The continuous ranked probability score (CRPS) assesses how well an estimated probability distribution $F$ aligns with an observation $x$. It is defined as the integral of the quantile loss $\Lambda_{\alpha}(q,z):=(\alpha-\mathbf{1}_{z<q})(z-q)$ over all quantile levels $\alpha\in[0,1]$:

$$ $\text{CRPS}(F^{-1},x)=\int_{0}^{1}2\Lambda_{\alpha}(F^{-1}(\alpha),x)\,d\alpha$ $$

where $\mathbf{1}$ represents the indicator function. We then calculated quantile losses for quantile levels discretized in 0.05 increments. Thus, we approximated CRPS as follows:

$$ $\text{CRPS}(F^{-1},x)\approx\frac{1}{19}\sum_{i=1}^{19}2\Lambda_{i\cdot 0.05}( F^{-1}(i\cdot 0.05),x).$ $$

Next, we computed the normalized average CRPS for all features and time steps:

$$ $\frac{\sum_{k,l}\text{CRPS}(F^{-1}_{k,l},x_{k,l})}{\sum_{k,l}|x_{k,l}|}$ $$

where $k$ and $l$ denote the features and time steps of the imputation targets, respectively.

CRPSsum measures CRPS for the distribution $F$ of the sum of all $K$ features, calculated by:

$$ $\frac{\sum_{l}\text{CRPS}(F^{-1},\sum_{k}x_{k,l})}{\sum_{k,l}|x_{k,l}|}$ $$

where $\sum_{k}x_{k,l}$ is the total of the forecasting targets for all features at time point $l$.

## Appendix E Synthetic Generation

We use 80% of all data for training and evaluate on the same data. For the air quality dataset, previous methods did not carefully the -200 values as placeholder for missing values. In our experiment, we masked all the -200 values for TimeDiT and baselines that support masks. For baselines that do not support mask, we replace -200 with the mean value. Minmax scaler is used for all models. Figure 4, 3,5,6 shows the PCA plots for all datasets and baselines. The visual comparison also validates the superiority of TimeDiT.

Figure: Figure 4: PCA plot for sine dataset.
Refer to caption: x5.png

Figure: Figure 5: PCA plot for air quality dataset.
Refer to caption: x6.png

Figure: Figure 6: PCA plot for energy dataset.
Refer to caption: x7.png

## Appendix F Imputation

**Table 10: Full imputation result.**
| Methods | TimeDiT | GPT2(3) | TimesNet | PatchTST | ETSformer | LightTS | DLinear | FEDformer | Stationary | Autoformer | Informer | Reformer |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Mask | Ratio | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
| $ETTh1$ | 12.5% | 0.025 | 0.101 | 0.043 | 0.140 | 0.057 | 0.159 | 0.093 | 0.201 | 0.126 | 0.263 | 0.240 | 0.345 | 0.151 | 0.267 | 0.070 | 0.190 | 0.060 | 0.165 | 0.074 | 0.182 | 0.114 | 0.234 | 0.074 | 0.194 |
| 25% | 0.034 | 0.122 | 0.054 | 0.156 | 0.069 | 0.178 | 0.107 | 0.217 | 0.169 | 0.304 | 0.265 | 0.364 | 0.180 | 0.292 | 0.106 | 0.236 | 0.080 | 0.189 | 0.090 | 0.203 | 0.140 | 0.262 | 0.102 | 0.227 |  |
| 37.5% | 0.047 | 0.143 | 0.072 | 0.180 | 0.084 | 0.196 | 0.120 | 0.230 | 0.220 | 0.347 | 0.296 | 0.382 | 0.215 | 0.318 | 0.124 | 0.258 | 0.102 | 0.212 | 0.109 | 0.222 | 0.174 | 0.293 | 0.135 | 0.261 |  |
| 50% | 0.063 | 0.166 | 0.107 | 0.216 | 0.102 | 0.215 | 0.141 | 0.248 | 0.293 | 0.402 | 0.334 | 0.404 | 0.257 | 0.347 | 0.165 | 0.299 | 0.133 | 0.240 | 0.137 | 0.248 | 0.215 | 0.325 | 0.179 | 0.298 |  |
| Avg | 0.042 | 0.135 | 0.069 | 0.173 | 0.078 | 0.187 | 0.115 | 0.224 | 0.202 | 0.329 | 0.284 | 0.373 | 0.201 | 0.306 | 0.117 | 0.246 | 0.094 | 0.201 | 0.103 | 0.214 | 0.161 | 0.279 | 0.122 | 0.245 |  |
| $ETTh2$ | 12.5% | 0.025 | 0.107 | 0.039 | 0.125 | 0.040 | 0.130 | 0.057 | 0.152 | 0.187 | 0.319 | 0.101 | 0.231 | 0.100 | 0.216 | 0.095 | 0.212 | 0.042 | 0.133 | 0.044 | 0.138 | 0.305 | 0.431 | 0.163 | 0.289 |
| 25% | 0.037 | 0.129 | 0.044 | 0.135 | 0.046 | 0.141 | 0.061 | 0.158 | 0.279 | 0.390 | 0.115 | 0.246 | 0.127 | 0.247 | 0.137 | 0.258 | 0.049 | 0.147 | 0.050 | 0.149 | 0.322 | 0.444 | 0.206 | 0.331 |  |
| 37.5% | 0.046 | 0.149 | 0.051 | 0.147 | 0.052 | 0.151 | 0.067 | 0.166 | 0.400 | 0.465 | 0.126 | 0.257 | 0.158 | 0.276 | 0.187 | 0.304 | 0.056 | 0.158 | 0.060 | 0.163 | 0.353 | 0.462 | 0.252 | 0.370 |  |
| 50% | 0.062 | 0.173 | 0.059 | 0.158 | 0.060 | 0.162 | 0.073 | 0.174 | 0.602 | 0.572 | 0.136 | 0.268 | 0.183 | 0.299 | 0.232 | 0.341 | 0.065 | 0.170 | 0.068 | 0.173 | 0.369 | 0.472 | 0.316 | 0.419 |  |
| Avg | 0.042 | 0.139 | 0.048 | 0.141 | 0.049 | 0.146 | 0.065 | 0.163 | 0.367 | 0.436 | 0.119 | 0.250 | 0.142 | 0.259 | 0.163 | 0.279 | 0.053 | 0.152 | 0.055 | 0.156 | 0.337 | 0.452 | 0.234 | 0.352 |  |
| $ETTm1$ | 12.5% | 0.016 | 0.083 | 0.017 | 0.085 | 0.023 | 0.101 | 0.041 | 0.130 | 0.096 | 0.229 | 0.093 | 0.206 | 0.080 | 0.193 | 0.052 | 0.166 | 0.032 | 0.119 | 0.046 | 0.144 | 0.063 | 0.180 | 0.042 | 0.146 |
| 25% | 0.019 | 0.091 | 0.022 | 0.096 | 0.023 | 0.101 | 0.044 | 0.135 | 0.096 | 0.229 | 0.093 | 0.206 | 0.080 | 0.193 | 0.052 | 0.166 | 0.032 | 0.119 | 0.046 | 0.144 | 0.063 | 0.180 | 0.042 | 0.146 |  |
| 37.5% | 0.025 | 0.102 | 0.029 | 0.111 | 0.029 | 0.111 | 0.049 | 0.143 | 0.133 | 0.271 | 0.113 | 0.231 | 0.103 | 0.219 | 0.069 | 0.191 | 0.039 | 0.131 | 0.057 | 0.161 | 0.079 | 0.200 | 0.063 | 0.182 |  |
| 50% | 0.032 | 0.115 | 0.040 | 0.128 | 0.036 | 0.124 | 0.055 | 0.151 | 0.186 | 0.323 | 0.134 | 0.255 | 0.132 | 0.248 | 0.089 | 0.218 | 0.047 | 0.145 | 0.067 | 0.174 | 0.093 | 0.218 | 0.082 | 0.208 |  |
| Avg | 0.023 | 0.098 | 0.028 | 0.105 | 0.027 | 0.107 | 0.047 | 0.140 | 0.120 | 0.253 | 0.104 | 0.218 | 0.093 | 0.206 | 0.062 | 0.177 | 0.036 | 0.126 | 0.051 | 0.150 | 0.071 | 0.188 | 0.055 | 0.166 |  |
| $ETTm2$ | 12.5% | 0.016 | 0.065 | 0.017 | 0.076 | 0.018 | 0.080 | 0.026 | 0.094 | 0.108 | 0.239 | 0.034 | 0.127 | 0.062 | 0.166 | 0.056 | 0.159 | 0.021 | 0.088 | 0.023 | 0.092 | 0.133 | 0.270 | 0.108 | 0.228 |
| 25% | 0.022 | 0.078 | 0.020 | 0.080 | 0.020 | 0.085 | 0.028 | 0.099 | 0.164 | 0.294 | 0.042 | 0.143 | 0.085 | 0.196 | 0.080 | 0.195 | 0.024 | 0.096 | 0.026 | 0.101 | 0.135 | 0.272 | 0.136 | 0.262 |  |
| 37.5% | 0.027 | 0.089 | 0.022 | 0.087 | 0.023 | 0.091 | 0.030 | 0.104 | 0.237 | 0.356 | 0.051 | 0.159 | 0.106 | 0.222 | 0.110 | 0.231 | 0.027 | 0.103 | 0.030 | 0.108 | 0.155 | 0.293 | 0.175 | 0.300 |  |
| 50% | 0.031 | 0.099 | 0.025 | 0.095 | 0.026 | 0.098 | 0.034 | 0.110 | 0.323 | 0.421 | 0.059 | 0.174 | 0.131 | 0.247 | 0.156 | 0.276 | 0.030 | 0.108 | 0.035 | 0.119 | 0.200 | 0.333 | 0.211 | 0.329 |  |
| Avg | 0.024 | 0.083 | 0.021 | 0.084 | 0.022 | 0.088 | 0.029 | 0.102 | 0.208 | 0.327 | 0.046 | 0.151 | 0.096 | 0.208 | 0.101 | 0.215 | 0.026 | 0.099 | 0.029 | 0.105 | 0.156 | 0.292 | 0.157 | 0.280 |  |
| $ECL$ | 12.5% | 0.051 | 0.148 | 0.080 | 0.194 | 0.085 | 0.202 | 0.055 | 0.160 | 0.196 | 0.321 | 0.102 | 0.229 | 0.092 | 0.214 | 0.107 | 0.237 | 0.093 | 0.210 | 0.089 | 0.210 | 0.218 | 0.326 | 0.190 | 0.308 |
| 25% | 0.061 | 0.163 | 0.087 | 0.203 | 0.089 | 0.206 | 0.065 | 0.175 | 0.207 | 0.332 | 0.121 | 0.252 | 0.118 | 0.247 | 0.120 | 0.251 | 0.097 | 0.214 | 0.096 | 0.220 | 0.219 | 0.326 | 0.197 | 0.312 |  |
| 37.5% | 0.074 | 0.181 | 0.094 | 0.211 | 0.094 | 0.213 | 0.076 | 0.189 | 0.219 | 0.344 | 0.141 | 0.273 | 0.144 | 0.276 | 0.136 | 0.266 | 0.102 | 0.220 | 0.104 | 0.229 | 0.222 | 0.328 | 0.203 | 0.315 |  |
| 50% | 0.090 | 0.202 | 0.101 | 0.220 | 0.100 | 0.221 | 0.091 | 0.208 | 0.235 | 0.357 | 0.160 | 0.293 | 0.175 | 0.305 | 0.158 | 0.284 | 0.108 | 0.228 | 0.113 | 0.239 | 0.228 | 0.331 | 0.210 | 0.319 |  |
| Avg | 0.069 | 0.174 | 0.090 | 0.207 | 0.092 | 0.210 | 0.072 | 0.183 | 0.214 | 0.339 | 0.131 | 0.262 | 0.132 | 0.260 | 0.130 | 0.259 | 0.100 | 0.218 | 0.101 | 0.225 | 0.222 | 0.328 | 0.200 | 0.313 |  |
| $Weather$ | 12.5% | 0.029 | 0.033 | 0.026 | 0.049 | 0.025 | 0.045 | 0.029 | 0.049 | 0.057 | 0.141 | 0.047 | 0.101 | 0.039 | 0.084 | 0.041 | 0.107 | 0.027 | 0.051 | 0.026 | 0.047 | 0.037 | 0.093 | 0.031 | 0.076 |
| 25% | 0.031 | 0.033 | 0.028 | 0.052 | 0.029 | 0.052 | 0.031 | 0.053 | 0.065 | 0.155 | 0.052 | 0.111 | 0.048 | 0.103 | 0.064 | 0.163 | 0.029 | 0.056 | 0.030 | 0.054 | 0.042 | 0.100 | 0.035 | 0.082 |  |
| 37.5% | 0.034 | 0.037 | 0.033 | 0.060 | 0.031 | 0.057 | 0.035 | 0.058 | 0.081 | 0.180 | 0.058 | 0.121 | 0.057 | 0.117 | 0.107 | 0.229 | 0.033 | 0.062 | 0.032 | 0.060 | 0.049 | 0.111 | 0.040 | 0.091 |  |
| 50% | 0.031 | 0.041 | 0.037 | 0.065 | 0.034 | 0.062 | 0.038 | 0.063 | 0.102 | 0.207 | 0.065 | 0.133 | 0.066 | 0.134 | 0.183 | 0.312 | 0.0377 | 0.068 | 0.037 | 0.067 | 0.053 | 0.114 | 0.046 | 0.099 |  |
| Avg | 0.031 | 0.036 | 0.031 | 0.056 | 0.030 | 0.054 | 0.060 | 0.144 | 0.076 | 0.171 | 0.055 | 0.117 | 0.052 | 0.110 | 0.099 | 0.203 | 0.032 | 0.059 | 0.031 | 0.057 | 0.045 | 0.104 | 0.038 | 0.087 |  |

## Appendix G Anomaly Detection

**Table 11: Anomaly Detection result on 100-length multivariate time series. We calculate Precision, Recall, and F1 score as % for each dataset. ’.’ notation in model name stands for transformer. Bold indicates best result, Underline indicates the second best result.**
| Methods | MSL | SMAP | SWaT | SMD | PSM | 1st Pl |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Metrics | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1 | Count |
| TimeDiT | 91.54 | 87.23 | 89.33 | 93.35 | 98.61 | 95.91 | 93.64 | 99.46 | 96.46 | 78.83 | 88.26 | 83.28 | 97.36 | 97.79 | 97.57 | 11 |
| GPT(6) | 82.00 | 82.91 | 82.45 | 90.60 | 60.95 | 72.88 | 92.20 | 96.34 | 94.23 | 88.89 | 84.98 | 86.89 | 98.62 | 95.68 | 97.13 | 1 |
| TimesNet | 89.54 | 75.36 | 81.84 | 90.14 | 56.40 | 69.39 | 90.75 | 95.40 | 93.02 | 87.91 | 81.54 | 84.61 | 98.51 | 96.20 | 97.34 | 0 |
| PatchTST | 88.34 | 70.96 | 78.70 | 90.64 | 55.46 | 68.82 | 91.10 | 80.94 | 85.72 | 87.26 | 82.14 | 84.62 | 98.84 | 93.47 | 96.08 | 0 |
| ETSformer | 85.13 | 84.93 | 85.03 | 92.25 | 55.75 | 69.50 | 90.02 | 80.36 | 84.91 | 87.44 | 79.23 | 83.13 | 99.31 | 85.28 | 91.76 | 1 |
| FEDformer | 77.14 | 80.07 | 78.57 | 90.47 | 58.10 | 70.76 | 90.17 | 96.42 | 93.19 | 87.95 | 82.39 | 85.08 | 97.31 | 97.16 | 97.23 | 0 |
| LightTS | 82.40 | 75.78 | 78.95 | 92.58 | 55.27 | 69.21 | 91.98 | 94.72 | 93.33 | 87.10 | 78.42 | 82.53 | 98.37 | 95.97 | 97.15 | 0 |
| DLinear | 84.34 | 85.42 | 84.88 | 92.32 | 55.41 | 69.26 | 80.91 | 95.30 | 87.52 | 83.62 | 71.52 | 77.10 | 98.28 | 89.26 | 93.55 | 0 |
| Autoformer | 77.27 | 80.92 | 79.05 | 90.40 | 58.62 | 71.12 | 89.85 | 95.81 | 92.74 | 88.06 | 82.35 | 85.11 | 99.08 | 88.15 | 93.29 | 0 |
| AnoTransformer | 79.61 | 87.37 | 83.31 | 91.85 | 58.11 | 71.18 | 72.51 | 97.32 | 83.10 | 88.91 | 82.23 | 85.49 | 68.35 | 94.72 | 79.40 | 2 |

### G.1 SR processing for Anomaly Detection

#### Spectral Residue

The SR Transformation involves the following equations. Table 11 shows the full anomaly detection results.

$$ $A(f)=\text{Amplitude}(F(x))$ (18) $$

$$ $P(f)=\text{Phase}(F(x))$ (19) $$

$$ $L(f)=\log(A(f))$ (20) $$

$$ $AL(f)=h_{q}(f)\cdot L(f)$ (21) $$

$$ $R(f)=L(f)-AL(f)$ (22) $$

$$ $S(x)=F^{-1}(\exp(R(f)+iP(f)))$ (23) $$

## Appendix H Physics Equations

The Burgers Equation is:

$$ $\frac{\partial u}{\partial t}+u\frac{\partial u}{\partial x}-v\frac{\partial^{ 2}u}{\partial x^{2}}=0$ (24) $$

where $v$ is the diffusion term. We set the $v$ (diffusion term) as 0.1 and randomly sample a combination of sine waves as initial status

The Advection Equation is:

$$ $\frac{\partial u}{\partial t}+c\frac{\partial u}{\partial x}=0$ (25) $$

where $c$ is the advection speed. We set the $c$ as 1.0 and randomly placed Gaussian peaks as initial status

The diffusion-reaction Equation is:

$$ $\frac{\partial u}{\partial t}-D\frac{\partial^{2}u}{\partial x^{2}}-R(u)=0$ (26) $$

where $D$ is the diffusion coefficient and $R(u)$ is the reaction term. Here, we apply a linear reaction term $R(u)=-k\cdot u$, where $k$ is the reaction speed. We set the $D$ as 1.0, $k$ as 0.1, and a Gaussian distribution with random parameters as initial status.

The Kolmogrov Flow is a specific case of NS equation. More specifically, it is described by:

$$ $\mathbf{u}(x,y,z,t)=\left(-\frac{\partial\psi}{\partial y},\frac{\partial\psi} {\partial x},0\right)$ (27) $$

where the $psi$ is the flow function. It is usually set as:

$$ $\psi(x,y,z,t)=A\sin(kx)\cos(zy+\omega t)$ (28) $$

where $A,k,w$ are hyperparameters.

## Appendix I Proof of the Theorem

###### Proof.

Let us consider the objective function:

$$ $\displaystyle O(q(y|x))$ $\displaystyle=\mathbb{E}_{y\sim q(y|x)}K(y)-\alpha D_{KL}(q(y|x)||p(y|x))$ (29) $\displaystyle=\mathbb{E}_{y\sim q(y|x)}K(y)-\alpha\int_{y}q(y|x)\log(\frac{q(y |x)}{p(y|x)})dy$ $\displaystyle=\int_{y}q(y|x)[K(y)+\alpha\log p(y|x)-\alpha\log q(y|x)]dy$ $$

We try to find the optimal $q(y|x)$ through Lagrange multipliers. The constraint of the above objective function is that $q(y|x)$ is a valid $\int_{y}q(y|x)dy=1$. Thus, the Lagrangian is:

$$ $\displaystyle L(q(y|x),\lambda)$ $\displaystyle=\int_{y}q(y|x)[K(y)+\alpha\log p(y|x)-\alpha\log q(y|x)]dy- \lambda(\int_{y}q(y|x)dy-1)$ (30) $\displaystyle=\int_{y}q(y|x)[K(y)+\alpha\log p(y|x)-\alpha\log q(y|x)-\lambda q (y|x)]dy+\lambda$ $$

We define $f(q(y|x),y,\lambda)=q(y|x)[K(y)+\alpha\log p(y|x)-\alpha\log q(y|x)-\lambda]+
\lambda h(y)]$, where $h(y)$ can be the density function of any fixed distribution defined on the support set of $y$. Therefore, $L(q(y|x),\lambda)=\int_{y}f(q(y|x),y,\lambda)dy$.
According to Euler-Lagrange equation, when the above Lagrangian achieve extreme point, we have:

$$ $\displaystyle\frac{\partial f}{\partial q}=K(y)+\alpha\log p(y|x)-\alpha\log q (y|x)-\lambda-\alpha=0$ (31) $$

Thus, we have:

$$ $\displaystyle\alpha\log q(y|x)$ $\displaystyle=K(y)+\alpha\log p(y|x)-\log q(y|x)-\lambda-\alpha$ (32) $\displaystyle q(y|x)$ $\displaystyle=\exp(\frac{1}{\alpha}K(y)+\log p(y|x)-\frac{\lambda}{\alpha}-1)$ $\displaystyle=\frac{1}{\exp(\frac{\lambda}{\alpha}+1)}\exp(\frac{1}{\alpha}K(y )+\log p(y|x))$ $$

Meanwhile, since $\int_{y}q(y|x)dy=1$, we have:

$$ $\displaystyle\int_{y}\exp(\frac{1}{\alpha}K(y)+\log p(y|x)-\frac{\lambda}{ \alpha}-1)dy$ $\displaystyle=1$ (33) $\displaystyle\frac{1}{\exp(\frac{\lambda}{\alpha}+1)}\int_{y}\exp(\frac{1}{ \alpha}K(y)+\log p(y|x))dy$ $\displaystyle=1$ $$

Thus, we have $\exp(\frac{\lambda}{\alpha}+1)=\int_{y}\exp(\frac{1}{\alpha}K(y)+\log p(y|x))
dy=Z$, leading to:

$$ $q(y|x)=\frac{1}{Z}\exp(K(y)+\alpha\log p(y|x)),Z=\int\exp(K(y)+\alpha\log p(y| x))dy$ (34) $$

∎

###### Proof.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. The scales of the model and training data are crucial to the performance and generalizability of foundation model. Can the author provide a summary of the datasets used in pre-training and fine-tuning and the scale of models in the number of parameters? A good reference could be Table 2 in this survey work[1] of foundation models for time-series. This detail would be critical to assess the contribution of the work.

2. Incorporating external physics knowledge would require having knowledge of the partial differential equation (PDE) that characterizes the underlying dynamics. The problem setting and practical values of the approach itself require further justification. Since we already have access to the ground truth dynamics, can we directly generate training samples by solving the PDE numerically and training a separate model? If it is feasible, how does it compare to the Physics-Informed TimeDiT in the work?

References

[1] Ye, Jiexia, et al. “A Survey of Time Series Foundation Models: Generalizing Time Series Representation with Large Language Mode.” arXiv preprint arXiv:2405.02358 (2024).
soundness: 3
strengths: The work exhibits the following notable strengths:
1. The work effectively demonstrated the value of integrating masked reconstruction with diffusion models in the domain of time-series data.
2. Extensive experiments are conducted to illustrate the potential of this approach as a foundational model for the proposed methodology.
3. The incorporation of PDE-based external physics knowledge into diffusion models for time-series is a novel and intriguing concept.
summary: The proposed work presents a method for training a diffusion model for time-series based on masked reconstruction. This model models the distribution of masked time series values conditioned on the observed values. Various masking strategies with different inductive biases are designed for pre-training and task-specific fine-tuning. Additionally, external physics knowledge on time-series dynamics in the form of partial differential equations (PDEs) can be incorporated into samples generated by the model without re-training the model through changes to the sampling dynamics. The proposed model demonstrates impressive performance across various time-series tasks, including imputation, forecasting, and anomaly detection. Furthermore, it exhibits generalizability across datasets in zero-shot forecasting. The experimental results suggest the potential of a foundation model for time-series data.
weaknesses: The work presents several notable weaknesses:

1. The lack of detailed information regarding the datasets utilized for pre-training and fine-tuning the model poses a significant concern. If each task necessitates a specific subset of datasets for pre-training, the work risks over-claiming its role as a foundational model for time-series.

2. The primary methodology contribution of the work, which combines masked reconstruction with diffusion models, lacks novelty as similar ideas have been explored in the image domain ([1, 2]).

3. Sequence-level classification is a crucial task in time-series modeling ([3, 4]). Consequently, a foundational model for time-series should demonstrate its capability in handling this task. Notably, this aspect is absent from the experimental results presented in the work.

References

[1] Wei, Chen, et al. “Diffusion models as masked autoencoders.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Gao, Shanghua, et al. “Masked diffusion transformer is a strong image synthesizer.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Kidger, Patrick, et al. “Neural controlled differential equations for irregular time series.” Advances in Neural Information Processing Systems 33 (2020): 6696-6707.

[4] Morrill, James, et al. “Neural rough differential equations for long time series.” International Conference on Machine Learning. PMLR, 2021.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * How sensitive is model performance to the choice of mask ratios during pretraining?
* Could you please provide further information about how TimeDiT processes inputs with different channel dimensions?
* With physics-informed sampling, could you comment on how one manages the tension between the demands introduced by physical constraints on the one hand and learned distributions on the other?
* Concerning the adaptive layer normalization AdaLN, how does it stack up compared to other conditioning methods one may have tried?
* Which was the rational for choosing those very specific transformer architecture details, e.g., number of heads and layer depth?
* For the forecasting experiments that involve missing values, how were the missing patterns generated? Are results robust against different missing data mechanisms, MCAR, MAR, MNAR?
* In the anomaly detection task, how sensitive are the results against the choice of 99th percentile threshold shown, did you experiment with others?
* How do you control that the generated samples in synthetic generation experiments are temporally coherent?
* Which part of the architecture is most responsible for the performance improvement over baselines?
* When TimeDiT is not outperforming baselines, what general features does such a setting have?
* Please provide more intuition about Theorem 4.1 and its practical implication.
* How does the choice of the diffusion schedule influence the incorporation of physical constraints?
* Which methods were explored for stabilizing the training of this combined diffusion model and transformer?
* How was the trade-off done between sample quality and the number of sampling steps?
soundness: 3
strengths: * Techniques to integrate diffusion models and transformers for time series
* Extensive experimental validation on a variety of tasks and datasets
* Usability in real-world along with the ability to address the actual challenges
* Theoretical properness with physics-informed sampling
summary: This paper introduces TimeDiT, a general-purpose foundation model for time series analysis, which leverages transformer architecture with the diffusion model.

* A unified masking strategy that allows the handling of multiple tasks, such as forecasting, imputation, and anomaly detection.
* Direct handling of missing values and multi-resolution data using adaptive mechanisms in input processing.
* Integration of physical knowledge into the sampling via an energy-based prior
* It provides state-of-the-art performance on various benchmarks and practical scenarios.
weaknesses: * Limited discussion about the computational costs of the models, and efficiency in training.
* More could be done by way of analysis on model scalability for very long sequences.
* Limitations Relatively short discussion of
* Providing deeper analysis about failure cases would be good.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: See Weaknesses
soundness: 3
strengths: - Overall, the work has strong experimental results.
- This work introduces new method to perform cross-dataset training on multiple tasks.
summary: In this work, the authors introduce Time Diffusion Transformer (TimeDiT), a foundational model for time-series data that leverages a diffusion transformer architecture. The authors also propose a novel approach for integrating physical knowledge into foundational time-series models, enhancing their applicability to real-world scenarios. Extensive experiments are conducted across multiple tasks—including forecasting, imputation, and anomaly detection—demonstrating the model's potential effectiveness across a variety of applications
weaknesses: The following aspects of the paper require clarification and improvement to enhance its reproducibility, understanding, and relevance.

### Methodological Clarity and Missing Technical Details

Several essential technical details are missing in the Method section, making it challenging to fully understand the proposed approach:

- Token Definition: The authors should specify what is considered a "token" in the model, as this definition is critical for understanding the data representation.
- Embedding Architecture: The architecture used to encode each time point $x_i$ (referred to as the "embedder") is unclear. For example, is it a simple linear projection, a convolutional layer, or another design? The appendix also lacks implementation details about this embedder. Given that the paper proposes a new architecture, the engineering choices for this component should be discussed.
- Noise Embedding Justification: The rationale behind embedding the noised sample instead of the original observation is not clearly explained. It would be valuable to discuss the intuition behind embedding noise, as opposed to embedding the original observation and applying the diffusion process in the embedded space. This additional context could help readers understand the reasoning and potential advantages of this approach.

### Irregular Time-Series Data Claims and Evaluation
- In the introduction, the authors claim that their method can handle irregular time-series data, supported by real-world examples. However, the experimental section only evaluates the model on an imputation task, which does not fully align with the challenges posed by irregular time-series data. In irregular time-series modeling, the input often contains missing data by default, and the model learns from partially observed inputs, as demonstrated in related works like [1] and [2].

- To substantiate the claim that TimeDiT can handle irregular time-series, it would be beneficial to include experiments that specifically address irregular data scenarios, such as handling incomplete observations in generation tasks.

### Comparison with Established Benchmarks
The paper does not include a comparison with well-established benchmarks for time-series forecasting, such as the one proposed in [3]. Including such a comparison would allow readers to better understand how the proposed method performs relative to existing approaches in the field. This is particularly important given the paper’s goal to demonstrate generalizability and robustness across various time-series tasks.

### Adaptability to Different Channel Numbers and Input Resolutions
The adaptability of the proposed model to inputs of different resolutions and channel numbers is not clearly addressed.

Channel Adaptation: Does the model accommodate varying channel counts via padding or another method? The paper suggests a predefined maximum resolution for signal input, but it is unclear how the model would handle cases where the input resolution might exceeds this maximum, such as in a zero-shot setting.
It would be helpful if the authors could elaborate on the selection criteria for the maximum resolution. How was this limit determined?



*[1] "GT-GAN: General Purpose Time Series Synthesis with Generative Adversarial Networks", Jinsung Jeon et al.*

*[2] "Generative Modeling of Regular and Irregular Time Series Data via Koopman VAEs", Ilan Naiman et al.*

*[3] "One Fits All:Power General Time Series Analysis by Pretrained LM", Tian Zhou et al.*

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: **Question 1:** 

We argue that TimeDiT simply couples multiple tasks within a single framework via a masking paradigm, and does not substantially address differences in the level of inductive bias across tasks (see weaknesses for details). We consider this paper as incremental work rather than an innovative contribution. 

Indeed, if we splice a trainable cls-token after the sequence as an identifier for the classification task, can we claim to have proposed a base model that takes into account both classification and prediction tasks?

**Question 2:** 

Consider that the random masking strategy in the pre-training phase destroys the coherence of the time series data, which leads to the loss of information such as trend-season, multi-resolution, and multi-periods. Recent research (SimMTM[8]) has shown that pre-training a foundation model with sufficient cross-domain generalization based on the masked self-supervision strategy often requires a large amount of unlabeled data and a slow learning process. 

Furthermore, fine-scale and coarse-scale supervision usually contain low-level and high-level information from the observation, respectively, and TimeDiT only applies to single-scale supervision for reconstruction, widely ignoring multiscale supervision information from the inputs, how can TimeDiT overcome these challenges?

**Question 3:** 

Although the authors claim that TimeDiT "enables direct processing of multivariate inputs", the methodology lacks a design that establishes cross-channel connectivity.   Specifically, instead of designing novel structures to replace the Channel Independent and Patch Embedding designs, TimeDiT still adopts the primitive ‘each moment as a token’ strategy, yet PatchTST and a series of models (OneFitsAll, TimeLLM, Timer, Moment) based on the ‘Patch Embedding’ design have achieved significant success.  

According to existing research, a single Time point has limited temporal information, so this naive "Time as Token" strategy has been replaced by "Patch as Token" strategy.  The authors are expected to design more ablation experiments to verify the superiority of the proposed modelling approach over the Channel Independent and Patch Embedding strategies.

**Question 4:** 

See Weakness for our concerns about the experimental results, and we believe that the introduction of competitive and up-to-date baselines is considered necessary.

**Question 5:** 

The results of Baseline presented in Table 6 are inconsistent with the results presented in the original paper, e.g., the Discriminative Score of the Diffusion-TS model under the Stocks dataset in Table 6 is 0.1869, whereas it is reported as 0.067 in the original paper. In fact, **the results of all the Baselines in Table 6 have a large discrepancy**. The authors must give further experimental details to explain the potential reasons for these deviations.

**Question 6:** 

In Section 5.7, the authors introduced frequency and category information to align textual and temporal data, which significantly improves the zero-sample capability of TimeDiT. However multimodal alignment is a complex and complete endeavor, has TimeDiT designed a unique structure to accommodate the introduction of textual information? Does it utilize pre-trained LLMs to help align text and temporal modalities? The fact that aligning textual and temporal information to guide the noise reduction process of the diffusion model does not yet exist, we think this is an interesting contribution, however unfortunately **the description of Multi-model TimeDiT is missing** in both the main text and the appendices, and we look forward to the authors discussing the implementation details of this part further.

**Question 7:** 

The authors claim to have proposed a generalized time series model based on a self-supervised paradigm, however, the main paper and appendix sections **lack a detailed description of a "unified pre-trained temporal dataset"**. We hope that the authors can give some guidance, such as how to combine a large number of small datasets (19 datasets are included in the Appendix section) into a general dataset, and in particular how the authors can deal with the fact that cross-domain datasets tend to have different channel counts, sampling rates, and data sizes. 

The established temporal basis models generally agree that establishing a unified dataset has a large impact on model performance, and it is expected that the authors will discuss more details of the experimental design.

**Question 8:** 

An inescapable and confusing problem is the seeming **lack of ablation experiments** in the main paper and appendix sections, which makes it difficult to judge the effectiveness of the individual components in TimeDiT. In fact, the authors claim to have introduced a number of unique designs in TimeDiT, including 1) Transformer-based Condition Injection; 2) Time Series Mask Unit; and 3) Physics-Informed. Adequate ablation experiments are considered important in order to facilitate the reader to further understand the impact of each component on the overall performance.

**Reference:** 

8) Dong, Jiaxiang et al. “SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modeling.” NIPS 2023.
soundness: 1
strengths: This paper discusses the potential challenges that may be faced in establishing a generic diffusion-based time series model, such as variable channel sizes across domains, missing values, and multi-resolution characteristics prevalent in the real world. Furthermore, the ideal foundation model is expected to be simultaneously compatible with a wide range of downstream tasks in the time series domain, such as forecasting, imputation, anomaly detection and classification. Due to their unique ability to capture uncertainty and randomness from time series domain, diffusion models are seen as promising theories for establishing foundation models. In addition, exploiting multimodal textual information in order to improve the performance of foundation models in zero-shot forecasting is also seen as promising work.
summary: This paper proposes TimeDiT, a generalized model for time series based on the diffusion, where masking units are designed for task-independent pre-training and task-specific sampling, and allow for the flexible integration of external knowledge during the sampling process in order to create models that can seamlessly deal with the diversity and complexity of time series data with varying lengths of history and characteristics. In addition, TimeDiT incorporates physical knowledge to further improve model performance and interpretability, and utilizes the Transformer's inductive bias to capture time-dependent and diffusive processes to generate high-quality candidate samples.
weaknesses: **Weakness 1:** 

According to the authors, TimeDiT "can unify diverse time series tasks within a single generative framework". Since imputation and anomaly detection tasks are implemented utilizing the same mask-reconstruction strategy, TimeDiT is essentially self-supervised pre-training first, and then sampling on specific tasks without fine-tuning.

However, there are differences between masking and forecasting: 

* the masking task instructs the model to capture contextual information from multiple randomly masked and discontinuous subsequences for reconstructing the masked portion. 

* the forecasting task requires the model to capture global dependencies and local fluctuation patterns, such as seasonal-trend, multi-scale, and multi-period information, from the observed sequences, which are often corrupted by random masking operations.

* the reconstruction paradigm allows the use of local supra- and infra- information to generate the masked portion, whereas the prediction paradigm allows the use of observed sequences only to predict future sequences.

These differences make it difficult for the temporal representations learned in the masking task to be used directly to model observed sequences.

**Weakness 2:** 

TimeDiT's Transformer backbone still adopts the traditional Encoder-only architecture consisting of Input Embedding Layer, Multi-layer Encoder, Output Projection Layer, which is in line with most of the existing articles (PatchTST, iTransformer, UniTime, Timer, Moment, Moirai) and thus lacks innovation. Moreover, this architecture is unable to cope with the inherent challenges of temporal data, such as multi-scale and multi-period temporal characterization.

**Weakness 3:** 

In the denoising process, the conditional information captured from the observation sequences is directly spliced into the input sequences, however, recent studies (ControlNet[1]) have shown that such a simple scheme cannot guarantee a high multivariate temporal correlation between past observations and future predictions. 

Indeed, many diffusion prediction methods (mr-Diff[2], MG-TSD[3]) are dedicated to efficient architectural design on conditionally denoised networks. Introducing consistency complementation in the progressively denoised diffusion training process will help to improve the generation accuracy and generalization ability of diffusion models for tasks such as time series prediction and interpolation. 

Specifically, if the observation sequence contains multi-resolution features, an intuitive design is to construct a multi-level representation extractor and subsequently reconstruct coarse- and fine-grained fluctuation patterns successively during the denoising process. 
In summary, we argue that the plain model architecture and conditional denoising strategy are the main reasons for the lack of novelty contribution to TimeDiT.

**Weakness 4:** 

The authors devised a gradient derived from physical laws represented by Partial Differential Equations (PDEs) to guide the denoising process in the inference. However, the experiments related to PDEs (Table 2) demonstrate the performance of TimeDiT in solving classical partial differential equations, rather than introducing PDE theory into time series prediction. the authors do not explain in detail how this contributes to the "incorporate physics knowledge as an energy-based prior for TimeDiT during inference". The authors' claim that TimeDiT “can integrate physics-informed knowledge into time series forecasting” is not compounded, and we believe that TimeDiT's contribution to the field of time series forecasting is overstated.

**Weakness 5:** 

The experimental part generally needs improvement, there is a lack of state-of-the-art baselines to verify that TimeDiT is sufficiently competitive, and TimeDiT shows really limited performance improvement.

* Table 1 is expected to introduce the most recent baseline under each paradigm, e.g., the data-specific model TimeMixer(ICLR2024)[4], LLM-based model TimeLLM(ICLR2024), Diffusion-based model MG-TSD (ICLR2024)[3], mr-Diff(ICLR2024)[2]. In addition, an important issue is that in Table 1, Diffusion-TS(ICLR2024) exhibits far worse performance than CSDI(NIPS2019), which is a confusing result. Meanwhile, TimeDiT exhibits worse performance in PhysioNet (a) and NASDAQ compared to CSDI (NIPS2019), and TimeDiT shows less than 10% performance improvement in MIMIC-III and PhysioNet (b). Does this suggest that TimeDiT obtained from pre-training based on the masking paradigm cannot directly demonstrate competitive predictive power?

* In Table 2, the performance improvement of TimeDiT is less than 5% in most of the PDEs, e.g., the average improvement in Advection, Navier-Stokes and Diffusion Sorption is 2.27%, 2.7% and 3.88%, respectively. Further, Table 2 compares the performance of TSDiff, PPDM and PPIM for solving Partial Differential Equations only, where TSDiff as a temporal prediction model does not have a priori the ability to solve PDEs, and where PPDM and PPIM are designed for vision generation tasks. Furthermore, to the best of our knowledge, TimeDiT was not the first to propose a method for solving PDEs using diffusion models. We hope that the introduction of TimeDiT with PDE-Diffusion (ICLR2024)[5] and DiffusionPDE (ICML2024)[6] in Table 2 will help validate the effectiveness of TimeDiT.

* Similar problems still exist for the zero-shot forecasting task, and Table 3 is considered to be missing state-of-the-art baselines, such as: 1) Timer(ICLR2024), a foundation model pre-trained in multiple temporal domains, 2) GPD(2024)[7], a recent unified diffusion model for time series and TimeLLM(ICLR2024). These models also show good potential, and further experiments are considered necessary. Indeed, similar problems exist in Tables 4 and 5, where the best SOTA model shown is OneFItsALL (NIPS2023), which is significantly outdated for the rapidly evolving field.

**Reference:** 

1) Zhang, Lvmin et al. “Adding Conditional Control to Text-to-Image Diffusion Models.” 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (2023): 3813-3824.

2) Shen, Lifeng et al. “Multi-Resolution Diffusion Models for Time Series Forecasting.” ICLR 2024.

3) Fan, Xinyao et al. “MG-TSD: Multi-Granularity Time Series Diffusion Models with Guided Learning Process.” ICLR 2024.

4) Wang, Shiyu et al. “TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting.” ICLR 2024.

5) Chonghan, Gao et al. "PDE-Diffusion: Physic guided diffusion model for solving partial derivative equations."

6) Huang, Jiahe et al. “DiffusionPDE: Generative PDE-Solving Under Partial Observation.” ICML 2024

7) Yang, Jiarui et al. “Generative Pre-Trained Diffusion Paradigm for Zero-Shot Time Series Forecasting.” ArXiv abs/2406.02212 (2024): n. pag.

Meta-review: This paper introduces Time Diffusion Transformer (TimeDiT), a diffusion-based generative model for time series. The authors provide experiments on a variety of tasks, such as forecasting, imputation, and anomaly detection and claim that their work provides a “general foundation model for time series”. They introduce new physics-based constraints on diffusion and show how this can be used in different settings.

While the reviewers agreed upon the importance and timeliness of the work, the reviewers also raised a number of concerns. In particular, there were some questions about whether the claims that the work is a foundation model were substantiated, as the pretraining is not conducted on multiple datasets from different domains and then finetuned on new datasets. The authors response to this question about the datasets used during pretraining was inadequate and unclear. 

Additionally, there was concern about the lack of ablations, missing baselines, and the novelty of the work. During the rebuttal period, the authors provided a new ablation analysis and added additional experiments. This was helpful and led some reviewers to increase their scores to borderline acceptance. 

In the end, because the model is not pretrained on multiple datasets nor finetuned on new datasets, the claims around the method being a general-purpose foundation model have not been demonstrated. In fact, the different tasks and datasets employ different pretraining and thus there is not even a common method for pretraining used throughout. The method thus is more well positioned as a general self-supervised approach for time series modeling that can be used on different downstream tasks.

Decision: Reject