UTSD: Unified Time Series Diffusion Model

Authors: Xiangkai Ma, Xiaobin Hong, Wenzhong Li, Sanglu Lu
Primary area: learning on time series and dynamical systems
Keywords: time series anlysis, foundation model, cross-doman, diffusition model

Abstract:
Transformer-based architectures have achieved unprecedented success in time series analysis. However, facing the challenge of across-domain modeling, existing studies utilize statistical prior as prompt engineering fails under the huge distribution shift among various domains. In this paper, a Unified Time Series Diffusion (UTSD) model is established for the first time to model the multi-domain probability distribution, utilizing the powerful probability distribution modeling ability of Diffusion. Unlike the autoregressive models that capture the conditional probabilities of the prediction horizon to the historical sequence, we use a diffusion denoising process to model the mixture distribution of the cross-domain data and generate the prediction sequence for the target domain directly utilizing conditional sampling. The proposed UTSD contains three pivotal designs: (1) The condition network captures the multi-scale fluctuation patterns from the observation sequence, which are utilized as context representations to guide the denoising network to generate the prediction sequence; (2) Adaptor-based fine-tuning strategy, the multi-domain universal representation learned in the pretraining stage is utilized for downstream tasks in target domains; (3) The diffusion and denoising process on the actual sequence space, combined with the improved classifier free guidance as the conditional generation strategy, greatly improves the stability and accuracy of the downstream task. We conduct extensive experiments on mainstream benchmarks, and the pre-trained UTSD outperforms existing foundation models on all data domains, exhibiting superior zero-shot generalization ability. After training from scratch, UTSD achieves comparable performance against domain-specific proprietary models. In particular, UTSD shows stable and reliable time series generation, and the empirical results validate the potential of UTSD as a time series foundational model. The source codes of UTSD are publicly available on https://anonymous.4open.science/r/UTSD-1BFF.

Decision: reject
Reviewer scores: 5, 6, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730482002673

thread_type:
official_review

<thread_object>
creator: reviewer_evfc

<content>
summary:
This paper proposes a time series diffusion model for time series forecasting in multi-domain settings. The proposed model consists of the condition net, adapter, and denoising net. Different from prior autoregressive models, a diffusion denoising process is applied to capture the mixture distribution of cross-domain data and generate predictions directly. Experimental results show the effectiveness of the proposed method on probabilistic time series forecasting tasks.

strengths:
According to the authors' claim, this is the first work to incorporate diffusion models into time series foundation models.

weaknesses:
- The paper is generally not well-written. There are many typos and formatting issues.

- Example of typos: adaptor in the abstract

- Example of formatting issues: Text size is not consistent in the abstract, and citep is not used.

- There are false statements when criticizing prior works to motivate the proposed method. For example, while UniTime [Liu et al., 2024a] and Moirai [Woo et al., 2024] are non-autoregressive, authors claim that they "rely on an auto-regressive mechanism" in L085-088.

- Different from the statement around key contributions, the idea of denoising with condition has already been studied in many prior works, including [Fan et al.], [Shen et al., 2024], [Yuan and Qiao, 2024].

[Fan et al.] MG-TSD: Multi-Granularity Time Series Diffusion Models with Guided Learning Process. In ICLR, 2024.

- L114-116: "each iteration of the denoising stage causes an accumulation of errors in the latent space, which are further amplified during the alignment of the latent space to the actual sequence space." -> Is there any evidence on this statement?

- Not enough justification on the effectiveness of the proposed components. The ablation study in Table 5 is insufficient to explain why they should work.

- L177-179: Any supporting theory or experiments on the inherent drawbacks stated here?

- While it is natural to consider large-scale datasets for "unified" time series diffusion model, the datasets used in this paper is limited to small single-task datasets.

- Average performance should not be counted when ranking methods.

- Please distinguish citet and citep.

- Zero-shot forecasting is done in an unusual way; while the source and target datasets are similar ETT variations in this work, they usually have a significant gap in the standard zero-shot forecasting settings.

- Table 3 caption does not make sense.

- In Table 5, it is not clear what happens if condition net is missing, which implies that no input is given.

questions:
Please address concerns above.

---
I appreciate the response from authors, they addressed some of my concerns. I raised my rating accordingly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730548343813

thread_type:
official_review

<thread_object>
creator: reviewer_pg2f

<content>
summary:
This work introduces a conditional diffusion model for MTS prediction task. Different from previous non-Diffusion based methods, it employs a complicated denoised process to model the latent discrepancies in both temporal and channels (domains as described in this work). Different from the diffusion-based methods, it is flexible since it can be easily scalable for N time steps input and M time steps output scenario thanks to its well-designed interaction between conditional embeddings and denoised outputs. In general, the proposed method achieves promising results in various MTS prediction tasks such as prediction and cross-domain prediction.

strengths:
1. The motivation of designing conditional diffusion model for flexible N time steps input and M time steps output is clear and reasonable, the model architecture is also well-designed.
2. Experimental results are soundness and promising.

weaknesses:
1. The dimension transform is not clearly presented, which may make readers confused. For example, given N time steps input lookback, how to obtain a M time steps output prediction? Although it might be discussed in the manuscript, however, it is still not so easy to follow.
2. As claimed by the authors in line 201-206 that “existing conditional diffusion models often use simple NN layer to capture … demonstrates low accuracy”. However, according to Sec. 3, as far as I understand that the authors use CNN or Transformer based modules to transform features along the time embedding dimension of input series. For example, in denoising net of Fig. 2, the dimension of condition $\overline{h}$ is $P_L \times P_d$ while the dimension of denoised series before decoder block-a is also $P_L \times P_d$. The time dimension transformation happens at the very last block in denoising net, decoder block-a. Therefore, it should still be a simple NN layer conceptually, which is conflicted with the authors claim.
3. How do the authors obtain a so called pretrained models? It is not clear in the manuscript.
4. The authors mentioned many times in the manuscript on down stream tasks of MTS, it might be unsuitable since they only concern MTS prediction task, although including traditional prediction and cross-domain prediction, but it still belongs to MTS prediction task, not involving other down stream MTS tasks, such as anomaly detection or impainting.
5. In Fig. 4, the authors show results of some short-term periodic series. How about the results for long-term periodic series and short-term non-periodic series? These should be crucial for practical use.

questions:
See the weakness above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730595084956

thread_type:
official_review

<thread_object>
creator: reviewer_ybbt

<content>
summary:
The model presents a Unified Time Series Diffusion (UTSD) model, which is established for the first time to model the multi-domain probability distribution, utilizing the powerful probability distribution modeling ability of Diffusion.

strengths:
1 The condition-denoising architecture is novel.
2 The performance improvement compared to listed baselines.
3 The paper presents cross-domain generalizability.

weaknesses:
1 The comparison did not include some interesting baselines.
2 The cross-domain transfer is limited to similar datasets. 
3 I did not see obvious advantages compared to DiffusionTS.

questions:
1 The evaluation is very limited int he time series prediction. As listed in DiffusionTS, we should include other different metrics (like Discriminative Score and so on) to evaluate the performance. MSE/MAE evaluation is very biased evaluation metrics. For example, a straight average line achieve reasonable performances but is not meaningful.
2 The model should compare with DiffusionTS in Table 1/2, which I did not see in this paper. I would highly recommend the comparison since it is also diffusion based methods.
3 DiffusionTS can handle both conditional and unconditional generation. Is there a way for this method also be applicable to unconditional time series generation?
4 The paper claims this paper to be a foundation model for TS area. I would like to see the performance of a short forcasting model applied to long context prediction and vice versa.
5 For Table 3, the performance of ETHh1 to ETHh2 should be needed.
6 The performances of the other two datasets: Solar Forecasting and Mujoco Imputation should be included since it is widely benchmarked.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730621000839

thread_type:
official_review

<thread_object>
creator: reviewer_aaqn

<content>
summary:
This paper introduces UTSD (Unified Time Series Diffusion), a novel diffusion-based model for multi-domain time series forecasting. The key contributions include:
- A condition-denoising architecture that captures multi-scale temporal patterns
- Direct modeling in actual sequence space rather than latent space
- An improved classifier-free guidance strategy for conditional generation
- A transfer-adapter module for efficient domain adaptation
The model achieves state-of-the-art results on multiple benchmarks in different forecasting scenarios including across-domain training, zero-shot learning, and probabilistic forecasting.

strengths:
This study introduce an innovative condition-denoising architecture along with an actual-space diffusion and denoising process, complemented by a classifier-free guidance strategy for conditional generation. This study represents the first establishment of a foundational time series model leveraging diffusion techniques. The model demonstrates exceptional cross-domain generalization capabilities, indicating its potential to redefine paradigms in time series analysis.And the Unified Time Series Diffusion (UTSD) model achieves state-of-the-art (SOTA) results across various benchmarks. 
The originality of this model, coupled with its high-quality outcomes, clarity in methodology, and significant implications for the field, underscores the importance of this findings.

weaknesses:
- There are too few types of evaluation indicators, as a model generated by the diffuison process, more indicators such as those used in the DiffusionTS paper, such as Context-FID Score, Correlational Score, Discriminative Score, Predictive Score, etc.
- Theoretical analysis of why the model works better in actual sequence space vs latent space could be stronger
- Memory and computational complexity analysis is missing
- Limited discussion of model limitations and failure cases
- Some architectural choices (like number of blocks) need better justification
- Comparison with some recent relevant works could be expanded

questions:
1. Can you please add evaluation indicators other than MSE and MAE?
2. As the author said in the paper “the average MSE is reduced by 17.9%, 18.6% and 22.4% compared to the existing TimeLLM, LLM4TS and GPT4TS, which indicates that the proposed method can fully utilize a small amount of data for efficient training.” Could you please explain why a decrease in MSE means that less data can be used for training? 
3. In order to train a across-domain model, a mixed dataset needs to be established. However, will the training cost increase after the dataset is enlarged?
4. How does the computational complexity compare to existing approaches?
5. What are the main limitations/failure cases of the model?
6. How sensitive is the model to hyperparameter choices?
7. Can the authors provide theoretical justification for why modeling in actual sequence space is better?
8. How does the model perform on extremely long sequences?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730746736107

thread_type:
official_review

<thread_object>
creator: reviewer_jtdw

<content>
summary:
The paper proposes a Unified Time Series Diffusion (UTSD) model for time series tasks (mainly focusing on forecasting tasks). The model utilizes a diffusion-based approach rather than traditional autoregressive methods. In particular, the conditional network, adaptor fine-tuning strategy are considered. Numerical experiments and sample codes are provided.

strengths:
1. This paper considers an interesting research topic which trying to apply the diffusion model into time-series analysis domain.

2. Various experimental tasks are considered and cover diverse domains and performance comparisons against both foundation models and domain-specific models.

weaknesses:
1. My primary concern lies in the suitability of employing a non-autoregressive diffusion model for forecasting tasks. Forecasting is typically a regression problem rather than a generative one, and autoregressive behavior is often observed in certain domains, such as climate and finance. Given this, the current draft would benefit from further justification regarding the choice of a non-autoregressive diffusion model in this context.

2. The architecture of the proposed model requires further discussion. Section 3 describes the proposed model as a combination of convolution and attention blocks. However, existing literature shows that models based solely on either convolutional or attention networks are already effective for forecasting tasks. Could the authors clarify why a hybrid structure is necessary here? Specifically, how does this configuration address any limitations of diffusion models in forecasting?

Minor Issue: In Appendix E.2, the benchmarks also include imputation and classification tasks, but these seem to be missing from the main draft.

questions:
Please see the weakness section.

At this stage, the novelty of the work is insufficient for acceptance at a top-tier machine learning conference, so I tend to recommend rejection. However, I am open to reconsidering my evaluation based on further discussion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732286950197

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions

comment:
We sincerely thank all the reviewers for their insightful reviews and valuable comments, which are instructive for improving our paper further.

In this paper, a **Unified Time Series Diffusion (UTSD)** model is established **for the first time to model the multi-domain probability distribution**, utilizing the powerful probability distribution modeling ability of Diffusion. Unlike existing models that capture the conditional probabilities of the prediction horizon to the historical sequence, we use a diffusion denoising process to model the mixture distribution of the cross-domain data and generate the prediction sequence for the target domain directly utilizing conditional sampling. The proposed UTSD contains several pivotal designs: The **condition network** captures the multi-scale fluctuation patterns from the observation sequence, which are utilized as context representations; The **adaptor-based fine-tuning strategy** is utilized for downstream tasks in target domains before multi-domain joint pretraining, and supports flexible observation and prediction horizons through token-level alignment. We conduct extensive experiments on mainstream benchmarks, and UTSD outperforms existing foundation models on all data domains, exhibiting superior zero-shot generalization ability. Empirical results validate the potential of UTSD as a time series foundational model.

The reviewers generally expressed positive opinions of our paper, noting that the method proposed "**an interesting research topic,**" coupled with "**its high-quality outcomes, clarity in methodology, innovative architecture,**" demonstrates "**significant implications for the field,**" and achieves "**state-of-the-art performance.**" Additionally, they highlighted its "**potential to serve as the time-series unified diffusion model.**"

The reviewers also raised insightful and constructive concerns. We have made every effort to address all the concerns by providing sufficient evidence and requested results. Here is the summary of the major revisions:

1. **Employing non-autoregressive diffusion model for forecasting (Reviewers jtdw, eVfc)**: Diffusion models exhibit good cross-domain generalisation by modelling integrated conditional probability distributions across multiple domains, rather than temporal dependencies within individual domains. Extensive experiments indicate the suitability of our approach for forecasting tasks that involve complex, multi-domain data.
2. **Add multi-tasking experiment results (Reviewers jtdw, YBBT)**: The UTSD model is designed to be a versatile tool for various time-series tasks, following the generative-based pretrain paradigm. We conduct additional experiments on imputation and unconditional cross-domain generation tasks to assess UTSD's performance.
3. **Add more evaluation metrics and baseline models (Reviewers AAQN, YBBT)**: To evaluate the quality of generated time series, we refer to the metrices from DiffusionTS and perform the comparison experiments.
4. **Compare our method and previous Diffusion4TS models (Reviewers YBBT, eVfc)**: We provide a comprehensive analysis of the differences of our method and other Diffusion4TS models, focusing on cross-domain generalization, efficient fine-tuning paradigm, flexible timesteps of input and output.
5. **More visualisation results for challenging time series forecasting (Reviewers AAQN, pg2F)**: We introduce more visualisation results in the revision appendix section of the revised version. UTSD also shows good performance in extremely long sequence prediction, which verifies that UTSD has the ability to capture extremely long term dependencies, which is crucial for practical applications. In addition, UTSD also shows satisfactory prediction results in short-term non-periodic sequences.

The valuable suggestions from reviewers are very helpful for us to improve our paper. **All the changes I've made in our revision (highlighted in blue)**. We'd be very happy to answer any further questions.

Looking forward to the reviewer's feedback.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732609952850

thread_type:
official_comment

<thread_object>
creator: area_chair_jgvu

<content>
title:
Encouragement to Actively Participate in the Discussion Phase

comment:
Dear Reviewers,

Thank you for your valuable contributions to the review process so far. As we enter the discussion phase, I encourage you to actively engage with the authors and your fellow reviewers. This is a critical opportunity to clarify any open questions, address potential misunderstandings, and ensure that all perspectives are thoroughly considered.

Your thoughtful input during this stage is greatly appreciated and is essential for maintaining the rigor and fairness of the review process.

Thank you for your efforts and dedication.
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Summary of Scientific Claims and Findings
This paper introduces the Unified Time Series Diffusion (UTSD) model for multi-domain time series tasks. UTSD leverages a diffusion denoising process to model joint probability distributions across diverse domains, bypassing the autoregressive approach. The paper proposes three key contributions:
A condition network to capture multi-scale patterns from observation sequences as context for denoising.
An adapter-based fine-tuning strategy to enhance domain adaptability while reusing universal representations learned in pretraining.
A diffusion process performed in the actual sequence space with classifier-free guidance to improve stability and performance.
Extensive experiments demonstrate that UTSD outperforms baseline models in forecasting, imputation, and other tasks, with claims of superior zero-shot and cross-domain generalization.

(b) Strengths of the Paper
Novelty: UTSD introduces a novel approach by integrating diffusion models into time series foundational models and emphasizing sequence-space denoising.
Extensive Experiments: The authors present results across multiple benchmarks and tasks, showcasing improved performance over existing baselines.
Practical Contributions: The adapter-based fine-tuning mechanism provides computational efficiency during domain-specific adaptation.
Visualization and Analysis: The paper includes visualizations and ablation studies to support claims about the model’s effectiveness in long- and short-term dependencies.

(c) Weaknesses of the Paper
Theoretical Insufficiency: The paper lacks rigorous theoretical justification for the advantages of sequence-space modeling over latent-space alternatives. While some empirical evidence is provided, the theoretical foundation remains weak.
Limited Cross-domain Generalization: The model's cross-domain performance is restricted to datasets with similar characteristics, reducing its applicability in more diverse, real-world scenarios.
Inadequate Comparisons: While comparisons to certain baselines (e.g., DiffusionTS) are included, the absence of benchmarks with other state-of-the-art models like MOIRAI or UniTS undermines the claim of being a comprehensive foundational model.
Evaluation Metrics: The initial reliance on MSE/MAE as the primary evaluation metrics is problematic for generative models. Although additional metrics were added during rebuttal, this should have been included from the outset.
Computational Overhead: While UTSD optimizes diffusion processes through design choices like de-Markovization, diffusion models inherently require higher computational resources, especially for long sequences and multiple iterative denoising rounds.
Presentation and Clarity: Some sections, particularly on the hybrid architecture and diffusion process, lack sufficient explanation or empirical backing, leading to questions about the necessity and effectiveness of the design.

(d) Reasons for Rejection
Scope and Applicability: The UTSD model is innovative but lacks generalizability to a broad range of time series tasks. Its reliance on diffusion models limits its practical impact.
Theoretical Gaps: The paper does not provide sufficient theoretical insights into the proposed approach, particularly regarding the choice of sequence-space modeling and hybrid architectures.
Benchmarking Limitations: Missing comparisons with key state-of-the-art models and insufficient diversity in datasets weaken the empirical claims.
Reviewer Concerns: Despite an active rebuttal period, several critical concerns raised by reviewers—including cross-domain generalization, theoretical contributions, and computational efficiency—remain inadequately addressed.