TSGGuide: Recommendation Guide for Multivariate Time Series Generation

Authors: Beining Zhang, Kai Wu, Xiaoyu Zhang, Handing Wang, Jing Liu, Bei Wang
Primary area: learning on time series and dynamical systems
Keywords: time series generation, diffusion model, Channel Independence

Abstract:
Multivariate Time Series Generation (MTSG) plays a crucial role in time series analysis, supporting tasks such as data augmentation and anomaly detection. While several methods exist for MTSG, recommending the most suitable method for new scenarios remains a significant challenge. Although prior work by  (Ang et al., 2023a) provides guidance for selecting MTSG methods, it lacks coverage of recent diffusion-based methods and has limited exploration of channel-independent frameworks. We address these gaps by improving the recommendation guide, highlighting the effectiveness of a central discriminator within the channel-independent framework. Our revised guide makes three key recommendations: 1) VAE-based methods excel on small-scale datasets; 2) a channel-independent framework with the newly designed central discriminator is optimal in most cases; and 3) a diffusion-based method is preferable when ample data and computational resources are available.

Decision: reject
Reviewer scores: 5, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730719499237

thread_type:
official_review

<thread_object>
creator: reviewer_5rtz

<content>
summary:
This paper addresses three issues in the existing MSTG recommendation guidance: inadequate evaluation, limited independent exploration of channels, and the inability of CCD to capture dependencies. The authors propose an improved solution for MSTG. The innovation lies in introducing an enhanced CCD that demonstrates strong practicality in small sample scenarios. Compared to previous recommendation guidance, the proposed enhanced CCD places greater emphasis on the independent channel framework and diffusion model methods, aiming to provide more refined recommendations for different contexts.

strengths:
1. The paper is well-written, with clear and fluent language.
2. The experiments are comprehensive, and the details are thoroughly presented (exclude key hyperparameters though).
3. The overall methodology is easy to understand, and the enhanced CCD performs well.

weaknesses:
1. Some equations are not clearly presented; for example, the notation in Eq. 4 is not standard.
2. Since hyperparameters are crucial for time series-related tasks, the lack of details on hyperparameters (including those for your method and the compared methods) makes replication difficult.
3. The paper feels somewhat lacking in originality. Notably, in Section 5, while three synthetic datasets are used, the motivation for using these datasets is insufficient.
4. In Section 5, the paper primarily explains the advantages of CCD on small sample data from the perspective of time series classification, but it lacks a discussion of other possible reasons. Moreover, it does not sufficiently explore the strengths and weaknesses of the diffusion model method and the CCD module in different scenarios.

questions:
Plz see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730733699540

thread_type:
official_review

<thread_object>
creator: reviewer_vfbr

<content>
summary:
This paper aims at providing guidelines to researchers on how to select different Multivariate Time Series Generation models and metrics based on the scenatios that researchers are working on. More specifically, the authors provide their empirical findings across dataset characteristics, computational resources, and metrics. Based on their own findings and motivations on analyzing the perofrmance on small datasets with GAN-based model, the authors further developed CCD, a light-scale architecture for small datasets. It successfully improve the performance of GAN-based model on this task and make them close, even comparable to VAE-based models, which is dominant on this task. Moreover, the efficiency of CCD-based GAN is better than diffusion based model.

strengths:
This paper's content covers a pretty wide range. Its empirical findings include:

1. Understanding how we should select our models based on our dataset characteristics and computational resources

2. Improving the performance of GAN-based models on this task

The recommendations are useful for researchers to apply in their research

weaknesses:
1. The experiments, especially compared to the main benchmark paper TSGBench, is not comprehensive enough. Many datasets included in TSGBench are not covered. The TSGBench uses 10 datasets, while this paper only make use of the Stock and Energy dataset. The generalizability of the recommendation on other domains is questionable. Some baselines are also missing.

2. The writing is so confusing. I seldom comment a paper with this issue. But this paper's contribution is quite unclear. I spent several days trying to understand the central contribution of this paper. In abstract, the authors said that they would like to provide a recommendation. I understand it as a kind of more instructive benchmark. But then they start to present CCD, which is a margnially novel improvement on the GAN-based model (a new kind of discriminator in GAN). After that they again start to do analysis. My understanding is that the authors propose a relatively new improvement on GAN-based models. But then they found that the proposed model is still defeated by VAE-based model on small-scale dataset and diffusion models when resource is sufficient. So they try to sell their work by embedding it into a so-called recommendation.

3. Some results in this recommendation is well-known. The finding 4 about diffusion model's large training and inference load is well-known.

questions:
1. What is the main contribution of this paper? Do you want to sell a new method or provide a recommendation?

2. What are the performances of the CCD on other domains such as air quality?

3. About the training time, I hope to know more detailed analysis. For example, if method A is with longer training time of method B, is it because A need more iterations, or because its computational load on one sample is higher?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730734069986

thread_type:
official_review

<thread_object>
creator: reviewer_qryw

<content>
summary:
The paper provides a recommendation framework for selecting suitable multivariate time series generation (MTSG) methods across various scenarios. It addresses existing gaps in previous frameworks, particularly by incorporating recent diffusion-based methods and enhancing the channel-independent framework using a central discriminator (CCD). The paper conducts extensive experiments on several datasets, including Stock, Energy, and EEG data, which varifies the effectiveness of the proposed model.

strengths:
S1. The paper is well-organized, and each module is clearly explained.

S2. The CCD introduces frequency-domain transformations and a central discriminator to capture inter-channel dependencies.

weaknesses:
W1. The technical contribution need to be clarified. The proposed method reshapes time series data and applies a 2D convolutional framework, followed by optimization with a basic GAN architecture. However, this approach lacks novelty and may fall short in capturing the complex inter-channel correlations inherent to multivariate time series. Reshaping the data and applying 2D convolutions does not seem to address the unique and complicated dependencies between channels. A more comprehensive module that learns correlations directly to capture correlations and periodicity might be more effective and adaptive. Moreover, The CCD relies on 2D convolutions, a standard technique in time series modeling (such as TCN). The paper does not sufficiently justify what specific innovation CCD brings beyond typical 2D convolutional methods.

W2. The experiments focus on stock, energy, and EEG datasets, which may not sufficiently represent the diversity of real-world MTSG applications. The results would be more convincing if the proposed approach were evaluated on datasets covering additional scenarios of spatial-temporal forecasting (such as OD flow and traffic forecasting), which would better validate the method’s generalizability and effectiveness.

W3. The baselines used for comparison are not up-to-date, with the latest being from 2023. More recent methods like TimeGAE [1] and DLT [2] should be included to conduct sufficient experiment. What’s more, the paper lacks a detailed hyperparameter experiments, such as the weighting factor gamma in Equation 6. 

W4. The authors do not specify how the datasets are split into training, validation, and test sets in the experiments to ensure the reproducibility of results.
[1] Bai, Zhao, et al. "TimeGAE: A Multivariate Time-Series Generation Method via Graph Auto Encoder." International Conference on Database Systems for Advanced Applications. Singapore: Springer Nature Singapore, 2024.
[2] Feng, Shibo, et al. "Latent diffusion transformer for probabilistic time series forecasting." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 11. 2024.

questions:
Please refer to Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730738459638

thread_type:
official_review

<thread_object>
creator: reviewer_u8ef

<content>
summary:
This paper introduces TSGGuide, an updated recommendation guide for multivariate time series generation, aimed at refining the selection of generation methods. Through comprehensive analysis, the authors identified limitations in existing frameworks, specifically the limited exploration of channel-independent frameworks and a lack of evaluation for diffusion-based models. By enhancing the central discriminator within the channel-independent framework, incorporating methods such as TimesNet and attention mechanisms, and introducing CCD, the study demonstrates notable performance improvements in small-sample scenarios.

strengths:
- This work improves upon previous guidelines by prioritizing both channel-independent and diffusion-based generative frameworks, providing users with a more comprehensive and practical TSGGuide for selecting time series generation frameworks.

- By enhancing the central discriminator in the channel-independent framework with advanced techniques (e.g., TimesNet, attention mechanisms, CCD), the study demonstrates clear performance gains in small-sample scenarios, contributing strong methodological advancements.

- The proposed framework shows effective performance in enhancing CCD and CCD-based models, as supported by experimental results.

weaknesses:
- **Motivation:** The motivation for this work is not fully justified. The rationale for addressing limitations in channel-independent methods and diffusion models in MTSG for small-sample datasets seems somewhat forced. Small-sample datasets tend to yield low-quality results in diffusion models due to limited information, challenging the model’s ability to learn data distributions. Despite this, the paper shows that CCDGAN’s performance is comparable to that of Diffusion-TS, suggesting a lack of theoretical and empirical support for this motivation.
- **Dataset Limitation:** Although the paper claims TSGGuide is a highly flexible mechanism capable of adapting to diverse tasks, it only evaluates stock, energy, and medical datasets. The limited selection constrains the demonstration of the module’s plug-and-play capability, which may impact the credibility of its claimed flexibility across various domains.
- **Channel-Independent Approach:** The paper’s choice to adopt a channel-independent framework raises questions when applied to stock datasets, which exhibit strong inter-channel correlations (e.g., stocks in the same sector often follow similar trends). While the model demonstrates effectiveness, it may overlook inherent characteristics of financial data by disregarding these correlations.

questions:
- Given that only stock, energy, and medical datasets were used, how might this affect the generalizability and credibility of TSGGuide’s flexibility across different domains?

- In small-sample contexts, GANs generally outperform diffusion-based models. Yet, as noted in the paper, CCDGAN does not significantly outperform Diffusion-TS. Can the authors provide additional evidence to substantiate their theoretical claims?

- Is CCDGAN essentially a combination of the CCD mechanism with GAN? The methodology section lacks detail here; could you provide more on how CCDGAN was developed?

- Given the emphasis on a channel-independent framework, why are there no comparisons with channel-mixed framework methods? Since channel-independent frameworks ignore inter-sequence interactions, empirical evidence is needed to validate their benefits, especially for financial datasets where sequences are highly correlated.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732529025172

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thank you for your insightful feedback. In response, we have revised the manuscript and uploaded the updated PDF, with changes highlighted in blue. Key revisions include:

**Clarification of Motivation:** To enhance clarity, we emphasize that the primary objective of this work is to analyze current time series generation methods and develop an effective recommendation framework, providing users with informed guidance across various scenarios. To align with this goal, we have moved the validation of the Central Convolution Discriminator (CCD) to the supplementary materials, ensuring it does not overshadow the core motivation.

**Increased Dataset Diversity:** To improve the robustness and accuracy of the proposed guidelines, we have incorporated datasets from diverse domains, including DLG (traffic data) and Air. Additionally, we generated 100 synthetic datasets using the methodology of Meidani et al. (2023) for comprehensive evaluation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732764216986

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

As the author-reviewer discussion period nears its deadline, we kindly ask you to review our responses to your comments, concerns, and suggestions. If you have any additional questions or feedback, we will do our best to address them promptly before the discussion period concludes. If our responses have satisfactorily addressed your concerns, we would greatly appreciate it if you could update your evaluation of our work accordingly.

Thank you once again for your valuable time and thoughtful feedback.

Sincerely, 

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a recommendation guide for MTSG methods, incorporating diffusion-based techniques and a channel-independent framework. The paper’s strengths include a comprehensive analysis, clear explanations, and improved guidelines for small-sample scenarios. However, it suffers from limited dataset diversity, insufficient exploration of channel-independent frameworks, and a lack of technical novelty. 


Despite the authors’ rebuttal, the paper’s weaknesses, such as unclear equations, limited datasets, and insufficient originality, remain. The reviewers appreciated the clear language and comprehensive experiments but found the main contribution unclear and the writing confusing. Given these issues, I recommend rejecting the paper as it does not meet the necessary standards for acceptance.