Grassmannian Geometry Meets Dynamic Mode Decomposition in DMD-GEN: A New Metric for Mode Collapse in Time Series Generative Models

Authors: Yassine ABBAHADDOU, Amine M. Aboussalah
Primary area: learning on time series and dynamical systems
Keywords: Time series, Generative models, Mode Collapse

Abstract:
Generative models like Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) often fail to capture the full diversity of their training data, leading to mode collapse. While this issue is well-explored in image generation, it remains underinvestigated for time series data. We introduce a new definition of mode collapse specific to time series and propose a novel metric, DMD-GEN, to quantify its severity. Our metric utilizes Dynamic Mode Decomposition (DMD), a data-driven technique for identifying coherent spatiotemporal patterns, and employs Optimal Transport between DMD eigenvectors to assess discrepancies between the underlying dynamics of the original and generated data. This approach not only quantifies the preservation of essential dynamic characteristics but also provides interpretability by pinpointing which modes have collapsed. We validate DMD-GEN on both synthetic and real-world datasets using various generative models, including TimeGAN, TimeVAE, and DiffusionTS. The results demonstrate that DMD-GEN correlates well with traditional evaluation metrics for static data while offering the advantage of applicability to dynamic data. This work offers for the first time a definition of mode collapse for time series, improving understanding, and forming the basis of our tool for assessing and improving generative models in the time series domain.

Decision: reject
Reviewer scores: 3, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730145776108

thread_type:
official_review

<thread_object>
creator: reviewer_cbwd

<content>
summary:
This work focuses on the time series generation problem. Specifically, they focus on the issue of "mode collapse" in time series generative model and propose a new notation of mode collapse and a new evaluation metric based on Dynamic Mode Decomposition (DMD) to quantify time series model collapse. They claim the proposed metric has connection/motivation/benefits from Grassmannian geometry and optimal transport. They provide some experiments on simple time series datasets to support their claims.

strengths:
-

weaknesses:
The main theme of this paper is confusing. The proposed metric seems to trailed for dynamical systems, with "highest frequency $\Delta t$" (`line 135`) known. Yet, the authors claim it to be a generic metric and test it on common time series datasets. Moreover, there are clear logic gaps in conveying the motivation/theoretical ground of the proposal. I find it hard to understand, and not very convincing. I detail reasons and clarification questions in below.

* Why Grassmannian?
* How does the propose method/metric handle the inherent noise in time series? Can you elaborate this from a data-driven perspective? (i.e., how the number of sample affect the noise robustness?)
* The experiments are not convincing. Please include more summary statistics of the generated data, and compare with baselines. For example, it's well-known that stock data have many important  summary statistics like volatility, moving average...e.t.c. It's only fair to show that the proposal achieve more similar summary statistics to ensure its efficacy. 
* Is Thm. 3.4 your own result or from previous work? If it's from previous work, can you use lemma and specify source in lemma title instead?

questions:
see above Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730468422369

thread_type:
official_review

<thread_object>
creator: reviewer_avzd

<content>
summary:
The paper proposes DMD-GEN, a method for evaluating mode collapse in timeseries generative models.
The method is based on evaluating Grassmanian geometry for dynamic mode decomposition (DMD) for timeseries.
In experiments, DMD-GEN is compared to baselines on synthetic and real timeseries.

strengths:
1. Novelty. To my knowledge, this is the first paper which studies directly a problem of mode collapse in timeseries.
2. A diverse set of timeseries generative models is studied (TimGAN, DiffusionTS, TimeVAE).

weaknesses:
1. line 136: "Assuming uniform sampling in time, we approximate the dynamical system linearly as $x_{k+1} \approx A x_k$."
It seems to be an unrealistic assumption for real data.
2. Limited novelty. DMD was proposed before.
3. I don't understand, why DMD-GEN in superior to alternatives in section 4.5. For example, ContextFID also changes monotonically.
4. A comparison with an MTopDiv (Barannikov et al. 2021) is missing. MTopDiv can also be applied to evaluation of timeseries generative models.
5. You claim "This approach not only quantifies the preservation of essential dynamic characteristics but also provides interpretability by pinpointing which modes have collapsed." I can't find a proof that your method can pinpoint "which modes have collapsed. 
6. You claim that "This work offers for the first time a definition of mode collapse for time series..." 
The definition of mode collapse is quite basic - it means that a particular mode is present in training data, but absent in generated data.

Barannikov, S., Trofimov, I., Sotnikov, G., Trimbach, E., Korotin, A., Filippov, A., & Burnaev, E. (2021). Manifold Topology Divergence: a Framework for Comparing Data Manifolds. Advances in neural information processing systems, 34, 7294-7305.

questions:
1.  "The results demonstrate that DMD-GEN correlates well with traditional evaluation metrics for static data while offering the advantage of applicability to dynamic data". What is the difference between dynamic and static data?
2. What does "||" mean in line 146, line 203?
3. Do you study univariate or multivariate timeseries?
4. What is the purpose of "x" in generators in section 4.5?

**Conclusion**.
Overall, I think that the paper raises an important research question: how to evaluate mode collapse in timeseries generative models.
But experiments providing the superiority w.r.t. baselines are not convincing and there are only few of them.
Some claims in abstract are not supported with strong evidence.
Overall, my opinion is that the paper contains some interesting ideas (application of DMD to evaluation of timeseries generative models), 
but it must be accompanied with a stronger empirical evidences, including not only synthetic mode collapse (see examples of such experiments in Barannikov et al., 2021).


**Post rebuttal**. I am thankful to authors for the detailed responses and conducting additional experiments. Sorry for the delay in my answers.
I think that the discussion helped to improve that manuscript. I am updating my rating. The final rating is defined by the overall novelty and impact of your method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684375158

thread_type:
official_review

<thread_object>
creator: reviewer_ue8p

<content>
summary:
This paper presents DMD-GEN, a novel metric for assessing mode collapse in generative models for time series data, specifically using Dynamic Mode Decomposition (DMD). The study addresses a gap in mode collapse research, which has predominantly focused on image data, by proposing a time-series-specific approach. DMD-GEN leverages Optimal Transport to measure dynamic similarities between original and generated data, providing an interpretable metric to quantify the extent of mode collapse. The authors validate DMD-GEN on synthetic and real-world datasets using various generative models, demonstrating its consistency with traditional metrics offering computational efficiency without additional training.

strengths:
The introduction of DMD-GEN as a time-series-specific metric for mode collapse is a **novel** contribution. This approach brings a new perspective to evaluating time series generative models by capturing coherent dynamic patterns, which is less explored.

The methodology is robust and **comprehensive**, integrating techniques from DMD and Optimal Transport. The authors provide a clear mathematical foundation for their approach, along with thorough experimental validation across diverse datasets. The authors also provide the **public code** of the DMD-GEN implementation for practical use.

DMD-GEN offers both **computational efficiency** and interpretability, making it a potentially valuable tool in time-series modeling. Its ability to detect and quantify mode collapse without retraining the model makes it practical for real-time applications.

weaknesses:
While the experiments conducted demonstrate the effectiveness of DMD-GEN, the paper could benefit from a **broader range of datasets and generative models** to strengthen its generalizability. More extensive empirical validation could enhance the paper's claims about the metric's superiority.

While DMD-GEN is computationally efficient compared to metrics that require retraining, calculating DMD-GEN could still be resource-intensive on **large datasets** or **high-dimensional** time series. Discussing the limitations of DMD-GEN in terms of scalability would be valuable.

questions:
Would the authors consider evaluating DMD-GEN on larger datasets and more complex and longer time-series data? This would strengthen the paper's claims regarding the metric's applicability.

Can the authors discuss potential scalability issues when applying DMD-GEN to larger or more complex datasets? For example, it could be useful to measure memory footprint and computational times as dataset size changes, while employing a specific set of computational resources.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730724830864

thread_type:
official_review

<thread_object>
creator: reviewer_yiss

<content>
summary:
This paper addresses the significant challenge of mode collapse in time series generative models. The authors introduce a novel definition of mode collapse tailored specifically to time series data and propose a new metric, DMD-GEN, for its quantitative evaluation. DMD-GEN leverages Dynamic Mode Decomposition (DMD) to identify coherent spatio-temporal patterns and uses Optimal Transport to quantify discrepancies between real and generated data. The metric is tested across various recent time series generative models. Experimental results demonstrate DMD-GEN's effectiveness in evaluating the time series generative models.

strengths:
- The paper studies the critical and underexplored problem of evaluating mode collapse in time series generative models. It also highlights the absence of specific metrics designed to address the dynamic characteristics of time series data.
- The proposed metric DMD-GEN is novel to the best of my knowledge. Furthermore, it provides valuable insights into the dynamics of generated time series, allowing for a deeper analysis in future research on time series modeling.
- DMD-GEN is computationally efficient as no additional training is required, which is very practical in real-time applications.

weaknesses:
- The proposed metric effectively demonstrates that Diffusion-TS captures the target time series without mode collapse. However, the paper lacks a comparative analysis with models that are prone to mode collapse. It’s essential to provide such comparisons to verify the metric’s capability to discern between generative models that do and do not suffer from mode collapse.
- The robustness across different types of time series data is not thoroughly discussed, particularly for non-stationary time series with a low signal-to-noise ratio given the results on the stock dataset (please refer to Question 1 for a detailed concern).
- The presentation of results in figures and tables could be more self-contained. Specifically, the captions of Figures 1 and 2 should briefly explain the depicted results. Additionally, Table 1 should clearly indicate whether higher or lower values represent better performance.

questions:
1. The differences between epoch 0 and the final epoch are significant in the comparisons of DMD eigenvalues of the original and generated time series across various datasets, with an exception noted in the stock dataset. Specifically, in Figure 2, the generated samples closely match the actual data from the start at epoch 0, with a slight improvement after the final epoch. Does it suggest that the DMD-GEN metric might not be as effective for highly non-stationary time series with low signal-to-noise ratio like stocks, compared to other types of data?
2. Do the results on the stock dataset in Figure 2 differ from Figure 6?
3. I am interested in exploring how the proposed DMD-GEN metric performs when applied to time series generated using traditional time series generation methods, e.g., Bootstrap. Could you extend your experiments to include this technique and provide the corresponding results?
4. Can you discuss the limitations of the DMD-GEN metric? Are there particular characteristics of time series datasets that might limit DMD-GEN's applicability, based on your current insight?
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes DMD-GEN, a novel metric based on Dynamic Mode Decomposition (DMD) and Grassmannian geometry to evaluate mode collapse in time series generative models. The authors claim it captures temporal dynamics more effectively than static metrics. Strengths include the novelty of addressing mode collapse in time series data, computational efficiency without retraining, and the provision of interpretability by identifying collapsed modes. However, weaknesses raised by reviewers include limited empirical validation, lack of comparisons with models prone to mode collapse, insufficiently addressed noise robustness, and unclear presentation of results and theoretical contributions. Ultimately, the lack of strong empirical evidence and clarity led to a borderline rejection recommendation.