Title: Sequence Modeling with Spectral Mean Flows
Venue: NeurIPS 2025
Accepted: yes
Keywords: sequence modeling

time series

hidden Markov models

mean embeddings

linear operators

maximum mean discrepancy

gradient flows
Abstract: A key question in sequence modeling with neural networks is how to represent and learn highly nonlinear and probabilistic state dynamics. Operator theory views such dynamics as linear maps on Hilbert spaces containing mean embedding vectors of distributions, offering an appealing but currently overlooked perspective. We propose a new approach to sequence modeling based on an operator-theoretic view of a hidden Markov model (HMM). Instead of materializing stochastic recurrence, we embed the full sequence distribution as a tensor in the product Hilbert space. A generative process is then defined as maximum mean discrepancy (MMD) gradient flow in the space of sequences. To overcome challenges with large tensors and slow sampling convergence, we introduce spectral mean flows, a novel tractable algorithm integrating two core concepts. First, we propose a new neural architecture by leveraging spectral decomposition of linear operators to derive a scalable tensor network decomposition of sequence mean embeddings. Second, we extend MMD gradient flows to time-dependent Hilbert spaces and connect them to flow matching via the continuity equation, enabling simulation-free learning and faster sampling. We demonstrate competitive results on a range of time-series modeling datasets.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Spectral Mean Flow (SMF), a new sequence modeling framework that combines an operator‑theoretic view of Hidden Markov Models, a tensor‑network decomposition of the sequence mean embedding derived from the spectral decomposition of the underlying operator, and a time‑dependent Maximum Mean Discrepancy (MMD) gradient flow that is trained with the Flow‑Matching objective. The resulting model is fully differentiable, avoids simulation during training, and samples by solving an ODE. Empirically, SMF attains competitive scores on six multivariate time‑series data sets compared to diffusion, VAE, and GAN baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n* The paper unifies three previously disjoint ideas (operator theory, tensor networks, flow matching) into a coherent generative framework. In particular, the extension of MMD flows to time‑dependent RKHSs (Eq. 3.15) and its connection to flow matching appear original.\n* The authors discuss complex‑valued parameterization, low‑rank sharing, and stability tricks (SwiGLU, RMS‑LN, sharing most of the computations), which will be valuable for practitioners. \n\n\n**Weaknesses:**\n\n* The paper does not convincingly articulate the practical advantages of this new paradigm over standard flow-matching or diffusion models. Because the operator is not modeled explicitly, potential benefits such as interpretability through mode analysis remain out of reach, and the added complexity may not translate into clear gains.\n* All empirical studies are restricted to fixed-length sequences of 24 time steps. The authors do not investigate scalability to longer or variable-length sequences, nor do they discuss any practical limitations in this regard.\n* A detailed complexity analysis is missing: the authors do not report wall-clock training or inference times, FLOPs, or parameter counts, nor do they compare these metrics with competing baselines.\n* Important recent baselines, such as SDFormer [1], ImagenTime [2], FiDE [3], KoVAE [4] and other recent works, are omitted, making it hard to judge the proposed method’s relative performance.\n\n[1] Chen, Zhicheng, et al. \"Sdformer: Similarity-driven discrete transformer for time series generation.\" NeurIPS 2024\n\n[2] Naiman, Ilan, et al. \"Utilizing image transforms and diffusion models for generative modeling of short and long time series.\" NeurIPS 2024\n\n[3] Galib, Asadullah Hill, Pang-Ning Tan, and Lifeng Luo. \"FIDE: Frequency-Inflated Conditional Diffusion Model for Extreme-Aware Time Series Generation.\" NeurIPS 2024.\n\n[4] Naiman, Ilan, et al. \"Generative modeling of regular and irregular time series data via Koopman VAEs.\" ICLR 2024.\n\nOther related work:\n\n[5] Coletta, Andrea, et al. \"On the constrained time-series generation problem.\" NeurIPS 2023.\n\n[6] Jeon, Jinsung, et al. \"GT-GAN: General purpose time series synthesis with generative adversarial networks.\" NeurIPS 2022\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide a novel method for generating sequences using a technique called spectral mean flows. This uses a neural network parameterization of a spectral decomposition of linear operators to define a generative model over the space of sequences. They demonstrate this on toy data sets and multiple timeseries dat sets showing good distribution match between ground truth and generated data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe authors provide adequate background grounding their extensions. This method makes no assumptions on characteristics of the timeseries being modeled (in comparison to others which use Fourier transforms to explicitly parameterize periodicity). The methods are original.\nThe authors recognize a main limitation that methods to generate conditioned on a partial sequence are not provided.\nWeaknesses:\nThe authors do not comment on whether a timeseries produced through irregular or informative sampling would be handled well by this approach. For example, a system with a measurement generator function M(h) which emits measurements of value x at time t according to the value of some hidden state h such that  both the value x and the time difference between x_t and x_{t+1} are dependent on h.\n\nQUESTIONS:\nFollowing from the potential weakness above, can this method incorporate information from an informatively sampled system?\n\nThe authors comment that “G is closed under point wise multiplication“ and “in the infinite width limit, multi-layer perceptrons (MLPs) are norm equivalent to Sobolev spaces”. How well do finite width and depth MLPs approximate this space? Does this lead to unique scaling performance with MLP size?\n\nLIMITATIONS:\nThe authors recognize a main limitation that methods to generate sequences conditioned on a partial sequence are not provided.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper utilizes an operator-theoretic interpretation of hidden Markov models to propose a novel sequence modeling algorithm. The proposed method, spectral mean flows, utilizes a neural architecture that leverages the spectral decomposition of linear operators for scalability and extends the maximum mean discrepancy (MMD) gradient flows to time-dependent Hilbert spaces. The authors validate the method on both simulated and real-world time-series datasets.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: The submission is technically sound. The background and methods sections are especially technically sound, going into the details. However, I am not fully convinced that the spectral mean flow improves much from other baselines. For example, the predictive scores of spectral mean flow are on par or worse than other baselines across all datasets.\n\nClarity: The paper is mostly clearly written and well organized. It has clear methods, background, and related work sections. However, I think the figure and table captions can be improved, providing more explanations or intuition, especially for Figure 1.\n\nSignificance: Although the idea is novel, and the technical details in the paper are sound, the paper lacks experimental results to back up the proposed method. I think a more careful demonstration of the advantage of using spectral mean flow over other baseline methods will improve the significance of the paper.\n\nOriginality: The work provides new insight through an operator-theoretic view of a hidden Markov model.\n\nQUESTIONS:\nCould you provide more explanation and intuition on the right panel of Figure 2? More specifically, why does the curve look as it does (initially it goes up)? Moreover, why does it take much longer for the MMD of spectral mean flow to decrease as much as TD-RBF?\n\nCould you provide visualizations for some of the generated sequences in Section 5.2? I think that could help me understand the results from Table 1 better.\n\nThe paper makes claims on faster convergence and scalability of the spectral mean flow. Could you run ablation studies that show how each part of your method contributes to this efficiency?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper introduces a sequence modeling algorithm grounded in an operator-theoretic interpretation of Hidden Markov Models. Rather than directly implementing stochastic recurrence, the authors embed distributions into Hilbert spaces. The model further incorporates a time-dependent MMD gradient flow for sampling and employs a flow-matching learning algorithm trained via double backpropagation.

STRENGTHS AND WEAKNESSES:
Strengths: The paper is clearly written and easy to follow. The motivation behind the work is well-articulated, and the related literature is discussed. The proposed method is explained in detail, with clear examples of its integration into state-of-the-art graph learning models.

Weaknesses: 
1) The paper lacks discussion and experimental analysis on the robustness of the proposed method with respect to real word datasets. Also the results in Table 1 is very marginal.
2) The framework currently lacks a principled approach for conditioning the generative process on partial observations.
3) The learned model makes it difficult to extract identifiable components like the observation operator or left eigenfunctions, which limits its interpretability.

QUESTIONS:
Q1) Do the authors have any preliminary insights on how kernel Bayesian inference or Laplace transforms of conditional mean operators could be practically incorporated into their framework for conditioning on partial data?

Q2) Have the authors explored methods to recover or approximate interpretable components from the end-to-end model?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03