Title: Shallow Flow Matching for Coarse-to-Fine Text-to-Speech Synthesis
Venue: NeurIPS 2025
Accepted: yes
Keywords: text-to-speech

speech synthesis

flow matching

coarse-to-fine

generative model
Abstract: We propose Shallow Flow Matching (SFM), a novel mechanism that enhances flow matching (FM)-based text-to-speech (TTS) models within a coarse-to-fine generation paradigm. Unlike conventional FM modules, which use the coarse representations from the weak generator as conditions, SFM constructs intermediate states along the FM paths from these representations. During training, we introduce an orthogonal projection method to adaptively determine the temporal position of these states, and apply a principled construction strategy based on a single-segment piecewise flow. The SFM inference starts from the intermediate state rather than pure noise, thereby focusing computation on the latter stages of the FM paths. We integrate SFM into multiple TTS models with a lightweight SFM head. Experiments demonstrate that SFM yields consistent gains in speech naturalness across both objective and subjective evaluations, and significantly accelerates inference when using adaptive-step ODE solvers. Demo and codes are available at https://ydqmkkx.github.io/SFMDemo/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors present a novel Shallow Flow Matching (SFM) mechanism inspired by DiffSinger's [27] Shallow Diffusion Mechanism. SFM constructs intermediate paths along the Flow Matching's (FM) path, using coarse output representations i.e, it introduces a way of starting the ODE solver from the intermediate state in the flow path instead of Gaussian noise $N(0, I)$ as done by previous SOTA methods. Using the principle of shallow diffusion mechanism, they introduce an adaptive timestep $t_h$ which can be used as a pre-informed point to initiate solving the vector field ODE. During training, authors propose to use an orthogonal projection (by adding an SFM head to the encoder) method to predict the temporal parameter and use it to estimate the segment's starting point (coarse output). During inference, this coarse output is used as the initial point instead of isotropic Gaussian noise. Evaluations confirm that SFM improves both subjective, and objective evaluations and accelerates inference when using an **adaptive ODE solver**.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Evaluating the proposed method on three different SOTA architectures, mainly consisting of UNets, and DiT, demonstrates the effectiveness of SFM and decouples the advantage of proposed changes with previous design choices.\n- Novel and principled design improvement, by formalising piecewise-single segment flow and then evaluating it in practice.This grounds the advantage of using SFM especially when using adaptive ODE solver, showing that model can take longer steps confidently when solving the ODE as error is smaller than using just OT-CFM.\n\n### Weakness\n* While in theory method improves the solution of ODE when using an adaptive solver. In practice none of the SOTA models use an adaptive solver. Most use either order 1 or order 2 solvers and perceptually it makes little to no difference.\n* Because of an additional hyperparameter $\\alpha$ which is tuned for each architecture each dataset, the method adds an additional overhead to optimise this parameter.\n* It would have been better if the authors could quantify Number of Function Evaluation (NFE), a commonly used evaluation term in diffusion/flow matching literature.\n\nQUESTIONS:\n* What are $\\hat{t_g}$ and $\\hat{\\sigma_g}$ in Table 2? it is not concretely defined in the paper; are they already scaled? \n* Practically, most of these architectures (at least all three of Matcha-TTS, Stable-TTS, and CosyVoice {which uses Matcha-TTS's U-Net}), in the architecture of their flow prediction network, concatenate the noise and coarse spectrogram ($\\mu$). This essentially means the initial point is sampling from Gaussian noise centered around  $N(\\mu, I)$. How is this different from such modifications? Is it that SFM also predicts a $\\sigma$ so the noise is scaled at the coarse spectrogram?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-23"}

comments: SUMMARY:
The authors propose a Shallow Flow Matching (SFM) algorithm that can inference from the intermediate state rather than pure noise and reduce computation cost and time for the Flow matching generation.  An orthogonal projection method to adaptively determine the temporal position of the intermediate states, and a single-segment piecewise flow is proposed to generate remaining states. Experiments show that SFM consistently improves the naturalness of synthesized speech in both objective and subjective evaluations. It is a solid work.

STRENGTHS AND WEAKNESSES:
Strengths:
- An improved flow matching algorithm is proposed to reduce the computation.
- A proof is provided for the proposed algorithm
- Good results are achieved compared with multiple baselines

Weaknesses:
- The computation cost baseline only include the ablated systems it would be helpful if the baselines results are included too.

QUESTIONS:
1 In Table 3, the proposed systems are outperform than the baseline systems, which is derived from pure noise. Can we state the generation outputs are better if they were generated the intermediate states instead of noise states? Any explanation?

2. Table 4 and 10 compare the RTFs for different ODE solvers under different $\alpha$. They only compared with ablated systems, how about the comparison with the baselines mentioned?

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-06-26

comments: SUMMARY:
This paper proposes Shallow Flow Matching (SFM), a mechanism designed to improve flow matching (FM)-based text-to-speech (TTS) models within a coarse-to-fine generation paradigm. SFM constructs intermediate states along the FM paths using coarse output representations. During training, it employs an orthogonal projection method to adaptively determine the temporal position of these states and applies a principled construction strategy based on a single-segment piecewise flow. During inference, SFM starts from this intermediate state instead of pure noise, focusing computation on the later stages of the FM paths. The authors integrate SFM into multiple TTS models.

STRENGTHS AND WEAKNESSES:
Strengths:
The motivation is clear. The problem that to speed up the FM-basd TTS models is worth to be explored.

Weaknesses:
1. As the authors mentioned it, the idea of shallow diffusion is proposed by DiffSinger. The unique challenge of introduce such idea into FM model has not be explained clearly.
2. The experimental settings is simple. Only three small-scale English dataset (LJSpeech, VCTK, and LibriTTS) are included. The authors could also introduce the single-stage FM model such as E2TTS or F5TTS as the base models.
3. Some results of Table 3 displayed that using SFM could lead to the negative effect for WER.
4. The presentation of the efficency benefits after using SFM is not clear. There should be also the RTF results in Table 3 in addition to Table 4.

QUESTIONS:
NA

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Shallow Flow Matching (SFM) mechanism to enhance flow matching (FM)-based Text-to-Speech (TTS) models that operate within a coarse-to-fine generation paradigm. The core idea is to leverage the coarse representations produced by a weak generator to construct an intermediate starting point for the FM process, rather than starting from pure noise. The authors demonstrate that integrating SFM into several TTS models improves both the sampling quality and inference speed.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper provides a theoretical validation for the proposed SFM method, with theorems and proofs.\n* The method demonstrates improved performance compared to its baseline counterparts in subjective evaluations of naturalness and similarity.\n* The significant improvement in inference speed when using the proposed method is thoroughly analyzed.\n\nWeaknesses:\n* The experimental results on small datasets like LJSpeech and VCTK do not adequately justify the method's scalability. While LibriTTS is a comparatively larger set, the experiments do not convincingly demonstrate how this approach would perform on the massive, diverse datasets (10k+ hours) used by many state-of-the-art zero-shot TTS models (e.g., VALL-E, Voicebox, MELLE, DiTTo-TTS, E2-TTS, F5-TTS, and CosyVoice 2).\n* The method has limited applicability in its current form. While the authors show its applicability to CosyVoice, a two-stage coarse-to-fine generative model, it is unclear if or how SFM could be applied to other prominent TTS methods, such as single-stage large diffusion/flow-matching models (e.g., DiTTo-TTS, F5-TTS) or discrete-token generative models (e.g., Llasa, MaskGCT).\n* The paper does not compare the proposed acceleration technique to other well-known methods for speeding up generative model inference, such as consistency models. This makes it difficult to assess the significance of the speed improvements in the broader context of fast sampling techniques.\n \n[1] Wang, Yuancheng, et al. \"MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer.\" The Thirteenth International Conference on Learning Representations.\n\nQUESTIONS:\nThe main issues I would like to see addressed are already listed in the weaknesses section above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-03"}