Venue: NAACL 2025

# WaveFM: A High-Fidelity and Efficient Vocoder Based on Flow Matching

Tianze Luo, Xingchen Miao, Wenbo Duan

Tsinghua Universtiy

{ltz22,miuxc22,dwb22}@mails.tsinghua.edu.cn

###### Abstract

Flow matching offers a robust and stable approach to training diffusion models. However, directly applying flow matching to neural vocoders can result in subpar audio quality. In this work, we present WaveFM, a reparameterized flow matching model for mel-spectrogram conditioned speech synthesis, designed to enhance both sample quality and generation speed for diffusion vocoders. Since mel-spectrograms represent the energy distribution of waveforms, WaveFM adopts a mel-conditioned prior distribution instead of a standard Gaussian prior to minimize unnecessary transportation costs during synthesis. Moreover, while most diffusion vocoders rely on a single loss function, we argue that incorporating auxiliary losses, including a refined multi-resolution STFT loss, can further improve audio quality. To speed up inference without degrading sample quality significantly, we introduce a tailored consistency distillation method for WaveFM. Experiment results demonstrate that our model achieves superior performance in both quality and efficiency compared to previous diffusion vocoders, while enabling waveform generation in a single inference step. 1

Footnote 1: Our codes are available at [https://github.com/luotianze666/WaveFM](https://github.com/luotianze666/WaveFM).

## 1 Introduction

Recent advancements in network architectures and training algorithms have greatly enhanced the ability of deep generative models to produce high-fidelity audio in speech synthesis Lee et al. (2021); Siazdak (2023); Huang et al. (2023); Wang et al. (2023); Nguyen et al. (2024); Ju et al. (2024); Kumar et al. (2024). The initial breakthrough came with the autoregressive generation of waveforms from mel-spectrograms Oord et al. (2016); Kalchbrenner et al. (2018), which provided high audio fidelity but suffered from slow inference speeds. To enable real-time high-fidelity speech synthesis, a variety of non-autoregressive models have been introduced, classified broadly into three categories: flow-based models, generative adversarial networks (GANs), and diffusion models.

Flow-based models utilize invertible neural networks to generate waveforms from a selected prior distribution, such as a Gaussian distribution, estimating log-likelihoods during training Ping et al. (2020); Prenger et al. (2019). While these intricately designed models maintain invertibility and evaluate determinants, this complexity limits their flexibility, and consequently the quality of the audio output.

Generative Adversarial Networks (GANs) provide greater flexibility than flow-based models and can generate waveforms with high fidelity more efficiently Kumar et al. (2019); Kong et al. (2020); Kim et al. (2021); Jang et al. (2021); Lee et al. (2022); Siuzdak (2023). Their success stems from the generators' large receptive fields and the discriminators' ability to detect noise across various scales and periods. For instance, Kumar et al. (2019) introduced multi-scale discriminators, while Kong et al. (2020) developed a multi-receptive field (MRF) generator alongside multiple multi-period discriminators, leading to substantial improvements. Moreover, Lee et al. (2022) further enhanced sample quality by utilizing the snake activation function and integrating the anti-aliased multi-periodicity (AMP) composition module.

Denoising diffusion probabilistic models (DDPMs) have recently gained significant popularity for their ability to transform a simple prior distribution into a complex ground truth distribution through a Markov chain process Kong et al. (2020); Lam et al. (2022); Huang et al. (2023); Nguyen et al. (2024). These models rely on a parameter-free noise-adding diffusion process to generate training data for the denoising generator, eliminating the need for auxiliary networks like discriminators or autodecoders during training.



However, the inference phase of diffusion models tends to be time-consuming. To mitigate this issue, Kong et al. (2020); Lam et al. (2022), and Huang et al. (2023) introduced several fast-sampling algorithms that speed up waveform generation, though with a minor compromise in sample quality. Consistency models Song et al. (2023); Song and Dhariwal (2023) have been proposed to enhance the efficiency of diffusion models by directly predicting the endpoint of the probability flow ordinary differential equation (PF-ODE) at each step, enabling single-step inference. These models surpass previous distillation approaches in image synthesis tasks and achieve higher distillation efficiency by aligning along ODE trajectories, thereby avoiding the need to numerically solve the entire ODE.

In this study, we propose WaveFM for the mel-spectrogram conditioned speech synthesis task. Firstly, since the mel-spectrogram records the energy information in waveforms, an appropriately conditioned distribution can significantly improve sample quality. Additionally, we adopt a reparameterized flow matching method that directly predicts the waveform, allowing us to apply several auxiliary losses to the original flow-matching loss to further enhance the model's performance.

To better supervise the model's output waveforms regarding phase angles, we also incorporate a multi-resolution phase loss into our model. Furthermore, gradient and Laplacian operators are utilized on the real and generated spectrograms. Minimizing the mean square losses associated with these operations enables the model to better learn edge details and structural patterns in the spectrograms.

Finally, we propose a tailored consistency distillation method for WaveFM to further accelerate the model's inference speed while maintaining audio quality. The subjective and objective experiment results indicate that WaveFM outperforms previous diffusion models in terms of sample quality and efficiency, and generalizes better on out-of-distribution musical mel-spectrograms.

## 2 Related Works

### Flow Matching and Rectified Flow Models

Flow matching Lipman et al. (2022) and rectified flow Liu et al. (2022) models share a similar training objective, and diffusion models can also be interpreted within this framework. Here, we briefly introduce their mathematical principles using simpler, self-contained notation.

**Theorem 1**: Let \({\mathbf{x}_{t}}\) be a continuously differentiable random process on \(t\in[0,1]\) and \(p({\mathbf{x}},t)\) be its probability density function. We denote the prior distribution as \({\mathbf{x}_{0}}\) and the ground truth distribution as \({\mathbf{x}_{1}}\). If the conditional expectation is locally Lipschitz, we let

\[{\mathbf{v}}({\mathbf{x}},t)=\mathbb{E}\left[\frac{\mathrm{d}{\mathbf{x}_{t}}}{\mathrm{d} t}\middle|{\mathbf{x}_{t}}={\mathbf{x}}\right]. \tag{1}\]

Then samples from the data distribution \({\mathbf{x}_{1}}\) can be obtained by sampling from the prior distribution \({\mathbf{x}_{0}}\) and solving the following ODE with an initial value \({\mathbf{x}_{0}}\) at time \(t=0\):

\[\frac{\mathrm{d}{\mathbf{x}}}{\mathrm{d}t}={\mathbf{v}}({\mathbf{x}},t). \tag{2}\]

The detailed proof is available in Appendix A. The conditional expectation can be expressed as a simple mean square training objective:

\[\min_{{\mathbf{v}}}\mathbb{E}\left\lVert\frac{\mathrm{d}{\mathbf{x}_{t}}}{\mathrm{d} t}-{\mathbf{v}}({\mathbf{x}_{t}},t)\right\rVert^{2}. \tag{3}\]

In practice, straight trajectories generally imply lower transportation costs. Thus, we take

\[{\mathbf{x}_{t}}=t{\mathbf{x}_{1}}+(1-t){\mathbf{x}_{0}},\quad t\in[0,1], \tag{4}\]

which leads to the following objective for the neural network \({\mathbf{v}}({\mathbf{x}},t)\). After the training process, data samples can be generated by numerically solving the ODE according to **Theorem 1**.

\[\min_{{\mathbf{v}}}\mathbb{E}\left\lVert{\mathbf{x}_{1}}-{\mathbf{x}_{0}}-{\mathbf{v}}({\mathbf{ x}_{t}},t)\right\rVert^{2}. \tag{5}\]

### Consistency Distillation

Consistency Distillation (CD) Song et al. (2023) is an efficient method for distilling diffusion models to enable one-step generation. In the original paper, the authors adopt the following forward Stochastic Differential Equation (SDE) to diffuse data:

\[\mathrm{d}{\mathbf{x}}=\sqrt{2t}\,\mathrm{d}{\mathbf{w}},\quad t\in[\epsilon,T], \tag{6}\]

where \(\epsilon=0.002\) and \(T=80\). The corresponding backward SDE and PF-ODE are given by:

\[\mathrm{d}{\mathbf{x}}=-2t\nabla_{{\mathbf{x}}}\log p({\mathbf{x}},t)\, \mathrm{d}{\mathbf{t}}+\sqrt{2t}\,\mathrm{d}{\mathbf{w}},\] (7) \[\qquad\mathrm{d}{\mathbf{x}}=-t\nabla_{{\mathbf{x}}}\log p({\mathbf{x}},t)\, \mathrm{d}{\mathbf{t}}.\] (8) 

In CD, the time steps are discretized by

\[t_{i}=\epsilon^{1/\rho}+\frac{i-1}{N-1}(T^{1/\rho}-\epsilon^{1/\rho}), \tag{9}\]

where \(N\) is the total number of discretization steps, \(\rho=7\), and \(i\in\{1,\cdots,N\}\). According to the forward SDE, CD samples \(n\sim\mathcal{U}\{1,2,\ldots,N-1\}\), \(\mathbf{x}_{t_{n+1}}\sim\mathcal{N}\left(\mathbf{x}_{\text{clean}},t_{n+1}^{2}\mathbf{ I}\right)\), and then the pretrained teacher score network is used to compute \(\hat{\mathbf{x}}_{t_{n}}\) by numerically solving the PF-ODE, where any type of ODE solver can be chosen for this purpose. The student network in CD aims to predict the endpoint at time \(t=\epsilon\) of the PF-ODE trajectory at any position and time, parameterized as follows:

\[\mathbf{f_{\theta}}(\mathbf{x},t)=c_{\text{skip}}(t)x+c_{\text{out}}(t)\mathbf{F_{\theta} }(\mathbf{x},t), \tag{10}\]

where \(c_{\text{skip}}(t)=\frac{\sigma_{\text{data}}^{2}}{(t-\epsilon)^{2}+\sigma_{ \text{data}}^{2}}\), \(c_{\text{out}}(t)=\frac{\sigma_{\text{data}}(t-\epsilon)}{\sqrt{\sigma_{ \text{data}}^{2}+t^{2}}}\), \(\sigma_{\text{data}}=0.5\) and \(\mathbf{F_{\theta}}(\mathbf{x},t)\) is the neural network. The loss function of CD is given by

\[\lambda(t_{n})d\left(\mathbf{f_{\theta}}(\mathbf{x}_{t_{n+1}},t_{n+1}),\mathbf{f_{\theta ^{-}}}(\hat{\mathbf{x}}_{t_{n}},t_{n})\right), \tag{11}\]

where \(\lambda(t_{n})\) is a scale function, and \(d(\cdot,\cdot)\) is a distance function such as L2 distance. The parameters \(\mathbf{\theta}^{-}\) are updated using Exponential Moving Average (EMA), where \(\mu\) is the EMA decay rate.

\[\mathbf{\theta}^{-}\leftarrow\mathrm{stopgrad}(\mu\mathbf{\theta}^{-}+(1-\mu)\mathbf{ \theta}). \tag{12}\]

## 3 Methodology

### Mel-Conditioned Prior Distribution

Mel-conditioned prior distributions have been applied to diffusion models Lee et al. (2021); Koizumi et al. (2022), but they do not closely approximate the audio distribution due to the necessity of stabilizing diffusion training objectives. For instance, Lee et al. (2021) utilize \(\mathcal{N}(\mathbf{\mu},\mathbf{\Sigma})\) as the diffusion prior distribution, with their training objective defined as

\[\mathbf{x}_{t}=\sqrt{\bar{\alpha}_{t}}(\mathbf{x}_{0}-\mathbf{\mu})+\sqrt{1-\bar{\alpha}_ {t}^{2}}\mathbf{\epsilon}, \tag{13}\]

\[\min_{\mathbf{\epsilon}}(\mathbf{\epsilon}-\mathbf{\epsilon_{\theta}}(\mathbf{x}_{t},t))^{ \top}\mathbf{\Sigma}^{-1}(\mathbf{\epsilon}-\mathbf{\epsilon_{\theta}}(\mathbf{x}_{t},t)), \tag{14}\]

where \(\mathbf{x}_{0}\sim p_{data}\) and \(\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{\Sigma})\). They set \(\mathbf{\mu}=0\) and \(\mathbf{\Sigma}\) as a diagonal matrix derived from the mel-spectrogram. Nonetheless, to stabilize the training process, they need to clamp the standard deviations between 0.1 and 1, which increases the distance between the prior and the audio distribution.

According to **Theorem 1**, however, to train a flow matching model, we only need to sample from two marginal distributions without requiring their analytical forms. This means that WaveFM could utilize a prior distribution with much smaller variance without compromising training stability. We choose \(\mathcal{N}(\mathbf{0},\mathbf{\Sigma})\) as the prior distribution with a diagonal \(\mathbf{\Sigma}\). We utilize the logarithmic mel-spectrograms as inputs to the neural network, as the raw values span a wide range of \([0,32768]\). Since the mel-spectrogram captures the energy of the audio signal, the square root of the sum across the frequency dimension is a suitable choice for the standard deviation of the prior distribution. We normalize it by dividing it by \(\sqrt{\texttt{mel-bands}}\times 32768\) to ensure that it falls within \([0,1]\), apply linear interpolation to align its shape with the audio, and clamp the values with a minimum of \(10^{-3}\). Given that the values in a mel-spectrogram are typically much smaller than the potential maximum value, the standard deviation can indeed approach \(10^{-3}\) in nearly silent regions. This suggests that our prior distribution is aligned with the audio distribution more closely. Our ablation study indicates that the adopted prior distribution enhances the sample quality of WaveFM.

### Training Objective

We follow the notation in subsection 2.1. The original objective in Equation 5 aims to estimate a random derivative, which not only prevents the incorporation of auxiliary losses, such as the mel-spectrogram loss, but also complicates the design of a neural network with periodic inductive bias, as the random noise can disrupt periodic patterns. Experiment results demonstrate that this original objective can result in inferior sample quality for our network. Therefore, we hope the neural network to directly generate audio from noise, rather than predicting random derivatives. To achieve this, we choose to reparameterize the original mean square objective as follows:

\[\mathbf{x}_{1}-\mathbf{x}_{0}=\frac{\mathbf{x}_{0}-\mathbf{x}_{t}}{1-t},\quad t\in[0,1). \tag{15}\]

\[\mathbf{v}(\mathbf{x},t)=\mathbb{E}[\mathbf{x}_{1}-\mathbf{x}_{0}|\mathbf{x}_{t}]=\frac{\mathbb{E} [\mathbf{x}_{1}|\mathbf{x}_{t}]-\mathbf{x}}{1-t}. \tag{16}\]

\[\Leftrightarrow\min_{v^{\prime}}\mathbb{E}\left\lVert\mathbf{x}_{1}-\mathbf{v}^{\prime} (\mathbf{x}_{t},t)\right\rVert^{2},t\in[0,1). \tag{17}\]

We can now directly utilize a neural network to predict clean audio from mel-spectrograms, similar 

to GANs, allowing for the straightforward addition of auxiliary losses to the mean square loss:

\[\begin{split}&\min_{v^{\prime}}\Big{(}\frac{1}{1-t} \mathbb{E}\left\|\mathbf{x}_{1}-\mathbf{v}^{\prime}(\mathbf{x}_{t},t)\right\|^{2}\\ &+\lambda_{0}\mathbb{E}\left[\mathrm{STFTLoss}(\mathbf{x}_{1},\mathbf{v}^ {\prime}(\mathbf{x}_{t},t))\right]\\ &+\lambda_{1}\mathbb{E}\left\|\mathrm{mel}(\mathbf{x}_{1})-\mathrm{ mel}(\mathbf{v}^{\prime}(\mathbf{x}_{t},t))\right\|_{1}\Big{)}.\end{split} \tag{18}\]

The total loss function employed by WaveFM is defined as above, with \(\lambda_{0}=0.02\), \(\lambda_{1}=0.02\). To prevent factor \(\frac{1}{1-t}\) from approaching infinity and stabilize training, we set the coefficient to 10 for \(t\in[0.9,1)\). The first auxiliary loss function is a multi-resolution STFT loss, initially introduced by Parallel WaveGAN (Yamamoto et al., 2019). They apply the short-time Fourier transform (STFT) at three resolutions to both the clean audio and the generated audio, with FFT, hop, and window sizes set to \((1024,2048,512)\), \((120,240,50)\), and \((600,1200,240)\), respectively. The spectral convergence loss \(L_{\text{sc}}\) and log STFT magnitude loss \(L_{\text{mag}}\) are computed as follows:

\[L_{\text{sc}}(\mathbf{x},\hat{\mathbf{x}})=\frac{\||\mathrm{STFT}(\mathbf{x})|-|\mathrm{ STFT}(\hat{\mathbf{x}})|\|_{F}}{\||\mathrm{STFT}(\mathbf{x})|\|_{F}}, \tag{19}\]

\[L_{\text{mag}}(\mathbf{x},\hat{\mathbf{x}})=\frac{1}{N}\left\|\log\frac{|\mathrm{STFT }(\mathbf{x})|}{|\mathrm{STFT}(\hat{\mathbf{x}})|}\right\|_{1}, \tag{20}\]

where \(\mathbf{x},\hat{\mathbf{x}}\) denote the clean and generated audios, respectively; \(N\) is the number of elements in the STFT spectrogram; \(\|\cdot\|_{F},\|\cdot\|_{1}\) denote the Frobenius and L1 norms; operators to the STFT-shaped matrices inside the norms are element-wise. Thus, the total loss function equals

\[\mathrm{STFTLoss}(\mathbf{x},\hat{\mathbf{x}})=\frac{1}{3}\sum_{m=1}^{3}(L_{\text{sc} }^{(m)}+L_{\text{mag}}^{(m)})(\mathbf{x},\hat{\mathbf{x}}) \tag{21}\]

Notably, the original multi-resolution STFT loss leverages only the magnitude information in STFT spectrograms. So we replace \(L_{\text{sc}}\) with a phase angle loss \(L_{\text{pha}}\), defined as

\[\Delta P=\mathrm{Phase}(\mathrm{STFT}(\mathbf{x}))-\mathrm{Phase}(\mathrm{STFT}( \hat{\mathbf{x}})), \tag{22}\]

\[L_{\text{pha}}(\mathbf{x},\hat{\mathbf{x}})=\frac{\left\|\mathrm{atan2}(\sin\Delta P, \cos\Delta P)\right\|_{1}}{N}, \tag{23}\]

where \(\mathrm{atan2}\) is used to wrap the phase difference into \((-\pi,\pi]\). We do not compute the phase angle loss where the squared magnitude is less than \(1\times 10^{-6}\), as the phase angles there are insignificant and can produce excessively large gradients that destabilize the model. In our implementation, we add a small constant of \(1\times 10^{-6}\) to the squared magnitude for the computation of \(L_{\text{mag}}^{(m)}\), and we adjust the FFT, hop, and window sizes to \((1024,2048,512)\), \((128,256,64)\), and \((512,1024,256)\), respectively.

Finally, to further enhance the detection capability of our multi-resolution STFT loss, we apply temporal gradient, frequency gradient, and Laplacian operators to the magnitude of both the clean and generated spectrograms. These operators are defined respectively as follows:

\[\frac{1}{4}\begin{bmatrix}-1&1\\ -2&2\\ -1&1\end{bmatrix},\frac{1}{4}\begin{bmatrix}-1&-2&-1\\ 1&2&1\end{bmatrix},\frac{1}{8}\begin{bmatrix}-1&-1&-1\\ -1&8&-1\\ -1&-1&-1\end{bmatrix}. \tag{24}\]

We then compute their mean square errors, scaled by 4, 4 and 2, respectively, to help the model learn the regular patterns within the spectrograms. These operators enhance the visibility of edge information in the spectrograms, enabling the model to capture finer details more effectively.

Ablation studies demonstrate that our multi-resolution STFT loss significantly improves sample quality in one-step generation for WaveFM. This is further illustrated in Figure 1, where the spectrograms of audio generated using the original loss exhibit lower accuracy compared to those generated using our proposed loss.

The second auxiliary loss function is the L1 loss of the mel-spectrograms. Both subjective and objective experiment results indicate that these two auxiliary losses significantly improve sample quality. Besides, we provide the Pytorch implementation details of our multi-resolution STFT loss in Appendix C.

It is worth noting that our reparameterization diverges at time \(t=1\). Therefore, during distillation, we restrict the range of \(t\) to \([0,0.99]\), as a \(t\) that

Figure 1: Spectrograms of a clean audio and audios generated by WaveFM-6 Steps using the original STFT loss and our proposed STFT loss, from left to right.

 

is too large is ineffective since the waveforms are already sufficiently clean. The training process is summarized in **Algorithm**1, where \({\rm prior}(m)\) denotes the diagonal covariance matrix derived from the mel-spectrogram \(m\). For simplicity, we denote the reparameterized \(\mathbf{v^{\prime}}\) as \(\mathbf{v_{\theta}}\) in the algorithm.

```
Input: neural network \(\mathbf{v_{\theta}}\), mel-spectrogram \(m\), time step \(t\sim U[0,1]\), \(\lambda_{0}=0.02,\lambda_{1}=0.02\) repeat \(\mathbf{x}_{1}\sim p_{\rm data}(\mathbf{x}|m),\mathbf{x}_{0}\sim\mathcal{N}(\mathbf{0},{ \rm prior}(m))\) \(\mathbf{x}_{t}=t\mathbf{x}_{1}+(1-t)\mathbf{x}_{0},\mathbf{v}_{0}=\mathbf{v_{\theta}}(\mathbf{x}_{t}, t,m)\) \({\rm Loss}=\frac{1}{\min(0.1,1-t)}\left\lVert\mathbf{x}_{1}-\mathbf{v}_{0}\right\rVert^{2}\) \(+\lambda_{0}\operatorname*{STFTLoss}(\mathbf{x}_{1},\mathbf{v}_{0})\) \(+\lambda_{1}\left\lVert{\rm mel}(\mathbf{x}_{1})-{\rm mel}(\mathbf{v}_{0})\right\rVert _{1}\) Take gradient descent according to loss. until WaveFM converges
```

**Algorithm 1** Train WaveFM

### Distillation Objective

The conventional inference method using numerical ODE solvers typically requires numerous steps to generate waveforms, which contradicts our efficiency demand. Inspired by consistency distillation Song et al. (2023) for SDEs, we propose a specialized consistency distillation algorithm for our model, summarized in **Algorithm**2.

```
Input: student network \(\mathbf{v_{\theta}}\), teacher network \(\mathbf{v^{\prime}_{\theta^{\prime}}}\), EMA decay rate \(\mu=0.999\), mel-spectrogram \(m\), distance \(d(\cdot,\cdot)\), time duration \(\Delta t=0.01\) Initialize EMA parameters \(\mathbf{\theta}^{-}=\mathbf{\theta}\) repeat \(\mathbf{x}_{1}\sim p_{\rm data}(\mathbf{x}|m),\mathbf{x}_{0}\sim\mathcal{N}(\mathbf{0},{ \rm prior}(m))\) \(t\sim\widetilde{\mathcal{N}}\left(0,0.33^{2}\right)\), where \(\widetilde{\mathcal{N}}\) refers to the truncated Gaussian distribution into \([0,0.99]\) \(\mathbf{x}_{t}=t\mathbf{x}_{1}+(1-t)\mathbf{x}_{0}\) if\(t+\Delta t>0.99\)then \({\rm target}=\mathbf{x}_{1}\) else \(\mathbf{x}_{t+\Delta t}={\rm ODESOLVE}(\mathbf{v^{\prime}_{\theta^{\prime}}},\mathbf{ x}_{t},t,\Delta t)\) \({\rm target}=\mathbf{v_{\theta^{-}}}(\mathbf{x}_{t+\Delta t},t+\Delta t,m)\) endif \({\rm Loss}=d(\mathbf{v_{\theta}}(\mathbf{x}_{t},t,m),{\rm target})\)  Take gradient descent to loss to update \(\mathbf{\theta}\) \(\mathbf{\theta}^{-}={\rm stopgrad}(\mu\mathbf{\theta}^{-}+(1-\mu)\mathbf{\theta})\) untilWaveFM converges
```

**Algorithm 2** Distill WaveFM

In the algorithm, the student network is initialized with the pretrained model, and we calculate the exponential moving average (EMA) of the student network parameters to produce the consistency training target, which is essential for stabilizing the distillation procedure. During the distillation process, we sample \(t\) from a truncated \(\mathcal{N}\left(0,0.33^{2}\right)\) within the range \([0,0.99]\), rather than from \(\mathcal{U}(0,0.99)\). This choice is made because the error at time steps near \(t=0\) is more critical in one-step generation. The distance function \(d(\cdot,\cdot)\) in our algorithm serves as shorthand for the training loss between network outputs and targets, comprising four terms as previously mentioned. The \({\rm ODESOLVE}\) can utilize any numerical solver, and we employ the Euler method in our implementation. Notably, since we reparameterize the original objective, it is necessary to reconstruct the original function for the numerical ODE solver. Furthermore, instead of parameterizing the consistency function in a continuously differentiable manner as in Song et al. (2023), we directly set the targets to be clean audios at time points close to 1 and use the neural network to predict results at smaller time points. This represents a significant deviation from traditional consistency models, as the quality of generated audio would severely degrade if we parameterized our model conventionally. Additionally, our method is compatible with our dynamic prior distribution, requiring only sampling from it, in contrast to the original consistency models that rely on a fixed prior distribution determined by the solutions to their forward SDEs.

### Network Architecture

The function \(\mathbf{v}(\mathbf{x},t)\) is predicated on a 19.5M-parameter asymmetric U-Net model, adopting multi-receptive field modules, which are first introduced in the Hifi-GAN Kong et al. (2020a) generator, referred to as ResBlocks and ResLayers in Figure 2. In ResBlock ConvId takes "same" padding. Each ResLayer is defined with a kernel list and a dilation list, and their outer product defines both the ResBlock matrix, and the kernel width and dilation of convolutional layers in each ResBlock.

Given that mel-spectrograms provide detailed conditions, the neural network's primary task is to upsample the mel-spectrogram and refine the waveform step by step. Consequently, we allocate most of our parameters and FLOPs to the upsampling process of the U-Net rather than distributing them equally between both sides. Additionally, we employ dilated convolutional layers with larger kernel sizes in the upsampling process to reconstruct 

audio from mel-spectrograms, while the downsampling process features simpler convolutional layers. On the left column of Figure 2 are downsampling ResLayers, each containing a \(4\times 1\) ResBlock matrix, while on the right columns are upsampling ResLayers, each containing a \(3\times 3\) ResBlock matrix, following the structure from Hifi-GAN. In each ResBlock the number of channels remains unchanged from layers to layers.

Inside the multi-receptive field modules, we adopt the snake-beta activation function from BigVGAN, defined with channel-wise log-scale parameters \(\alpha\) and \(\beta\):

\[{\rm snake}(x)=x+\frac{1}{e^{\beta}+\epsilon}\sin^{2}(e^{\alpha}x), \tag{25}\]

where \(\epsilon=10^{-8}\) to ensure numerical stability. Note that this activation function, with its periodic inductive bias, is ineffective for the original flow matching model, as the noise in the random derivatives disrupts periodic patterns in the audio.

The downsampling and upsampling processes are implemented using strided and transposed convolutions, respectively. This design choice reflects our goal of generating waveforms directly from mel-spectrograms, where the additional information from downsampling features is less critical and primarily serves as a controller for the upsampling process.

For time representation, we follow Kong et al. (2020b) by embedding \(t\in[0,1]\), scaled by 100 to align its magnitude with diffusion models, into a 128-dimensional positional encoding vector

\[\begin{split}&\left[\,\sin\left(100t\cdot 10^{\frac{0\times 4}{63}} \right),\ldots,\sin\left(100t\cdot 10^{\frac{63\times 4}{63}}\right),\right.\\ &\left.\cos\left(100t\cdot 10^{\frac{0\times 4}{63}}\right),\ldots, \cos\left(100t\cdot 10^{\frac{63\times 4}{63}}\right)\,\right]\end{split} \tag{26}\]

These 128-dim time embeddings are first expanded to 512-dim after two linear-SiLU layers, then reshaped to the desired shape of each resolution, and finally added to the hidden layers during the downsampling process.

## 4 Experiments

### Datasets

To ensure fair and reproducible comparisons with other competing methods, we employ the LibriTTS dataset Zen et al. (2019), a large-scale corpus of read English speech comprising over 350,000 audio clips at 24,000 Hz, spanning approximately 1,000 hours of recordings from multiple speakers. All models are trained using the full dataset, including train-clean-100, train-clean-360, and train-other-500. For our mel-spectrograms, we generate 100-band mel-spectrograms with a frequency range of [0, 12] kHz. The FFT size, Hann window size, and hop size are set to 1024, 1024, and 256, respectively.

To evaluate the model's ability to generalize in out-of-distribution scenarios, we use the MUSDB18-HQ music dataset Rafii et al. (2017). This multi-track dataset includes original mixture audio, along with four separated tracks: vocals, drums, bass, and other instruments.

### Training and Evaluation Metrics

The detailed architectures and configurations of the models can be found in subsection 3.4. For training, the model is run on a single NVIDIA RTX

Figure 2: Network architecture. ConvId and ConvTransposeld are set with parameters (output channel, kernel width, dilation, padding).

 

4090 GPU, starting with an initial learning rate of \(7.5\times 10^{-5}\) and a batch size of 16. The learning rate decays according to a cosine annealing schedule, with the final learning rate set to \(5\times 10^{-6}\), and the training process spans 1 million steps. We adopt the AdamW optimizer, setting the betas to (0.9, 0.99) and the weight decay rate to \(5\times 10^{-4}\). The distillation stage, however, only requires 25,000 steps, with the initial learning rate reduced to \(2\times 10^{-5}\). During distillation, we adjust the weight decay rate to \(1\times 10^{-2}\) and set the betas to (0.8, 0.95). The time duration during distillation is set to 0.01. The training process requires approximately two days, while the distillation process is completed in about two hours. Given the need to evaluate performance on out-of-distribution data, we conduct a 5-point Similarity Mean Opinion Score (SMOS) test as described in BigVGAN Lee et al. (2022). This subjective evaluation is carried out by ten volunteers, and the reported SMOS scores include a 95% confidence interval. To ensure evaluation accuracy, 150 audio samples are generated per dataset for testing, with six different workers rating each sample.

Additionally, we incorporate objective automatic metrics to assess sample quality, including Multi-resolution STFT (M-STFT) loss Yamamoto et al. (2019), Perceptual Evaluation of Speech Quality (PESQ) RIX (2001), Mel-cepstral distortion ( MCD) with dynamic time warping Kubichek (1993), Periodicity error, and the F1 score for voiced/unvoiced (V/UV) classification Morrison et al. (2021). Details on the implementation of these metrics are provided in Appendix B .

Moreover, we compute the real-time factor (RTF) using the same RTX 4090 GPU, defined as the ratio of total generated audio duration to inference time. It is important to note that the inference time excludes data loading and saving times.

### Comparison With Other Models

We conduct a series of experiments on speech synthesis tasks to evaluate our model. Models we have compared with are listed below:

**Hifi-GAN V1 Kong et al. (2020a)**, **BigVGANbase**Lee et al. (2022), two well-known GAN vocoders; **DiffWave**Kong et al. (2020b), **PriorGrad**Lee et al. (2021), **FastDiff**Huang et al. (2022), **FreeGrad**Nguyen et al. (2024), four diffusion probabilistic models: all proved to be high-fidelity. We train these models for 1M steps with a batch size of 16 on LibriTTS following the setups

\begin{table}
\begin{tabular}{c|l c c c c c} \hline
**Model** & **SMOS (\(\uparrow\))** & **M-STFT (\(\downarrow\))** & **PESQ (\(\uparrow\))** & **MCD (\(\downarrow\))** & **Period (\(\downarrow\))** & **V/UV F1 (\(\uparrow\))** \\ \hline Ground Truth & 4.41\(\pm\)0.06 & 0.000 & 4.644 & 0.000 & 0.000 & 1.000 \\ Hifi-GAN V1 & 4.09\(\pm\)0.08 & 0.995 & 2.943 & 1.942 & 0.163 & 0.928 \\ Diffwave-6 Steps & 4.07\(\pm\)0.09 & 1.279 & 2.956 & 2.675 & 0.154 & 0.936 \\ PriorGrad-6 Steps & 4.12\(\pm\)0.10 & 1.832 & 3.161 & 2.519 & 0.159 & 0.937 \\ FreGrad-6 Steps & 4.08\(\pm\)0.09 & 1.893 & 3.148 & 2.573 & 0.165 & 0.932 \\ FastDiff-6 Steps & 4.06\(\pm\)0.08 & 2.181 & 2.889 & 3.264 & 0.156 & 0.937 \\ BigVGAN-base & 4.17\(\pm\)0.09 & 0.876 & 3.503 & 1.316 & 0.130 & 0.945 \\ \hline WaveFM-1 Step & 4.11\(\pm\)0.08 & 0.872 & 3.514 & 1.355 & 0.138 & 0.943 \\ WaveFM-6 Steps & **4.19\(\pm\)**0.10 & **0.841** & **3.882** & **1.150** & **0.116** & **0.956** \\ \hline \end{tabular}
\end{table}
Table 1: Subjective results with 95% confidence interval and objective evaluation results on LibriTTS dev set.

\begin{table}
\begin{tabular}{c|l l l l l l} \hline
**Model** & **SMOS (\(\uparrow\))** & **M-STFT (\(\downarrow\))** & **PESQ (\(\uparrow\))** & **MCD (\(\downarrow\))** & **Period (\(\downarrow\))** & **V/UV F1 (\(\uparrow\))** \\ \hline Ground Truth & 4.38\(\pm\)0.08 & 0.000 & 4.644 & 0.000 & 0.000 & 1.000 \\ Hifi-GAN V1 & 3.81\(\pm\)0.11 & 1.288 & 2.635 & 2.469 & 0.182 & 0.924 \\ Diffwave-6 Steps & 3.85\(\pm\)0.10 & 1.354 & 2.731 & 3.875 & 0.171 & 0.929 \\ PriorGrad-6 Steps & 3.92\(\pm\)0.09 & 1.925 & 3.156 & 3.439 & 0.168 & 0.931 \\ FreGrad-6 Steps & 3.87\(\pm\)0.10 & 1.960 & 2.953 & 3.325 & 0.180 & 0.924 \\ FastDiff-6 Steps & 3.79\(\pm\)0.08 & 2.257 & 2.659 & 4.331 & 0.179 & 0.923 \\ BigVGAN-base & 3.95\(\pm\)0.09 & 1.262 & 3.170 & 1.597 & 0.151 & 0.942 \\ \hline WaveFM-1 Step & 3.96\(\pm\)0.09 & 1.034 & 3.287 & 1.589 & 0.148 & 0.947 \\ WaveFM-6 Steps & **4.05\(\pm\)**0.08 & **0.968** & **3.639** & **1.342** & **0.135** & **0.952** \\ \hline \end{tabular}
\end{table}
Table 2: Subjective results with 95% confidence interval and objective evaluation results on MUSDB18-HQ.

 

as in the original papers.

The results in Table 1 show that our models demonstrate superior performance over various previous models in terms of sample quality. In the table, WaveFM-1 Step and WaveFM-6 Steps audios are generated by our distilled model and undistilled with 6 steps Euler solver respectively. According to the data, our 6 steps model achieves the best performance on M-STFT, PESQ, MCD, Periodicity error and V/UV F1 score and our consistency distillation algorithm doesn't compromise too much sample quality in order to achieve one step generation.

Besides, our model has advantages in terms of synthesis speed. The RTF results in Table 3 have shown that our distilled model enjoys a inference speed close to Hifi-GAN V1, which is much higher than the previous diffusion models since they need several steps to generate waveforms.

### Out-of-Distribution Situation

We demonstrate the generalizability of WaveFM using the musical dataset MUSDB18-HQ. The SMOS test is conducted by uniformly sampling from the five tracks: drums, bass, vocal, others, and mixture. Since automatic evaluators are primarily designed for speech analysis, we use vocal track samples for automatic evaluation, where audio segments with high silence ratios are removed. The results in Table 2 indicate our model exhibits commendable performance in unseen scenarios, exceeding the performance of the baseline models. To be specific, WaveFM-6 Steps model performs significantly better than previous diffusion models, and even distilled WaveFM-1 Step model can generate acceptable waveforms compared to other methods, which implies that our models generalize well on out of distribution data. We can further illustrate this point by visualizing the spectrograms of the generated audio. Figure 3 shows that our model's spectrogram is closer to ground truth spectrogram, and comparing to our model, PriorGrad tends to overestimate low frequency components and underestimate high frequency components, while BigVGAN-base fails to generate the regular components in the clean spectrogram neatly. Figuratively speaking, PriorGrad's spectrogram is too light at the bottom and a little dark at the top, and BigVGAN's spectrogram fails to keep the regular lines that can be found in clean spectrogram.

### Ablation Study

In order to show that our structural designs are effective, we have conducted seve

\begin{table}
\begin{tabular}{c|l} \hline
**Model** & **SMOS (\(\uparrow\))** \\ \hline Ground Truth & \(4.41\pm 0.06\) \\ WaveFM-1 Step & \(\mathbf{4.11\pm 0.08}\) \\ w/o Snake Activation & \(4.03\pm 0.07\) \\ w/o Conditioned Prior & \(3.79\pm 0.08\) \\ with Original STFTLoss & \(4.05\pm 0.07\) \\ w/o Auxiliary Losses & \(3.97\pm 0.09\) \\ w/o Reparameterization & \(3.88\pm 0.07\) \\ \hline \end{tabular}
\end{table}
Table 4: Ablation study results on LibriTTS Test set.

Figure 3: Spectrograms of a music clip (Ground Truth, WaveFM-6 Steps, PriorGrad-6 Steps, BigVGAN-base)



on LibriTTS Test set. Here are the observations:

1. After removing the random derivative term, the snake activation function with periodic inductive bias, as used in BigVGAN, improves the sample quality of WaveFM;
2. The mel-conditioned prior distribution significantly improves the one-step sample quality, indicating that reducing unnecessary distribution transportation costs is effective;
3. For our model, replacing the spectral convergence loss \(L_{\text{sc}}\) by the phase angle loss \(L_{\text{pha}}\) and incorporating gradient and Laplacian operators can improve sample quality, which is consistent with the visual results in Figure 1.
4. The auxiliary losses are important to improve the sample quality, which aligns with the experiment results of GANs. Moreover, the results suggest that additional losses cannot be efficiently applied to the original flow matching objective unless it is reparameterized to directly predict waveforms; thus, the reparameterization is indeed crucial.

## 5 Conclusion

We propose WaveFM, a high-fidelity vocoder for speech synthesis conditioned on mel-spectrograms. First, it utilizes the energy information from mel-spectrograms to generate a prior distribution with low variance. Additionally, reparameterizing the original flow matching objective not only introduces periodic inductive bias into the neural network, but also enables the inclusion of auxiliary losses. Specifically, we design a multi-resolution STFT loss function to enhance sample quality for our model. Finally, our consistency distillation algorithm allows WaveFM to produce audio in one step without significantly sacrificing sample quality. Together, these techniques greatly improve both the quality and efficiency of WaveFM, with SMOS tests and automatic evaluators confirming that it performs competitively against previous diffusion models.

## 6 Limitations and Potential Risks

The main limitation of our work lies in the noticeable gap in sample quality between multi-step and single-step models. While single-step models offer faster synthesis, their quality still lags behind, which restricts their usability for generating high-quality waveforms at scale. This trade-off remains a challenge, as improving the performance of single-step models without resorting to adversarial training requires further exploration and insights.

While our proposed model improves the accessibility of high-fidelity speech synthesis, it also introduces potential risks. By lowering the technical barriers, our approach could inadvertently facilitate misuse, such as more convincing voice spoofing or impersonation in media, customer service, or telephone scams. This raises concerns about the ethical implications of deploying such technology without proper safeguards.

## References

* Huang et al. (2022) Rongjie Huang, Max WY Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. 2022. Fastdiff: A fast conditional diffusion model for high-quality speech synthesis. _arXiv preprint arXiv:2204.09934_.
* Huang et al. (2023) Rongjie Huang, Yi Ren, Ziyue Jiang, Chenye Cui, Jinglin Liu, and Zhou Zhao. 2023. Fastdiff 2: Revisiting and incorporating gans and diffusion models in high-fidelity speech synthesis. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 6994-7009.
* Jang et al. (2021) Won Jang, Dan Lim, Jaesam Yoon, Bongwan Kim, and Juntae Kim. 2021. Univnet: A neural vocoder with multi-resolution spectrogram discrimininators for high-fidelity waveform generation. _arXiv preprint arXiv:2106.07889_.
* Ju et al. (2024) Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. 2024. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. _arXiv preprint arXiv:2403.03100_.
* Kalchbrenner et al. (2018) Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. 2018. Efficient neural audio synthesis. In _International Conference on Machine Learning_, pages 2410-2419. PMLR.
* Kim et al. (2021) Ji-Hoon Kim, Sang-Hoon Lee, Ji-Hyun Lee, and Seong-Whan Lee. 2021. Fre-gan: Adversarial frequency-consistent audio synthesis. _arXiv preprint arXiv:2106.02297_.
* Koizumi et al. (2022) Yuma Koizumi, Heiga Zen, Kohei Yatabe, Nanxin Chen, and Michiel Bacchiani. 2022. Specgrad: Diffusion probabilistic model based neural vocoder with adaptive noise spectral shaping. _arXiv preprint arXiv:2203.16749_.
* Kong et al. (2020a) Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020a. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. _Advances in 

Neural Information Processing Systems_, 33:17022-17033.
* Kong et al. (2020b) Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. 2020b. Diffwave: A versatile diffusion model for audio synthesis. _arXiv preprint arXiv:2009.09761_.
* Kubichek (1993) Robert Kubichek. 1993. Mel-cepstral distance measure for objective speech quality assessment. In _Proceedings of IEEE pacific rim conference on communications computers and signal processing_, volume 1, pages 125-128. IEEE.
* Kumar et al. (2019) Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre De Brebisson, Yoshua Bengio, and Aaron C Courville. 2019. Melgan: Generative adversarial networks for conditional waveform synthesis. _Advances in neural information processing systems_, 32.
* Kumar et al. (2024) Rittesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. 2024. High-fidelity audio compression with improved rvogan. _Advances in Neural Information Processing Systems_, 36.
* Lam et al. (2022) Max WY Lam, Jun Wang, Dan Su, and Dong Yu. 2022. Bddm: Bilateral denoising diffusion models for fast and high-quality speech synthesis. _arXiv preprint arXiv:2203.13508_.
* Lee et al. (2021) Sang-gil Lee, Heeseung Kim, Chaehn Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie-Yan Liu. 2021. Priorgrad: Improving conditional denoising diffusion models with data-dependent adaptive prior. _arXiv preprint arXiv:2106.06406_.
* Lee et al. (2022) Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. 2022. Bigvgan: A universal neural vocoder with large-scale training. _arXiv preprint arXiv:2206.04658_.
* Lipman et al. (2022) Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. 2022. Flow matching for generative modeling. _arXiv preprint arXiv:2210.02747_.
* Liu et al. (2022) Xingchao Liu, Chengyue Gong, and Qiang Liu. 2022. Flow straight and fast: Learning to generate and transfer data with rectified flow. _arXiv preprint arXiv:2209.03003_.
* Morrison et al. (2021) Max Morrison, Rithesh Kumar, Kundan Kumar, Prem Seetharaman, Aaron Courville, and Yoshua Bengio. 2021. Chunked autoregressive gan for conditional waveform synthesis. _arXiv preprint arXiv:2110.10139_.
* Nguyen et al. (2024) Tan Dat Nguyen, Ji-Hoon Kim, Youngjoon Jang, Jaehu Kim, and Joon Son Chung. 2024. Fregrad: Lightweight and fast frequency-aware diffusion vocoder. _arXiv preprint arXiv:2401.10032_.
* van den Oord et al. (2016) Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. _arXiv preprint arXiv:1609.03499_.
* Ping et al. (2020) Wei Ping, Kainan Peng, Kexin Zhao, and Zhao Song. 2020. Waveflow: A compact flow-based model for raw audio. In _International Conference on Machine Learning_, pages 7706-7716. PMLR.
* Prenger et al. (2019) Ryan Prenger, Rafael Valle, and Bryan Catanzaro. 2019. Waveglow: A flow-based generative network for speech synthesis. In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 3617-3621. IEEE.
* Rafii et al. (2017) Zafar Rafii, Antoine Liutks, Fabian-Robert Stoter, Stylianos Ioannis Mimilakis, and Rachel Bittner. 2017. Musdb18-a corpus for music separation.
* RIX (2001) A RIX. 2001. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In _Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2001_, pages 73-76.
* Siuzdak (2023) Hubert Siuzdak. 2023. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. _arXiv preprint arXiv:2306.00814_.
* Song and Dhariwal (2023) Yang Song and Prafulla Dhariwal. 2023. Improved techniques for training consistency models. _arXiv preprint arXiv:2310.14189_.
* Song et al. (2023) Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. 2023. Consistency models. _arXiv preprint arXiv:2303.01469_.
* Steinmetz and Reiss (2020) Christian J Steinmetz and Joshua D Reiss. 2020. auraloss: Audio focused loss functions in pytorch. In _Digital music research network one-day workshop (DMRN+ 15)_.
* Wang et al. (2023) Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. 2023. Neural codec language models are zero-shot text to speech synthesizers. _arXiv preprint arXiv:2301.02111_.
* Yamamoto et al. (2019) Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. 2019. Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. _arXiv preprint arXiv:1910.11480_.
* Zen et al. (2019) Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. 2019. Librits: A corpus derived from librispech for textbook-speech. _arXiv preprint arXiv:1904.02882_ 

Proof of Theorem 1

**Theorem 1** Let \({\mathbf{x}_{t}}\) be a continuously differentiable random process on \(t\in[0,1]\) and \(p({\mathbf{x}},t)\) be its probability density function. We denote the prior distribution as \({\mathbf{x}_{0}}\) and the ground truth distribution as \({\mathbf{x}_{1}}\), if the conditional expectation is locally Lipschitz, let

\[{\mathbf{v}}({\mathbf{x}_{t}},t)=\mathbb{E}\left[\frac{\mathrm{d}{\mathbf{x}_{ t}}}{\mathrm{d}t}\bigg{|}{\mathbf{x}_{t}}={\mathbf{x}}\right], \tag{27}\]

then we can draw samples from data distribution \({\mathbf{x}_{1}}\) by drawing samples from prior distribution \({\mathbf{x}_{0}}\) and then solving the following ODE with an initial value \({\mathbf{x}_{0}}\) at time \(t=0\).

\[\frac{\mathrm{d}{\mathbf{x}}}{\mathrm{d}t}={\mathbf{v}}({\mathbf{x}},t) \tag{28}\]

To generate data by drawing samples from prior distribution \({\mathbf{x}_{0}}\) and then solving the following ODE with an initial value \({\mathbf{x}_{0}}\) at \(t=0\):

\[\frac{\mathrm{d}{\mathbf{x}}}{\mathrm{d}t}={\mathbf{v}}({\mathbf{x}},t), \tag{29}\]

we only need to check that the probability density function \(p({\mathbf{x}},t)\) satisfies the probability flow equation below with given \({\mathbf{v}}({\mathbf{x}},t)\) since when solving an ODE with locally Lipschitz condition from time \(t=0\) to \(t=1\) with initial value \({\mathbf{x}_{0}}\), the evolution of probability density is described by the probability flow equation.

\[\frac{\partial p}{\partial t}({\mathbf{x}},t)+\nabla_{{\mathbf{x}}}(p({ \mathbf{x}},t){\mathbf{v}}({\mathbf{x}},t))=0 \tag{30}\]

We can prove this by multiplying any finite supported continuously differentiable function \(h({\mathbf{x}})\) and then apply integral:

\[\int h({\mathbf{x}})\!\left(\!\frac{\partial p}{\partial t}({\mathbf{x}},t)+\nabla_{ {\mathbf{x}}}(p({\mathbf{x}},t){\mathbf{v}}({\mathbf{x}},t))\!\right)\!\mathrm{d}{\mathbf{x}}=0 \tag{31}\] \[\qquad\frac{\partial}{\partial t}\int h({\mathbf{x}})p({\mathbf{x}},t) \,\mathrm{d}{\mathbf{x}}\] (32) \[= -\int h({\mathbf{x}})\nabla_{{\mathbf{x}}}(p({\mathbf{x}},t){\mathbf{v}}({\mathbf{x} },t))\,\mathrm{d}{\mathbf{x}}\]

Integrating by parts to the right hand side, since \(h\) is finite supported, we have

\[\begin{split}&\frac{\mathrm{d}}{\mathrm{d}t}\int h({\mathbf{x}})p({ \mathbf{x}},t)\,\mathrm{d}{\mathbf{x}}\\ =&\int(p({\mathbf{x}},t){\mathbf{v}}({\mathbf{x}},t))^{\top} \nabla_{{\mathbf{x}}}h({\mathbf{x}})\,\mathrm{d}{\mathbf{x}}\end{split} \tag{33}\] \[\frac{\mathrm{d}}{\mathrm{d}t}\mathbb{E}[h({\mathbf{x}_{t}})]=\mathbb{ E}\left[v({\mathbf{x}_{t}},t)^{\top}\nabla_{{\mathbf{x}}}h({\mathbf{x}_{t}})\right] \tag{34}\]

By the tower property of expectation, we have

\[\begin{split}&\mathbb{E}\left[\mathbb{E}\left[\frac{\mathrm{ d}{\mathbf{x}_{t}}}{\mathrm{d}t}\bigg{|}{\mathbf{x}_{t}}\right]^{\top}\nabla_{{\mathbf{ x}}}h({\mathbf{x}_{t}})\right]\\ =&\mathbb{E}\left[v({\mathbf{x}_{t}},t)^{\top}\nabla_{ {\mathbf{x}}}h({\mathbf{x}_{t}})\right]\end{split} \tag{36}\]

since we choose

\[v({\mathbf{x}},t)=\mathbb{E}\left[\frac{\mathrm{d}{\mathbf{x}_{t}}}{\mathrm{d}t} \bigg{|}{\mathbf{x}_{t}}={\mathbf{x}}\right]. \tag{37}\]

Thus, the equations hold for any finite supported continuous differentiable function \(h\). We have

\[\frac{\partial p}{\partial t}({\mathbf{x}},t)+\nabla_{{\mathbf{x}}}(p({ \mathbf{x}},t){\mathbf{v}}({\mathbf{x}},t))=0, \tag{38}\]

since we can arbitrarily choose \(h\). Then **Theorem 1** is proved and we can draw samples by solving the ODE numerically.

## Appendix B Implementations of Metrics

### M-STFT:

We use the implementation in Auraloss(Steinmetz and Reiss, 2020) with codes from [https://github.com/csteinmetz1/auraloss](https://github.com/csteinmetz1/auraloss).

**PESQ:** We resample the audios from 24,000 Hz to 16,000 Hz and pick the wideband version of PESQ from [https://github.com/ludlows/PESQ](https://github.com/ludlows/PESQ).

**MCD:** We use the implementation at [https://github.com/jasminsternkopf/mel_cepstral_distance](https://github.com/jasminsternkopf/mel_cepstral_distance) with DTW enabled.

**Periodicity and V/UV F1:** Both are provided in CARGAN (Morrison et al., 2021) at [https://github.com/descriptinc/cargan](https://github.com/descriptinc/cargan).

 

Pytorch Implementation of Our Multi-resolution STFT Loss

import torch.nn.functional as F

def filterTime(input): input = F.pad(input, pad={1,0,1,},mode="constant") weight = torch.tensor([
[~1.0,1.0], [~2.0,2.0], [~1.0,1.0]].to( input.device).reshape(1,1,3,2)/4 deltaT = torch.convzd{ imput.unsqueeze(1), weight=weight, returndeltaT.squeeze(1) defilterFree(input): input = F.pad(input, pad={1,1,1,0},mode="constant") weight = torch.tensor([
[~1.0,2.0,1.0]], intout device).reshape(1,1,2,3)/4 deltar = torch.convzd( input.unsqueeze(1),weight=weight) returndeltaF.squeeze(1) deffilterLaplacian(input): input = F.pad(input, pad={1,1,1},mode="constant") weight = torch.tensor([~1.0,-1.0,-1.0], [~1.0,8.0,-1.0], [~1.0,-1.0,-1.0]]).to( input.device).reshape(1,1,3,3)/8 laplicaian = torch.convzd( input.unsqueeze(1),weight=weight) returnlaplacian.squeeze(1) defgetSTFTLoss(answer,predict,fft_sizes=(1024,2048,512), hop_sizes=(128,256,64), win_lengths=(512,1024,256), window=torch.hann_window, ): loss = @ for i inrange(len(fft_sizes)): ansStft = torch.view_as_real( torch.sft( answer.squeeze(1), n_fft=fft_sizes[i], hop_length=hop_sizes[i], win_length=win_lengths[i], device=answer.device), return_complex=True) predStrt = torch.view_as_real( torch.strf{ predict.squeeze(1), n_fft=fft_sizes[i], hop_length=hop_sizes[i], win_length=win_lengths[i], device=answer.device), return_complex=True) predStrt = torch.view_as_real( torch.stff{ predict.squeeze(1), n_fft=fft_sizes[i], hop_length=hop_sizes[i], win_length=win_lengths[i], window=window( win_lengths[i], device=predict.device), return_complex=True) predStrtMag = (ansStiftLag = torch.sqrt( ansSttfMag + magMin) predStrtMag = torch.sqrt( predStrtMag + magMin) ansStftPha = torch.atan2( ansStft[...,1][mask], ansStft[...,0][mask]) predStrtPha = torch.atan2( predStrt[...,1][mask], predStrt[...,0][mask]) deltaPase = (ansStftPla - predStrtpha) loss += torch.atan2( torch.sin(deltaPhase), torch.cos(deltaPhase), ).abs().mean() loss += (ansStftMag.log() - predStrtMag.log() ).abs().mean() ansStftMagD7 = filterTime( ansStftMag) ansStftMagDF = FilterFreeq( ansStftMagDF = filterFreeq( ansStftMag) assStftMagD2 = filterLaplacian( ansStftMag) predStrtMagDT = filterTime( predStrtMag) predStrtM