RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction

## Abstract

Abstract Recent advancements in generative modeling have significantly enhanced the reconstruction of audio waveforms from various representations. While diffusion models are adept at this task, they are hindered by latency issues due to their operation at the individual sample point level and the need for numerous sampling steps. In this study, we introduce RFWave, a cutting-edge multi-band Rectified Flow approach designed to reconstruct high-fidelity audio waveforms from Mel-spectrograms or discrete acoustic tokens. RFWave uniquely generates complex spectrograms and operates at the frame level, processing all subbands simultaneously to boost efficiency. Leveraging Rectified Flow, which targets a straight transport trajectory, RFWave achieves reconstruction with just 10 sampling steps. Our empirical evaluations show that RFWave not only provides outstanding reconstruction quality but also offers vastly superior computational efficiency, enabling audio generation at speeds up to 160 times faster than real-time on a GPU. An online demonstration is available at: https://rfwave-demo.github.io/rfwave/ .

## 1 Introduction

Audio waveform reconstruction significantly enhances the digital interactions by enabling realistic voice and sound generation for diverse applications. This technology transforms low-dimensional features, derived from raw audio data, into perceptible sounds, improving the audio experience on various platforms such as virtual assistants and entertainment systems.
Autoregressive models and Generative Adversarial Networks (GANs) (Goodfellow et al., ) have been applied to this task, greatly advancing audio quality beyond traditional signal processing methods (Kawahara et al., ; Morise et al., ). Autoregressive methods, while effective, are hindered by slow generation speeds due to their sequential prediction of sample points (Oord et al., ; Kalchbrenner et al., ; Valin & Skoglund, ). In contrast, GANs predict sample points in parallel, resulting in faster generation speeds and maintaining high-quality output (Kumar et al., ; Yamamoto et al., ; Kong et al., ; Siuzdak, ; Du et al., ). Consequently, GAN-based methods deliver impressive performance and are extensively utilized in real-world audio generation applications.

Despite the advancements, GAN-based waveform reconstruction models face challenges such as the necessity for complex discriminator designs and issues like instability or mode collapse (Thanh-Tung et al., ). In response, diffusion models for waveform reconstruction have been explored, offering stability during training and the ability to reconstruct high-quality waveforms (Chen et al., ; Kong et al., ; gil Lee et al., ; Nguyen et al., ; Huang et al., ; Koizumi et al., ). However, these models are at least an order of magnitude slower compared to GANs. The slow generation speed in these diffusion-based waveform reconstruction models is primarily due to two factors: (1) the requirement of numerous sampling steps to achieve high-quality samples, and (2) the operation at the waveform sample point level. The latter often involves multiple upsampling operations to transition from frame rate resolution to sample rate resolution, increasing the sequence length and consequently leading to higher GPU memory usage and computational demands.

In this paper, we propose RFWave, a diffusion-type waveform reconstruction method designed to match the speed of GAN-based methods while maintaining the training stability and high sample quality of diffusion models. To overcome the challenge of slow sampling, we employ Rectified Flow (Liu et al., ; Lipman et al., ; Albergo & Vanden-Eijnden, ), which connects data and noise along a straight line, thereby enhancing sampling efficiency. To address the GPU memory and computational demands of sample point-level modeling, our model operates at the level of Short-Time Fourier Transform (STFT) frames, enabling more efficient processing and reducing GPU memory usage.
RFWave with only 10 sampling steps can generate high-quality audio, achieving an inference speed of up to 160 times real-time on an NVIDIA GeForce RTX 4090 GPU. Additionally, the incorporation of three enhanced loss functions and an optimized sampling strategy further elevates the overall quality of the reconstructed waveforms. To our knowledge, RFWave stands as the fastest diffusion-based audio waveform reconstruction model, and it delivers superior audio quality. Furthermore, it can reconstruct waveforms from Mel-spectrograms and discrete acoustic tokens, enhancing its versatility and applicability in various audio generation tasks. Our main contributions are as follows:

- 1.
By integrating the Rectified Flow and 3 enhanced loss functions – energy-balanced loss, overlap loss, and STFT loss – our model can reconstruct high-quality waveforms with a drastically reduced number of sampling steps.
- 2.
We utilize a multi-band strategy, coupled with the high-efficiency ConvNeXtV2 (Woo et al., ) backbone, to generate different subbands concurrently. This not only assures audio quality by circumventing cumulative errors, but also boosts the synthesis speed.
- 3.
Our model operates at the level of STFT frames, not individual waveform sample points. This approach significantly accelerates processing and reduces GPU memory usage.
- 4.
We propose a technique for selecting sampling time points based on the straightness of the Rectified Flow transport trajectories, which enhances sample quality for free.

## 2 Background

In this section, we describe the basic formulation of the Rectified Flow and present releated works, while also detailing our correlation and distinction.

#### Rectified Flow

Rectified Flow (Liu et al., ) presents an innovative Ordinary Differential Equation (ODE)-based
framework for generative modeling and domain transfer.
It introduces a method to learn a mapping that connects two distributions,
$\pi_{0}$ and $\pi_{1}$ on $\mathbb{R}^{d}$, based on empirical observations:

$$ $\frac{\mathrm{d}Z_{t}}{\mathrm{d}t}=v(Z_{t},t),\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \text{initialized from $Z_{0}\sim\pi _{0}$, such that $Z_{1}\sim\pi_{1}$},$ (1) $$

where $v\colon\mathbb{R}^{d}\times[0,1]\to\mathbb{R}^{d}$ represents a velocity field. The learning of this field involves minimizing a mean square objective function,

$$ $\min_{v}\mathbb{E}_{(X_{0},X_{1})\sim\gamma}\left[\int_{0}^{1}\mid\mid\frac{ \mathrm{d}}{\mathrm{d}t}X_{t}-v(X_{t},t)\mid\mid^{2}\mathrm{d}t\right], \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode \nobreak\ \text{with}\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode \nobreak\ \leavevmode\nobreak\ X_{t}=\phi(X_{0},X_{1},t),$ (2) $$

where $X_{t}=\phi(X_{0},X_{1},t)$ represents a time-differentiable interpolation between $X_{0}$ and $X_{1}$,
with $\frac{\mathrm{d}}{\mathrm{d}t}X_{t}=\partial_{t}\phi(X_{0},X_{1},t)$.
The $\gamma$ represents any coupling of $(\pi_{0},\pi_{1})$.
An illustrative instance of $\gamma$ is the independent coupling $\gamma=\pi_{0}\times\pi_{1}$,
which allows for empirical sampling based on separately observed data from $\pi_{0}$ and $\pi_{1}$.
Liu et al. () recommended a simple choice of

$$ $X_{t}=(1-t)X_{0}+tX_{1}\implies\frac{\mathrm{d}}{\mathrm{d}t}X_{t}=X_{1}-X_{0}.$ (3) $$

This simplification results in linear trajectories, which are critical for accelerating the inference process.
Typically, the velocity field $v$ is represented using a deep neural network. The solution to (2) is approximated through stochastic gradient methods.
To approximate the ODE presented in (1), numerical solvers are commonly employed.
A prevalent technique is the forward Euler method. This approach computes values using the formula

$$ $Z_{t+\frac{1}{n}}=Z_{t}+\frac{1}{n}v(Z_{t},t),\leavevmode\nobreak\ \leavevmode \nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode \nobreak\ \leavevmode\nobreak\ \forall t\in\{0,\ldots,n-1\}/n,$ (4) $$

where the simulation is executed with a step interval of $\epsilon=1/n$ over $n$ steps.

The velocity field has the capacity to incorporate conditional information, which is particularly essential in applications such as text-to-image generation and waveform reconstruction from compressed acoustic representation.
Consequently, in such contexts, $v(Z_{t},t)$ in (2) is modified to $v(Z_{t},t\mid\mathcal{C})$, where $\mathcal{C}$ represents the conditional information pertinent to the corresponding $X_{1}$.

#### Diffusion Models for Audio Waveform Reconstruction

Diffusion models (Song et al., ; Ho et al., ) have become the de-facto choice for high-quality generation in the realm of generative models. Diffwave (Kong et al., ) and WaveGrad (Chen et al., ) were pioneering efforts to reconstruct waveforms using diffusion models, achieving performance comparable to autoregressive models and GANs. PriorGrad (gil Lee et al., ) enhances both speech quality and inference speed by employing a data-dependent prior distribution. Meanwhile, FreGrad (Nguyen et al., ) simplifies the model and reduces denoising time through the use of Discrete Wavelet Transform (DWT). Multi-Band Diffusion (MBD) (Roman et al., ) leverages a diffusion model to reconstruct waveforms from discrete EnCodec (Défossez et al., ) tokens. Despite these advancements, the current fastest generation speed of these methods is only about 10 to 20 times faster than real-time, which limits their application for real-time use, especially when combined with large-scale Transformer-based acoustic models (Wang et al., ; Du et al., ). In contrast, our method achieves speeds up to 160 times faster than real-time, significantly enhancing real-time applicability.

#### Estimating Complex Spectrograms

Waveform reconstruction from complex spectrograms can be effectively achieved using the ISTFT. Notably, Vocos (Siuzdak, ) and APNet2 (Du et al., ), utilizing GANs as their model framework, estimate magnitude and phase spectrograms from the input Mel-spectrograms,
which can be transformed to complex spectrograms effortlessly.
Both models operate at the frame level, enabling them to achieve significantly faster inference speeds compared to HiFi-GAN (Kong et al., ), which uses multiple upsampling layers and operates at the level of waveform sample points. Morever, these models preserve the quality of the synthesized waveform, demonstrating their superiority in both speed and fidelity without a trade-off.
In this paper, we directly estimate complex spectrograms using Rectified Flow and focus on frame-level operations, aiming to enhance both the efficiency and quality of our waveform synthesis process. Notably, the distributions of real and imaginary parts of the complex spectrograms appear more homogeneous in comparison to the distributions magnitude and phase.

#### Multi-band Audio Waveform Reconstruction

Both Multi-band MelGAN (Yang et al., ) and Multi-band Diffusion (Roman et al., ) employ multi-band strategies, albeit for different purposes within their respective frameworks. Multi-band MelGAN, specifically, uses Pseudo-Quadrature Mirror Filters (PQMF) (Johnston, ) to divide frequency bands. This division results in each subband’s waveform being a fraction of the original waveform’s length, based on the number of subbands.
By reshaping these subbands into feature dimensions and utilizing a unified backbone for modeling, Multi-band MelGAN is able to operate on considerably shorter signals. This strategy significantly enhances the efficiency of the model, leading to accelerated training and inference processes.
Multi-band Diffusion utilizes an array of band-pass filters to separate the frequency bands and models each subband with a distinct model. This approach ensures that errors in one band do not negatively impact the others.
In our research, we simplify the process of frequency band division by directly choosing the appropriate dimensions from the complex spectrograms. Furthermore, we enhance efficiency by modeling all subbands together in parallel with a single, unified model. This strategy improves the processing speed and also helps in reducing error accumulation across different subbands.

## 3 Method

Our model utilizes a multi-band Rectified Flow to directly predict the complex spectrogram.
It operates at the STFT frame level and incorporates a highly efficient ConvNeXtV2 (Woo et al., )
backbone. With only 10 steps of sampling, the model is capable of producing high-quality waveforms.
In this section, we present the structure of the multi-band Rectified Flow model, which can operate in either the time or frequency domain. Additionally, we describe the corresponding normalization techniques, three enhanced loss functions, and the strategy for selecting sampling time points.

Figure: Figure 1: The overall structure for RFWave. $i_{sb}$ is the subband index, C is the conditional input, which can be Encodec token or Mel-spectrogram, and $i_{bw}$ is the EnCodec bandwidth index. Modules enclosed in a dashed box, as well as dashed arrows, are considered optional.
Refer to caption: x1.png

### 3.1 Multi-band Rectified Flow

#### Model Structure

The model structure is depicted in Figure 1.
All frequency bands, each distinguished by a unique subband index, share the same model. The subbands of a given sample are grouped together into a single batch for processing, which facilitates simultaneous training or inference. This significantly reduces inference latency.
Moreover, independently modeling the subbands reduces error accumulation. As discussed in (Roman et al., ), conditioning higher bands on lower ones can lead to an error accumulation, which means inaccuracies in the lower bands can adversely affect the higher bands during inference.

For each subband, the corresponding noisy sample ($X_{t}$) is fed into the ConvNeXtV2 backbone
to predict the velocity ($v_{t}$) conditioned on time $t$, the subband index ($i_{sb}$),
conditional input ($C$, the Mel-spectrogram or the EnCodec (Défossez et al., ) tokens),
and an optional EnCodec bandwidth index ($i_{bw}$).
The detailed structure of the ConvNeXtV2 backbone is shown in Figure 1.
We employ Fourier features as described in (Kingma et al., ).
The noisy sample, Fourier features, and conditional inputs are concatenated and
then passed through a linear layer, forming the input that is fed into a series of ConvNeXtV2 blocks.
The sinusoidal time embedding, along with the optional EnCodec bandwidth index embedding,
are element-wise added to the input of each ConvNeXtV2 block.
Furthermore, the subband index is incorporated via an adaptive layer normalization module,
which utilizes learnable embeddings as described in (Siuzdak, ; Xu et al., ).
The other components are identical to those within the ConvNeXtV2 architecture,
details can be found in (Woo et al., ).

Our model, utilizing frame-level features, demonstrates greater memory efficiency compared to diffusion vocoders like PriorGrad that operate on waveform sample points. While PriorGrad (gil Lee et al., ) can train on 6-second(^1^11The duration of an audio clip is calculated as the batch size multiplied by the number of segment frames and hop length, divided by the sampling rate.) audio clips at 44.1 kHz within 30 GB of GPU memory, our model is capable of handling 177-second clips with the same memory resources.

#### Operating in Time Domain and Waveform Equalization

Our model is designed to function at the STFT frame level and possesses the capability to operate in the time domain. Assuming that the noisy sample and velocity reside in the time domain, the use of STFT and ISTFT, as illustrated in Figure 1, becomes necessary. The dimensions of noise and velocity adhere to $[1,T]$, where $T$ represents the waveform length in sample points(^2^22For simplicity, the batch dimension is not included in the discussion.).
After the STFT operation, we extract the subbands by equally dividing the full-band complex spectrogram. Each subband $i$ is then processed by the backbone as an independent sample. Subsequently, these processed subbands are merged back together before the ISTFT operation.
The feature processed by the backbone thus has dimensions of $[d_{s},F]$,
where $d_{s}$ denotes the dimension of a subband’s complex spectrum and $F$ the number of frames.
The real and imaginary parts of each subband are interleaved to form a $d_{s}$-dimensional feature.

White Gaussian noise has uniform energy across frequency bands, but waveform energy profiles vary significantly. For example, speech energy decays exponentially with frequency, while music maintains a consistent distribution. These differences challenge diffusion model training, making energy equalization across bands beneficial (Roman et al., ).
In the time-domain model, a bank of Pseudo-Quadrature Mirror Filters (PQMF) is employed to decompose the
input waveform into subbands. Subsequently, these subbands are equalized and then recombined to
form the equalized waveform.
The performance of the PQMF bank exhibits a modest enhancement compared to
the array of band-pass filters employed in (Roman et al., ).
It’s important to note that the PQMF is solely utilized for waveform equalization and holds no association with the division of the complex spectrogram into subbands.
For waveform equalization, mean-variance normalization is employed, utilizing the exponential moving average of mean and variance of each waveform subband computed during training.
This approach ensures that the transformation can be effectively inverted
using the same statistics.

#### Alternative Approach: Operating in Frequency Domain and STFT Normalization

Assuming that noisy sample and velocity reside in the frequency domain, the use of STFT and ISTFT, as depicted in Figure 1, becomes unnecessary. The subbands of noisy sample and velocity are then directly extracted from their full-band feature by selecting the corresponding dimensions, resulting in a shape of $[d_{s},F]$.
In the frequency-domain model, the waveform is transformed to a complex spectrogram without equalization.
Subsequent processing involves the dimension-wise mean-variance normalization of the complex spectrogram.

During the inference stage, the model operating in the time domain requires two additional operations—STFT and ISTFT—at each sampling step. In contrast, the model operating in the frequency domain requires only a single ISTFT operation after sampling.
However, despite the additional operations, the time-domain model demonstrates slightly superior performance, particularly in capturing high-frequency details according to our experiment results.

### 3.2 Loss functions

#### Energy-balanced Loss

In preliminary experiments, we noticed low-volume noise in expected silent regions.
We attribute this to the property of Mean Square Error (MSE) used in (2).
The MSE measures the absolute distortion between the predicted values and the ground truth.
Since the values in silent regions are close to zero,
even a minor absolute distortion in predictions can lead to a significant relative error.
Consequently, models trained with the MSE produce small absolute distortions in silent regions, which are then perceived as noise.

We propose energy-balanced loss to mitigate this problem.
Our energy-balanced loss is designed to weight errors differently
depending on the region’s volume (or energy) accross the time-axis.
Specifically, for each frequency subband,
we compute the standard deviation along the feature dimension of the ground truth velocity to construct a weighting coefficient of size $[1,F]$.
This vector is reflective of the frame-level energy of the respective subband, as depicted in Figure A.1.
Subsequently, both the ground truth and predicted velocity are divided
by this vector before proceeding to the subsequent steps.
For the frequency domain model, the training objective defined in (2) is adjusted as follows:

$$ $\displaystyle\min_{v}$ $\displaystyle\mathbb{E}_{X_{0}\sim\pi_{0},(X_{1},\mathcal{C})\sim D}\left[\int _{0}^{1}\mid\mid(X_{1}-X_{0})/\sigma-v(X_{t},t\mid\mathcal{C})/\sigma\mid\mid^ {2}\mathrm{d}t\right],$ (5) $\displaystyle\text{with}\leavevmode\nobreak\ \leavevmode\nobreak\ \sigma=\sqrt {\text{Var}_{1}(X_{1}-X_{0})}\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ \text{and}\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\ X_{t}=tX_{1}+(1-t)X_{0},$ $$

where $D$ represents the dataset with paired $X_{1}$ and $\mathcal{C}$, and $\text{Var}_{1}$ calculates the variance along the feature dimension. For the time domain model, this energy balancing operation precedes the ISTFT process.
This approach helps to minimize the relative error in low-volume regions.
Our experimental results demonstrate that this method enhances overall performance,
benefiting not just the silent parts.

Energy-balanced loss is related to PriorGrad (gil Lee et al., ), yet they are fundamentally different. PriorGrad uses a data-dependent diffusion prior to narrow the gap between the prior and data distribution, thereby reducing sampling steps. This prior is manually chosen based on specific datasets and tasks. In contrast, our method employs standard Gaussian noise as the diffusion prior and adjusts the loss function according to the target’s standard deviation. Importantly, our approach requires no manual selection.

#### Overlap Loss

Within the multi-band structure, each subband is predicted independently, potentially resulting in inconsistencies among them.
To mitigate these inconsistencies, we introduce an overlap loss.
This involves maintaining overlaps between the subbands when dividing the full-band complex spectrograms. A detailed illustration of this scheme is provided in Figure 2 and described in detail in Appendix Section A.3.
In this paper, we employ an 8-dimensional overlap.
During the training phase, the MSEs of the overlapped predictions is minimized.
In the inference phase, the overlaps are removed, and all subbands are merged to recreate the full-band complex spectrograms.

As each subband is predicted, the model internally maintains consistency between its overlap section and the main section.
The overlap serves as an anchor to maintain consistency among the subbands.
Modeling higher bands based on lower bands rather than predicting all subbands in parallel can increase consistency between subbands. However, at the inference stage, if the lower bands are incorrectly predicted, the higher bands conditioned on them would also be inaccurate (Roman et al., ). RFWave addresses this by using overlap loss to maintain consistency between subbands during training. At the inference stage, subbands are predicted independently, so any error occurring in one subband does not negatively affect the others.

Figure: Figure 2: An illustration of dividing complex spectrograms into subbands. The area highlighted in pink represents a subband, while the section enclosed by the two dashed vertical lines indicates the main section.
Refer to caption: x2.png

#### STFT Loss

The magnitude spectrogram derived loss (Arik et al., ) is extensively utilized in GAN-based vocoders, such as HiFi-GAN (Kong et al., ) and Vocos (Siuzdak, ), both of which leverage a Mel-spectrogram loss. Nevertheless, its use in diffusion-based vocoders is relatively rare. This might stem from its lack of direct compatibility with the formalization of a noise prediction diffusion model.
Here we adopt STFT loss for RFWave. According to (5),
the model’s output serves as an approximation for velocity:

$$ $v(X_{t},t\mid\mathcal{C})\approx\frac{\mathrm{d}}{\mathrm{d}t}X_{t}=X_{1}-X_{0}.$ (6) $$

Hence, at time $t$, an approximation for $X_{1}$ is:

$$ $\overset{\sim}{X_{1}}\approx X_{0}+v(X_{t},t\mid\mathcal{C}).$ (7) $$

The STFT loss can be applied on the approximation $\overset{\sim}{X_{1}}$,
incorporating the spectral convergence loss and log-scale STFT-magnitude loss as proposed in (Arik et al., ), which are detailed in Appendix Section A.4.
According to our experimental findings, the STFT loss effectively reduces artifacts in the presence of background noise.

### 3.3 Selecting Time Points for Euler Method

Liu et al. () employ equal step intervals for the Euler method, as illustrated in (4) . Here, we propose selecting time points for sampling based on the straightness of transport trajectories. The time points are chosen such that the increase in straightness is equal across each step. The straightness of a learned velocity filed $v$ is defined as
$S(v)=\int_{0}^{1}E\mid\mid(X_{1}-X_{0})-v(X_{t},t\mid\mathcal{C})\mid\mid^{2}
\mathrm{d}t$, which describes the deviation of the velocity along
the trajectory. A smaller
$S(v)$ means straighter trajectories.
This approach ensures that the difficulty of each Euler step remains consistent, allowing the model to take more steps in more challenging regions. Using the same number of sampling steps, this method outperforms the equal interval approach. The implementation details is provided in Appendix Section A.5. We refer to this approach as equal straightness.

## 4 Experiments and Analysis

### 4.1 Experimental Setup

#### Overview

We evaluate RFWave using both Mel-spectrograms and discrete EnCodec tokens as inputs. For Mel-spectrogram inputs, we first benchmark RFWave against existing diffusion vocoders to demonstrate its superiority. We then compare it with widely-used GAN models to highlight its practical applicability and advantages. For discrete EnCodec tokens, we evaluate RFWave’s efficiency in reconstructing high-quality audio from compressed representations across diverse domains.
Finally, we conduct ablation studies and further analysis to examine the effects of the individual components of RFWave.

#### Data

For Mel-spectrogram inputs, we conduct two evaluations. When benchmarking against diffusion vocoders, we train and test on LibriTTS (Zen et al., ) (speech), MTG-Jamendo (Bogdanov et al., ) (music), and Opencpop (Wang et al., ) (vocal) datasets to ensure comprehensive comparison across various audio categories. When comparing to widely used GAN-based models, we train on LibriTTS and test on both LibriTTS and MUSDB18 (Rafii et al., ) test subset, evaluating in-domain performance and out-of-domain generalization ability respectively.

For discrete EnCodec token inputs, we follow convention by training a universal model on a large-scale dataset. This dataset combines Common Voice 7.0 (Ardila et al., ) and clean data from DNS Challenge 4 (Dubey et al., ) for speech, MTG-Jamendo (Bogdanov et al., ) for music, and FSD50K (Fonseca et al., ) and AudioSet (Gemmeke et al., ) for environmental sounds. Recognizing the lack of a comprehensive test set for universal audio codec models, we constructed a unified evaluation dataset comprising 900 test audio samples from 15 external datasets, covering speech, vocals, and sound effects. Detailed information about this test set is provided in Table A.4.

#### Baseline and Evaluation Metrics

We conducted a comprehensive benchmark comparison of RFWave against a series of state-of-the-art models. For Mel-spectrogram inputs, we utilized PriorGrad (gil Lee et al., ) and FreGrad (Nguyen et al., ) as baselines for diffusion-based models. Vocos (Siuzdak, ) and BigVGAN (gil Lee et al., ) served as baselines for GAN-based methods, reflecting their widespread use in real-world applications. For discrete EnCodec token inputs, we compared with EnCodec (Défossez et al., ) and MBD (Roman et al., ).

We trained PriorGrad and FreGrad on LibriTTS, Opencpop, and MTG-Jamendo using their open-source details. For Vocos, BigVGAN, EnCodec, and MBD, we used public pre-trained models. We followed the authors’ recommended sampling steps: 6 for PriorGrad, 50 for FreGrad, and 20 for MBD. Model sources are in Table A.2.

For objective evaluation, we use ViSQOL (Chinen et al., ) to assess perceptual quality, employing speech-mode for 22.05/24 kHz waveforms and audio-mode for 44.1 kHz waveforms. Additional metrics include the Perceptual Evaluation of Speech Quality (PESQ) (Rix et al., ), the F1 score for voiced/unvoiced classification (V/UV F1), and the periodicity error (Periodicity) (Morrison et al., ).
For subjective evaluation, we conduct crowd-sourced assessments, employing a 5-point Mean Opinion Score (MOS) to determine the audio naturalness, ranging from 1 (‘poor/unnatural’) to 5 (‘excellent/natural’).

#### Implementation

We use the time-domain model with three enhanced loss functions as the default.
The RFWave backbone contains 8 ConvNeXtV2 blocks, and the complex spectrogram is divided into 8 equally spanned subbands.
For evaluation, we use 10 sampling steps.
Table A.3 lists the parameters used for extracting Mel-spectrograms and complex spectrograms from datasets with different sample rates.
Further implementation details and computational resource requirements are provided in the Appendix Sections A.1 and A.2, respectively.

### 4.2 Experimental Results on Mel-spectrograms Input

#### Comparasion with Diffusion-based Method

In this portion, we compare RFWave(^3^33This evaluation used an earlier RFWave version without the optimized sampling method (equal straightness) from Section 3.3. Despite this, the results convincingly demonstrate RFWave’s superiority over other diffusion vocoders.) with PriorGrad and FreGrad on LibriTTS, MTG-Jamendo, and Opencpop datasets. Table 1 presents overall results, while Table 6 provides detailed metrics for each audio category.
RFWave consistently outperforms PriorGrad and FreGrad in both objective and subjective metrics. The superior MOS score achieved by RFWave is likely due to its ability to produce clearer and more consistent harmonics, particularly in high-frequency ranges, which contributes to better overall audio quality.
A collection of spectrograms and their corresponding analyses can be found in Appendix Section A.6.

#### Comparasion with GAN-based Method

When comparing RFWave to widely used state-of-the-art GAN-based models (BigVGAN, Vocos), we used models trained on LibriTTS. Besides evaluating on the LibriTTS testset, to assess the models’ robustness and extrapolation capabilities, following the methodology in (gil Lee et al., ), we also evaluated these models’ performance on the MUSDB18 dataset. It can be observed in Tables 2 that on the in-domain test set (LibriTTS), RFWave is generally on par with state-of-the-art GAN-based models. However, on the out-of-domain test set (MUSDB18), according to the results in Table 3, RFWave shows significant advantages over BigVGAN and Vocos, even though BigVGAN employed Snake activation (Ziyin et al., ) to enhance its out-of-domain data generation capabilities. This demonstrates that RFWave, as a diffusion-type model, offers clear advantages in generalization and robustness compared to GAN-based models. For detail, we observe that GAN-based methods tend to generate horizontal lines in the high-frequency regions of the spectrograms, as exemplified in Figure A.7. In contrast, RFWave consistently produces clear high-frequency harmonics, even when applied to out-of-domain data.

**Table 1: Average Mean Opinion Score (MOS) and objective evaluation metrics for RFWave, PriorGrad and FreGrad across various test sets. MOS is provided with 95% confidence interval.**
| Model | MOS ↑ | PESQ ↑ | ViSQOL ↑ | V/UV F1 ↑ | Periodicity ↓ |
| --- | --- | --- | --- | --- | --- |
| RFWave | 3.95$\pm$0.09 | 4.202 | 4.456 | 0.979 | 0.070 |
| PriorGrad | 3.75$\pm$0.09 | 3.612 | 4.347 | 0.974 | 0.082 |
| FreGrad | 2.99$\pm$0.14 | 3.640 | 4.179 | 0.973 | 0.087 |
| Ground truth | 4.00$\pm$0.09 | - | - | - | - |

**Table 2: MOS and objective evaluation metrics for RFWave, BigVGAN and Vocos on LibriTTS.**
| Model | MOS ↑ | PESQ ↑ | ViSQOL ↑ | V/UV F1 ↑ | Periodicity ↓ |
| --- | --- | --- | --- | --- | --- |
| RFWave | 3.82$\pm$0.12 | 4.304 | 4.579 | 0.967 | 0.091 |
| BigVGAN | 3.78$\pm$0.11 | 4.240 | 4.712 | 0.978 | 0.067 |
| Vocos | 3.74$\pm$0.10 | 3.660 | 4.696 | 0.958 | 0.104 |
| Ground truth | 3.91$\pm$0.10 | - | - | - | - |

**Table 3: MOS for RFWave, BigVGAN and Vocos on MUSDB18.**
| Model | Vocals | Drums | Bass | Others | Mixture | Average |
| --- | --- | --- | --- | --- | --- | --- |
| RFWave | 3.46$\pm$0.14 | 3.65$\pm$0.10 | 3.62$\pm$0.10 | 3.54$\pm$0.11 | 4.04$\pm$0.11 | 3.67$\pm$0.05 |
| BigVGAN | 3.42$\pm$0.13 | 3.68$\pm$0.09 | 3.50$\pm$0.12 | 3.33$\pm$0.13 | 3.58$\pm$0.10 | 3.51$\pm$0.05 |
| Vocos | 3.04$\pm$0.15 | 3.54$\pm$0.11 | 2.96$\pm$0.12 | 2.88$\pm$0.15 | 3.08$\pm$0.14 | 3.10$\pm$0.06 |
| Ground truth | 3.62$\pm$0.14 | 3.71$\pm$0.14 | 3.75$\pm$0.12 | 3.79$\pm$0.12 | 4.17$\pm$0.10 | 3.80$\pm$0.05 |

### 4.3 Experimental Results on Discrete Encodec Tokens Input

We evaluated RFWave against EnCodec and MBD across various bandwidths for discrete EnCodec tokens inputs. For RFWave, we used Classifier-Free Guidance (CFG) (Ho & Salimans, ) with a 2.0 coefficient. CFG showed little improvement for Mel-spectrogram inputs, but effective for EnCodec tokens, likely due to CFG tending to be more impactful when the input is more compressed.

Table 4 displays the average MOS and objective metrics for EnCodec tokens with varying bandwidths across different auditory contexts. Meanwhile, Table A.6 offers the detailed results for each category. RFWave excels in all metrics, achieving optimal scores, except for ViSQOL. While using a larger bandwidth generally improves performance, for RFWave, an increase in bandwidth from 6.0 kbps to 12.0 kbps results in only a slight enhancement in MOS. EnCodec’s GAN-based decoder attains the highest ViSQOL. Different model families may produce distinct waveform footprints, with ViSQOL showing a subtle bias for GAN-based models.

**Table 4: Average MOS and objective evaluation metrics for RFWave, EnCodec and MBD across various test sets.**
| Bandwidth | Model | MOS ↑ | PESQ ↑ | ViSQOL ↑ | V/UV F1 ↑ | Periodicity↓ |
| --- | --- | --- | --- | --- | --- | --- |
| 1.5 kbps | RFWave(CFG2) | 3.17$\pm$0.22 | 1.797 | 3.108 | 0.914 | 0.193 |
| EnCodec | 2.23$\pm$0.23 | 1.708 | 3.518 | 0.906 | 0.199 |  |
| MBD | 3.01$\pm$0.19 | 1.699 | 2.982 | 0.901 | 0.212 |  |
| 3.0 kbps | RFWave(CFG2) | 3.52$\pm$0.25 | 2.444 | 3.570 | 0.939 | 0.145 |
| EnCodec | 2.79$\pm$0.25 | 1.934 | 3.793 | 0.930 | 0.166 |  |
| MBD | 3.06$\pm$0.23 | 2.310 | 3.402 | 0.922 | 0.171 |  |
| 6.0 kbps | RFWave(CFG2) | 3.69$\pm$0.16 | 2.936 | 3.892 | 0.954 | 0.117 |
| EnCodec | 3.10$\pm$0.15 | 2.432 | 4.091 | 0.951 | 0.126 |  |
| MBD | 3.43$\pm$0.15 | 2.488 | 3.582 | 0.929 | 0.168 |  |
| 12.0 kbps | RFWave(CFG2) | 3.73$\pm$0.16 | 3.270 | 4.124 | 0.965 | 0.099 |
| EnCodec | 3.55$\pm$0.15 | 2.892 | 4.291 | 0.963 | 0.105 |  |
| MBD | - | - | - | - | - |  |
| Ground truth | 4.07$\pm$0.14 | - | - | - | - |  |

### 4.4 Analysis

#### Ablations

We evaluate the model’s performance in both frequency and time domains, then incrementally add the three loss functions and equal straightness to the better-performing model for further analysis. We conduct ablation studies on LJSpeech (Ito & Johnson, ), using 250 test sentences for Mel-spectrogram-based waveform reconstruction, summarized in Table 5. Despite the metrics not consistently correlating with human evaluations, they accurately captured the quality improvements brought about by design modifications (Roman et al., ). The model operating in the time domain outperforms its frequency domain counterpart, as the ISTFT operation in the former (Figure 1) introduces periodic signals. The inverse Discrete Fourier Transform (IDFT) matrix comprises sinusoidal functions of frequencies $[0,1/N,...,(N-1)/N]$, where $N$ is the number of FFT points (Oppenheim & Schafer, ). This mechanism is similar to the Snake activation (Ziyin et al., ) in BigVGAN (gil Lee et al., ), which also introduces periodic signals into the generator.
Energy-balanced loss and equal straightness improves the overall performance.
The STFT loss enhances PESQ scores but adversely affects ViSQOL and periodicity error. This occurs because the model tends to prioritize magnitude over phase when applying STFT loss. This trade-off is essential for effectively eliminating artifacts, especially in the presence of background noise. Although overlap loss slightly decreases PESQ, omitting it results in a noticeable transition between subbands. Example spectrograms illustrating the effects of omitting STFT loss and overlap loss are provided in Figures A.8 and A.9, respectively.

**Table 5: Objective metrics assess the model’s performance in the frequency domain and the time domain, where three loss functions and equal straightness are applied incrementally.**
| Setting | PESQ ↑ | ViSQOL ↑ | V/UV F1 ↑ | Periodicity ↓ |
| --- | --- | --- | --- | --- |
| frequency | 3.872 | 4.430 | 0.948 | 0.144 |
| time | 4.127 | 4.551 | 0.957 | 0.124 |
| + energy-balanced loss | 4.181 | 4.598 | 0.965 | 0.110 |
| + overlap loss | 4.158 | 4.599 | 0.959 | 0.122 |
| + STFT loss | 4.211 | 4.578 | 0.961 | 0.119 |
| + equal straightness | 4.275 | 4.654 | 0.966 | 0.100 |

#### Futher Analysis

Using the model with an 8-layer ConvNeXtV2 backbone, time domain, and three loss functions as the baseline, we first experimented with the DDPM (Ho et al., ) approach. Utilizing the noise schedule from DiffWave and PriorGrad, the performance with 50 sampling steps fell short of the baseline, as demonstrated in Table 6. This highlights that Rectified Flow is critical for our model to efficiently generate high-quality audio samples.
Next, we experimented with a ResNet (He et al., ) backbone with a similar number of parameters. This configuration slightly reduced efficiency and performance in objective metrics (Table 6), indicating that the ConvNeXtV2 backbone enhances both efficiency and audio quality.
Finally, we experimented with models of different sizes. Specifically, we tested ConvNextV2 backbones of half and double sizes, with detailed configurations provided in A.1. Increasing the backbone size improves performance but decreases efficiency (Table 6).

**Table 6: Objective metrics for further analysis. xRT stands for the speed at which the model can generate speech in comparison to real-time.**
| Setting | GPU xRT ↑ | PESQ ↑ | ViSQOL ↑ | V/UV F1 ↑ | Periodicity ↓ |
| --- | --- | --- | --- | --- | --- |
| baseline | 162.59 | 4.211 | 4.578 | 0.961 | 0.119 |
| DDPM, 50 steps | 34.40 | 2.790 | 4.499 | 0.939 | 0.174 |
| ResNet | 150.95 | 3.865 | 4.509 | 0.955 | 0.128 |
| half size | 228.70 | 4.128 | 4.553 | 0.959 | 0.120 |
| double size | 111.62 | 4.285 | 4.636 | 0.968 | 0.100 |

### 4.5 Inference Speed

We perform inference speed benchmark tests using an NVIDIA GeForce RTX 4090 GPU. The implementation was done in PyTorch (Paszke et al., ), and no specific hardware optimizations were applied. The inference was carried out with a batch size of 1 sample, utilizing the LJSpeech test set, resampled to model’s sampling rate. Table 7 displays the model size, synthesis speed and GPU memory consumption of the models.
RFWave is more than twice as fast as BigVGAN and consumes less GPU memory, thereby eliminating latency as a barrier for practical applications. This speed advantage becomes even more pronounced when synthesizing high-resolution audio (44.1/48 kHz). Frame-level models like RFWave significantly outperform sample-point-based models in high-resolution audio synthesis in terms of speed.
RFWave can easily adjust the complex spectrogram’s window lenght and hop length for 44.1 kHz sampling, maintaining low computational complexity, whereas BigVGAN requires additional upsampling blocks.
Vocos serves as a strong baseline, given its efficiency in requiring only a single forward pass and operating at the frame level.

**Table 7: Model footprint and synthesis speed. xRT stands for the speed at which the model can generate speech in comparison to real-time.**
| Model | Parameters (M) | GPU xRT $\uparrow$ | GPU Memory (MB) | Sampling steps |
| --- | --- | --- | --- | --- |
| RFWave | 18.1 | 162.59 | 780 | 10 |
| BigVGAN | 107.7 | 72.68 | 1436 | 1 |
| Vocos(ISTFT) | 13.5 | 2078.20 | 590 | 1 |
| PriorGrad | 2.6 | 16.67 | 4976 | 6 |
| FreGrad | 1.7 | 7.50 | 2720 | 50 |
| MBD | 411.0 | 4.82 | 5480 | 20 |
| RFWave(44.1KHz) | 20.5 | 152.58 | 902 | 10 |
| BigVGAN(44.1KHz) | 116.5 | 39.03 | 1740 | 1 |

## 5 Conclusion

In this study, we propose RFWave, a multi-band Rectified Flow approach for audio waveform reconstruction.
The model has been carefully designed to overcome the latency issues associated with diffusion models.
RFWave stands out for its ability to generate complex spectrograms by operating at the frame level, processing all subbands concurrently. This concurrent processing significantly enhances the efficiency of the waveform reconstruction process.
The empirical evaluations conducted in this research have demonstrated that RFWave achieves exceptional reconstruction quality. Moreover, it has shown superior computational efficiency by generating audio at a speed that is 160 times faster than real-time, comparable to GAN-based methods, making it practical for real-world applications.

## Appendix A Appendix

### A.1 Implementation Details

The RFWave backbone contains 8 ConvNeXtV2 blocks. Within each ConvNeXtV2 block, the depth-wise convolutional layer featuring a large kernel utilizes a kernel size of 7 and has a channel dimension of 512. The first and last 1x1 point-wise convolutional layers in the sequence possess channel dimensions of 512 and 1536, respectively.
In Subsection 4.4, the half-size variant contains 8 ConvNeXtV2 blocks with point-wise convolutional layers of 384 and 1152, while the double-size variant features 16 ConvNeXtV2 blocks with point-wise convolutional layers of 512 and 1536.

During the extraction of complex coefficients for the model, we use the orthonormal Fast Fourier Transform (FFT) and its inverse (IFFT), with the normalization convention of dividing by $1/\sqrt{N}$ for both operations, here $N$ is the FFT size. This approach ensures the spectrogram extracted is within a more reasonable range for modeling.

In terms of loss functions, we assign a weight of 1 to either the Rectified Flow loss or its energy-balanced variant. If employed, the overlap loss and STFT loss are assigned a weight of 0.01. These weights are determined by aligning the L2-norm of the gradients from the overlap loss and STFT loss with approximately 1/10 of that from the Rectified Flow loss.

Training details are also worth mentioning. Audio samples are randomly cropped to lengths of 32512 and 65024 for 22.05/24 kHz and 44.1 kHz waveforms, respectively. This is equivalent to a crop window of 128 frames for both sampling rates. We use a batch size of 64. The model optimization is performed using the AdamW optimizer with a starting learning rate of 2e-4 and beta parameters of (0.9, 0.999). A cosine annealing schedule is applied to reduce the learning rate to a minimum of 2e-6.

### A.2 Computational Resource Required for the Experiments

The bulk of the experiments were carried out on personal computers and GPU servers, which were sourced from cloud service providers. Primarily, we utilized Nvidia-4090-24G and Nvidia-A100-80G GPUs for these tasks.

The specific hardware used and the corresponding time taken for training RFWave on various datasets are as follows:

**Table A.1: GPU configurations and training duration for various datasets**
| Dataset | GPU configuration | Training duration |
| --- | --- | --- |
| LJspeech | 1$\times$4090 | 2 days |
| LibriTTS | 2$\times$A100 | 5 days |
| Opencpop | 1$\times$4090 | 1 day |
| MTG-Jamendo | 2$\times$A100 | 7 days |
| EnCodec Training | 4$\times$A100 | 10 days |

Figure: Figure A.1: The energy and weighting coefficients, represented by $\sigma$, display a consistent variation throughout the frames.
Refer to caption: extracted/5905878/volume.png

### A.3 Deviding into Subbands

The complex spectrogram is circularly padded in the feature dimension with a size of $(d_{ol},d_{ol}-1)$, where $d_{ol}$ signifies the overlap size.
The dimension of each subband’s main section, denoted as $d_{m}$, is calculated by dividing $d-1$ by the total number of subbands.
Here, $d$ represents the dimension of the complex spectrograms.
The last subband is an exception, having an extra feature dimension but one less padding dimension.
Subbands are extracted by applying a sliding window along the feature dimension, where the window has a size of $d_{m}+2d_{ol}$ and shifts by $d_{m}$(^4^44The feature dimension of each subband, represented as $d_{s}$, equals $2(d_{m}+2d_{ol})$, accounting for the interleave of real and imaginary parts.).

### A.4 STFT Loss

#### Spectral convergence (SC) loss

$$ ${\||X_{1}|-|\overset{\sim}{X_{1}}|\|_{F}}/{\||X_{1}|\|_{F}},$ (8) $$

where $X_{1}$ is the ground truth and $\overset{\sim}{X_{1}}$ is the estimated complex spectrogram using (7). The Frobenius norm, $\|\cdot\|_{F}$, applied over time and frequency, emphasizes large spectral components in SC loss.

#### Log-scale STFT-magnitude loss

$$ $\|\log(|X_{1}|+\epsilon)-\log(|\overset{\sim}{X_{1}}|+\epsilon)\|_{1},$ (9) $$

where the $L^{1}$ norm, $\|\cdot\|_{1}$, along with a small constant $\epsilon$, is used to accurately capture small-amplitude components in the log-scale STFT-magnitude loss.

### A.5 Selecting Time Points for Euler Method

Straightness is calculated from a single batch with a size of 96. The Euler method uses 100 equal interval steps to estimate the integral in the straightness definition. Time points are selected such that the increase in straightness remains consistent across each interval. An example with 10 Euler steps is provided in Figure A.2. The time points are calculated only once per model, resulting in negligible computational load.

Figure: Figure A.2: Deviation (top) and straightness (bottom) over time: red dots mark Euler method time points, with constant increase in straightness across intervals.
Refer to caption: extracted/5905878/straightness.png

### A.6 Spectrogram Examples for Comparison with Diffusion-based Method

Figure A.3 showcases spectrogram examples generated by various models using the Opencpop dataset. When compared to the ground truth spectrogram, RFWave is seen to produce clean and stable harmonics, whereas the harmonics generated by other models exhibit minor discontinuities.

Figure A.4 exhibits spectrogram examples generated by different models utilizing the LibriTTS dataset. RFWave generate clear high-frequency harmonics, while PriorGrad and FreGrad result in blurred high-frequency harmonics.

Figure A.5 displays spectrogram examples generated by diverse models from the Jamendo dataset. Both RFWave and PriorGrad generate commendable spectrograms, with RFWave edging out slightly in the high-frequency range.

Figure: (a) Ground truth
Refer to caption: extracted/5905878/spec_images/opencpop/gt.png

Figure: (a) Ground truth
Refer to caption: extracted/5905878/spec_images/libritts/gt.png

Figure: (a) Ground truth
Refer to caption: extracted/5905878/spec_images/jamendo/gt.png

### A.7 Spectrogram Examples for Comparison with GAN-based Method

Figure A.6 illustrates that RFWave, BigVGAN and Vocos are capable of generating high-quality spectrograms when applied to the LibriTTS dataset. This demonstrates the effectiveness of each approach in handling in-distribution data. The spectrograms exhibit well-defined harmonics and minimal artifacts, highlighting the robustness of the models within their familiar domain.

Figure A.7 presents spectrogram examples generated on the MUSDB18 dataset, with the models having been trained on the LibriTTS dataset. A notable observation is that both BigVGAN and Vocos exhibit a tendency to produce horizontal lines in the high-frequency regions of the spectrograms. These artifacts are perceptually problematic, as they often result in a metallic sound quality that detracts from the naturalness of the audio. In stark contrast, RFWave consistently generates clear and well-defined high-frequency harmonics. This capability underscores RFWave’s superior ability to maintain spectral fidelity, even when applied to out-of-distribution data, thereby enhancing the overall audio quality and realism.

Figure: (a) Ground truth
Refer to caption: extracted/5905878/gan_libritts/gt.png

Figure: (a) Ground truth
Refer to caption: extracted/5905878/gan_musdb/gt.png

Figure: (a) Ground truth
Refer to caption: extracted/5905878/stft_loss/gt.png

Figure: (a) Ground truth
Refer to caption: extracted/5905878/overlap/gt.png

### A.8 Tables

**Table A.2: Code repositories for model training.**
| Model | repository |
| --- | --- |
| PriorGrad | [https://github.com/microsoft/NeuralSpeech/tree/master/PriorGrad-vocoder](https://github.com/microsoft/NeuralSpeech/tree/master/PriorGrad-vocoder) |
| FreGrad | [https://github.com/signofthefour/fregrad](https://github.com/signofthefour/fregrad) |
| Multi-Band Diffusion (MBD) | [https://github.com/facebookresearch/audiocraft](https://github.com/facebookresearch/audiocraft) |
| EnCodec | [https://github.com/facebookresearch/encodec](https://github.com/facebookresearch/encodec) |
| Vocos | [https://github.com/gemelo-ai/vocos](https://github.com/gemelo-ai/vocos) |
| BigVGAN | [https://github.com/NVIDIA/BigVGAN](https://github.com/NVIDIA/BigVGAN) |

**Table A.3: Parameters for extracting Mel-spectrograms and complex spectrograms.**
| Dataset | Sample rate (kHZ) | Window length | Hop length | FFT size | Mel bins |
| --- | --- | --- | --- | --- | --- |
| LJSpeech | 22.05 | 1024 | 256 | 1024 | 100 |
| LibriTTS | 24 | 1024 | 256 | 1024 | 100 |
| Opencpop | 44.1 | 2048 | 512 | 2048 | 100 |
| MTG-Jamendo | 44.1 | 2048 | 512 | 2048 | 100 |
| EnCodec Training | 24 | 1280 | 320 | 1280 | - |

**Table A.4: Construction of the general Encodec test dataset**
| Category | Subdirectory | Samples | Description |
| --- | --- | --- | --- |
| Speech | Expresso-test | 50 | From the Expresso dataset test set (Nguyen et al., ), randomly select 50 samples, segmenting any audio longer than 30 seconds into 10-second clips beforehand. |
| HiFiTTS-test | 50 | Randomly select 50 samples from the test set of the Hi-Fi TTS dataset (Bakhturina et al., ). |  |
| LibriTTS-test | 50 | Randomly select 50 samples from the test set of LibriTTS (Zen et al., ). |  |
| Aishell3-test | 50 | Randomly select 50 samples from the test set of the Aishell3 dataset (Shi et al., ). |  |
| JVS | 50 | Randomly select 50 samples from the entire JVS Corpus. (Takamichi et al., ). |  |
| CML-TTS-test | 50 | Randomly select 50 samples from the test set of the CML-TTS dataset (Oliveira et al., ). |  |
| Vocal | Musdb-test-vocal | 60 | Segment the vocal track audio from the Musdb test set (Rafii et al., ) into 10-second clips, then randomly select 60 samples. |
| CSD | 20 | Segment all audio in the CSD (Choi et al., ) into 10-second clips, then randomly select 20 samples. |  |
| Opencpop | 20 | Randomly select 20 samples from the Opencpop dataset (Wang et al., ). |  |
| ChineseOpera-monophonic | 60 | From the monophonic subset of the Chinese Opera Singing Dataset (Black et al., ), segment any audio longer than 30 seconds into 10-second clips, then randomly select 60 samples. |  |
| JVS-Music | 60 | Randomly select 60 samples from the JVS-Music Corpus (Tamaru et al., ). |  |
| RAVDESS | 60 | Randomly select 60 samples from the song subset of the RAVDESS corpus (Livingstone & Russo, ). |  |
| Ccmusic-demo-vocal | 20 | From the demo audios of the Ccmusic Dataset (Liu & Li, ), select the vocal parts. Then, randomly select 20 samples from these for the test set, segmenting any audio longer than 30 seconds into 10-second clips beforehand. |  |
| Sound Effect | ESC-50 | 150 | Randomly select 150 samples from the ESC-50 (Piczak, ). |
| Musdb-test-accompaniment | 40 | Segment the accompaniment track audio from the test set of the Musdb dataset (Rafii et al., ) into 10-second clips, then randomly select 40 samples from these clips. |  |
| Musdb-test-mixture | 20 | Segment the mixture track audio from the test set of the Musdb dataset (Rafii et al., ) into 10-second clips, then randomly select 20 samples from these clips. |  |
| ChineseOpera-polyphonic | 20 | From the polyphonic subset of the Chinese Opera Singing Dataset (Black et al., ), segment any audio longer than 30 seconds into 10-second clips, then randomly select 20 samples. |  |
| OpenMIC-test | 40 | Randomly select 40 samples from the test set of OpenMix 2018 (Humphrey et al., ). |  |
| Ccmusic-demo-music | 30 | From the demo audios of the Ccmusic Dataset (Liu & Li, ), select the non-vocal parts. Then, randomly select 20 samples from these for the test set, segmenting any audio longer than 30 seconds into 10-second clips beforehand. |  |

**Table A.5: MOS and Objective evaluation metrics for RFWave, PriorGrad and FreGrad across various datasets.(^6^66The MOS of the ground truth on LibriTTS differs from that presented in Table 2 because these two MOS tests were conducted separately.)**
| Dataset | Model | MOS ↑ | PESQ ↑ | ViSQOL ↑ | V/UV F1 ↑ | Periodicity ↓ |
| --- | --- | --- | --- | --- | --- | --- |
| LibriTTS | RFWave | 3.83$\pm$0.14 | 4.228 | 4.595 | 0.968 | 0.090 |
| PriroGrad | 3.70$\pm$0.16 | 3.820 | 4.134 | 0.960 | 0.100 |  |
| FreGrad | 3.68$\pm$0.15 | 3.758 | 4.278 | 0.960 | 0.099 |  |
| Ground truth | 3.88$\pm$0.15 | - | - | - | - |  |
| Opencpop | RFWave | 4.26$\pm$0.17 | 4.176 | 4.564 | 0.990 | 0.049 |
| PriroGrad | 3.79$\pm$0.21 | 3.404 | 4.512 | 0.988 | 0.064 |  |
| FreGrad | 3.73$\pm$0.20 | 3.522 | 4.507 | 0.986 | 0.074 |  |
| Ground truth | 4.33$\pm$0.18 | - | - | - | - |  |
| MTG-Jamendo | RFWave | 3.90$\pm$0.15 | - | 4.317 | - | - |
| PriroGrad | 3.79$\pm$0.14 | - | 4.412 | - | - |  |
| FreGrad | 1.89$\pm$0.17 | - | 3.765 | - | - |  |
| Ground truth | 3.93$\pm$0.15 | - | - | - | - |  |

**Table A.6: MOS and objective evaluation metrics for RFWave, EnCodec and MBD across various test sets.**
| Test set | Bandwidth | Model | MOS | PESQ ↑ | ViSQOL ↑ | V/UV F1 ↑ | Periodicity↓ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Speech | 1.5 kbps | RFWave(CFG2) | 3.29$\pm$0.47 | 1.774 | 3.102 | 0.913 | 0.178 |
| EnCodec | 2.29$\pm$0.35 | 1.515 | 3.310 | 0.870 | 0.231 |  |  |
| MBD | 2.94$\pm$0.29 | 1.659 | 2.793 | 0.901 | 0.195 |  |  |
| 3.0 kbps | RFWave(CFG2) | 3.85$\pm$0.39 | 2.421 | 3.582 | 0.935 | 0.137 |  |
| EnCodec | 2.85$\pm$0.47 | 1.967 | 3.746 | 0.919 | 0.166 |  |  |
| MBD | 2.88$\pm$0.33 | 2.194 | 3.219 | 0.915 | 0.166 |  |  |
| 6.0 kbps | RFWave(CFG2) | 4.05$\pm$0.26 | 2.974 | 3.913 | 0.952 | 0.109 |  |
| EnCodec | 3.19$\pm$0.31 | 2.554 | 4.048 | 0.945 | 0.121 |  |  |
| MBD | 3.51$\pm$0.30 | 2.372 | 3.410 | 0.924 | 0.161 |  |  |
| 12.0 kbps | RFWave(CFG2) | 3.96$\pm$0.22 | 3.393 | 4.158 | 0.965 | 0.089 |  |
| EnCodec | 3.52$\pm$0.22 | 3.104 | 4.250 | 0.961 | 0.095 |  |  |
| MBD | - | - | - | - | - |  |  |
| Ground truth | 4.13$\pm$0.23 | - | - | - | - |  |  |
| Vocal | 1.5 kbps | RFWave(CFG2) | 3.06$\pm$0.34 | 1.820 | 3.317 | 0.914 | 0.208 |
| EnCodec | 1.94$\pm$0.38 | 1.900 | 3.931 | 0.941 | 0.166 |  |  |
| MBD | 2.84$\pm$0.31 | 1.739 | 3.221 | 0.901 | 0.228 |  |  |
| 3.0 kbps | RFWave(CFG2) | 3.47$\pm$0.34 | 2.467 | 3.793 | 0.942 | 0.152 |  |
| EnCodec | 2.63$\pm$0.37 | 1.900 | 3.931 | 0.941 | 0.166 |  |  |
| MBD | 3.26$\pm$0.45 | 2.426 | 3.655 | 0.929 | 0.175 |  |  |
| 6.0 kbps | RFWave(CFG2) | 3.77$\pm$0.23 | 2.897 | 4.080 | 0.956 | 0.124 |  |
| EnCodec | 2.92$\pm$0.23 | 2.310 | 4.206 | 0.956 | 0.131 |  |  |
| MBD | 3.30$\pm$0.26 | 2.604 | 3.840 | 0.933 | 0.175 |  |  |
| 12.0 kbps | RFWave(CFG2) | 3.83$\pm$0.31 | 3.147 | 4.273 | 0.964 | 0.109 |  |
| EnCodec | 3.40$\pm$0.29 | 2.679 | 4.373 | 0.965 | 0.114 |  |  |
| MBD | - | - | - | - | - |  |  |
| Ground truth | 4.27$\pm$0.24 | - | - | - | - |  |  |
| Soudn Effect | 1.5 kbps | RFWave(CFG2) | 3.24$\pm$0.43 | - | 2.906 | - | - |
| EnCodec | 2.62$\pm$0.42 | - | 3.314 | - | - |  |  |
| MBD | 3.33$\pm$0.36 | - | 2.932 | - | - |  |  |
| 3.0 kbps | RFWave(CFG2) | 3.11$\pm$0.59 | - | 3.334 | - | - |  |
| EnCodec | 2.89$\pm$0.48 | - | 3.703 | - | - |  |  |
| MBD | 3.11$\pm$0.54 | - | 3.331 | - | - |  |  |
| 6.0 kbps | RFWave(CFG2) | 3.27$\pm$0.27 | - | 3.682 | - | - |  |
| EnCodec | 3.17$\pm$0.24 | - | 4.020 | - | - |  |  |
| MBD | 3.44$\pm$0.23 | - | 3.496 | - | - |  |  |
| 12.0 kbps | RFWave(CFG2) | 3.33$\pm$0.28 | - | 3.942 | - | - |  |
| EnCodec | 3.72$\pm$0.26 | - | 4.249 | - | - |  |  |
| MBD | - | - | - | - | - |  |  |
| Ground truth | 3.78$\pm$0.28 | - | - | - | - |  |  |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Here I list questions and typos:
- Line 174: “can be” to “can be an”
- Line 184:  Fourier features of what? Of the noisy sample? How we concatenate the conditional inputs? They share same dimensionality?
- Line 205: If interleaved then shouldn’t be $2d_s$?
- Line 219: “For waveform equalization… computed during training” This part as well refers to PQMF? Or is distinct?
- Line 228: “Subsequent processing involves the dimension-wise mean-variance normalization of the complex spectrogram.” As well here there is the mean-variance normalization as in time domain?
- Line 246: Why compute the standard deviation and not the energy altogether? Is there any reason for this?
- Line 252: Space after the $\min$.
- Line 295: Isn’t $X_0$ the data point and $X_1$ the noisy point? So shouldn’t we approximate  the STFT Loss using $X_0$?
- Line 307: “filed” is “field”.  In the integral use $\mathbb{E}$. Also I think the straightness should have argument a time $s$ (see Weakness section).
- Line 382: Why snake activation enhances out-of-domain data generation capabilities?
- Line 384: Isn’t this imputable to the architecture choice more than the type of generative model?  Especially upsampling layers as empirically shown here: https://arxiv.org/pdf/2010.14356 ?
soundness: 4
strengths: - While methodologically the authors borrow many ideas from previous papers, e.g., the model architecture ConvNeXtV2 is not a novelty of the paper, and training in the STFT domain is a classic choice in audio deep learning, it is very interesting how the combination of those ideas can bring us closer on surpassing the fundamental limitations in efficiency of diffusion models from a practical point of view. Seeing in Table 7 that actually performing 10 inference step of RFWave we have a way higher realtime factor than a BigVGAN (while having similar quality performance on LibriTTS and improved OOD performance on MUSDB18) is a strong indication that the inference time gap between diffusion models and GAN models is finally disappearing. And the best thing here is that one obtains this reduction without having to resort to distillation techniques (which are notably difficult to perform, especially consistency distillation), but only resorting to reasonable design choices, especially the choice of the architecture.
- The results in the experimental section are very good for the proposed model, putting it as a novel strong baseline in the audio reconstruction setting. Additionally, the ablations in Tables 5 and 6 give a detailed view on why the design choices in the paper contribute to the overall performance.
- The proposed losses and the idea of equal straightness intervals are valuable not only in the audio reconstruction context but also more generally when dealing with other types of STFT based models / other generative settings for rectified flow models.
summary: In this paper the authors propose a new model called RFWave (Rectified Flow Wave) for achieving audio reconstruction from Mel Spectrograms or Encodec tokens. They propose training an efficient ConvNeXtV2
 backbone (a state-of-the-art convolutional architecture) using the rectified flow (diffusion) paradigm, that operate on STFT spectrograms bands for improved efficiency (than training directly in waveform domain). Each spectrogram band is processed independently by the network, and are concatenated before the inverse STFT (differently than the previous Multi-band Diffusion conditional approach of Roman et al. 2023, in which error accumulates from low to high frequency bands). The model can be trained both by performing the noising in time-domain, where is data is mapped first to STFT and back via the iSTFT before applying the loss, or trained
directly in STFT domain (authors show improved results on time-domain nosing approach). Methodologically the authors propose three new losses and new sampling algorithm for rectified flow. The losses are: a weighted loss which scales the STFT band by the variance of the ground truth velocity band (which is proportional to its energy), such that nearly-silent portions of data do not result in noisy outputs, an overlap loss which equalises
the outputs of different bands at their boundaries and an STFT loss computed via the rectified flow prediction at time $t$. The novel sampling algorithm choses sampling times by subdividing the flow trajectory in intervals
 of equal straightness (the integral of the difference between the ground truth velocity direction and the predicted velocity). The authors perform a solid experimental evaluation, comparing both with recent diffusion-based models and SOTA GAN-based models, showcasing very good results.
weaknesses: - The paper in some parts is not very clear in its explanations. For example in the Energy-balanced Loss part of Section 3.2 (Loss Functions), from first (and repeated) readings, I could not understand what was the  problem with the MSE loss. After different readings I interpreted it in the following way: small absolute errors in silent regions contribute little to the overall MSE loss, so the model doesn't prioritize eliminating them. This means the model doesn't effectively suppress minor deviations in silent areas, resulting in perceptible noise. Thus larger errors in high-amplitude regions significantly impact the MSE loss, causing the model to focus more on reducing those errors during training. Another part that was not clear was all Section 3.3 (Selecting Time Points for Euler Method). Here, first, straightness should be defined for a time $s$, as $S(v, s)$, setting the definite integral from $0$ to $s$ (using $s$ to leave the $t$ in $dt$). I believe authors use this because they took it from the original paper on rectified flow. Then writing “allowing to take more steps in more challenging regions” is pretty confusing. The term “allowing” is not ideal, I would use “requiring”. I understood the concept thinking that if in an interval the straightness does not change much, it means that we can take bigger steps, saving on the overall number of steps. I please ask the authors to re-write these parts in a more understandable way.
- While the experimental sections is done in a good way as mentioned in the Strengths part, the fact that the authors trained RFWave on their proposed large-scale dataset (combining Common Voice 7.0, DNS Challenge 4, MTG-Jamendo, FSD50K and AudioSet) and that Vocos, BigVGAN, EnCodec, and MBD are evaluated using the public pre-trained models is somehow problematic: it could be that Vocos, BigVGAN, EnCodec, and MBD are under-trained with respect to the authors dataset. I would like the author to comment on this and at least show that the training sources are comparable.
- It would have been interesting to compare also with latent diffusion models, where the network operates in another domain that is more compressed as the waveform domain, maybe a fine-tuned version of Stable Audio Open, given their widespread adoption.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - I find it difficult to understand the rationale behind dividing into subbands when working in the STFT domain. If we are considering the STFT, it seems that the signal is already inherently divided into frequency components. 
  - Also, this paper mentions introducing a loss related to overlap in order to mitigate inconsistencies caused by subband division. However, inconsistencies between subbands are not limited to adjacent bands but may also impact the coherence of harmonic components. This could also be considered a potential drawback of subband division. 
  - The paper claims improved performance on harmonics compared to conventional methods with some empirical examples, but it is not clearly justified why it is effective.

- The STFT introduces redundancy in the representation due to overlapping frames compared to the original waveform. Consequently, the assertion that complex STFT spectrograms could be generated much faster than waveforms is not immediately intuitive. A more detailed and clearer explanation is required to substantiate this claim.
soundness: 2
strengths: - Experimental results have demonstrated superior sound quality across multiple metrics compared to conventional methods.
- The strength of this paper appears to lie in the application of the latest technique, rectified flow, which enables efficient sampling. This aspect of the work stands out as a significant contribution.
- This work introduces three additional loss functions to improve performance. The additional loss functions appear to consistently improve performance.
summary: An efficient diffusion-based vocoder is proposed. The strengths of this paper are claimed to be that it generates the complex spectrogram of STFT at the frame level and that it can be generated in 10 denoising steps.
weaknesses: - The task addressed in this paper is rather conventional, using standard datasets and common evaluation methods. As a result, the work comes across as a minor variation of existing research. While this is certainly important work, it lacks a clear element of originality. It is unfortunate that, in a field where many similar approaches already exist, this work appears to be another study that only slightly improves existing benchmarks.
- Moreover, the approach is not theoretically groundbreaking either. At first glance of the contribution part, the proposed method appears to be a combination of existing ideas, making its originality and novelty less immediately clear.
- Some spectrograms are shown, but the differences are not immediately apparent, making it unclear what aspects should be evaluated. It would be helpful to provide a little more explanation rather than leaving the interpretation up to the reader saying see the high-frequency components.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See above
soundness: 2
strengths: Audio reconstruction from Mel-spectrogram is achieved with a few sampling steps, and audio is generated much faster than in real-time.
summary: This paper considers a method for reconstructing the waveform from Mel-spectrogram or discrete acoustic tokens. The method generates complex spectrograms and performs reconstruction on a frame level.  Reconstruction is performed using a straight transport trajectory, which is achieved in a few sampling steps. Audio generation can be achieved much faster than in real time.
weaknesses: 1. Algorithm Flow and Domain Clarity:
The paper suffers from readability issues, making it challenging to understand the overall flow of the proposed algorithm. Specifically, it is unclear whether the algorithm operates in the time domain, frequency domain, or both, as it seems to suggest that transitions between these domains are possible but lacks a clear explanation or visual aids (e.g., a flowchart). This lack of clarity weakens the reader's comprehension and hinders an appreciation of the technical merits of the algorithm. It would benefit the authors to provide a visual or detailed description to convey these aspects effectively.

2. Assumptions about Input Data (Mel-Spectrogram):
The paper assumes a mel-spectrogram input to produce waveform output. However, it does not clarify how this mel-spectrogram is initially obtained or what assumptions are made about the input data. Understanding the starting point and any assumptions about preprocessing steps is crucial to assessing the practicality of the proposed algorithm. The authors should explicitly state these assumptions and briefly explain how mel-spectrograms are typically derived in relevant application contexts.

3. Frequency and Time Domain Operations:
Whether the multiband rectified flow is used to reconstruct the complex spectrogram or the waveform directly remains ambiguous. While the paper mentions using the short-time Fourier Transform (STFT), which implies frequency-domain operations, it also suggests waveform reconstruction capabilities. This lack of specificity raises questions about whether the algorithm operates exclusively in one domain or both. A more precise discussion of domain-specific processes, including the role and interplay of STFT in this context, would enhance the reader’s understanding and the algorithm's technical robustness.

4. Role of ConvNeXt V2 and Learning Velocity:
The role of ConvNeXt V2 in learning the velocity field is unclear. Given that ConvNeXt V2 is a recent architecture capable of supervised and self-supervised learning, understanding how it contributes to the velocity-based learning within the algorithm is essential for evaluating the approach's novelty and effectiveness. A clear outline of ConvNeXt V2’s integration and function in this context, especially concerning velocity learning, would clarify the methodology and strengthen the reader's ability to assess its value.

5. Waveform Generation from Complex Spectrogram:
The method for generating the final waveform from a complex spectrogram is not well-explained. Although the paper references GAN-based methods, it lacks a detailed description of how this process is implemented in the proposed model. An explicit explanation of the techniques used to transition from the complex spectrogram to the waveform would clarify the reconstruction process, making it easier to evaluate the proposed method's strengths and limitations.

6. Independent Subband Reconstruction:
The decision to reconstruct subbands independently raises concerns about consistency across subbands. While the paper suggests that independent reconstruction mitigates error accumulation, it overlooks the strong correlations typically present among subbands, which, if neglected, may compromise audio quality. Discussing the potential drawbacks of independent subband reconstruction and any strategies for inter-subband consistency checks would address this issue, making the approach more comprehensive and practical.

7. Reconstruction Speed-Up Justification:
Although the algorithm claims to achieve a reconstruction speed over 100 times faster than real-time, it is unclear whether such a significant speed-up is necessary or beneficial for practical applications. While computational efficiency is generally advantageous, the paper could explore the trade-offs between speed, model complexity, and audio quality to justify the need for such extreme speed-ups in relevant application scenarios. A deeper analysis here could reinforce the practical relevance of the proposed method.

8. Novelty and Engineering-Based Solutions:
The paper's novelty is in question, as it mainly combines existing components (multiband rectified flow and ConvNeXt V2) rather than proposing a fundamentally new approach. While this engineering solution may have merit in specific applications, the lack of novel contributions limits the paper's impact. Strengthening the theoretical foundation or providing a unique methodological contribution would be beneficial in highlighting the algorithm’s distinct value.

In summary, the paper presents an engineering-focused approach for accelerated waveform reconstruction. Still, several aspects lack clarity, including domain transitions, assumptions about input data, the role of ConvNeXt V2, and consistency in subband reconstruction. Addressing these issues and offering more insight into the necessity of the speed-up and potential applications would significantly improve the paper's quality and technical rigor.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: - BigVGAN repository provides several versions of the model. Did you compare the performance of RFWave with the smaller ones in terms of sound quality and inference speed? Adding more flavors of BigVGAN into Table 7 would give better understanding of the strengths of your method.
soundness: 3
strengths: - The paper is well written and easy to follow;
- The introduced heuristics are intuitive and reasonable;
- The experiments clearly demonstrate better performance of the proposed method compared to diffusion-based baselines and give good understanding of pros and cons compared to GAN-based ones.
summary: The paper introduces RFWave, a new method for reconstructing audio waveforms from Mel-spectrograms or discrete acoustic tokens. The proposed method operates on Mel-spectrogram frames and produces spectral subbands in parallel, boosting synthesis speed. Additionally, the paper presents several heuristics aimed at speeding up inference and improving quality, namely: scaling flow-matching loss based on frame energy, adding MSE loss on overlapping parts of predicted frames, incorporating STFT loss on the estimation 𝑋_1​ at time 𝑡, and selecting optimal points for the Euler method. The model consistently outperforms diffusion-based baselines and performs on par with GAN-based methods. The experiments demonstrate that RFWave achieves a good trade-off between generation speed, memory consumption, and sound quality.
weaknesses: - Interpretation of the experimental results aren’t always supported by the data provided in the corresponding tables. 
For instance, 1) according to the authors, Table 3 demonstrates significant advantages over GAN-based methods but the confidence intervals overlap in all scores accept one (“Mixed”, which, however, may be the most important one). I think that a significance test may help to emphasize the superiority of your method. 2) the same problem with overlapping MOS intervals appears in Table 4. At least, MOS evaluation doesn’t support the claim that RFWave “excels” in all metrics being very close to MBD in the key subjective metric. 
- Although Table 3 demonstrates better performance of RFWave on out-of-domain data, I still have doubts that such experimental setting  is practical: in a real-life scenario one would rather re-train a waveform generation model on the target domain (music in this case) than use a model trained on an out-of-domain data (speech). I would consider testing performance of BigVGAN fine-tuned on small amount of music data to get better understanding of advantages of RFWave brought by its better performance on out-of-domain data.
- Minor issues: “Tables 2” p.8:378, “velocity filed”, p.6:308

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: * Were any methods trained on multiple types of audio (speech, music, sound) at the same time, or was a specialized model prepared of each RFWave and retrained baseline for *every* audio type? This is not clear to me from line 344, which would suggest that baselines were trained on multiple audio types at the same time, whereas Appendix A.2 suggests that RFWave was trained on every single dataset separately. If the authors compare dataset-specialized RFWave against multi-dataset baselines, such a comparison would be unfair.

* For completeness and better understanding, I would suggest adding a pointwise metric such as a simple L2 distance (or SNR in dB) of the reconstruction compared to the ground truth. I am asking since the presented metrics (MOS, PESQ, ViSQOL, F1, Periodicity) do not provide much insight in how the methods compare regarding pointwise distortion, but this is an important property, especially for the decoding task.

* The authors do not use the rectification step that a rectified flow model enables in principle, correct? Can they provide a reasoning for why they do not perform this step, which in theory should help improve the inference efficiency by further reducing the necessary number of sampling steps?

* Lines 47, 54-56, 70-71: It is a priori unclear why operating on STFT frames should be more efficient than operating on a waveform, since the STFT is redundant and introduces more data points to process. This seems to be more about frame-parallel processing of the DNN architecture rather than the STFT itself, could the authors clarify what they mean?

* In Section 3.3, what is meant precisely by timepoints being "chosen such that the increase in straightness is equal across each step"? I would have liked a more concrete description of the method used to obtain the time steps here - was this optimization-based, or somehow estimated in closed-form? What was the formal metric used to judge "equal straightness" across all intervals?

* What do the authors really mean by one model variant being capable of "operating in the time domain" (line 197-199)? In the abstract, the authors state that RFWave "generates complex spectrograms" -- in a flow/diffusion setting, shouldn't then the feature-space input also be a complex spectrogram? In this case, an STFT/iSTFT is used, and the ConvNeXt model performs 2D convolutions, and operates in the time-frequency domain, not in the time-domain, right? Please clarify this point here and also in the paper.

Minor points:

* In lines 204-206, the authors state that they interleave real and imaginary parts of each subband. Wouldn't this lead to a $2 d_s$-dimensional feature rather than a $d_s$-dimensional feature? (Same comment in line 227 where this notation is repeated)

* When the authors say "dimension-wise" (line 228), do they mean only $d_s$, or only $F$, or both $d_s$ and $F$?

* Can the authors add a reference for the claim that "speech energy decays exponentially with frequency, while music maintains a consistent distribution"? (lines 207-209)

* I do not follow the argument made in 3.2, lines 237-242, regarding MSE-based training. If "minor absolute distortions lead to significant relative error", shouldn't a model trained by minimizing MSE *avoid* such small absolute distortions rather than producing them? To me, the observation that the model exhibits residual noise sounds rather like a time-weighting or noise schedule issue than a problem with the MSE used within the training target?

* In the experimental setup section (4.1), I would suggest explicitly referring to Appendix A.1 which lists further training hyperparameters - I could not find such a reference and was missing these details in the main text before finding them in the appendix by chance.

* In the appendix, please provide (visually or quantitatively) the dynamic range used for producing each selection of shown spectrograms.

Typos:

* Lines 364 and 372: "Comparasion" -> "Comparison"
* Line 511: "lenght" -> "length"
* Table A.5: "PriroGrad" -> "PriorGrad"
soundness: 3
strengths: The presented method is original in this application context, combining several existing works and ideas (rectified flow, ConvNeXt architecture, sub-band processing) with some problem-specific novel losses (energy-balanced loss, subband overlap loss) and a specialized time-point selection method for the ODE solver used for inference.

The evaluations show that the method offers a new exciting point along the tradeoff between inference speed and quality. It outperforms most baselines in audio quality and performs similarly to, or better than, BigVGAN while having only ~20% of the parameters, running more than twice as fast, and using ~40% less memory. The authors further conduct convincing ablation experiments on their proposed components.

The paper is overall structured well and written clearly, with both the clarity of the technical description and the use of language being good in general.
summary: The present paper proposes RFWave, a rectified flow model for reconstructing audio waveforms from mel spectrograms or discrete tokens (EnCodec codes). The authors design their model to operate on slightly overlapping subbands, allowing for parallelized inference at greatly improved computational efficiency without introducing artifacts at the borders between subbands. The authors propose an energy-balanced loss and several auxiliary losses to enhance the output quality. The method shows highly competitive performance on the evaluated tasks compared to SOTA baselines, at an improved tradeoff between quality and inference speed.
weaknesses: Some claims and described methodology lack clarity and should be expanded upon or rewritten more clearly.

* The description of the conducted subjective listening experiment is extremely terse with only a single sentence (lines 352-354). In my opinion, this is insufficient as several details here are key for the reproducibility and reliability of these results. It is necessary to add details on the used platform/service for evaluation, the number of listeners, the number and type of samples per listener, and ideally also the audio equipment used by listeners.

* Figure 1 is insufficient for understanding the model architecture. Specifically, I did not understand how the fusion of the $X_t$ and $C$ occurs before the linear block. The EnCodec bandwidth index $i_{bw}$ also occurs without further motivation or explanation here and is somewhat unclear.

* The relationship between the PQMF bank (lines 207-221) to the complex spectrogram feature-space representation (see abstract) is somewhat unclear. What is really used as input/output of each model variant?

* I appreciate that the authors provide an online demonstration, which is essential for the trustworthiness of the results. However, the selection of samples is suboptimal, particularly for speech where the ground truth signals are of low quality and affected by artifacts. 
Please replace these or add new samples of higher ground-truth quality.

* The authors argue that using lower-frequency bands to predict higher-frequency bands can negatively impact the higher-frequency bands (lines 142-146), but later also argue that doing so can potentially increase consistency between bands (lines 280-281). Conditioning higher on lower bands is also not the same as processing all bands jointly (since there, also lower bands are estimated based on higher bands), so I do not fully follow the argument made here. The exact argument pro/contra processing low- and high-frequency bands jointly (or not) is somewhat unclear - please clarify.

* The authors seem to have convincing results for their "equal straightness" variant (Table 5), but somehow do not include these (at least) in Tables 6 and the comparison with diffusion-based methods, as they also note in the footnote on page 7. It is unclear to me why this is. Please update all metric results to present what the final RFWave variant is consistently, or list both the equal-straightness and non-equal-straightness variants in all relevant experiments for comparison.

* While the objective and subjective metrics speak for the method, from the online demo I find that the authors' method can introduce a kind of colorization and phasy artifacts that are not present in other methods, which are however not discussed in the paper. For transparency, could the authors add a short discussion of this limitation and potential reasons for it?

Meta-review: The paper proposes a generative model for audio signals named RFWave, a diffusion-type solution. Inspired by rectified flow studies, the authors introduce an enhanced sampling strategy and novel loss functions designed to generate high-quality audio signals, addressing challenges such as ensuring that nearly-silent portions of the signal do not produce noisy outputs. RFWave employs a convolutional architecture based on ConvNeXtV2 and outperforms various baseline methods, including GAN and diffusion-based solutions, across a comprehensive set of experiments. Achieving these results with only ten sampling steps makes it a particularly remarkable solution.  
The paper was reviewed by five experts, and the consensus among reviewers was to accept the work. The AC agrees with the reviewers' evaluation and recommends acceptance.  **Congratulations** on this excellent work! Please revise the study based on the author-reviewer discussions. Once again, congratulations!

Decision: Accept (Poster)