RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction
Authors: Peng Liu
Dongyang Dai
Zhiyong Wu
Venue: ICLR 2025 Poster (2025)
Keywords: Rectified Flow
Audio Waveform Reconstruction
Multi-band audio generation，Real-time diffusion  Vocoder
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: RFWave, a frame-level multi-band Rectified Flow model, achieves high-fidelity audio waveform reconstruction from Mel-spectrograms or discrete tokens, with generation speeds up to 160 times faster than real-time on a GPU.

Abstract: Recent advancements in generative modeling have significantly enhanced the reconstruction of audio waveforms from various representations. While diffusion models are adept at this task, they are hindered by latency issues due to their operation at the individual sample point level and the need for numerous sampling steps. In this study, we introduce RFWave, a cutting-edge multi-band Rectified Flow approach designed to reconstruct high-fidelity audio waveforms from Mel-spectrograms or discrete acoustic tokens. RFWave uniquely generates complex spectrograms and operates at the frame level, processing all subbands simultaneously to boost efficiency. Leveraging Rectified Flow, which targets a straight transport trajectory, RFWave achieves reconstruction with just 10 sampling steps. Our empirical evaluations show that RFWave not only provides outstanding reconstruction quality but also offers vastly superior computational efficiency, enabling audio generation at speeds up to 160 times faster than real-time on a GPU. Both an online demonstration and the source code are accessible.

Published as a conference paper at ICLR 2025
RFWAVE: MULTI-BAND RECTIFIED FLOW FOR AUDIO WAVEFORM RECONSTRUCTION
Peng Liu
Dongyang Dai Individual Researcher accum, dai dai@gmaill com
TEX AI, Transsion
peng. www.siiutrrssiiinncoom
Zhiyong Wu
Shenzhen International Graduate School, Tsinghua University zywuusz. t tginguua www.gageuuu.... edu.cn
ABSTRACT
Recent advancements in generative modeling have significantly enhanced the recon- struction of audio waveforms from various representations. While diffusion models are adept at this task, they are hindered by latency issues due to their operation at the individual sample point level and the need for numerous sampling steps. In this study, we introduce RFWave, a cutting-edge multi-band Rectified Flow approach designed to reconstruct high-fidelity audio waveforms from Mel-spectrograms or discrete acoustic tokens. RFWave uniquely generates complex spectrograms and operates the frame level, processing all subbands simultaneously boost efficiency. Leveraging Rectified Flow, which targets a straight transport trajectory, RFWave achieves reconstruction with just 10 sampling steps. Our empirical eval- uations show that RFWave not only provides outstanding reconstruction quality but also offers vastly superior computational efficiency, enabling audio genera- tion at speeds up to 160 times faster than real-time on a GPU. Both an online demonstration and the source code are accessible'.
INTRODUCTION
Audio waveform reconstruction significantly enhances the digital interactions by enabling realistic voice and sound generation for diverse applications, This technology transforms low-dimensional features, derived from raw audio data, into perceptible sounds, improving the audio experience on various platforms such as virtual assistants and entertainment systems. Autoregressive models and Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have been applied to this task, greatly advancing audio quality beyond traditional signal processing methods (Kawahara et al., 1999; Morise et al., 2016). Autoregressive methods, while effective, are hindered by slow generation speeds due to their sequential prediction of sample points (Oord ct al., 2016; Kalchbrenner et al., 2018; Valin & Skoglund, 2019). In contrast, GANs predict sample points in parallel, resulting in faster generation speeds and maintaining high-quality output (Kumar et al., 2019; Yamamoto al., 2020; Kong et al., 2020a; Siuzdak, 2023; Du et al., 2023). Consequently, GAN-based methods deliver impressive performance and are extensively utilized in real-world audio generation applications.
Despite the advancements, GAN-based waveform reconstruction models face challenges such as the necessity for complex discriminator designs and issues like instability mode collapse (Thanh-Tuung et al., 2018). In response, diffusion models for waveform reconstruction have been explored, offering stability during training and the ability to reconstruct high-quality waveforms (Chen et al., 2020; Kong et al., 2020b; gil Lee et al., 2022; Nguyen et al., 2024; Huang et al., 2022; Koizumi et al., 2022). However, these models are least an order of magnitude slower compared to GANs. The slow generation speed in these diffusion-
waveform reconstruction models is primarily due to two factors: (1) the requirement numerous sampling steps to achieve high-quality samples, and (2) the operation at the waveform sample point level. The latter often involves multiple upsampling
Demo: https: http:///waaeaeee deneveeddmm... demo. gf thub, ia/rfwave Code: https: / wtt.ttt.bb...t/////////tt com /bfsl 1 Ffaaveee

---

Published as a conference paper at ICLR 2025
operations to transition from frame rate resolution to sample rate resolution, increasing the sequence length and consequently leading to higher GPU memory usage and computational demands.
In this paper, propose RFWave, diffusion-type waveform reconstruction method designed to match the speed of GAN-based methods while maintaining the training stability and high sample quality of diffusion models. To overcome the challenge of slow sampling, employ Rectified Flow (Liu et al., 2023; Lipman et al., 2023; Albergo & Vanden- BindennEijnde,, 2023), which connects data and noise along straight line, thereby enhancing sampling efficiency. To address the GPU memory and computational demands of sample point-level modeling, our model operates at the level of Short- Time Fourier Transform (STFT) frames, enabling more efficient processing and reducing GPU memory usage. RFWave with only 10 sampling steps can generate high high-quality audio, achieving an inference speed of up to 160 times real-time on an NVIDIA GeForce RTX 4090 GPU. Additionally, the incorporation of three enhanced loss functions and an optimized sampling strategy further elevates the overall quality of the reconstructed waveforms. To our knowledge, RFWave stands as the fastest diffusion-based audio waveform reconstruction model, and it delivers superior audio quality, Furthermore, it can reconstruct waveforms from Mel-spectrograms and discrete acoustic tokens, enhancing its versatility and applicabilityy in various audio generation tasks. Our main contributions are as follows:
1. By integrating the Rectified Flow and enhanced loss functions energy-balance loss, overlap loss, and STFT loss our model can reconstruct high-quality waveforms with drastically reduced number of sampling steps.
2. We utilize multi-band strategy, coupled with the high-efficiency ConvNeXrV2 (Woo et al., 2023) backbone, to generate different subbands concurrently. This not only assures audio quality by circumventing cumulative errors, but also boosts the synthesis speed.
3. Our model operates at the level of STFT frames, not individual waveform sample points. This approach significantly accelerates processing and reduces GPU memory usage.
4. We propose technique for selecting sampling time points based on the straightness of the Rectified Flow transport trajectories, which enhances sample quality for free.
2 BACKGROUND
In this section, we describe the basic formulation of the Rectified Flow and present releated works, while also detailing our correlation and distinction.
Rectified Flow (Liu et al., 2023) presents innovative Ordinary Differential Equation (ODE)-based framework for generative modeling and domain transfer. It introduces a method learn a mapping that connects two distributions, mo and ni on Rd based empirical observations:
Rectified Flow
d7 dt
v( t),,t), initialized from Zo ~ To. such that Z₁ ~ Ri,
where v: Rd [0, + Rd represents a velocity field. The learning of this field involves minimizing a mean square objective function,
[h d di x, -
min Exxo
with X. = ((xo (),
(2)
where x;
= 6 6X,, X1, represents a dime-diffeeentiable interpolation between X, and X1, with d dxx, = à, t). The y represents any coupling of (To, π1). An illustrative instance of y is the independent coupling 7 no Ti, which allows for empirical sampling based separately observed data from xo and n Liu et al. (2023) recommended a simple choice of
d X, x Xo- di
X, = (1 l)x, +X1
This simplification results in linear trajectories, which are critical for accelerating the inference process. Typically, the velocity field is represented using a deep neural network. The solution to (2) is approximated through stochastic gradient methods. To approximate the ODE presented in (1),
2

---

Published as a conference paper at ICLR 2025
numerical solvers commonly employed. A prevalent technique is the forward Euler method. This approach computes values using the formula
Z = Zz 1 ().,)) 71
14 (0,..., (o,...,n-1]// 13/n,
where the simulation is executed with step interval of = 1/n over n steps.
The velocity field has the capacity to incorporate conditional information, which is particularly essen- tial in applications such as text-to-image generation and waveform reconstruction from compressed acoustic representation. Consequently, in such contexts, v [ t) in (2) is modified to 1 (Z C). where C' represents the conditional information pertinent to the corresponding X1.
Diffusion Models for Audio Waveform Reconstruction Diffusion models (Song et al., 2020; Ho et al., 2020) have become the de-facto choice for high-quality generation in the realm of generative models. Diffwave (Kong et al., 2020b) and WaveGrad (Chen et al., 2020) were pioneering efforts to reconstruct waveforms using diffusion models, achieving performance comparable to autoregressive models and GANs. PriorGrad (gil Lee al., 2022) enhances both speech quality and inference speed by employing a data-dependent prior distribution. Mcanwhile, FreGrad (Nguyen et al., 2024) simplifies the model and reduces denoising time through the use of Discrete Wavelet Transform (DWT). Multi-Band Diffusion (MBD) (Roman et al., 2023) leverages a diffusion model to reconstruct waveforms from discrete EnCodec (Défossez et al., 2022) tokens. Despite these advancements, the current fastest generation speed of these methods is only about 10 to 20 times faster than real- time, which limits their application for real-time use, especially when combined with large-scale Transformer-basee acoustic models (Wang et al., 2023; Du et al., 2024). In contrast, our method achieves speeds up to 160 times faster than real-time, significantly enhancing real-time applicability.
Estimating Complex Spectrograms Waveform reconstruction from complex spectrograms can be effectively achieved using the ISTFT. Notably, Vocos (Siuzdak, 2023) and APNet2 (Du et al., 2023), utilizing GANs their model framework, estimate magnitude and phase spectrograms from the input Mel-spectrograms, which can be transformed to complex spectrograms effortlessly. Both models operate at the frame level, enabling them to achieve significantly faster inference speeds compared to HiFi-GAN (Kong et al., 2020a), which uses multiple upsampling layers and operates at the level of waveform sample points. Morever, these models preserve the quality of the synthesized waveform, demonstrating their superiority in both speed and fidelity without a trade-off. In this paper, we directly estimate complex spectrograms using Rectified Flow and focus on frame-level operations, aiming to enhance both the efficiency and quality of our waveform synthesis process. Notably, the distributions of real and imaginary parts of the complex spectrograms appear more homogeneous in comparison to the distributions magnitude and phase.
Multi-band Audio Waveform Reconstruction Both Multi-band MelGAN (Yang et al., 2021) and Multi-band Diffusion (Roman et al., 2023) employ multi-band strategies for different purposes. Multi- band MelGAN uses Secudo-Quadrature Mirror Filters (PQMF) (Johnston, 1980) to divide frequency bands. This division results in each subband's 's waveform being fraction of the original waveform's 's length, based on the number of subbands. By reshaping these subbands into feature dimensions and using a unified backbone, it operates on shorter signals, enhancing efficiency during training and inference. Multi-band Diffusion instead uses band-pass filters to separate frequency bands and models each subband distinctly, preventing cross-band error propagation. Our approach simplifies frequency band division by directly selecting appropriate dimensions from complex spectrograms. We improve efficiency by modeling all subbands together in parallel with a single unified model, increasing processing speed while reducing error accumulation across subbands.
3
METHOD
Our model utilizes a multi-band Rectified Flow with a ConvNeXtV2 (Woo et al., 2023) backbone, offering two alternative modeling choices with noisy samples in time or frequency domain while the neural network consistently operates at the STFT frame level. With only 10 steps of sampling, the model is capable of producing high-quality waveforms. In this section, present the structure of the multi-band Rectified Flow model, describe the corresponding normalization techniques, three enhanced loss functions, and the strategy for selecting sampling time points.
3

---

Published as a conference paper at ICLR 2025
ISTFT
Harga Subband
Puintoios Cort
Linwer
GELU
CureNeat122 Blocka
Pointvise
Ada Layar
Depttwets Corr
Divide Subbond
STFT
Embed
Embas
Figure 1: The overall structure for RFWave. is is the subhand index, C' is the conditional input, which can be an Encodee token or -ep-secctoo and few is the EnCodec handwidth index. Modules enclosed in dashed hox, well dashed arrows, are considered optional.
3.1 MULTI-BAND RECTIFIED FLOW
All frequency bands, each dis- tinguished by unique subband index, share the same model. The subbands of a given sample are grouped together into single batch for processing, which facilitates simultaneous training or inference. This significantly reduces inference latency. Moreover, independently modeling the subbands reduces error accumulation. As discussed in (Roman et al., 2023), conditioning higher bands on lower ones can lead to an error accumulation, which means inaccuracies in the lower bands can adversely affect the higher bands during inference.
Model Structure The model structure is depicted in Figure 1.
The model maps a noisy sample (X,) to its velocity (wt). For each subband, the subband's noisy sample (X²**) ) is fed into the ConvNeXtV2 backbone to predict its velocity (u/=*) conditioned on time (t), the subband index (iab), the conditional input (C, the Mel-spectrogram the EnCodec (Défossez et al., 2022) tokens), and an optional EnCodec bandwidth index (fbw). The detailed structure of the ConvNeXtV2 backbone is shown in Figure 1. We employ Fourier features described in (Kingma et al., 2021). The X1 C, and Fourier features concatenated along the channel dimension and then passed through linear layer, forming the input that is fed into a series of ConvNeXtV2 blocks. element wise added to
The sinusoidal embedding, along with the optional êbw embedding, the input of each ConvNeXrV2 block. The Vew is utilized during the decoding of EnCodec tokens, enabling a single model to support EnCodec tokens with various bandwidths. Furthermore, the is is incorporated via an adaptive layer normalization module, which utilizes Iearnable embeddings as described in (Siuzdak, 2023; Xu al., 2019). The other components identical to those within the ConvNeXrV2 architecture, details can be found in (Woo et al., 2023).
Our methodology offers two modeling options. The first involves mapping Gaussian noise to the waveform directly in the time domain, wherein Xo. X1. X, and all reside in the time domain. The second option maps Gaussian noise to the complex spectrogram, placing Xo, X,, X, and i2 in the frequency domain. Notably, X7' and u/ are consistently represented in the frequency domain, as detailed in the following paragraphs, ensuring that the neural network runs at the frame level. By processing frame-level features, our model achieves greater memory efficiency compared to diffusion vocoders like PriorGrad, which operate at the level of waveform sample points. While PriorGrad (gil Lee et al., 2022) can train on 6-second2 audio clips at 44.1 kHz within GB of GPU memory, our model is capable of handling 177-sccond clips with the same memory resources.
Operating with X, in the Time Domain and Waveform Equalization Since our model is designed to function at the frame level, when X, and v are in the time domain (specially, X, is the waveform and Xo noise of the identical shape, with X, and ot derived from (3)), the use of STFT and ISTFT, as illustrated in Figure 1, becomes necessary. The dimension of Xx and V, adhere to [1, 273, where T' is the waveform length in sample points. After the STFT operation, we extract the subbands by equally dividing the full-band complex spectrogram, getting X Each subband X
TThe duration of an audio clip is calculated as the batch size multiplied by numher of segment frames and hop length, divided hy the sampling rate.
For simplicity, the batch dimension is included in the discussion.
4

---

Published as a conference paper at ICLR 2025
is then processed independently by the backbone as individual sample to predict the corresponding oith. These predictions are subsequently merged back together before the ISTFT operation. The X/ processed by the backbone thus has dimensions of [2d,, F",, where dg represents the number of frequency bins in a subband's 's complex spectrum and F the number of frames. The real and imaginary parts of each subband are interleaved to form a 2d, -iimensional feature.
White Gaussian noise has uniform energy across frequency bands, but waveform energy profiles vary significantly. For example, speech energy decays exponentially with frequency, while music maintains a consistent distribution (Schnupp et al., 201 1). These differences challenge diffusion model training, making energy equalization across bands beneficial (Roman al., 2023). In the time-domain model, a bank of Pseudo-Quadratee Mirror Filters (PQMF) is employed to decompose the input waveform into subbands. Subsequently, these subbands are equalized and then recombined to form the equalized waveform. The performance of the PQMF bank exhibits modest enhancement compared to the array of Sand-pass filters employed in (Roman et al., 2023). It's important to note that the PQMF is solely utilized for waveform equalization and holds no association with the division of the complex spectrogram into subbands. For waveform equalization, mean-variance normalization employed, utilizing the exponential moving average of mean and variance of each waveform subband computed during training. This approach ensures that the transformation can be effectively inverted using the same statisties.
Alternative Approach: Operating with X; in the Frequency Domain and STFT Normalization When X2 and e reside in the frequency domain (i.e., X, is the waveform's 's complex spectrogram and Xo is noise of the identical shape), STFT and ISTFT, as shown in Figure 1, are unnecessary. The dimensions of X, and et are [2d, FJ. where d is the number of frequency bins in the complex spectrogram. By equally partitioning the full-band complex spectrogram, X is extracted, resulting in a shape of [2d,, F), which is then processed by the ConvNeXtV2 backbone. the frequency- domain model, the waveform is transformed to complex spectrogram without equalization. The preprocessing involves the dimension-we -iie mean-variance normalization of the complex spectrogram.
During inference, operating with X, and Us in the time domain requires both STFT and ISTFT at each sampling step. In contrast, when X, and oy are in the frequency domain, only a single ISTFT is needed after the entire sampling process. Despite this computational overhead, our experiments show that the time-domain configuration achieves slightly better performance, particularly in preserving nigh-frequency details. Sampling algorithms for the two distinct approaches, one in the time domain and the other in the frequency domain, are provided in Appendix Section A.9.1.
3.2 LOSS FUNCTIONS
Energy-balanced Loss In preliminary experiments, we noticed low-volume noise in expected silent regions. We attribute this to the property of Mean Square Error (MSE) used in (2). The MSE measures the absolute distortion between the predicted values and the ground truth. In silent regions, small absolute errors contribute minimally to the overall MSE loss, so the model does not prioritize climinating them during training. This results in the model's inability to effectively suppress minor deviations in silent areas, potentially leading to perceptible noise. In contrast, larger errors in high-amplitude regions have a significant impact on the MSE loss, causing the model to focus more on reducing errors in these areas during training.
We propose energy- balanced loss to mitigate this problem. Our mneayy-bllanced loss is designed to weight errors differently depending on the region's volume (or energy) accross the time-axis. Specifically, for each frequency subband, we compute the standard deviation along the feature dimension of the ground truth velocity to construct weighting coefficient of size [1, F). This vector is reflective of the frame-level energy of the respective subband, as depicted in Figure A.I. Subsequently, both the ground truth and predicted velocity are divided by this vector before proceeding to the subsequent steps. For the frequency domain model, the training objective defined in (2) is adjusted as follows:
v(X., min E Ex 1/' 11 (XXI-X)//o- - (C)/ 11
(5)
with o = =/ / (X1 Xo) and X, = =xx + (1 - t).o.
5

---

Published as a conference paper at ICLR 2025
padding
evertop
Right overlap
Right padding
Frames
Subbbend /
Channels
Figure An illustration of dividing complex spectrograms into subbands. The area highlighted pink represents a subband, while the section enclosed by the two dashed vertical lines indicates the main section.
where D represents the dataset with paired X1 and C, and Var, calculates the variance along the feature dimension. For the time domain model, this energy balancing operation precedes the ISTFT process. This approach helps minimize the relative error in low-volume regions. Our experimental results demonstrate that this method enhances overall performance, benefiting not just the silent parts.
Overlap Loss Within the multi-band structure, each subband is predicted independently, potentially resulting in inconsistencies among them. To mitigate these inconsistencies, introduce overlap loss. This involves maintaining overlaps between the subbands when dividing the full-band complex spectrograms. A detailed illustration of this scheme provided in Figure and described in detail in Appendix Section A.3. In this paper, we employ an 8-dimensional overlap. During the training phase, the MSEs of the overlapped predictions minimized. In the inference phase, the overlaps removed, and all subbands are merged to recreate the full-band complex spectrograms.
As each subband is predicted, the model internally maintains consistency between its overlap section and the main section. The overlap serves as an anchor to maintain consistency among the subbands. Modeling higher bands based on lower bands rather than predicting subbands parallel can increase consistency between subbands. However, at the inference stage, if the lower bands are incorrectly predicted, the higher bands conditioned on them would also be inaccurate (Roman et al., 2023). RFWave addresses this by using overlap loss to maintain consistency between subbands during training. At the inference stage, subbands are predicted independently, so any error occurring in one subband does not negatively affect the others.
STFT Loss The magnitude spectrogram derived loss (Arik et al., 2018) is extensively utilized in GAN-based vocoders, such as Hili-GAN (Kong et al., 2020a) and Vocos (Siuzdak, 2023), both of which leverage Mel-spectrogram loss. Nevertheless, its use in diffusion-based vocoders is relatively rare. This might stem from its lack of direct compatibility with the formalization of a noise prediction diffusion model. Here adopt STFT loss for RFWave. According (5), the model's 's output serves as an approximation for velocity:
uX,,, X, X1 -Xo-
(6)
di
Hence, at time t, an approximation for X2 is:
X1 to
The STFT loss can be applied on the approximation X,, incorporating the spectral convergence loss and log-scale STFT-magnitude loss as proposed in (Arik et al., 2018), which detailed in Appendix Section A.4. According to our experimental findings, the STFT loss effectively reduces artifacts in the presence of background noise.
3.3 SELECTING TIME POINTS FOR EULER METHOD
Liu et al. (2023) employ equal step intervals for the Euler method, as illustrated in (4). Here, we propose selecting time points for sampling based on the straightness of transport trajectories. The time points are chosen such that the increase in straightness is equal across each step. The straightness of learned velocity field is defined as S(v) = fo E || (X1 XX) u(,,, t C) I|2 dt, which describes the deviation of the velocity along the trajectory. A smaller 5(v) means straighter trajectories. This approach ensures that the difficulty of each Euler step remains consistent, requiring the model to take more steps in more challenging regions. Using the same number of sampling steps, this method outperforms the equal interval approach. The implementation details are provided in Appendix Section A.5, while the algorithm is presented in Appendix Section A.9.2. We refer to this approach equal straightness.

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS AND ANALYSIS
4.1 EXPERIMENTAL SETUP
Overview We evaluate RFWave using both -el-spectss and discrete EnCodec tokens as inputs. For Mel-spectrogram imputs, we first benchmark RFWave against existing diffusion vocoders to demonstrate its superiority. We then compare it with widely-used GAN models to highlight its practical applicability and advantages. For discrete EnCodec tokens, we evaluate RFWave's efficiency in reconstructing high-quality audio from compressed representations across diverse domains. Finally, we conduct ablation studies and further analysis to examine the effects of the individual components of RFWave.
Data For Mel-spectrogram inputs, we conduct two evaluations. When benchmarking against diffusion vocoders, we train separate models on LibriTTS (Zen et al., 2019) (speech), MTGG-Jamendo (Bogdanov et al., 2019) (music), and Openepop (Wang al., 2022) (vocal) datasets and test each model on its respective dataset to ensure comprehensive comparison across various audio categories. When comparing RFWave to widely used GAN-based models, we train a model on LibriTTS and evaluate its in-domain performance on the LibriTTS test set. Additionally, we assess the out-of- domain generalization ability of this LibriTTS-traine model by testing it on the MUSDB18 (Rafi et al., 2017) test subset.
For discrete EnCodee token inputs, we follow convention by training a universal model on a large- scale dataset. This dataset combines Common Voice 7.0 (Ardila et al., 2019) and clean data from DNS Challenge 4 (Dubey et al., 2022) for speech, MTG-Jamendo (Bogdanov et al., 2019) for music, and FSD50K (Fonseca et al., 2021) and AudioSet (Gemmeke et al., 2017) for environmental sounds. Recognizing the lack of a comprehensive test set for universal audio codec models, we constructed unified evaluation dataset comprising 900 test audio samples from 15 external datasets, covering speech, vocals, and sound effects. Detailed information about this test set is provided in Table A.4.
Baseline and Evaluation Metrics We conducted a comprehensive benchmark comparison of RFWave against a series of state-of-thh-arr models. For Mel-spectrogram inputs, we utilized PriorGrad (gil Lee et al., 2022) and FreGrad (Nguyen et al., 2024) as baselines for diffusion-basel models. Vocos (Siuzdak, 2023) and BiAV (gil Lee al., 2023) served as baselines for GAN-based methods, reflecting their widespread use in real-world applications. For discrete EnCodec token inputs, we compared with EnCodec (Défossez et al., 2022) and MBD (Roman et al., 2023).
We trained PriorGrad and FreGrad on LibriTTS, Opencpop, and MTG-Jamendo using their open- source details. For Vocos, BigVGAN, EnCodec, and MBD, we used public pre-trained models. We followed the authors' recommended sampling steps: 6 for PriorGrad, 50 for FreGrad, and 20 for MBD. Model sources are in Table A.2.
ViSQOL (Chinen et al., 2020) to assess perceptual quality, employing speech-mode for 22.05/24 kHz waveforms and audio-mode for 44.1 kHz waveforms. Additional metrics include the Perceptual Evaluation of Speech Quality (PESQ) (Rix et al., 2001), the F1 score for voiced/unvoiced classification (V/UV F1), and the periodicity error (Periodicity) (Morrison et al., 2022). For subjective evaluation, we conduct crowd-sourced assessments, employing 5-point Mean Opinion Score (MOS) to determine the audio naturalness, ranging from 1 ((poor/unnatural)) to 5 ("excellent/natural)) ), more details provided in the Appendix Section A.
For objective evaluation,
Implementation We use the time-domain model with three enhanced loss functions as the default. The RFWave backbone contains 8 ConvNeX(V2 blocks, and the complex spectrogram is divided into 8 equally spanned subbands. For evaluation, we use 10 sampling steps. Table A.3 lists the parameters used for extracting Mel-spects and complex spectrograms from datasets with different sample rates. Further implementation details and computational resource requirements are provided in the Appendix Sections A.I and A.2, respectively.
7

---

Published as a conference paper at ICLR 2025
4.2 EXPERIMENTAL RESULTS ON MEL-SPECTROGRAMS INPUT
Comparison with Diffusion-based Method In this portion, we compare RFWave* with PriorGrad and FreGrad by training separate models for each method on the LibriTTS, MTG-Jamendo, and Opencpop datasets and evaluating each model on its respective test set. Table 1 presents overall results, while Table A.5 provides detailed metrics for each audio category. RFWave consistently outperforms PriorGrad and FreGrad in both objective and subjective metrics. The superior MOS score achieved by RFWave is likely due to its ability to produce clearer and more consistent harmonies, particularly in feghffreuency ranges, which contributes to better overall audio quality. A collection of spectrograms and their corresponding analyses can be found in Appendix Section A.6.
Comparison with GAN-based Method When comparing RFWave to widely used state-of-the-an GAN-based models (BigVGAN, Vocos), we used models trained on LibriTTS. Besides evaluating on the LibriTTS testset, to assess the models' robustness and extrapolation capabilities, following the methodology in (gil Lee et al., 2023), we also evaluated these models' performance on the MUSDB 18 dataset. It can be observed in Table 2 that the in-domain test set (LibriTTS), RFWave is generally models. However, on the out-of-domain test set (MUSDB 18), on par with state-of-the-an art GAN- according to the results in Table 3, RFWave shows significant advantages over BigVGAN and Vocos, even though BigN employed Snake activation (Ziyin et al., 2020) to enhance its out-of-domain data generation capabilities. This demonstrates that RFWave, as a diffusion-type model, offers clear advantages in generalization and robustness compared to GAN-based models. For detail, we observe that GAN-based methods tend to generate horizontal lines in the high-frequency regions of the spectrograms, as exemplified in Figure A.7. In contrast, RFWave consistently produces clear fegg-freyuency harmonics, even when applied to oft-of-domain data.
Table I: Average Mean Opinion Score (MOS) and objective evaluation metrics for RFWave, PriorGrad and FreGrad across various test sets. MOS is provided with 95% confidence interval.
Model
MOS
PESQ f
ViSQOL
VUV F1 f
Periodicity
RFWave
3.95
4.202
4.456
0.979
0.070
PriorGrad
3.75
3.612
4.347
0.974
0.082
FreGrad
2.99:10.14
3.640
4.179
0.973
0.087
Ground truth
4.00.10.09
Table 2: MOS and objective evaluation metrics for RFWave, BigVGAN and Vocos on LibriTTS.
Model
MOS f
PESQ f
ViSQOL f
V/UV
Periodicity J
RFWave
3.82-10.12
4.304
4.579
0.967
0.091
BigVGAN
3.78
4.240
4.712
0.978
0.067
Vocos 3.74
3.660
4.696
0.958
0.104
Ground truth
3.91
Table MOS for RFWave, BigVGAN and Vocos MUSDB18.
Model
Vocals
Drums
Bass
Others
Mixture
Average
RFWave
3.46=0.14
3.65=0.10
3.62#0.10
3.54
4.04
3.67=0.05
BigVGAN
3.42+0.13
3.68+0.09
3.50+0.12
3.33+0.13
3.58
3.51+0005
Vocos
3.04.10.15
3.54
2.9610.12
2,8810.15
3.08
3.10.1006
Ground truth
3.62+0.14
3.71
3.75+0.12
3.79+0.12
4.17=0.10
3.80+0005
4.3 EXPERIMENTAL RESULTS ON DISCRETE ENCODEC TOKENS INPUT
We evaluated RFWave against EnCodec and MBD across various bandwidths for discrete EnCodec tokens inputs. For RFWave, used Mussifier--Fres Guidance (CFG) (Ho & Salimans, 2021) with
"Tii evaluation used an carlier RFWave version without equal straightness from Section 3.3. Despite this, the results convincingly demonstrate RFWave's superiority over other diffusion vocoders.
8

---

Published as a conference paper at ICLR 2025
2.0 coefficient. CFG showed little improvement for Mel-spectrogram inputs, but effective for EnCodec tokens, likely due to CFG tending to be more impactful when the input is more compressed.
Table shows average MOS and objective metrics for EnCodec tokens across bandwidths and auditory contexts, while Table A.6 provides detailed results for each category. RFWave achieves optimal scores in all metries except ViSQOL, where EnCodec's GAN-based decoder excels. ViSQOL shows a subtle bias toward GAN-based models, likely due to distinct waveform footprints. While using a larger bandwidth generally improves performance, increasing bandwidth from 6.0 12.0 kbps slightly improves MOS for RFWave.
Table 4: Average MOS and objective metrics for RFWave, EnCodec and MBD various test sets.
Bandwidth
Model
MOS
PESQ
ViSQOL
V/UV FI
Periodicity.
RFWave,cra) EnCodee
3.17=0.22
1.797
3.108
0.914
0,193
1.5 kbps
2.23=10.3
1.708
3,518
0.906
0.199
MBD
3.01
1.699
2.982
0.901
0.212
RFWavecrro)
3.52::0.25
2.444
3,570
0,939
0.145
3.0 kbps
EnCodec
2.79:00.25
1.934
3,793
0.930
0.166
MBD
3.06:20.23
2.310
3,402
0.922
0.171
RFWave
3.69=0.16
2.936
3.892
0.954
0.117
6.0 kbps
EnCodec
3.10
2.432
4.091
0,951
0.126
MBD
3.43
2.488
3.582
0.929
0.168
RFWavG,cKиz
3.73=0.16
3.270
4.124
0.965
0,099
12.0 kbps
EnCodee
3.55=0015
2.892
4.291
0,963
0.105
MBD
Ground truth
4.07
4.4 ANALYSIS
Ablations We evaluate the model in frequency and time domains, then analyze the impact of incrementally adding the three loss functions and equal straightness to the better-performing model. Ablation studies on LJSpeech (Ito & Johnson, 2017) (250 test sentences) are summarized Table Despite the metrics not consistently correlating with human evaluations, they accurately captured the quality improvements brought about by design modifications (Roman et al., 2023). The model operating in the time domain outperforms its frequency domain counterpart, as the ISTFT operation the former (Figure 1) introduces periodic signals. The inverse Discrete Fourier Transform (IDFT) matrix comprises sinusoidal functions of frequencies [0, 1/N, (N-1)/N) 1)/N]. where N the number of FFT points (Oppenheim & Schafer, 1975). This mechanism is similar to the Snake activation (Ziyin al., 2020) in BigVGAN (gil Lee al., 2023), which also introduces periodic signals into the generator. Energy- balanced loss and equal straightness improves the overall performance, STFT loss improves PESQ but degrades ViSQOL and periodicity due to its emphasis on magnitude over phase. This trade-off is essential for effectively eliminating artifacts, especially in the presence of background noise. While overlap loss slightly lowers PESQ, ensures smooth subband transitions. The effects visualized in Figures A.8 and A.9.
Table Objective metrics assess model's performance in the frequency domain the time domain, where three loss functions and equal straightness are applied incrementally.
Setting
PESQ 1
ViSQOL
VUV F1
Periodicity
frequency
3.872
4.430
0,948
0.144
time
4.127
4.551
0,957
0.124
+ cnergy-balanced loss
4.181
4.598
0.965
0.110
+ overlap loss
4.1.88
4.599
0.959
0.122
+ STFT
4.211
4.578
0.961
0.119
+ equal straightness
4.275
4.654
0,966
0.100
Futher Analysis Using the model with an 8-layer ConvNeXtV2 backbone, time domain, and three loss functions as the baseline, we first experimented with the DDPM (Ho al., 2020) approach.

---

Published as a conference paper at ICLR 2025
Utilizing the noise schedule from DiffWave and PriorGrad, the performance with 50 sampling steps fell short of the baseline, as demonstrated in Table 6. This highlights that Rectified Flow is critical for model to efficiently generate high-quality audio samples. Next, experimented with ResNet (He et al., 2016) backbone with a similar number of parameters. This configuration slightly reduced efficiency and performance in objective metrics (Table 6), indicating that the ConvNeXtVV backbone enhances both efficiency and audio quality. Finally, we experimented with models of different sizes, testing ConvNextV2 backbones half and double sizes, with configurations in A.I. Increasing backbone size improves performance but reduces efficiency (Table 6).
Table 6: Objective metrics for further analysis. xRT stands for the speed at which the model can generate speech in comparison to real-time.
Setting
GPU xRT
PESQ
ViSQOL f
VUVV FI f Periodicity J
baseline
162.59
4.211
4.578
0.961
0.119
DDPM, 50 steps
34,40
2.790
4.499
0.939
0.174
ResNet
150.95
3.865
4.509
0.955
0.128
half size
228.70
4.128
4.553
0.959
0.120
double size
111.62
4.285
4.636
0.968
0.100
4.5 INFERENCE SPEED
We perform inference speed benchmark tests using an NVIDIA GeForce RTX 4090 GPU. The implementation was done in PyTorch (Paszke et al., 2019), and no specific hardware optimizations were applied. The inference was carried out with batch size sample, utilizing the LJSpeech test set, resampled to model's sampling rate. Table 7 displays the model size, synthesis speed and GPU memory consumption of the models. RFWave is more than twice as fast as BigVGAN and consumes less GPU memory, thereby eliminating latency barrier for practical applications. This speed advantage becomes even more pronounced when synthesizing high-resolution audio (44. 1/48 kHz). Frame-level models like RFWave significantly outperform sample-point-haie baaedd models in high- resolution audio synthesis in terms of speed. RFWave can easily adjust the complex spectrogram's 's window length and hop length for 44.1 kHz sampling, maintaining low computational complexity, whereas Big VGAN requires additional upsampling blocks. Vocos serves as a strong baseline, given its efficiency in requiring only a single forward pass and operating at the frame level.
Table Model footprint and synthesis speed. xRT stands for the speed at which the model generate speech comparison to real-time.
Model
Parameters (M)
GPU xRT
GPU Memory (MB)
Sampling steps
RFWave
18.1
162.59
780
10
BiAN
107.7
72.68
1436
VocosasIFT
13.5
2078.20
590
PriorGrad
2.6
16.67
4976
6
FreGrad
1.7
7.50
2720
50
MBD
411.0
4.82
5480
20
RFWave IKH2)
20.5
152.58
902
10
Big (GAN
116.5
39.03
1740
5 CONCLUSION
this study, we propose RFWave, a multi-band Rectified Flow approach for audio waveform recon- struction. The model has been carefully designed to overcome the latency issues associated with diffusion models. RFWave stands out for its ability to generate complex spectrograms by operating at the frame level, processing all subbands concurrently. This concurrent processing significantly hances the efficiency of the waveform reconstruction process. The empirical evaluations conducted in this research have demonstrated that RFWave achieves exceptional reconstruction quality. Moreover, it has shown superior computational efficiency by generating audio at a speed that is 160 times faster than real-time comparable to GAN-based methods, making it practical for real-worle applications.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Michael S. Albergo and Eric Vanden-Fijnden. Building normalizing flows with stochastic inter- polants. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https////peeeeeeeeeeecoee net/
Eorum?id-117qeebbBR
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively- multilingual speech corpus. arXiv preprint arXiv: /9ti:.:9//..0 (e..//922.6667 2019.
Serean Arik, Heewoo Jun, and Gregory Diamos. Fast spectrogram inversion using multi-hews head convolutional neural networks. IEEE Signal Processing Letters, PP:1-1, 11 2018. doi: 10. 109/ S.P.2018.2880222
Evelina Bakhturina, Vitaly Lavrukhin, Boris Ginsburg, and Yang Zhang. Hi-fi multi multi-speaker english Its dataset. arXiv preprini arXiv:2104.01497, 2021.
Dawn AA Black, Ma Li, and Mi Tian. Automatic identification of emotional cues in chinese opera singing. ICMPC, Scoul, South Korea, 2014.
Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. The mtg- jamendo dataset for automatic music tagging. In Machine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML 2019), Long Beach, CA, United States, 2019. URL http:////// http//hhhhaaanneeeeeeeeeeeaaa handle wwt/ /021//2222//215.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.007 00713, 2020.
Michael Chinen, Felicia S. Lim, Jan Skoglund, Nikita Gureev, Feargus O' O'Gorman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric. In 2020 Twelfth International Conference on Quality of Multimedia Experience (QoMEX), pp. 1 1-6, 2020.
doi: 10. 1110906MEXX48332220000
Soonbeom Choi, Wonil Kim, Saebyul Park, Sangeon Yong, and Juhan Nam. Children's 's song dataset for singing voice research. In International Society for Music Information Retrieval Conference (ISMIR), volume 4, 2020.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. CoRR, abs/2210. 2022, doi: 10.46550/AXX..2100.55333 URL https: //dpii 48440855555XA 2210 13438.
Hui-Peng Du, Ye-Xin Lu, Yang Ai, and Zhen-Hua Ling. Apnet2: High-quality and high- efficiency neural vocoder with direct prediction of amplitude and phase spectra. arXiv preprint arXiv:2311.11545 2023.
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu. Ziyang Ma. Zhifu Gao, and Zhijie Yan. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. CORR, hix/2407.054001 2024. doi: 10. (0.48550//RXX1.2447...5
URL https: //doii http//....550/axxii 2407.
05407.
Harishchandra Dubey, Vishak Gopal, Ross Cutler, Ashkan Aazami, Sergiy Matusevych, Sebastian Braun, Scfik Emre Eskimez, Manthan Thakker, Takuya Yoshioka, Hannes Gamper, and Robert Aichner. Icassp 2022 deep noise suppression challenge, 2022.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions Audio, Speech, and Language Processing, 30:829 852, 2021.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
11

---

Published as a conference paper at ICLR 2025
Sang gil Lee, Heeseung Kim, Chachun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie-Yar Liu. Priorgrad: Improving conditional denoising diffusion models with data-dependent adaptive prior. In International Conference Learning Representations. 2022. URL https::///........ / openrevi ew net/ Barrrummid=_ANNAAATE
Sang gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. BigVGAN: A universal neural vocoder with large-scale training. In The Eleventh International Conference on Learning Representations, 2023, URL https: opppreeeeeee www.riewweeeeee..ee...e. forrmmoruum////mm DrrruumiidiiTTttMEEE.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Ad- vances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL https:// https://prooeedings neurips cc/ cc/paper.... fiwes/ Files//aperr2014/ file//casee12221558809994s4aaa.2
Papeer. www.teccff.aapp.p....... .pf.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Paltern Recognition (CVPR), pp. 770-778, 2016.
doi: 10.1 10.1 109/CVPR.0016.9000
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. URL https: openreview. net/ foru/fo qw BAKxfYbI.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
Rongjie Huang, Max W. Y. Lam, Wang, Dan Su, Dong Yu, Ren, and Zhou Zhao. Fastdiff: A fast conditional diffusion model for high-quality speech synthesis. In International Joint Conference on Artificial Intelligence, 2022. URL https: /api. 248300058.
org/ wwtg/orpprIID.......
Eric J. Humphrey, Simon Durand, and Brian McFee. Openmic-2018, September 2018. URL https: doo. wrggro.g........................ 1tt...221//eeeeco 1432 9.3.
Keith Ito and Linda Johnson. 1L-Speeca-Dattset/.200 2017.
The lj speech dataset.
httpss
Johnston. A filter family designed for use in quadrature mirror filter banks. In ICASSP '80. IEEE International Conference on Acoustics. Speech, and Signal Processing, volume 5, pp. 291-294, 1980. doi: 10. 109/CCSSP 1171025.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In International Conference on Machine Learning, pp. 2410-2419. PMLR, 2018
Hideki Kawahara, Ikuyo Masuda-Katsuse, and Alain De Cheveigne. Restructuring speech represen- tations using pitch-adaptive time-freeuue frequency smoothing and instantaneous-frequeenng based f0 extraction: Possible role repetitive structure in sounds. Speech communication, 27(3-4): 187-207, 1999.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, volume 34, pp. 21696-21707, Curran Asso- URL
ciates, /
neur ips cc /appeerrrppppeeeeepeeeeeeeeeee silesspaper. Pappeaaaaee...aaaaaaaae.......
Yuma Koizumi, Heiga Zen, Kohei Yatabe, Nanxin Chen, and Michiel Bacchiani. Specgrad: Diffusion probabilistic model based neural vocoder with adaptive noise spectral shaping. In Hanseok Ko and John H. L. Hansen (eds.), 23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022, pp. 803 807. ISCA, URL https: doi .or./org///oo ott.oooo//0.....991
2022. doi: 10.1137INNTERSPEECH..0022 Interspeech. 2012-301.
12

---

Published as a conference paper at ICLR 2025
Jungil Kong, Jachyeon Kim, and Jackyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-17033. 2020a.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint 20Xx::009.09761. 2020b.
Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre De Brebisson, Yoshua Bengio, and Aaron Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Repre- sentations, 2023, URL https: http///epeeeeeeeeeeeeeeeeeeeeeee .nw.//ee///et..... wtwpeeeeeeeeetteeeeeeeeeeeeeeeee doooumiidd-aa P
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representattinss ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https:///pperevilee net wttp////ddd----/-immd
Zhaorui Liu and Zijin Li. Music Data Sharing Platform for Computational Musicology Research (CCMUSIC DATASET), November 2021. URL https://doili http://d..o.orgo//// /10. 2281/zenodo, 5676893.
Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess): dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5)e0016339, 2018.
Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. World: rocoder-based high-quality speech synthesis system for real-time applications. IEICE TRANSACTIONS on Information and Systems, 99(7):1877-1844 2016.
Max Morrison, Rithesh Kumar, Kundan Kumar, Prem Seetharaman, Aaron Courville, and Yoshua Bengio. Chunked autoregressive GAN for conditional waveform synthesis. In International Conference on Learning Representations, 2022. URL https: /ttpa/://ppppeeeeeeeeee net /fotrm??m
id=v3aeIsY_vVX vVX.
Tan Dat Nguyen, Ji-Hoon Kim, Youngjoon Jang, Jaehun Kim, and Joon Son Chung. Fregrad: Lightweight and fast frequency-aware difftusion vocoder. International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024.
Tu Anh Nguyen, Wei-Ning Hsu, Antony d' Avirro, Bowen Shi, Itai Gat, Maryam Fazel-Zarani, Tal Remez, Jade Copet, Gabriel Synnaeve, Michael Hassid, et al. Expresso: A benchmark and analysis of discrete expressive speech resynthesis. arXiv preprint arXiv:2308.05725. 2023.
Frederico S Oliveira, Edresson Casanova, Arnalo Candido Junior, Anderson S Soares, and Arlindo R Galvão Filho. Cml-tts: A multilingual dataset for speech synthesis in low-resource languages. In International Conference on Text, Speech, and Dialogue, pp. 188-199. Springer, 2023.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint 1609::6609.000000 2016.
A. V. Oppenheim and R.W. Schafer. Digital Signal Processing. Prentice Hall international editions. Prentice-Hall, 1975. ISBN 9789380618388. URL https: /btoksooos////////// google. com/books" id=vSzuVLBbp6cC.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary Vito Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina
13

---

Published as a conference paper at ICLR 2025
Beygelzimer, Florence d' d'Alché-Buc Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8024-8035, 2019. URL https: /ttppps://prrocee http://poooeeenneeeeeeece hew.reerrennnnnnnneeeee..e...eee neurips .ttpppsc............ooo..ooooccc htcp/apppr2201100 htter///01//hhhhi 9/hahh/ bbbbc2888 fee ( -bs html.
Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference Multimedia, pp. 1015-1018, 2015.
Zafar Rafii, Antoine Liutkus, Fabian-Robert Stöter, Stylianos loannis Mimilakis, and Rachel Bittner. Musdb1 8-a corpus for music separation. 2017.
A. W.W. Rix, J.G. Beerends, M.P. Hollier, and A.P. Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 200 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221), volume 2, pp. 749-752 vol.2, 2001. doi: 10.1009/CCASPPP200...40000
Robin San Roman, Yossi Adi, Antoine Deleforge, Romain Serizel, Gabriel Synnaeve, and Alexandre Défossez. From discrete tokens to high-fidelity audio using multi-band diffusion. In Thirly-seventh Conference on Neural Information Processing Systems, 2023. URL https: https::/tppeaaaeeeett opeenevvie. net fet//oormmtii 2id 3onnkK
Jan Schnupp, Israel Nelken, and Andrew King. Auditory neuroscience: Making sense of sound. MIT press, 2011.
Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li. Aishell-3: A multi-speaker mandarin tts corpus and the baselines. arXiv preprint 1.Xi::0000.11667. 2020.
Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. CoRR, abs/2306.008144 2023, doi: 10:4550/ARXW.220660088 14.
URL https: /doi www.oogg//.... 48550/ htt.v..ivvivv//eee 2306. 00814.
Yang Song, Jascha Narain Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Er- mon, and Ben Poole. basee-based generative modeling through stochastic differential equations. ArXiv, abs/2011 10s/2011.113 13456, 2020. URL pattpss:/aplte semant icscholar. Worg/CorpusID. 227209335.
Shinnosuke Takamichi, Kentaro Mitsui, Yuki Saito, Tomoki Koriyama, Naoko Tanji, and Hiroshi Saruwatari. Jus corpus: free japanese multi-speaker voice corpus, arXiv preprint arXiv: 06248, 2019.
Hiroki Tamaru, Shinnosuke Takamichi, Naoko Tanji, and Hiroshi Saruwatari. Jvs-music: Japanese multispeaker singing-voice corpus. arXiv preprint arXiv:2001.077 0.00444 2020.
Hoang Thanh-Tung, Truyen Tran, and Svetha Venkatesh. On catastrophic forgetting and mode collapse in generative adversarial networks. CoRR, abs/1807.04015, 2018. URL http:// arxiv. org/ abs / 1807 04015.
Jean-Mare Valin and Jan Skoglund. Lpenet: Improving neural speech synthesis through linear prediction. In ICASSP 2019-2019 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5891-5899 IEEE, 2019.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech 2023.
synthesizers. arXiv preprint arXiv:2301.0211
Yu Wang, Xinsheng Wang, Pengcheng Zhu, Jie Wu, Hanzhao Li, Heyang Xue, Yongmao Zhang, Lei Xie, and Mengxiao Bi. Opencpop: A high-quality open source chinese popular song corpus for singing voice synthesis. In Hanseok Ko and John H. L. Hansen (eds.), Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pp. 4242 4246. ISCA, 2022. doi: 102147/INNTR///ECCL002222 URL https:/ //w.//do...////// htt.//ooo///oo/ooooooooo.ooooooo / 2...2.13777 Int./speeerreeeeeeeeeeeeeee 2022-48.
14

---

Published as a conference paper at ICLR 2025
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext V2: co-designing and scaling convnets with masked autoencoders. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 16133-16142. IEFE, 2023. doi: 10.1 (099/CVPR5279..222..0000 URL https::::... / /oi. org 10. 11 O9t.09//222222002 2023 01548.
Jingjig Xu, Xu Sun, Zhiyuan Zhang. Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. D'Alché-Bue E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL https: http://poooeeeddececeecc.oc nwur cc/paper_ftiicccccppp/eee/e lfpperrffles__ Paper.
Ryuichi Yamamoto, Eunwoo Song, and Jac-Min Kim. Parallel wavegan: A fast waveform generation model based generative adversarial networks with multi-resolution spectrogram. In ICASSP 1020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6199-6203, IEEE, 2020.
Geng Yang. Shan Yang, Kai Liu, Peng Fang, Wei Chen, and Lei Xie. Multi-band melgan: Faster waveform generation for high-quality text-to-speech. In 202/ IEEE Spoken I Language Technology Workshop (SLT), pp. 492-498, 2021. doi: 10.1109/SLT48890022000 20099000.022..... 19333551.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02880 2019.
Liu Ziyin, Tilman Hartwig, and Masahito Ueda. Neural networks fail to learn periodic functions and how fix In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 1583-1594. Curran Asso- ciates, Inc., 2020. URL https: /t/pattppe//prreeeess..eee 2020/File/1 22000
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 IMPLEMENTATION DETAILS
The RFWave backbone contains 8 ConvNeX(V2 blocks. Within each ConyNeXIV2 block, the depth-wise convolutional layer featuring large kernel utilizes kernel size of 7 and has channel dimension of 512. The first and last 1xl point-wise convolutional layers in the sequence possess channel dimensions of 512 and 1536, respectively. In Subsection 4.4, the half-size variant contains ConvNeX(V2 blocks with point-wise convolutional layers of 384 and 1 1152, while the double-size variant features 16 ConvNeXtV2 blocks with point- wise convolutional layers of 512 and 1536.
During the extraction of complex coefficients for the model, we use the orthonormal Fast Fourier Transform (FFT) and its inverse (IFFT), with the normalization convention of dividing by 1/VN for both operations, here is the FFT size. This approach ensures the spectrogram extracted is within a more reasonable range for modeling.
In terms of loss functions, we assign weight of either the Rectified Flow loss or its energy- balanced variant. If employed, the overlap loss and STFT loss are assigned weight of 0.01. These weights are determined by aligning the L2-norm the gradients from the overlap loss and STFT loss with approximately 1/10 of that from the Rectified Flow loss.
Training details are also worth mentioning. Audio samples are randomly cropped to lengths of 32512 and 65024 for 22.05/24 kHz and 44.1 kHz waveforms. respectively. This is equivalent to crop window of 128 frames for both sampling rates. We use batch size of 64. The model optimization is performed using the A optimizer with starting learning rate of 2e-4 and beta parameters of (0.9, 0.999). A cosine annealing schedule is applied to reduce the learning rate to a minimum 2e-6.
A.2 COMPUTATIONAL RESOURCE REQUIRED FOR THE EXPERIMENTS
The bulk of the experiments were carried out on personal computers and GPU servers, which were sourced from cloud service providers. Primarily, we utilized Nvidia-4090-24G and Nvidia-A100-80G GPUs for these tasks.
The specific hardware used and the corresponding time taken for training RFWave on various datasets are as follows:
Table A.1: GPU configurations and training duration for various datasets
Dataset
GPU configuration
Training duration
LJspecch
4090
days
LibriTTS
A100
days
Opencpop
4090
day
MTG-Jamendo
Ax
days
EnCodec mixed dataset
4XA100
10 days
A.3 DEVIDING INTO SUBBANDS
The complex spectrogram is circularly padded in the feature dimension with a size of (dat det - 1). where dar signifies the overlap size. The dimension of each subband's main section, denoted as dm calculated by dividing d - 1 by the total number of subbands. Here, d represents the dimension of the complex spectrograms. The last subband is an exception, having an extra feature dimension but one less padding dimension. Subbands are extracted by applying sliding window along the feature dimension, where the window has a size d., + 2d and shifts by d,.
'The feature dimension of each subband, represented as d,, equals (dm + 2dot), accounting for the interleave of real and imaginary parts.
16

---

Published as a conference paper at ICLR 2025
100
150
200
100
150
200
Suhband
Subband
100
150
200
Subband
dubbaed
100
150
200
Sa8band
100
150
200
energy
Figure A.1: The energy and weighting coefficients, represented by a, display a consistent variation throughout the frames.
A.4 STFT LOSS
Spectral convergence (SC) loss
|| /I||x
where X, is the ground truth and X is the estimated complex spectrogram using (7). The Frobenius norm, || [|,, applied over time and frequency, emphasizes large spectral components in SC loss.
Log-scale STFT-magnitude loss
log log( + +) log([Xi) + +)]l1,
where the I norm, || I1.. along with small constant is used to accurately capture small-amplitude components in the log-scale STFT-magnitude loss.
17

---

Published as a conference paper at ICLR 2025
A.5 SELECTING TIME POINTS FOR EULER METHOD
Straightness is calculated from single batch with a size of 96. The Euler method uses 100 equal interval steps estimate the integral in the straightness definition. Time points selected such that the increase in straightness remains consistent across each interval. An example with 10 Euler steps provided Figure A.2. The time points are calculated only once per model, resulting in negligible computational load.
2.0
ElIox; -x) vX.t|ee
1.0
0.5
0.4
0.6
Time
25
Straightness
10
0.0
0.2
0.4
0.6
0.8
1.0
Time
Figure A.2: Deviation (top) and straightness (bottom) over time: mark Euler method time points, with constant increase in straightness across intervals.
18

---

Published as a conference paper at ICLR 2025
A.6 SPECTROGRAM EXAMPLES FOR COMPARISON WITH DIFFUSION-BASE BASED METHOD
For all the spectrogram figures, we used Adobe Audition for spectrogram visualization, with the dynamic range being [-132, 0J dB.
Figure A.3 showcases spectrogram examples generated by various models using the Openepop dataset. When compared to the ground truth spectrogram, RFWave is seen to produce clean and stable harmonics, whereas the harmonics generated by other models exhibit minor discontinuities.
Figure A.4 exhibits spectrogram examples generated by different models utilizing the LibrÎTTS dataset. RFWave generate clear sigh-freuency harmonics, while PriorGrad and FreGrad result in blurred high- seghffreueency harmonics.
Figure A.5 displays spectrogram examples generated by diverse models from the Jamendo dataset. Both RFWave and PriorGrad generate commendable spectrograms, with RFWave edging out slightly the high-frequency range.
(a) Ground truth
(b) RFWave
(c) PriorGrad
(d) FreGrad
Figure A.3: Examples of spectrograms from Openepop. Within the region marked out by the green box, the harmonics generated through PriorGrad and FreGrad show minor discontinuities.
19

---

Published as a conference paper at ICLR 2025
(a) Ground truth
(b) RFWave
(c) PriorGrad
(d) FreGrad
Figure A.4: Examples of spectrograms from LibriTTS. Within the green-boxed region, RFWave yields hetter harmonics than PriorGrad and FreGrad.
20

---

Published as a conference paper at ICLR 2025
(a) Ground truth
(b) RFWave
(c) PriorGrad
(d) FreGrad
Figure A.5: Examples of spectrograms from MTG-Jamendo. Within the green-boxed region, RFWave generates more favorable high-frequency components compared to PriorGrad and FreGrad, Additionally, FreGrad gives rise to vertical line artifacts as highlighted by the top-right box.
21

---

Published as a conference paper at ICLR 2025
A.7 SPECTROGRAM EXAMPLES FOR COMPARISON WITH GAN-BASI METHOD
Figure A.6 illustrates that RFWave, BigVGAN and Vocos are capable of generating high-quality spectrograms when applied to the LibriTTS dataset. This demonstrates the effectiveness of each approach in handling in-distribution data. The spectrograms exhibit well-defined harmonics and minimal artifacts, highlighting the robustness of the models within their familiar domain.
Figure A.7 presents spectrogram examples generated on the MUSDBI 18 dataset, with the models having been trained on the LibriTTS dataset. A notable observation is that both BigVGAN and Vocos exhibit a tendency to produce horizontal lines in the high-frequency regions of the spectrograms. These artifacts are perceptually problematic, as they often result in metallic sound quality that detracts from the naturalness of the audio. stark contrast, RFWave consistently generates clear and well-defined high-frequency harmonics. This capabiliiy underscores RFWave's superior ability to maintain spectral fidelity, even when applied to distr--distiitution data, thereby enhancing the overall audio quality and realism.
Ground truth
(b) RFWave
(c) Big VGAN
(d) Vocas
Figure A.6: Examples of spectrograms from LibriTTS. the methods produce nigh-quality spectrograms.
22

---

Published as a conference paper at ICLR 2025
Ground truth
(b) RFWave
(c) BigVGAN
(d) Vacos
Figure A.7: Examples of spectrograms from MUSDB18. Within the green-hoxed region, both BigVGAN and Vocos have propensity to generate horizontal lines within high-frequent areas the spectrograms. These artifacts present perceptual issues since they frequently lead to a metallic sound quality that diminishes the naturalness of the audio. In sharp contrast, RFWave produces well defined high-frequency harmonics, even when applicd to out-of- nat-of-distribution data.
23

---

Published as a conference paper at ICLR 2025
(a) Ground truth
(b) w/ STFT Loss
(c) w/o STFT Loss
Figure A.8: The effect of STFT loss. There are some vertical patterns the spectrogram of waveforms generated by model without STFT loss, as highlighted by the green rectangular.
(a) Ground truth
w/ overlap Loss
(c) w/o overlap Loss
Figure A.9: The effect of overlap loss. Omitting it results a noticeable transition between subbands, as clearly illustrated by the green arrows.
24

---

Published as a conference paper at ICLR 2025
A.8
TABLES
Table A.2: Code repositories for model training.
Model
repository
https://gshhiib..cc////lcrrsttt
PriorGrad
leurralSpeechtttsee///ssser/
PrlorGrad-vocoder
FreGrad
fregrad
Multi-Bana Diffusion (MBD)
audiocraft:
EnCodec
encodec
Vocos
BigVGAN
Table A.3: Parameters for extracting Mel-spectrograms and complex spectrograms.
Dataset
Sample rate (kHZ)
Window length
Hop length
FFT size
Mel bins
LJSpeech
22.05
1024
256
1024
100
LibriTTS
24
1024
256
1024
100
Opencpop
44.1
2048
512
2048
100
MTG-Jamende
44.1
2048
512
2048
100
EnCodec Training
24
1280
320
1280
25

---

Published as a conference paper at ICLR 2025
Table A.4: Construction of the general Encodec test dataset
Category
Subdirectory
Samples
Description
Expresso dataset set (Nguyen 2023),
Expresso-test
50
randomly select 50 samples, segmenting any audio longer than 30 seconds into 10-sccond clips beforehand.
HiFiTTS-test
50
Randomly seloet 50 samples from the test set of the Hi-Fi TTS dataset (Bakhturina
2021).
LibriTTS-test
50
Randomly select 50 sumples from the text set of LibriTTS
Speech
(Zen al., 2019).
AishellB-test
50
Randomly
samples from
the
Aishell3 dataset (Shi al,, 2020).
JVS
50
Randomly select 50 samples from the entire JVS Corpus.
(lakamichi et al., 2019).
CML-TTS-test
50
Randomly select 50 samples from the test set of the CMI. TTS datasct (Oliveira et al., 2025).
Segment vocal
audio
Musdb
Musdb-test-vocal
60
(Rafii et al,, 2017) into 10-second clips, then randomly select 60 xamples.
CSD
20
Segment all audio in the CSD (Choi et al,, 2020) into 10- clips, randomly
samples. Randomly select 20 samples from the Opencpop dataset (Wang al., 2022).
Opencpop
20
munophonic subsel of Chinese Opera Singing Dataser (Black et al., 2014), segment any mudio longer than seconds into 10-sccond clips, then ran- domly select 60 samples.
ChinseoOporz--moopphoti
60
Vocal
JVS-Music
60
Randomly
samples
JVS-Music Corpus
(Tamaru el al., 2020).
RAVDESS
60
Randomly select 60 samples from the song subset of the
RAVDESS corpus (1 ('ivingsoon & Russo, 2018).
From the demo audins of the Cemusic Dataset (liu & 11, 202 D) select the vocal parts. Then, randomly select samples from these for the test segmenting any sudio
20
longer tham 30 seconds into 10 second clips beforehund.
ESC-50
150
Randomly select 150 samples from ESC-50 (Piczak, 2015).
Segment the acompaniment track audio of the Musdb dataset (Ralii et all, 2017) 7) into 10 second clips, randomly
Musiib-lest-accoomaainnnn
40
clips. Segment the mixture track audio from the test set of the Musdb dataset (Rafii al,, 2017) into 10-sccond clips, then randomly select 20 samples from these clips.
samples from
Musdb-test-mixture
20
From the polyphonic subset of the Chinese Opera Singing Dalaset (Black et 2014), segment any audio longer than seconds 10 ID-second clips, then randomly select samples.
Sound Effect
ChineseOpprz--polyphoniic
20
OpenMIC-test
40
Randomly select 40 samples from the test set of OpenMix 2018 (Humphrey et al., 2018).
From the demo audios of the Cemusic Dataset (Lio & 2021), select the non-vocal parts. Then, randomly select 20 samples from these for the test set, segmenting any audio longer than seconds 10-second clips beforchand.
Comusic-demo-musicc
30
26

---

Published as a conference paper at ICLR 2025
Table A.5: MOS and Ohjective evaluation metrics for RFWave, PriorGrad FreGrad across various datasets.
Dataset
Model
MOS
PESQ 1
ViSQOI. 1
VUVV F1
Periodicity +
RFWave
3.83:1014
4.228
4.595
0.968
0.090
PriorGrad
3.70+10.1
3.820
4.134
0.960
0.100
LibriTTS
FreGrad
3,68+0.15
3.758
4.278
0.960
0.099
Ground truth
3.88.00.1
RFWave
4.26:+0.17
4.176
4.564
0.990
0.049
PriorGrad
3.79:1021
3,404
4.512
0.988
0,064
Openepop
FreGrad
3.73:1:.20
3.522
4.507
0.986
0.074
Ground truth
4.3310.18
RFWave
3.904:.15
4.317
PriorGrad
3.7910.14
4,412
MTG-Jamendo
FreGrad
1.8940.17
3.765
Ground truth
3,93+t.11
'The MOS of the ground truth LibriTTS differs from that presented in Table 2 because these two MOS tests conducted separately.
27

---

Published as a conference paper at ICLR 2025
Table A.6: MOS and objective evaluation metrics for RFWave, EnCodec and MBD across various test sets.
Test
Bandwidth
Model
MOS
PESQ
ViSQOL
V/UV Periodicity
RFWavercro)
3.29=0.47
1.774
3.102
0,913
0.178
1.5 kbps
EnCodec
2.29
1.515
3.310
0,870
0.231
MBD
2.94
1.659
2.793
0,901
0.195
RFWave,cHo2)
3.85=0.39
2,421
3,382
0.935
0.137
3.0 kbps
EnCodec
2.85=0.47
1.967
3.746
0.919
0.166
MBD
2.88=0.33
2.194
3.219
0,915
0.166
Speech
RFWaveicrca
4.05=0026
2,974
3,913
0,952
0.109
6.0 kbps
EnCodec
3.19-0.31
2.554
4.048
0.945
0.121
MBD
3.51
2.372
3,410
0.924
0.161
RF RFWavccCRa)
3.96-0.22
3.393
4.158
0.965
0.089
12.0 kbps
EnCodec
3.52-0:22
3.104
4.250
0.961
0.095
MBD
Ground truth
4.13=0.23
RFWavoccoo))
3.06-0.34
1.820
3.317
0.914
0.208
1.5 kbps
EnCodec
1.94_0.38
1.900
3,931
0.941
0.166
MBD
2.84
1.739
3.221
0.901
0.228
RFWaveicraa)
3.7_
2.467
3.793
0,942
0.132
3.0 kbps
EnCodec
2.63=0.37
1.900
3,931
0,941
0.166
MBD
3.26=0045
2.426
3.655
0.929
0.175
Vocal
RFWaveicraa)
3.77=0.23
2.897
4.080
0,956
0.124
6.0 kbps
EnCodec
2.92-00.3
2.310
4.206
0,956
0.131
MBD
3.30-0.26
2.604
3.840
0.933
0.175
RF WaWaeccrr
3.83-0.31
3.147
4.273
0.964
0.109
12.0 kbps
EnCodec
3.40=029
2.679
4.373
0.965
0.114
MBD
Ground truth
4.27=024
RFWavacr
3.24-0.43
2.906
1.5 kbps
EnCodee
2.62-042
3,314
MBD
3.33
2.932
RI WFaave
3.11
3.334
3.0 kbps
EnCodec
2.89
3.703
MBD
3.11-0.54
3.331
Soudn Effect
RFWave,cra)
3.27-0.27
3.682
6.0 kbps
EnCodec
3.17=0.24
4.020
MBD
3.44
3.496
RF WaWaccrraa
3.33-0.28
3.942
12.0 kbps
EnCodee
3.72-0.26
4.249
MBD
Ground truth
3.78=0.28
28

---

Published as a conference paper at ICLR 2025
A.9 ALGORIHHMS
We present simplified version of the pseudocode along with a comprehensive Python implementation for the sampling algorithms. The detailed Python code delves into the intricacies of subband processing and inverse equalization, offering deeper insights into the algorithmic steps involved. Additionally, we provide Python implementation for selecting time points of equal straightness.
A.9.1 SAMPLING ALGORITIM
Following is the sampling algorithm (X, in time domain frequency domain) with its pseudocode and Python implementation.
For simplicity, in the pseudocode, we' ve omitted the subband-related operations. Here, x represents the combination of all X shown in Figure 1, and vi represents the combination of all 0/th shown Figure For comprehensive understanding of the processing steps, please refer to the Python implementation.
Algorithm Simplified Sampling Algorithm (X, in Time Domain)
Require:
onnmodel: pre-trained neural network model
2: C: conditional input
3: (4) time steps where 0 lu 4 ... < =1
Ensure:
4: Generated audio waveform wave
5:
6: Xp ~N(00, 1)
Initializ with Gaussian noise
7: X Xo
8:
9: for i to N- do
10: dt t t++1 tú
Calculate step interval X/ + STFT(X;) // Convert X₂ to frequency domain
11:
12:
mn-model.pprede
4i, C)) // ef in frequency domain
13:
// Convert vn to time domain 14: X + x + of dt // Euler step in time domain
ISTFT(u!)
end for
16:
X1 X
18: wave t- Xi
20: return wave
29

---

Published as a conference paper at ICLR 2025
Detailed Algorithm 1: Sample Time Domain
def cample_time_domalt (model, mel, ts):
:param medel: pro-trained time-domain REWavv modol.
sparam mel: A batch of mel-spoctzogram data, (batch size, channols num_framme).
:param ts: The selected sampl ing time points.
:return: The reconstructed audio waveform, [batch_size, Frames hop_length].
...
size, num frames = mel. shape mel sea.shape [2] noise_in_t torch (batch_size, size, model. nun_frames])
m
the STFT operation in Figure 1.
12
loise_in_ffo torch.stft (noise_in_t, s Divide the noise spectrogram into subbands and reshape into the batch dimension noise_in_f
14
15
repeat the mel mei-spectrogram to match the subbands accordingly.
16
mel = torch.
(mel, model num band, 0)
iin_t roise_in_s
19
__in_f noise_in_f
20
vs [] # used for selecting sampling time points
for i in range (len (ts) 1): ts[i]
23
dt ts 11 tslil $ the model runs at frame-level feature. v_in_f = model. preellppercctteeeeee f2_in_f, t, mel)
24
25
place the subbands to construct full band feature v_in_f place_aauslee subband (model, v_in_f) the ISTFT operation
28
Figure
29
v_in_t = Norch.istfe v_jin_,, **model stteeellsstt_ttarrree $ the guler step operates in the time-domain
in_t init + v_in_t at the STFT operation Figure 1. z_in_f torch (z_in_t, **model *=model.st stafeel.stft_kwargss z_in_f get_subband (model, (model,, z_in_f) as.append
32
35
34
append the straight line velocity vs. vs.append
37
noise_in, in_t)
38
inverse the waveform equalization.
39
ini__
inverso in_t)
return 2_in_t, vs
30

---

Published as a conference paper at ICLR 2025
Algorithm 2 Simplified Sampling Algorithm (X, in Frequency Domain)
Require:
1: mnn_model pre-trained neural network model
2: C: conditional input
3: {t4]i=o time steps where = to < t ... < =1
Ensure:
4: Generated audio waveform wave
5:
6: Xa N(0,1)id. // Initialize with Gaussian noise
7: x Xo
8:
9: for i 0to N N - do
Calculate step interval 4,, C) 12: X/ + x u. dt // Euler step
10:
t t,+1 - tú
11:
of +
13: end for
14:
15: X1 x!
16: wave ISTFT(X1)
17:
18: return wave
Detailed Algorithm 2: Sample Frequency Domain
def sample_frog__dmmal (model, mel, ts):
:param medel: : pre-trained frequency-domaia REWave model.
:param mel: A batch of mel-spectroggaa data, [batch_size, channels num__frame mramfra
:param ts: The sejected sampl ing time points.
:return: The reconstructed audic waveform, (batch size, num frames heop_eenghh/.
...
batch_size, num_frames mel. mel.shape[0] mel.shaped [2]
noise_in_f torch. randn siiee, model.n_fft. + 2, num__rraee noise_in_f model.get_sue get___baan_bbbdddeeeeeee (noise_in_f) mel torch repeat_ interlletteeeeeveeeee (mel, num bands, 0)
12
13
in = noise_in_f
vs [] $ used for selecting sampling time points
for i in range (len (ts) 1):
16
t ts[i]
17
dt tali -
18
# the model runs at Frame-level feature.
v_in_f model. predict (z_in_f, mel)
the Fuler step operates in the frequency-domain
21
v vs. ap.appedd (v (v_in__f)
zin z in
22
23
append the straight 1ine velocity vs (2_ noise_in_f)
25
place the subbands to construct full band spectrogram paaelp..aacessbbaade # inverse the stft normalization
26
z iin__
27
zii__
convert the compiex spectrogram to waveform
30
zin torch.istit in_in stooeel.sitt_kwe
31
return 2 in_t, vs
31

---

Published as a conference paper at ICLR 2025
A.9.2 SELECTING TIME POINTS OF EQUAL STRAIGHTNESS
Detailed Algorithm 3: Select Time Points
def select_time_pponta (model, sample_fn, mel, N1=100, N2=10):
This function is dosigned to solect time points of equal straightness.
:param model: pre-trained REWave model.
:param sample_fn, fn: sample_time_deee domain or sample frequency_doma :param mel: A batch of mel-spectrogran data, maatch_siee, channels framers
:param NI: The number of steps used for numerically estimating the straightness.
:param N2: The number of steps for equal straightness :reuurn: A list of selected time points.
12
Step 1: Numerically estimate the trajectory eq_ts torch.1inspace (0, 1, N1 + 1)
13
14
vs sample_fn (model, mel, eg_ts)
16
step 2: Calculate the deviation for each velocity step ds []
17
18
v_straight = vs[-1]
for v in v=[:-1]:
v_straight) view (v.size (0), -1)
21
d torch. (d, p='iro', dim=1) ds. ())
22
23
Step 3: calculate straightness c_cum[1] is the estimated straightness. cum torch.cumsure (torch, stack (ds), dim=0)
26
27
# s_ine is the increment of straightness for each step s_inc d_cum[-1] N2
30
step 4: Select the time points equal straightness ts [0.]
31
32
for i in (1, N2):
s_i - s_inc
idx torch abs [d_cum s_1) .argmi ()
35
ts (ipk N1)
36
return ts [1.]
32

---

Published as a conference paper at ICLR 2025
A.10 SUBJECTIVE EVALUATION
We conducted the subjective listening test on our self-developed website platform. The platform allows to play audio samples and rate the audio naturalness using a 5-point Mean Opinion Score (MOS) scale, where indicates "poor/unnatural" and 5 indicates "excellent/natural""
For each subjective evaluation, we invited 30 listeners to participate. For each comparison experiment to be evaluated, we randomly selected 20-40 groups of audio samples from the corresponding test for one listener to rate. Each group of audio includes ground truth audio, audio synthesized by RFwave, and audio synthesized by other models. The order of audio within each set was randomly shuffled.
All participating listeners were required to use headphones and give ratings from I to 5 based on the audio naturalness. The rating results were directly submitted to our evaluation website. Figure A.10 shows the interface of the evaluation website.
Vocoder Jamendo Evaluation
Pheme
mouethure svent qantl ouch wur folowing pringn accoring you toed iufeeiiee
adoo sad audi prosi ladd Audle contans nuttle AUMA,
Lost Nore Croce
Encodec Sound Effect Evaluation
Lecart
borealy lpedd kigg aboui jabee aeredd
Aey Soed Vay Solend
batae bocee'
Figure A.10: Interface of our subjective evaluation platform.
33


Reviews:
[{"review_id": "5kXUiU94d9", "reviewer": "Reviewer_2uM7", "summary": "In this paper the authors propose a new model called RFWave (Rectified Flow Wave) for achieving audio reconstruction from Mel Spectrograms or Encodec tokens. They propose training an efficient ConvNeXtV2\n backbone (a state-of-the-art convolutional architecture) using the rectified flow (diffusion) paradigm, that operate on STFT spectrograms bands for improved efficiency (than training directly in waveform domain). Each spectrogram band is processed independently by the network, and are concatenated before the inverse STFT (differently than the previous Multi-band Diffusion conditional approach of Roman et al. 2023, in which error accumulates from low to high frequency bands). The model can be trained both by performing the noising in time-domain, where is data is mapped first to STFT and back via the iSTFT before applying the loss, or trained\ndirectly in STFT domain (authors show improved results on time-domain nosing approach). Methodologically the authors propose three new losses and new sampling algorithm for rectified flow. The losses are: a weighted loss which scales the STFT band by the variance of the ground truth velocity band (which is proportional to its energy), such that nearly-silent portions of data do not result in noisy outputs, an overlap loss which equalises\nthe outputs of different bands at their boundaries and an STFT loss computed via the rectified flow prediction at time $t$. The novel sampling algorithm choses sampling times by subdividing the flow trajectory in intervals\n of equal straightness (the integral of the difference between the ground truth velocity direction and the predicted velocity). The authors perform a solid experimental evaluation, comparing both with recent diffusion-based models and SOTA GAN-based models, showcasing very good results.", "questions": "Here I list questions and typos:\n- Line 174: “can be” to “can be an”\n- Line 184:  Fourier features of what? Of the noisy sample? How we concatenate the conditional inputs? They share same dimensionality?\n- Line 205: If interleaved then shouldn’t be $2d_s$?\n- Line 219: “For waveform equalization… computed during training” This part as well refers to PQMF? Or is distinct?\n- Line 228: “Subsequent processing involves the dimension-wise mean-variance normalization of the complex spectrogram.” As well here there is the mean-variance normalization as in time domain?\n- Line 246: Why compute the standard deviation and not the energy altogether? Is there any reason for this?\n- Line 252: Space after the $\\min$.\n- Line 295: Isn’t $X_0$ the data point and $X_1$ the noisy point? So shouldn’t we approximate  the STFT Loss using $X_0$?\n- Line 307: “filed” is “field”.  In the integral use $\\mathbb{E}$. Also I think the straightness should have argument a time $s$ (see Weakness section).\n- Line 382: Why snake activation enhances out-of-domain data generation capabilities?\n- Line 384: Isn’t this imputable to the architecture choice more than the type of generative model?  Especially upsampling layers as empirically shown here: https://arxiv.org/pdf/2010.14356 ?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- While methodologically the authors borrow many ideas from previous papers, e.g., the model architecture ConvNeXtV2 is not a novelty of the paper, and training in the STFT domain is a classic choice in audio deep learning, it is very interesting how the combination of those ideas can bring us closer on surpassing the fundamental limitations in efficiency of diffusion models from a practical point of view. Seeing in Table 7 that actually performing 10 inference step of RFWave we have a way higher realtime factor than a BigVGAN (while having similar quality performance on LibriTTS and improved OOD performance on MUSDB18) is a strong indication that the inference time gap between diffusion models and GAN models is finally disappearing. And the best thing here is that one obtains this reduction without having to resort to distillation techniques (which are notably difficult to perform, especially consistency distillation), but only resorting to reasonable design choices, especially the choice of the architecture.\n- The results in the experimental section are very good for the proposed model, putting it as a novel strong baseline in the audio reconstruction setting. Additionally, the ablations in Tables 5 and 6 give a detailed view on why the design choices in the paper contribute to the overall performance.\n- The proposed losses and the idea of equal straightness intervals are valuable not only in the audio reconstruction context but also more generally when dealing with other types of STFT based models / other generative settings for rectified flow models.", "weaknesses": "- The paper in some parts is not very clear in its explanations. For example in the Energy-balanced Loss part of Section 3.2 (Loss Functions), from first (and repeated) readings, I could not understand what was the  problem with the MSE loss. After different readings I interpreted it in the following way: small absolute errors in silent regions contribute little to the overall MSE loss, so the model doesn't prioritize eliminating them. This means the model doesn't effectively suppress minor deviations in silent areas, resulting in perceptible noise. Thus larger errors in high-amplitude regions significantly impact the MSE loss, causing the model to focus more on reducing those errors during training. Another part that was not clear was all Section 3.3 (Selecting Time Points for Euler Method). Here, first, straightness should be defined for a time $s$, as $S(v, s)$, setting the definite integral from $0$ to $s$ (using $s$ to leave the $t$ in $dt$). I believe authors use this because they took it from the original paper on rectified flow. Then writing “allowing to take more steps in more challenging regions” is pretty confusing. The term “allowing” is not ideal, I would use “requiring”. I understood the concept thinking that if in an interval the straightness does not change much, it means that we can take bigger steps, saving on the overall number of steps. I please ask the authors to re-write these parts in a more understandable way.\n- While the experimental sections is done in a good way as mentioned in the Strengths part, the fact that the authors trained RFWave on their proposed large-scale dataset (combining Common Voice 7.0, DNS Challenge 4, MTG-Jamendo, FSD50K and AudioSet) and that Vocos, BigVGAN, EnCodec, and MBD are evaluated using the public pre-trained models is somehow problematic: it could be that Vocos, BigVGAN, EnCodec, and MBD are under-trained with respect to the authors dataset. I would like the author to comment on this and at least show that the training sources are comparable.\n- It would have been interesting to compare also with latent diffusion models, where the network operates in another domain that is more compressed as the waveform domain, maybe a fine-tuned version of Stable Audio Open, given their widespread adoption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k5EQDVYvC3", "reviewer": "Reviewer_5nT1", "summary": "An efficient diffusion-based vocoder is proposed. The strengths of this paper are claimed to be that it generates the complex spectrogram of STFT at the frame level and that it can be generated in 10 denoising steps.", "questions": "- I find it difficult to understand the rationale behind dividing into subbands when working in the STFT domain. If we are considering the STFT, it seems that the signal is already inherently divided into frequency components. \n  - Also, this paper mentions introducing a loss related to overlap in order to mitigate inconsistencies caused by subband division. However, inconsistencies between subbands are not limited to adjacent bands but may also impact the coherence of harmonic components. This could also be considered a potential drawback of subband division. \n  - The paper claims improved performance on harmonics compared to conventional methods with some empirical examples, but it is not clearly justified why it is effective.\n\n- The STFT introduces redundancy in the representation due to overlapping frames compared to the original waveform. Consequently, the assertion that complex STFT spectrograms could be generated much faster than waveforms is not immediately intuitive. A more detailed and clearer explanation is required to substantiate this claim.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Experimental results have demonstrated superior sound quality across multiple metrics compared to conventional methods.\n- The strength of this paper appears to lie in the application of the latest technique, rectified flow, which enables efficient sampling. This aspect of the work stands out as a significant contribution.\n- This work introduces three additional loss functions to improve performance. The additional loss functions appear to consistently improve performance.", "weaknesses": "- The task addressed in this paper is rather conventional, using standard datasets and common evaluation methods. As a result, the work comes across as a minor variation of existing research. While this is certainly important work, it lacks a clear element of originality. It is unfortunate that, in a field where many similar approaches already exist, this work appears to be another study that only slightly improves existing benchmarks.\n- Moreover, the approach is not theoretically groundbreaking either. At first glance of the contribution part, the proposed method appears to be a combination of existing ideas, making its originality and novelty less immediately clear.\n- Some spectrograms are shown, but the differences are not immediately apparent, making it unclear what aspects should be evaluated. It would be helpful to provide a little more explanation rather than leaving the interpretation up to the reader saying see the high-frequency components.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rVj9yblBcd", "reviewer": "Reviewer_c6cp", "summary": "This paper considers a method for reconstructing the waveform from Mel-spectrogram or discrete acoustic tokens. The method generates complex spectrograms and performs reconstruction on a frame level.  Reconstruction is performed using a straight transport trajectory, which is achieved in a few sampling steps. Audio generation can be achieved much faster than in real time.", "questions": "See above", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Audio reconstruction from Mel-spectrogram is achieved with a few sampling steps, and audio is generated much faster than in real-time.", "weaknesses": "1. Algorithm Flow and Domain Clarity:\nThe paper suffers from readability issues, making it challenging to understand the overall flow of the proposed algorithm. Specifically, it is unclear whether the algorithm operates in the time domain, frequency domain, or both, as it seems to suggest that transitions between these domains are possible but lacks a clear explanation or visual aids (e.g., a flowchart). This lack of clarity weakens the reader's comprehension and hinders an appreciation of the technical merits of the algorithm. It would benefit the authors to provide a visual or detailed description to convey these aspects effectively.\n\n2. Assumptions about Input Data (Mel-Spectrogram):\nThe paper assumes a mel-spectrogram input to produce waveform output. However, it does not clarify how this mel-spectrogram is initially obtained or what assumptions are made about the input data. Understanding the starting point and any assumptions about preprocessing steps is crucial to assessing the practicality of the proposed algorithm. The authors should explicitly state these assumptions and briefly explain how mel-spectrograms are typically derived in relevant application contexts.\n\n3. Frequency and Time Domain Operations:\nWhether the multiband rectified flow is used to reconstruct the complex spectrogram or the waveform directly remains ambiguous. While the paper mentions using the short-time Fourier Transform (STFT), which implies frequency-domain operations, it also suggests waveform reconstruction capabilities. This lack of specificity raises questions about whether the algorithm operates exclusively in one domain or both. A more precise discussion of domain-specific processes, including the role and interplay of STFT in this context, would enhance the reader’s understanding and the algorithm's technical robustness.\n\n4. Role of ConvNeXt V2 and Learning Velocity:\nThe role of ConvNeXt V2 in learning the velocity field is unclear. Given that ConvNeXt V2 is a recent architecture capable of supervised and self-supervised learning, understanding how it contributes to the velocity-based learning within the algorithm is essential for evaluating the approach's novelty and effectiveness. A clear outline of ConvNeXt V2’s integration and function in this context, especially concerning velocity learning, would clarify the methodology and strengthen the reader's ability to assess its value.\n\n5. Waveform Generation from Complex Spectrogram:\nThe method for generating the final waveform from a complex spectrogram is not well-explained. Although the paper references GAN-based methods, it lacks a detailed description of how this process is implemented in the proposed model. An explicit explanation of the techniques used to transition from the complex spectrogram to the waveform would clarify the reconstruction process, making it easier to evaluate the proposed method's strengths and limitations.\n\n6. Independent Subband Reconstruction:\nThe decision to reconstruct subbands independently raises concerns about consistency across subbands. While the paper suggests that independent reconstruction mitigates error accumulation, it overlooks the strong correlations typically present among subbands, which, if neglected, may compromise audio quality. Discussing the potential drawbacks of independent subband reconstruction and any strategies for inter-subband consistency checks would address this issue, making the approach more comprehensive and practical.\n\n7. Reconstruction Speed-Up Justification:\nAlthough the algorithm claims to achieve a reconstruction speed over 100 times faster than real-time, it is unclear whether such a significant speed-up is necessary or beneficial for practical applications. While computational efficiency is generally advantageous, the paper could explore the trade-offs between speed, model complexity, and audio quality to justify the need for such extreme speed-ups in relevant application scenarios. A deeper analysis here could reinforce the practical relevance of the proposed method.\n\n8. Novelty and Engineering-Based Solutions:\nThe paper's novelty is in question, as it mainly combines existing components (multiband rectified flow and ConvNeXt V2) rather than proposing a fundamentally new approach. While this engineering solution may have merit in specific applications, the lack of novel contributions limits the paper's impact. Strengthening the theoretical foundation or providing a unique methodological contribution would be beneficial in highlighting the algorithm’s distinct value.\n\nIn summary, the paper presents an engineering-focused approach for accelerated waveform reconstruction. Still, several aspects lack clarity, including domain transitions, assumptions about input data, the role of ConvNeXt V2, and consistency in subband reconstruction. Addressing these issues and offering more insight into the necessity of the speed-up and potential applications would significantly improve the paper's quality and technical rigor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XxgCU4KXLt", "reviewer": "Reviewer_kzjW", "summary": "The paper introduces RFWave, a new method for reconstructing audio waveforms from Mel-spectrograms or discrete acoustic tokens. The proposed method operates on Mel-spectrogram frames and produces spectral subbands in parallel, boosting synthesis speed. Additionally, the paper presents several heuristics aimed at speeding up inference and improving quality, namely: scaling flow-matching loss based on frame energy, adding MSE loss on overlapping parts of predicted frames, incorporating STFT loss on the estimation 𝑋_1​ at time 𝑡, and selecting optimal points for the Euler method. The model consistently outperforms diffusion-based baselines and performs on par with GAN-based methods. The experiments demonstrate that RFWave achieves a good trade-off between generation speed, memory consumption, and sound quality.", "questions": "- BigVGAN repository provides several versions of the model. Did you compare the performance of RFWave with the smaller ones in terms of sound quality and inference speed? Adding more flavors of BigVGAN into Table 7 would give better understanding of the strengths of your method.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written and easy to follow;\n- The introduced heuristics are intuitive and reasonable;\n- The experiments clearly demonstrate better performance of the proposed method compared to diffusion-based baselines and give good understanding of pros and cons compared to GAN-based ones.", "weaknesses": "- Interpretation of the experimental results aren’t always supported by the data provided in the corresponding tables. \nFor instance, 1) according to the authors, Table 3 demonstrates significant advantages over GAN-based methods but the confidence intervals overlap in all scores accept one (“Mixed”, which, however, may be the most important one). I think that a significance test may help to emphasize the superiority of your method. 2) the same problem with overlapping MOS intervals appears in Table 4. At least, MOS evaluation doesn’t support the claim that RFWave “excels” in all metrics being very close to MBD in the key subjective metric. \n- Although Table 3 demonstrates better performance of RFWave on out-of-domain data, I still have doubts that such experimental setting  is practical: in a real-life scenario one would rather re-train a waveform generation model on the target domain (music in this case) than use a model trained on an out-of-domain data (speech). I would consider testing performance of BigVGAN fine-tuned on small amount of music data to get better understanding of advantages of RFWave brought by its better performance on out-of-domain data.\n- Minor issues: “Tables 2” p.8:378, “velocity filed”, p.6:308", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2qeIdRziE8", "reviewer": "Reviewer_rq1Z", "summary": "The present paper proposes RFWave, a rectified flow model for reconstructing audio waveforms from mel spectrograms or discrete tokens (EnCodec codes). The authors design their model to operate on slightly overlapping subbands, allowing for parallelized inference at greatly improved computational efficiency without introducing artifacts at the borders between subbands. The authors propose an energy-balanced loss and several auxiliary losses to enhance the output quality. The method shows highly competitive performance on the evaluated tasks compared to SOTA baselines, at an improved tradeoff between quality and inference speed.", "questions": "* Were any methods trained on multiple types of audio (speech, music, sound) at the same time, or was a specialized model prepared of each RFWave and retrained baseline for *every* audio type? This is not clear to me from line 344, which would suggest that baselines were trained on multiple audio types at the same time, whereas Appendix A.2 suggests that RFWave was trained on every single dataset separately. If the authors compare dataset-specialized RFWave against multi-dataset baselines, such a comparison would be unfair.\n\n* For completeness and better understanding, I would suggest adding a pointwise metric such as a simple L2 distance (or SNR in dB) of the reconstruction compared to the ground truth. I am asking since the presented metrics (MOS, PESQ, ViSQOL, F1, Periodicity) do not provide much insight in how the methods compare regarding pointwise distortion, but this is an important property, especially for the decoding task.\n\n* The authors do not use the rectification step that a rectified flow model enables in principle, correct? Can they provide a reasoning for why they do not perform this step, which in theory should help improve the inference efficiency by further reducing the necessary number of sampling steps?\n\n* Lines 47, 54-56, 70-71: It is a priori unclear why operating on STFT frames should be more efficient than operating on a waveform, since the STFT is redundant and introduces more data points to process. This seems to be more about frame-parallel processing of the DNN architecture rather than the STFT itself, could the authors clarify what they mean?\n\n* In Section 3.3, what is meant precisely by timepoints being \"chosen such that the increase in straightness is equal across each step\"? I would have liked a more concrete description of the method used to obtain the time steps here - was this optimization-based, or somehow estimated in closed-form? What was the formal metric used to judge \"equal straightness\" across all intervals?\n\n* What do the authors really mean by one model variant being capable of \"operating in the time domain\" (line 197-199)? In the abstract, the authors state that RFWave \"generates complex spectrograms\" -- in a flow/diffusion setting, shouldn't then the feature-space input also be a complex spectrogram? In this case, an STFT/iSTFT is used, and the ConvNeXt model performs 2D convolutions, and operates in the time-frequency domain, not in the time-domain, right? Please clarify this point here and also in the paper.\n\nMinor points:\n\n* In lines 204-206, the authors state that they interleave real and imaginary parts of each subband. Wouldn't this lead to a $2 d_s$-dimensional feature rather than a $d_s$-dimensional feature? (Same comment in line 227 where this notation is repeated)\n\n* When the authors say \"dimension-wise\" (line 228), do they mean only $d_s$, or only $F$, or both $d_s$ and $F$?\n\n* Can the authors add a reference for the claim that \"speech energy decays exponentially with frequency, while music maintains a consistent distribution\"? (lines 207-209)\n\n* I do not follow the argument made in 3.2, lines 237-242, regarding MSE-based training. If \"minor absolute distortions lead to significant relative error\", shouldn't a model trained by minimizing MSE *avoid* such small absolute distortions rather than producing them? To me, the observation that the model exhibits residual noise sounds rather like a time-weighting or noise schedule issue than a problem with the MSE used within the training target?\n\n* In the experimental setup section (4.1), I would suggest explicitly referring to Appendix A.1 which lists further training hyperparameters - I could not find such a reference and was missing these details in the main text before finding them in the appendix by chance.\n\n* In the appendix, please provide (visually or quantitatively) the dynamic range used for producing each selection of shown spectrograms.\n\nTypos:\n\n* Lines 364 and 372: \"Comparasion\" -> \"Comparison\"\n* Line 511: \"lenght\" -> \"length\"\n* Table A.5: \"PriroGrad\" -> \"PriorGrad\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The presented method is original in this application context, combining several existing works and ideas (rectified flow, ConvNeXt architecture, sub-band processing) with some problem-specific novel losses (energy-balanced loss, subband overlap loss) and a specialized time-point selection method for the ODE solver used for inference.\n\nThe evaluations show that the method offers a new exciting point along the tradeoff between inference speed and quality. It outperforms most baselines in audio quality and performs similarly to, or better than, BigVGAN while having only ~20% of the parameters, running more than twice as fast, and using ~40% less memory. The authors further conduct convincing ablation experiments on their proposed components.\n\nThe paper is overall structured well and written clearly, with both the clarity of the technical description and the use of language being good in general.", "weaknesses": "Some claims and described methodology lack clarity and should be expanded upon or rewritten more clearly.\n\n* The description of the conducted subjective listening experiment is extremely terse with only a single sentence (lines 352-354). In my opinion, this is insufficient as several details here are key for the reproducibility and reliability of these results. It is necessary to add details on the used platform/service for evaluation, the number of listeners, the number and type of samples per listener, and ideally also the audio equipment used by listeners.\n\n* Figure 1 is insufficient for understanding the model architecture. Specifically, I did not understand how the fusion of the $X_t$ and $C$ occurs before the linear block. The EnCodec bandwidth index $i_{bw}$ also occurs without further motivation or explanation here and is somewhat unclear.\n\n* The relationship between the PQMF bank (lines 207-221) to the complex spectrogram feature-space representation (see abstract) is somewhat unclear. What is really used as input/output of each model variant?\n\n* I appreciate that the authors provide an online demonstration, which is essential for the trustworthiness of the results. However, the selection of samples is suboptimal, particularly for speech where the ground truth signals are of low quality and affected by artifacts. \nPlease replace these or add new samples of higher ground-truth quality.\n\n* The authors argue that using lower-frequency bands to predict higher-frequency bands can negatively impact the higher-frequency bands (lines 142-146), but later also argue that doing so can potentially increase consistency between bands (lines 280-281). Conditioning higher on lower bands is also not the same as processing all bands jointly (since there, also lower bands are estimated based on higher bands), so I do not fully follow the argument made here. The exact argument pro/contra processing low- and high-frequency bands jointly (or not) is somewhat unclear - please clarify.\n\n* The authors seem to have convincing results for their \"equal straightness\" variant (Table 5), but somehow do not include these (at least) in Tables 6 and the comparison with diffusion-based methods, as they also note in the footnote on page 7. It is unclear to me why this is. Please update all metric results to present what the final RFWave variant is consistently, or list both the equal-straightness and non-equal-straightness variants in all relevant experiments for comparison.\n\n* While the objective and subjective metrics speak for the method, from the online demo I find that the authors' method can introduce a kind of colorization and phasy artifacts that are not present in other methods, which are however not discussed in the paper. For transparency, could the authors add a short discussion of this limitation and potential reasons for it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)