PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generation
Authors: Sang-Hoon Lee
Ha-Yeong Choi
Seong-Whan Lee
Venue: ICLR 2025 Poster (2025)
Keywords: Conditional Flow Matching
Neural Vocoder
Speech Synthesis
Neural Audio Codec
Speech Language Models
Primary area: generative models
TL;DR: We propose PeriodWave, a novel universal waveform generator that can reflect different implicit periodic information when estimating the vector fields

Abstract: Recently, universal waveform generation tasks have been investigated conditioned on various out-of-distribution scenarios. Although one-step GAN-based methods have shown their strength in fast waveform generation, they are vulnerable to train-inference mismatch scenarios such as two-stage text-to-speech. Meanwhile, diffusion-based models have shown their powerful generative performance in other domains; however, they stay out of the limelight due to slow inference speed in waveform generation tasks. Above all, there is no generator architecture that can explicitly disentangle the natural periodic features of high-resolution waveform signals. In this paper, we propose PeriodWave, a novel universal waveform generation model from Mel-spectrogram and neural audio codec. First, we introduce a period-aware flow matching estimator that effectively captures the periodic features of the waveform signal when estimating the vector fields. Additionally, we utilize a multi-period estimator that avoids overlaps to capture different periodic features of waveform signals. Although increasing the number of periods can improve the performance significantly, this requires more computational costs. To reduce this issue, we also propose a single period-conditional universal estimator that can feed-forward parallel by period-wise batch inference. Additionally, we first introduce FreeU to reduce the high-frequency noise for waveform generation. Furthermore, we demonstrate the effectiveness of the proposed method in neural audio codec decoding task, and present the streaming generation framework of non-autoregressive model for speech language models. The experimental results demonstrated that our model outperforms the previous models in reconstruction tasks from Mel-spectrogram and discrete token, and text-to-speech tasks. Source code is available at https://github.com/sh-lee-prml/PeriodWave

Published as a conference paper at ICLR 2025
PERIODWAVE: MULTI-PERIOD FLOW MATCHING FOR HIGH-FIDELITY WAVEFORM GENERATION
Sang-Hoon 2.2,*
Ha-Yeong Choi
Seong-Whan Lee" 47
1 DDepartment of Software and Computer Engineering, Ajou University, Suwon, Korea
2 *Departeent of Artificial Intelligence. Ajou University, Suwon, Korea
3 Gen AI Lab, KT Corp., Seoul, Korea DDeparmment of Artificial Intelligence, Korea University, Seoul, Korea
ABSTRACT
Recently, universal waveform generation tasks have been investigated conditioned various out-of-distribution scenarios. Although one-step GAN-based methods have shown their strength in fast waveform generation, they are vulnerable to train-inference mismatch scenarios such as two-stage text-to-speech. Meanwhile, diffusion-based models have shown their powerful generative performance in other domains; however, they stay out of the limelight due to slow inference speed waveform generation tasks. Above all, there is no generator architecture that can explicitly disentangle the natural periodic features of high-resolution waveform signals. In this paper, propose PeriodWave, novel universal waveform gener- ation model from Mel-spectrogram and neural audio codec. First, we introduce period-aware flow matching estimator that effectively captures the periodic fea- tures of the waveform signal when estimating the vector fields. Additionally, we utilize multi-period estimator that avoids overlaps capture different periodic features of waveform signals. Although increasing the number of periods can improve the performance significantly, this requires more computational costs. To reduce this issue, we also propose single period-conditional universal estimator that can (eed-forward parallel by period-wise batch inference. Additionally, we first introduce FreeU to reduce the high- high-frequency noise for waveform generation. Furthermore, we demonstrate the effectiveness of the proposed method in neural audio codec decoding task, and present the streaming generation framework of non-autoregressive model for speech language models. The experimental results demonstrated that model outperforms the previous models in reconstruction tasks from Mel-spectrogram and discrete token, and text-to-speech tasks. Source code is available at htthtpss///gthhhh com sh- lee wtwpeeepmmmmeeeeeiiiee Peewiodaveieeeee
INTRODUCTION
Deep generative models have achieved significant success in high-fidelit waveform generation. In general, the neural waveform generation model which is called "Neural Vocoder" transforms a low-resolution acoustic representation such as Mel-spectrogram
linguistic representations into high-resolution waveform for regeneration learning (Tan et al. 2024). Conventional neural vocoders have been investigated for text-to-speech (Oord et al. | 2016 Shen et al. 2018 Ren et al. 2019 Kim ct al. 2020)/iang et al. (024) and voice conversion (Lee et al. 2021 Choi et al. (021). Furthermore, recent universal waveform generation models called "Universal Vocoder" are getting more attention
due to their various applicability in neural audio codec (Zeghidour et al. 2021 Détossez et al. 2023
Kumar et al. 2024 Ju et al. (2024] Zhang al. [204)), audio generation (Kreuk al. 20033] Roman
et al. 2023 Yang et al. 2023c Huang al. 2023; Liu al. (203),, and zero-shot voice cloning
systems (Lee et al. 2022d Huang et al. 2022c Wang et al. 2023] Li al. 2024 Le et al. 2024
Kim al. 2024 Shen et al. 2024 where models can generate high-fidelity waveform from the
"quual contribution
"ooresponning author

---

Published as a conference paper at ICLR 2025
highly compressed representations beyond the traditional acoustic features, Mel-spectrogram. In addition, universal vocoder requires generalization in various -ustrf-diittibutiio scenarios including unseen voice, instruments, and dynamic environments (Lee et al. 2023: Bak et al. 2023).
Previously, generative adversarial networks (GAN) models dominated the waveform generation tasks by introducing various discriminators that can capture the different characters of waveform signals. MelGAN (Kumar et al.] 2019) used the multi-scale discriminator to capture different features from the different scales of waveform signal. HiFi-GAN (Kong al. 2020) introduced the multi-period discriminator to capture the different periodic patterns of the waveform signal. UnivNet (Jang et al. 2021) utilized the multi-resolution spectrogram discriminator that can reflect the spectral features of waveform signal. BigVGAN (Lee et al. 2023) proposed the Snake activation function for the out-of-distribution modeling and scaled up the neural vocoder for universal waveform generation. Vocos (Siuzdak, 2024) significantly improved the efficiency of the neural vocoder without upsampling the time-axis representation. Although GAN-based models can generate the high-fidelity waveform signal fast, GAN models possess three major limitations: 1) they should utilize lot of discriminators to improve the audio quality, which increases training time; 2) this also requires hyper-paramete tuning to balance multiple loss terms; 3) they are vulnerable to train-inference mismatch scenarios such as two-state models, which induces metallic sound or hissing noise.
Recently, the multi-band diffusion (MBD) model (Roman et al. 2023) sheds light the effective- ness of the diffusion model for high-resolution waveform modeling, Although previous diffusion- based waveform models (Kong et al.] 2021 Chen et al., [021] existed, they could not model the high-frrequent information so the generated waveform only contains low-frequency information. Additionally, they still require lot of iterative steps to generate high-fidelity waveform signals. To reduce this issue, PriorGrad (Lee et al. (0225) introduced data-driven prior and FastDiff (Huang al., (022a) adopted an efficient structure and noise schedule predictor. However. they do not model the high-frequency information so these models only generate the low-frequency information well.
Above all, there no generator architecture to reflect the natural periodic features of high-resolution waveform signals. In this paper, we propose Pave novel waveform generation model that can reflect different implicit periodic representations. We also adopt the powerful generative model, flow matching that can estimate the vector fields directly using the optimal transport path for fast sampling. Additionally, we utilize a multi-period estimator by adopting the prime number to avoid overlaps. We observed that increasing the number of periods can improve the entire performance consistently. However, this also induces a slow inference speed. To simply reduce this limitation, we propose a period-conditional universal estimator that can feed-forward parallel by period-wise batch inference. For
information modeling, we investigate discrete wavelet transformation (DWT) (Lee al. 2022c) and FreeU in waveform generation tasks.
PeriodWave achieves a better performance in objective and subjective metrics than other publicly available strong baselines on both speech and -uttof-distribution samples. Specifically, the experimen- tal results demonstrated that our methods can significantly improve the pitch-related metrics including pitch distance, periodicity, and V/UV Fl score with unprecedented performance. Furthermore, we only train the models for only three days while previous GAN models require over three weeks.
Furthermore, we demonstrate the effectiveness of proposed method in neural audio codec decoding tasks using sate-of-the-ane neural audio codec, Mimi (Défossez et al.). Based on the tokens from Mimi, PeviodWave significantly improve audio quality compared to the original decoder of Mimi. Moreover, we propose the streaming generation method using PeriodWave trained by parallel generation, and PeriodWave can successfully decode in a stream manner with a minimal degradation.
The main contributions of this study are as follows:
We propose PeriodWave, a novel universal waveform generator that can reflect different implicit periodic information when estimating the vector fields.
We thoroughly analyze the limitation of high-frequency modeling, and we address this limitation by DWT and FreeU approach for uighfrrquency noise reduction.
PeriodWave outperformed the one-step GAN models in conventional two-stage TTS tasks.
Through iterative refinement, PeriodWave can reduce the train-inference mismatch problem.
Based on SOTA neural audio codec Mimi, we successfully demonstrate the effectiveness of PeriodWave in neural audio codec decoding task both in parallel and streaming generation.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
Neural Vocoder WaveNet (Oord et al. 2016) has successfully paved the way for high-quality neural waveform generation tasks. However, these auto-regressive (AR) models suffer from slow inference speed. To address this limitation, teacher-student distillation-bass
inverse AR flow methods (Oord ctt al. 2018, Ping et al. (019) have been investigated for parallel waveform generation. Flow-based models (Kim et al.. 2019; Prenger et al., 2019] Lee et al. (020) have also been utilized, which can be trained by simply maximizing the likelihood of the data using invertible transformation.
GAN-based Neural Vocoder MelGAN (Kumar et al. 2019) successfully incorporated generative adversarial networks (GAN) into the neural vocoder by introducing multi-scale discriminator to reflect different features from the different scales of waveform signal and feature matching loss for stable training. Parallel WaveGAN (Yamamoto et al. 2020 introduces multi-resolution STFT losses that can improve the perceptual quality and robustness of adversarial training. GAN-TTS (Binkowski et al. 2020) utilized an ensemble of random window discriminators that operate on random segments of waveform signal. GED (Gritsenko et al., 2020) proposed spectral energy distance with unconditional GAN for stable and consistent training. HiFi-GAN (Kong et al.) 2020) introduced novel discriminator, multi-period discriminator (MPD) that can capture different periodic features of waveform signal. UnivNet (Jang et al. 202T) employed adversarial feedback on the multi-resolution spectrogram capture the spectral representations at different resolutions.
Big BigVGAN (Lee et al., (203)) adopted periodic activation function and anti-aliased representation into the generator for generalization on -utto--distribution samples. Vocos (Siuzdak 2024) proposed an efficient waveform generation framework using ConvNeXt blocks and iSTFT head without any temporal domain upsampling. Meanwhile, neural codec models (Zeghidour et al. [202T] Défossez al. 2023 Kumar et al. 2024) and applications (Wang et al. 2023 Yang et al. 2023b) such as TTS and audio generation have been investigated together with the development of neural vocoder.
Diffusion-based Neural Vocoder DavfWave (Kong al., 2021) and WaveGrad (Chen al. 2021 introduced a Mel-conditional diffusion-based neural vocoder that can estimate the gradients of the data density. PriorGrad (Lee al., 2022b) improves the efficiency of the conditional diffusion model by adopting a data-dependent prior distribution for diffusion models instead of a standard Gaussian distribution. FastDiff (Huang et al. (022a) proposed a fast conditional diffusion model by adopting an efficient generator structure and noise schedule predictor. Multi-band Diffusion (Roman al. 2023) incorporated multi-band waveform modeling into diffusion models and it significantly improved the performance by band-wise modeling because previous diffusion methods could not model high-frequency information, which only generated the oww-frqueency representations. This model also focused on raw waveform generation from discrete tokens of neural codec model for various audio generation applications including speech, music, and environmental sound.
3 PERIODWAVE
The flow matching model (Lipman et al. 2022 Tong et al. 2023) has emerged as an effective strategy for the swift and simulation-fres -rree training of continuous normalizing flows (CNFs), producing optimal transport (OT) trajectories that are readily incorporable. We are interested in the use of flow matching models for waveform generation understand their capability manage complex transformations across waveform distributions. Hence, we begin with the essential notation to analyze flow matching with optimal transport, followed by detailed introduction to the proposed method.
3.1 PRELIMINARY: FLOW MATCHING WITH OPTIMAL TRANSPORT PATH
In the data space R consider an observation & E Rd sampled from an unknown distribution q(x:). CNFs transform a simple prior Po into a target distribution p1 = using time-dependent vector field
Vt. The flow o, is defined by the ordinary differential equation:
d
ग. [+(x) v. ())
<u(x) = x, x Po:
(1)
The flow matching objective, as introduced by (Lipman et al. 2022). aims to match the vector field
ve (x)) to an ideal Vector field un (x)) that would generate the destred probability path pt- The flow
matching training objective involves minimizing the loss function IF (0), which is defined by
regressing the model's vector field xt, to a target vector field u,(x) as follows:
LFM (0) Ee-[o.1]. (1, u (x)
(2)
3

---

Published as a conference paper at ICLR 2025
Ay ~ (0)
X1 Panta
step
Figure 1: Waveform generation using conditional flow matching and ODE solver
Fiad Block
Tretape Dh
1/v
Pertadity
Embeyd
URire 19, Tipe
Pariod aware Flow Matching Psttmator
Gannte
MBlokiik 52 x
DMtick 13) Mipt
Black
Transpoodd
FEEL
DBiwk Ri
SELU
Paleacco Coor
Felcedy
Layer
CoreNenter
Perood p
LB.L.T)
Condition:
Mod-spestoggem
Ported Matrhing Estmater
(6)
Figure 2: Overall architecture of PeriodWave
Given the impracticality of accessing u, and pt. conditional flow matching (CFM) is introduced:
Lora (0) (x-(oo.1_==pp(=1=)) (t, =<(+)>)11
(3)
Generalizing this with the noise condition To (O, 1), the OT-CFM loss is: oT (Ør (rx) | æ1) - u, (Ø: OT (zo) μ: 0)1
CPM (0) = (e
1|
(4)
where 6i (xx) (1 (1 - minntt) + 1x, and 4z ( (xp) x.)) = + (1 - min )ao This
approach efficiently manages data transformation and enhances training speed and efficiency by integrating optimal transport paths. The detailed formulas are described in Appendix A
3.2 PERIOD-AWARE FLOW MATCHING ESTIMATOR
In this work, we propose a period-awart flow matching estimator, which can reflect the different periodic features when estimating the vector field for high-quality waveform generation illustrated Figure First, we utilize time-conditional UNet-based structure for time-specific vector field estimation. Unlike previous UNet-based decoders. PaviodWave utilizes mixture of reshaped input signals with different periods as illustrated in Figure Similar to (Kong et al. 2020). we reshape the ID data sampled from (x(x) of length T' into 2D data of height T/p and width p. We will refer to this process as Periodify. Then, we condition the period embedding to indicate the specific period of each reshaped sample for period-awar feature extraction in single estimator. We utilize different periods of [1,2,3.5.7] that avoid overlaps to capture different periodic features from the utilize 2D convolution of down/upsampling layer and ResNet Blocks with a kernel size of 3 and dilation of 1, 2 for each UNet block. Specifically,
input signal.
downsample each signal by [4,4,4] so the representation of the middle block has height (pp 64) and width p. After extracting the representation for each period, we reshape the 2D representation into the original shape of the 1D signal for each period path. We sum all representations from all period paths. The final block estimates the vector fields from mixture of period representations.
For Mel-spectrogram conditional generation, we only add the conditional representation extracted from Mel-spectrogram to the middle layer representation of UNet for each period path. We utilize
4

---

Published as a conference paper at ICLR 2025
ConvNeXt V2 based Mel encoder to extract the conditional information for efficient time- freeffruyeecy modeling. Previously, Vocos (Siuzdak 2024) also demonstrated that ConvNeXt-basee time- -en-freuunncy modeling shows effectiveness on the low resolution features. In this works, we utilize the improved ConyNeXt V2 (Woo et al. 2023) blocks for Mel encoder, and the output of this block is fed to the period-aware flow matching estimator. Because we utilize hop size of 256, the Mel-spectrogram has a length of I 25.6. To align the conditional representation, we upsample it by 4x and downsample it by the different strides as periods of [1,2.3,5,7] to get shape of T /1/( (p 64).
To boost the inference speed, we introduce two methods: 1) period-wise batch inference that can feed-forward parallel for multiple periods by period-conditional universal estimator; 2) time-shared conditional representation extracted from Mel-spectrogram, which is utilized for every step.
3.3 FLOW MATCHING FOR WAVEFORM GENERATION
To the best of our knowledge, this is the first work utilize flow matching for waveform generation. In this subsection, we describe the problems we encountered and how to reduce these issues. First, we found that the it is crucial set the proper noise scale for Tg. In general, waveform signal is ranged with - to 1, so standard normal distribution AN 1) would be large for optimal path. This results in high-frequency information distortion, causing the generated sample to contain only low-frequency information. To reduce this issue, we scale down the z0 by multiplying a small value Although we successfully generate the waveform signal by small a, we observed that the generated sample sometimes contains a small white noise. We simply solve it by additionally multiplying temperature T on the 2n as analyzed in Table Furthermore, we adopt data-dependent prior (Lee et al.) (022b) to flow matching- based generative models. Specifically, simply extracted by averaging the Mel-spectrogram along the frequency axis. We set (000, 2) for the distribution of (=)),, and multiply Σ by small value of 0.5. All of them significantly improve the sample quality and boost the training speed.
utilize energy-based prior which can be
3.4 HIGH-FREQUENCY INFORMATION MODELING FOR FLOw MATCHING
Similar to the findings demonstrated by (Roman et al. 2023) based waveform generation models could not provide the high-frequency information well. To address this limitation, we adopt three approaches including multi-band modeling and FreeU (Si
also observed that flow matching-
et al. 2024
Multi-band Flow Matching with Discrete Wavelet Transform Previously, MBD (Roman et al. 2023) demonstrated that diffusion-basse models are vulnerable to high-frequency noise so they introduce the multi-band diffusion models by disentangling the frequency bands and introducing specialized denoisers for each band. Additionally, they proposed frequency equalizer (EQ) processor to reduce the white noise by regularizing the noise energy scale for each band Unlike MBD, we introduce discrete wavelet Transform based multi-bane modeling method which can disentangle the signal and reproduce the original signal without losing information PeriodWave-Mis consists of multiple vector field estimators for each band [0-3, 3-6, 6-9, 9-T2 kHz). Additionally, we first generate a lower band, and then concatenate the generated lower bands to the 2o to generate higher bands. We found that this significantly improve the quality even with small sampling steps. During training, we utilize a ground-trutt discrete wavelet Transform components for conditional information. Additionally, we also utilize a band-wise data-dependent prior by averaging Mel- spectrogram according to the frequency axis including overlapped frequency bands [0-61, 60-81, 80-93, 91-100 bins]. Moreover, we downsample each signal by [1.4,4] by replacing the first down/up- sampling with DWTADWT, and this also significantly reduce the computational cost by reducing time resolution.
Flow Matching with FreeU FreeU (Si et al. 2024) demonstrated that the features from the skip connection contain geghfrregency information in UNet-based diffusion models, and this could ignore the backbone semantics during image generation. We revisited this issue in high-resolution
'We entirely acknowledged (Roman et al. 2023) proposed novel pre-processing method and combined it with diffusion models well. However, do not use any pre-processing methods for a fair comparison. We introduce a novel architecture and method for high-fidelity waveform generation without any pre-processing. "We observed that using hand splitting of MBD without EQ processor results in white noise on the generated sample our preliminary study we introduce discrete wavelet Transform based multi-band modeling.
5

---

Published as a conference paper at ICLR 2025
waveform generation task. We also found that the skip features of our model contain large ratio of high-frequency information. Additionally, this also provided the noisy high- figh-freyeency information to the UBlock at the initial sampling steps. Hence, the accumulated high freeueeuuenccy noise prevents modeling the high-frequency information of waveform. To reduce this issue, we adopt FreeU by scaling down the skip features Zekip and scaling up the backbone features as follows:
(5) where we found the optimal typpe-parameters through grid search: 0.9 and B 1.1 at the Table 25 and this significantly improve the high-frequency modeling performance in terms of spectral distances. We also found that scaling up the backbone features could improve the perceptual quality by reducing the noisy sound which is included in ground-trute -uuth Mel-spectrogram.
I Saksp + ..v
4 EXPERIMENT AND RESULT
4.1 EXPERIMENTAL SETUP
Dataset We train the models using LJSpeech (Ito & Johnson 2017) and LibriTTS (Zen al.) 2019) datasets. LJSpeecc is high-quality single-speaker dataset with a sampling rate of 22,050 Hz. LibriTTS a multi-speaker dataset with a sampling rate of 24,000 Hz. Following (Lee et al. 2023). adopt the same configuration for Mel-spectrogram transformation. For the LJSpeech, we use the Mel-spectrogram of 80 bins. For the LibriTTS, we utilize the Mel-spectrogram of 100 bins.
Training For the LibriTTS dataset, train PeriodWave using the AdamW optimizer with
learning rate of 2x 10 batch size of 128 for IM steps on four NVIDIA A100 GPUs. Each band of PeriodWave-Me is trained using the AdamW optimizer with a learning rate of 2x10 batch
size of 64 for 1M steps on two NVIDIA A100 GPUs It only takes three days to train the model while GAN-based models take over three weeks. We do not apply any learning rate schedule. For the ablation study, we train the model with a batch size of 128 for 0.5M steps on four NVIDIA A100 GPUs.
Sampling For the ODE sampling, we utilize Midpoint methods with sampling steps of I Addi- tionally, we compared the ODE methods including Euler, Midpoint, and RK4 methods according to different sampling steps in Appendix D The experimental details are described in Appendix J and K
Table 1: Objective evaluation results on LJSpeech. We utilized the official checkpoints for all models.
BigVGAN models are trained with LJSpeech, VCTK, and LibriTTS datasets.
Method
Training Steps
Params (M)
M-STET (1) (T)Q
Periodicity (1)
VUV (t)
Pitch (1)
UTMOS (t)
Ground Truth
4,3804
HiFi-GAN (V1)
2.5M
14.01
10341
3.646
0.1064
0,9584
26.839
4.2691
BigVGANN-buse
5.0M
14.01
1.0016
3.868
0,1054
0,9597
25.142
4.1986
BigVGAN
5.0M
112.4
0.9369
4,210
0.0782
0.9713
19.019
4,2172
PriorGrad (50 steps)
3.0M
2.61
1,2784
3.918
0.0879
0,9661
17.728
3.6282
PreGrad (50 steps)
1.0
1.78
1,2913
3,275
0.1302
0,9490
27.317
3,1522
PerloWave-MB (16 steps)
0.5M
37,08x2
1.1722
4.276
0.0701
0.9730
4.2940
PeriodWave (16 steps)
LOM
29.23
1.1464
4.288
0.0744
0.9704
15,042
4.3243
PeriodWaver FreeU (16 steps)
LOM
29.73
1.1122
4,293
0.0749
0.9301
15.753
4.3578
4.2 LJSPEECI: HIGIN-QUALITY SINGLE SPEAKER DATASET WITH 22,050 Hz
We conducted an objective evaluation to compare the performance of the single-speaker dataset. We utilized the official implementation and checkpoints of HiFi-GAN, PriorGrad, and FreGrad (Nguyen et al. 2024). which have the same Mel- Mel-spectrogram configuration. Table shows that our model achieved a significantly improved performance all objective metrics without M-STFT. Additionally, GAN-based models take much more time to train the model due to the discriminators. Furthermore, our proposed methods require smaller sampling steps than diffusion-basee that diffusion-based model and flow matching-bass bbsed models could not model the high- figh-ffrequency information because their objective function does not guarantee the high- high-frequency information while GAN-based models utilize seessppctro loss and M-STFT- based discriminators. To reduce this issue, we utilize multi-band modeling and FreeU operation, and the results also show improved performance in most metrics.
models. We observed
DDee to the limited resources, we only used two GPUs for each band. "The results in Appendix D) show that increasing sampling steps improve the performance consistently.

---

Published as a conference paper at ICLR 2025
Table 2: Objective and subjective evaluation results LibriTTS. Following BigVGAN (Lee et al. 2023p, objective results are obtained from LibriTTS-dev subsets, and subjective results are obtained from LibriTTS-test subsets. We included the objective metrics of models' reported by BigVGAN. For MOS and Pitch, we utilize the official checkpoints of all models without UnivNet. Note that
BigVGAN-base and BigVGAN are trained for 5M steps while models are trained for 1M steps.
Method
Params (M) (L) PT)Q(1)
Periodicity (1) (T)
Pitch (U) MOS (T)
Ground Truth
3,9420.03
WaveGlow-256
99.43
1.3099
3.138
0.1485
0.9378
WaveFlow-128
22.58
1.1120
3.027
0,1416
0.9410
HiFi-GAN (V1)*
14.01
1.0017
2.947
0,1565
0,9300
UnivNet-c32
14.87
0.8947
3,284
0.1305
0.9347
$3.021
3.00
Vocos
13.53
0.8544
3.615
0.1113
0,9470
24.075
0.89
BigVGAN-base'
14.01
0,8788
3.519
0.1287
0,9459
24.432
3.91+0.03
BiANGAN
112.4
0.7997
4.027
0.1018
0.9598
25.651
3.92+0.03
PeriodWave-MB (16 steps)
37.08x4
0,9729
4.262
0,0704
0.9678
16.829
3,95+0.03
PoreodWave (16 steps)
29.80
1.2129
4.224
0.0762
0.9652
18.730
3.93#0.03
PeviodWave FreeU (16 steps)
1.0269
4.248
0.0765
0.9651
17.398
3.95+0.03
Table 3: Objective evaluation results with different training steps on LibriTTS dataset.
Methods
Training Steps
M-STFT (L)
PESQ (f)
Periodicity (1)
(1) UTMOS (1)
IM
0,9729
4.262
0.0704
0.9678
3.6534
PeriodWave-MMe
0.5M
0,9932
4.213
0,0745
0.9653
3.6142
(16 steps)
0.3M
1.0697
4.161
0.0777
0,9640
3,5641
0.15M
1.1003
4.020
0.0842
0.9580
3,4983
4.3 LIBRITTS: MULTI-SPEAKER DATASET WITH 24,000 Hz
We conducted objective and subjective evaluations to compare the performance of the multi-speaker dataset. We utilized the publicly available checkpoints of UnivNet, BigVGAN, and Vocos, which are trained with the LibriTTS dataset. Table 24 shows our model significantly improved performance in all metrics but the M-STFT metric. A Although other GAN-based models utilize Mel-spectrogram distance loss and multi-resolution spectrogram discriminators which can minimize the distance on the spectral domain, we only trained the model by minimizing the distance of the vector field on the waveform. However, our model achieved better performance subjective evaluation. Specifically, models have better performance the periodicity metrics, and this means that our period- -awar structure could improve the performance in terms of pitch and periodicity by significantly reducing the jätter sound. Both PeriodWave-Mo and PeriodWave demonstrated significantly lower pitch error distances compared to BigVGAN. Specifically, PeriodWave-MB and PeriodWave (FreeU) achieved pitch error distance of 16.829 and 18. 730 (17.398), respectively, while Big VGAN's pitch error distance was 25.651. Table 3 also demonstrated the fast training speed of PaviodWave. The model trained for 0.1 steps could achieve comparable performance compared to baseline models which are trained over 1M steps.
Table 4: Objective evaluation results with different temperature T.
Methods
Temperature:
M-STIT (1)
(f)
Periodicity (1)
VUV F1 (1)
UTMOS (f)
1.0
0.9363
4.152
0.0721
0.9679
3.5194
PrriodWave-MB
0.667
0.9729
4.262
0.0704
0.9678
3.6534
0.333
1.0915
4,278
0.0729
0.9668
3.5457
0.1
1.3062
3,847
0.0788
0.9634
3.1442
4.4 SAMPLING ROBUSTNESS, DIVERSITY, AND CONTROLLARILITY
We utilize flow matching model for PeriodWave, allowing it to generate diverse samples with different Gaussian noise. However, our goal is a conditional generation using the Mel-spectrogram. We need to decrease the diversity to improve the robustness of the model. To achieve this, we can multiply the small scale of temperature 7 to the Gaussian noise during inference. Table 4 shows that using 7 of 0.667 could improve the performance. We also observed that samples generated with of 1.0 contain a small amount of white noise, which decreases perceptual quality despite having the lowest lowest M-STFT metrics. Furthermore, we could control the energy for each band by using different scales of r. This approach could be utilized for a neural EQ that can generate the signal by reflecting the conditioned energy, not merely manipulating the energy of the generated samples.
7

---

Published as a conference paper at ICLR 2025
Table 5: Objective evaluation results out-of-distribution samples from MUSDBI8-HQ We
evaluated Periodicity, and V/UV Fl on vocal samples from MUSDBI8 18-HQ.
Method
M-STFT (1)
PESQ (1)
Periodicity (1)
(1)
UnivNet-c32
1.1377
1.678
0.1588
0.9186
Vocos
1,000
2.173
0.1305
0.9454
BigVGAN-basc
10132
2.315
0.1272
0.9307
BigVGAN
0.9062
2.862
0.0959
PeriodWave-MB (16 steps)
1,0900
3,120
0.0945
0.9524
PeriodWave steps. Midpoint)
PeriodWave + FreeU (16 steps. Midpoint)
1.1923
3,062
0.0994
0.9479
Table 5-scale SMOS results on out-of-distribution samples from MUSDBI8-HQ
Method
Vocal
Drums
Bass
Others
Mixture
Average
Ground Truth
3,85+0.11
4.00+0.11
3.83=0.11
4.01+0.11
4.0340.10
3.94
UnivNet-c32
3.3
3,40#0.16
2.8900116
2.92+0.18
2.80+0.15
3.06+0.07
Vacos
3.57
3.6420.13
2.89=0116
3.21
3.1640.13
3.29
BigVGAN-bong
3.6410.13
3.6810.13
3.0720.14
3.3120.15
3.51
3.4420.06
VIAN
3.6310.12
4.0110.12
3.1320.13
3.5310.15
3.5610.13
3.56110.66
PeviodWave (16 steps)
3.70+0.12
3.76+0.14
3.38+0.13 3.20+0.15
3.44+0.13
3.50+0.06
PeriodWave-MB (16 steps)
3,72+0.12
3.71+0.13
3.52=0.13
3.72+0.14
3.51
3,63=0.06
4.5 MUSDBIS H-H:: MULTI-TRACK MUSIC AUDIO DATASET FOR OOD ROBUSTNESS
To evaluate the robustness on the out-of-distributin samples, we measure performance the MUSDBIS-HQ dataset that consists multi-track music audio including vocals, drums, bass, others, and mixture. We utilize all test samples including 50 songs with 5 tracks, and randomly sample the 10-second segments for each sample. Table shows our model has better performance on all metrics without M-STFT. Table shows that Paveod Wave-MB outperformed the baseline models by improving the out-of-distribution robustness. Specifically, we significantly improve the performance of bass, the frequency range of which is known between 40 400 Hz. Additionally, we observed that our model significantly reduces the jitter sound in the out-of-distribution samples.
Table 7: Ablation study LibriTTS. All models trained for 0.5M steps.
Method
Period
M-STFT (1) (E)Q Periodicity (1) VUV FI (1)
UTMOS (f)
Ground Truth
3.8626
PeriodWave-MB
[1.2.3.5.7]
0,9932
4.213
0.0745
0.9653
3.6142
PeviodWave
[1.2.3,5.]]
L.737
4,072
0,0806
0.9627
3.5544
PeriodWave w/o Prior
[1,2.3.5.7]
L,3754
3,900
0.0930
0.9562
3,5352
PeriodWave Encoder
[122.3.5.7]
1,5194
2.511
0.1093
2.6737
PeveodWave
1,2588
3.795
0,0885
0,9572
3.4215
[1.1.1111]
I.1337
3,964
0.0888
0.9597
3.4728
[1,1,13,33
[1,1,13.9]]
1,2736
4.061
0.0830
0.9644
3.5057
[1,2,4.6.]]
1.1481
4.075
0.0782
0.9647
3.5468
[1,2.48.16]
L,463
4.124
0.0787
0,9639
3.5408
(1,23,5,,,1111331777
1.1617
4.125
0.0792
0.9610
3,5384
4.6 ABLATION STUDY
Different Periods We conduct ablation study for different periods at the same structure. Table shows that the model with period of shows the lowest performance. Increasing the number of periods could improve the entire performance in terms of most metrics, consistently. However, this also improves the computational cost and requires more training steps for optimizing various periods in single estimator so we fix the model with the period of [1,2,3,5.7]. Meanwhile, we compared the model with periods of [1,2,4,6,8] and [1,2,4,8, to demonstrate the effectiveness of the prime number for the period. We observed that using prime number could improve the UTMOS slightly and the model with periods of [1,2,466,8] and [1,2,4,8, 4,4,861,6 1,16 also have comparable performance, which reflect the different period representations of the waveform. We thought that the model with periods of [1,2,3,5,7,1 1,1,,17] requires more training steps. The model with periods of 1,11 showed slightly better performance than the model with a periods of [1]. However, significantly underperforms compared to other models using different size of periods. The models with periods of [1,1,1,3,3] and [1.1,1.3.9] have better performance than [1,1,1,1,1], and the results demonstrate that using different periods could perform better: This also demonstrates that our new waveform generator structure suitable for waveform generation.
8

---

Published as a conference paper at ICLR 2025
Prior PriorGrad demonstrated that data-dependent prior information could improve the performance and sampling speed for diffusion models. We also utilize the normalized energy which can be prior information. We observe that the data-dependent prior could improve the quality and sampling speed in flow matching based models. Meanwhile, although we failed to implement the quality reported by SpecGrad (Koizumi et al. 2022).
extracted -el-spectrogram
that the spectrogram based prior could improve the performance rather than the mnergy-basek prior.
Mel Encoder Our Mel encoder significantly improved the performance through efficient time- frequency modeling. This only requires a small increase in computation cost because we reused the extracted features which are fed the period- period-aware aware flow matching estimator for each sampling step.
Table 8: Comparison of Diffusion and CFM. All models are trained for 1M steps.
Method
Steps
M-STFT (4)
PESQ
Periodicity (1)
FI (1)
Pitch (4)
UTMOS (T)
PeriodWave w/ CFM
32
1.072
4.233
0.078
0,964
17.418
3.646
PeriodWave w/ CFM
25
1.159
4.233
0.078
0,964
17.420
3,650
PeriodWave w/ CFM
16
1,212
4,224
0.076
0,965
17.496
3,649
PeriodWave w/ CFM
6
1,379
4.178
0.082
0.959
23.223
3.628
PeriodWave w/ DDPM
50
1.159
4.151
0.084
0,961
23,046
3,377
PeriodWave w/ DDPM
6
1,233
3,541
0.095
0.958
24.351
2,953
CFM Diffusion We compared the model trained with diffusion and CFM to demonstrate the effectiveness of CFM for waveform generation. For a fair comparison, we utilize the same model architecture. Furthermore, we utilize a prior-based diffusion models following PriorGrad (Lee et al. 20226). We utilize the same noise scheduling method of PriorGrad. For time steps of 6, we utilized noise scheduling of [0.0001, 0.008, 0.010.05, 0.7, 0.9]. Table 8 shows that CFM outperformed diffusion in terms of all metrics. Furthermore, this indicates that CFM has several advantages in performance and efficiency where the model with CFM has better performance even with smaller sampling steps than the model trained with DDPM. Furthermore, our model was trained with a continuous time t while DDPM should be trained with discrete time t. We observed that training the diffusion-based model for continuous time t requires much more training times to optimize the models. However, while our model was trained with a continuous time t, our models show a faster convergence speed in that Table & indicated the effectiveness of CFM compared to diffusion.
4.7 MULTI SPEAKER TEXT- TS-PEECH
We conduct two-stage multi-speaker TTS experiments to further demonstrate the ro- bustness of the proposed models compared to previous large-scale GAN-based mod- els including BigVGAN and VSAN (Shibuya et al. 2004). Note that BiAN and VSAN were trained for 5M and 10M steps, respectively, We utilize ARDīT- TTS (Liu et al. 2024b) as zero-shot TTS model which was trained with LibriTTS dataset. We convert 500 samples of generated Mel-spectrogram into waveform signal by each model. The Table 9 shows that our model has better performance on the objective and subjective metrics in terms of UTMOS and MOS. Furthermore, Our model with FreeU has much better performance than others. We can discuss that FreeU could reduce the high-frequency noise resulting better perceptual quality.
Table 9: Zero-shot TTS Results. We utilized ARDIT- TTS trained with LibriTTS as TTS model.
Methods
MOS (1)
UTMOS (T)
BigVSAN BiAVGAN
3,99+0.01
3.9732
4.03+0.01
4.0424
PeriodWave (16 steps)
4.0620.01
4.2209
PeriodWave + FreeU (16 steps)
4.07#0.01
4.2621
We also discussed the train-inference mismatch problem of two-stage TTS and the effectiveness of iterative refinement to reduce this problem and added single-speaker TTS results in Appendix G
4.8 LIMITATION
Although our models could generate the waveform with small sampling steps, Table E shows that our models have slow synthesis speed compared to GAN-based models. To overcome this issue, we will explore distillation methods or adversarial training to reduce the sampling steps for much more fast inference by using our period-aware structure. Additionally, our models still show lack of robustness in terms of high-frequency information because we only train the model by estimating the vector fields on the waveform resolution. Although multi-band modeling reduced this issue, it requires more complex model designs. So, we have a plan to add a modified spectral objective function or blocks that can reflect the spectral representations when estimating vector fields by utilizing short-time Fourier convolution proposed in (Han & Lee (022) for audio super-resolution. Moreover, we see that classifier-free guidance could be adapted to our model to improve the audio quality.

---

Published as a conference paper at ICLR 2025
Table 10: Objective evaluation results of parallel generation from discrete token.
Mathod
Params (M)
CFR (1)
WER (1) ()STFT()) (T)
Perialicity (1)
VUVV (1)
(lich
UTMOS (t)
Ground Truth
0.92
2.94
3,8626
Mina (2 32)
79.30
1.28
3.81
L,1677
3,469
0.0886
0.9541
28,084
3,8005
(Q) 10)
79.30
1.35
3.81
1.2199
2.911
0.9345
34,863
3.7455
Mini ((2
79.
3.07
6.91
1.3509
2,266
0.9144
50.679
3,5068
PerkodWave (2- - 8, stepe)
35.80
2.50
5.56
1,2651
2.293
0.1413
0.9244
44,889
3.8877
PeriodWave (O- 8,4 steps)
35.86
2.40
5.45
1.2432
2.256
0.1424
0.9254
44.691
3.334
Table 11: Objective evaluation results of streaming generation from discrete token.
Method
CER (1)
WER
(LSTFT
(T)
Periodicity (1)
VUVV (f)
Piihh UTMOS
(Q-
79.
3.05
6.93
1.3522
0.9104
50.686
PeriodWave (Q= 8, stepe)
35.86
2.45
5.53
1,2716
2:233
0.1429
0.9228
40.634
3.850
AUDIO GENERATION FROM DISCRETE TOKEN
We conducted experiments for parallel and streaming generation from discrete tokens. We used Mimi of Moshi (Défossez et al.). state-of-the-art neural audio codec that operates at 12.5 Hz. Note that the number of codebooks in Mimi can be up to 32, but Moshi utilized a Q = 8 quantizer for speech language models, we also used the same discrete tokens from the eight quantizer as input to model instead of Mel-spectrogram. Specifically, we used the post-quantized latent representation as input for PeviodWave. For streaming generation, introduce single token delayed generation. Although the model was trained for parallel generation, the samples with streaming generation show minimal degradation. The details of streaming generation are described in Appendix I
and Figure 4 We trained the model using segments of 48,000 frames (2s waveform and 25 token embeddings). We modified the Mel-encoder by adding an additional upsampling layer to upsample a high- tigh-compeessed codec at 12.5 Hz to waveform signal with sampling rate of 24,000 Hz. Since energy can not be extracted from discrete tokens, we remove the energy-based prior. Furthermore, we fine-tuned Period- Wave by fixing the number of iteration steps to accelerate the inference speed for streaming generation, and utilized an adversarial training to improve the performance without using reconstruction loss.
We first conducted CER and WER evaluations using Whisper-large-v3-trie (Radford et al. 2023) to evaluate the pronunciation and semantic consistency of generated samples from low-bitrate discrete tokens. Table T0 shows that our model can generate speech with better pronunciation by preserving semantic information. Also, our models with Q = 8 quantizer have better naturalness than Mimi with Q = 32 quantizer in terms of UTMOS. PaviodWave can enhance the audio quality by increasing the iteration steps in parallel generation. Furthermore, Table [T demonstrates our proposed streaming generation can synthesize the speech experiments of streaming generation are reported in Appendix | and Table 20)
streaming manner with a minimal degradation. Additional
We believe that our model can be used for alternative decoder of neural audio codec models. We see that our model with parallel generation can facilitate the high-quality dialogue data collection from speech language models thanks to enhanced audio quality. However, there is room for improvement of efficiency in streaming audio generation. In future. we have plan to fine-tune the model with chucked auto-regressive or masking-infilllig for in-context streaming generation.
6 CONCLUSION
In this work, we proposed PeriodWave, a novel universal waveform generation model with conditional flow matching. Motivated by the multiple periodic characteristics of high-resolution waveform signals, we introduce the period-aware flow matching estimators which can reflect different implicit periodic representations when estimating vector fields. Furthermore, we observed that increasing the number of periods improve the performance, and we introduce a period-condittonal universal estimator for efficient structure. By adopting this, we also implement a period-wiss -wiee batch inference for efficient inference. The experimental results demonstrate the superiority of our model in high- quality waveform generation and OOD robustness. Furthermore, we demonstrate the effectiveness of proposed model in neural audio codec decoding tasks both parallel and streaming generation. GAN-based models still hold great potential and have shown strong performance but require multiple loss functions, resulting in complex training and long training times. On the other hand, we introduced flow matching based approach using single loss function, which offers notable advantages in both efficiency and performance.
We released all source code and checkpoints, hoping that approach will facilitate research in waveform generation. We believe that its efficiency and ficxibility make an ideal backbone, easily adaptable to domain-specifie personalized data, and fine-tuning with adversarial training.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
We'd d like to thank Yeongtae Hwang for helpful discussion and contributions to our work suggesting FreeU to address high-frequency error of diffusion models. We sincerely thank Zhijun Liu, the author of ARDiT-TTS for providing the Mel spectrograms of ARDiT-TTS, which enabled us to perform the second stage of TTS synthesis. This work was supported by Institute of Information communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) SUTTP-2025-R520222 0023-00255968, the Artificial Intelligence Convergence Innovation Human Resources Development and No. 2021-0-02063 Artificial Intelligence Innovation Hub) and Artificial intelligence industrial convergence cluster development project funded by the Ministry of Science and ICT(MSIT, Korea)&GGaangi) Metropolitan City.
REFERENCES
Anonymous. RFWave: Multi-band rectified flow for audio waveform reconstruction. In Submitted to The Thirteenth International Conference on Learning Representations, 2024. URL https under review.
net / -iw.rrmmmmmddggggtttttt
Taejun Bak, Junmo Lee, Hanbin Bae, Jinhyeok Yang, Jae-Sung Bae, and Young-Sun Joo. Avocodo: Generative adversarial network for artifact- free vocoder. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 12562-12555 12570, 2023.
Mikolaj Birikowski, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks, In International Conference on Learning Representations, 2020. URL [RRL/tttt////peeeeeee
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2021. URL /Rtl htt.eeeewweeeeoooooaaaaaaaaaaaar
Hyeong-Sof Srecong-Sook&ao Choi, Juheon Lee, Wansoo Kim, Jie Lee, Hoon Heo, and Kyogu Lee. Neural analysis and synthesis: Reconstructing speech from self-supervised representations, Advances in Neural Information Processing Systems, 34: 1121111661266 2021.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL Featured Certification, Reproducibility
//openreview. net. www.t//etttteinoooooooo Certification.
forum? iimmmii==iiee wtw.rrmiidiiiiooo2ssmmm
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint #xXiv:2210.334333 2022.
Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung Chung-Hsien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang. Xu Tan, E2 tts: Embarrassingly easy fully non- autoregressive zero-shot tts. arXiv preprint arXiv:2406.18000 2024.
Alexey Gritsenko, Tim Salimans, Rianne van den Berg, Jasper Snoek, and Nal Kalchbrenner. A spectral energy distance for parallel speech synthesis. Advances in Neural Information Processing Systems, 33:13062-13077 13072, 2020.
Yicheng Gu, Xueyao Zhang, Liumeng Xue, and Zhizheng Wu. Multi-scale sub-band constant-q transform discrininator for high-fidelity vocoder. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing ( (ICASSP), pp. 10616-10620. IEBE, 2024.
Seungu Han and Junhyeok Lee. NU-eave 2: A General Neural Audio Upsampling Model for Various Sampling Rates. In Proc. Interspeech 2022, pp. 4401-4405, 2022. doi: 10.21437//mtesseecch. 2022-45.
11

---

Published as a conference paper at ICLR 2025
Rongji Huang. Max W. Y. Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. Fastdiff: A fast conditional diffusion model for high-quality speech synthesis. In Lud De Raedt (ed.), Proceedings of the Thirty-Firt First International Joint Conference Artificial Intelligence, IICAI-22. pp. 4157-41633 International Joint Conferences on Artificial Intelligence Organization, 2022a. 2022/577 wtw.o/////..22......
doi: 1.229663/jjaai22222/577 URL
Main Track.
Rongjie Huang. Max WY Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao. Fastdiff: A fast conditional diffusion model for high-quality speech synthesis. arXiv preprint arXiv:2204.09334, 2022b.
Rongjie Huang. Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech. Advances in Neural Information Processing Systems, 11110970-19983, 2022c.
Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pp. 13916-13929 13932. PMLR, 2023.
Keith Ito and Linda Johnson. 2017.
The lj speech dataset.
htttp:://eithitto com/
L LJ- MW-Speech-Dataseee
Won Jang, Dan Lim, Jaesam Yoon, Bongwan Kim, and Juntae Kim. UnivNet: A Neural Vocoder with
Spectrogram Discriminators for High-Fidelity Waveform Generation. In Proc. Interspeech 2021, pp. 2207-2211, 2021. doi: 10.2 10221457/meespeecch000-- 202111100......1166...
Won Jang, Dan Lim, and Heayoung Park. FastFit: Towards Real-Time Iterative Neural Vocoder by Replacing U-Net Encoder With Multiple STFTs, In Proc. INTERSPEECH 2023, pp. 4364-4368, 2023. doi: 10221433/mieerppcchh2222333
Shengpeng Ji, Ziyue Jiang, Xize Cheng. Yifu Chen, Minghui Fang, Jialong Zuo, Qian Yang, Ruiqi Li, Ziang Zhang, Xiaoda Yang, et al. Wavtokenizer: efficient acoustic discrete codec tokenizer for audio language modeling. arXiv preprint arXiv:2408. 16532, 2024.
Ziyue Jiang, Jinglin Liu, Yi Ren, Jinzheng He, Zhenhui Ye, Shengpeng Ji, Qian Yang. Chen Zhang, Pengfei Wei, Chunfeng Wang, Xiang Yin, Zejun MA, and Zhou Zhao. Mega-TTS 2: Boosting prompting mechanisms for zero-shot speech synthesis. In The Twelfth International Confer- ence on Learning Representations, 2024. URL pttttp:://ppereviiie net www.:/wwwwt//tootmt/e/ forum? d
mvMT 3N4AvD
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et Naturalspeech 3: Zero-shot speech synthesis with factorized 2024.
codec and diffusion models. arXiv preprint
Takuhiro Kaneko, Kou Tanaka, Hirokazu Kameoka, and Shogo Seki. istfinct: Fast and lightweight mel-spectrogram vocoder incorporating inverse short-time fourier transform. In ICASSP 2022- 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6207-6211 IEEE, 2022.
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 33:8067-8077, 2020.
Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pp. 5530-5540. PMLR, 2021.
Sungwon Kim, Sang- Gil Lee, Jongyoon Song, Jachyeon Kim, and Sungroh Yoon. Flowavenet: A generative flow for raw audio. In International Conference on Machine Learning, pp. 3370-3378. PMLR, 2019.
12

---

Published as a conference paper at ICLR 2025
Sungwon Kim, Kevin Shih, Joao Felipe Santos, Evelina Bakhturina, Mikyas Desta, Rafael Valle, Sungroh Yoon, Bryan Catanzaro, et al. P-flow: A fast and data-efficient zero-shot tts through speech prompting. Advances in Neural Information Processing Systems, 36, 2024.
Yuma Koizumi, Heiga Zen, Kohei Yatabe, Nanxin Chen, and Michiel Bacchiani. SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping. In Proc. Interspeech 2022, pp. 803-807, 2022. doi: 10.2443/hnecspecch222--00
Yuma Koizumi, Kohei Yatabe, Heiga Zen, and Michiel Bacchiani. Wavefit: An iterative and non- autoregressive neural vocoder based on fixed-point iteration. In 2022 IEEE Spoken Language Technology Workshop (SLT), pp. 884-891. IEEE, 2023.
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33:17022-17033, 2020.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. https:// http::::///peerrvviiiee
net/ forum? d=a XF K8 Ymz5J x
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations, 2023. URL https http://ooommiidxx//eee-ee
//openreview
Kundan Kumar, Rithesh Kumar, Thibault De Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre De Brebisson, Yoshua Bengio, and Aaron C Courville. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32, 2019.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Systems, 36, 2024.
Adrian Lańcucki. Fastpitch: Parallel text-to-speech with pitch prediction. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6592. IEEE, 2021.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances neural information processing systems, 36, 2024.
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, et al. Direct speech-to-speech translation with discrete units. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3327-3339, 2022a.
Keon Lee, Dong Won Kim, Jachyeon Kim, and Jaewoong Cho. Ditto-tts: Efficient and scalable zero-shot text-to-speech with diffusion transformer. arXiv preprint arXiv:2406. 1147, 2024a.
Sang-gil Lee, Sungwon Kim, and Sungroh Yoon. Nanoflow: Scalable normalizing flows with sublinear parameter complexity. Advances in Neural Information Processing Systems, 33:14058- 14067, 2020.
Sang-gil gil Lee, Heeseung Kim, Chachun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, and Tie-Yan Liu. Priorgrad: Improving conditional denoising diffusson models with data-dependent adaptive prior. In International Conference on Learning Representations, 2022b. URL hhtttps::/ppperrtttttet
ew n forttm/ormm? i d= BNe__NNiNN11 1
Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. BigVGAN: A universal neural vocoder with large-scale training. In The Eleventh International Conference on Learning Representations, 2023. URL htt.hhttts::/ppeeeeeeee httprrrrreeeeeeeeeooorddd-dddd-d T
13

---

Published as a conference paper at ICLR 2025
Sang-Hoon Lee, Ji-Hoon Kim, Hyunseung Chung, and Seong- Whan Lee. Voicemixer: Adversarial voice style mixup. Advances in Neural Information Processing Systems, 34:294-308, 2021.
Sang-Hoon Lee, Ji-Hoon Kim, Kang-Eun Lee, and Seong-Whan Whan Lee. Fre-gan 2: Fast and efficient frequency- conssssecyy audio synthesis. In ICASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6192-6196. IEEE, 2022c.
Sang-Hoon Lee, Scung-Bing Bin Kim, Ji-Hyun Lee, Eunwoo Song, Min-Jae Hwang, and Seong- Whan Lee. Hierspeech: Bridging the gap between text and speech by hierarchical variational inference using self-supervised representations for speech synthesis. Advances in Neural Information Processing Systems, 35:16624-16636 2022d.
Sang-Hoon Lee, Ha- Yeong Choi, and Seong- Whan Lee. Accelerating high-fidelity waveform generation via adversarial flow matching optimization. arXiv preprint arXiv:2408.0N) 2024b.
Yinghao Aaron Li, Cong Han, Vinay Raghavan, Gavin Mischler, and Nima Mesgarani. Styletts 2:
Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. Advances in Neural Information Processing Systems, 36, 2024.
Dan Lim, Sunghee Jung, and Eesung Kim. JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech. In Proc. Interspeech 2022, pp. 21-25, 2022. doi: 10.214377/nterspeechh 1029-.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint 02X2022200.02747, 2022.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 21450 214744. PMLR, 23-29 Jul 2023.
Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley. Semanticodee: An ultra low bitrate semantic audio codec for general sound. arXiv preprint arXiv:2405.00233, 2024a.
Zhijum Liu, Shuai Wang. Sho Inoue, Qibing Bai, and Haizhou Li. Autoregressive diffusion transformer for text-to-speech synthesis. arXiv preprint arXiv:2406.0555/ 2024b.
Max Morrison, Rithesh Kumar, Kundan Kumar, Prem Seetharaman, Aaron Courville, and Yoshua Bengio. Chunked autoregressive GAN for conditional waveform synthesis. International Conference Learning Representations, 2022. URL httttts::/pppreee///// nww./orunfrrum
id=v3aeTsY_vo VVX
Tan Dat Nguyen, Ji-Hoon Kim, Youngjoon Jang, Jaehun Kim, and Joon Son Chung. Fregrad: Lightweight and fast frequency-aware diffusion vocoder. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10736-10740. IEEE, 2024.
Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukeuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fidelity speech synthesis. In International conference on machine learning, pp. 3918-3926. PMLR, 2018.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint
2016.
Wei Ping, Kainan Peng, and Jitong Chen. Clarinet: Parallel wave generation in end-to-end text- to-speech. In International Conference on Learning Representations, 2019. URL https //openrevi ew n bobormm?i-HH. 1 Y120cYm)
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3617-3621. IEBE, 2019.
14

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pp. 18492-28518 PMLR, 2023.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech: Fast, robust and controllable text to speech. Advances in neural information processing systems, 32, 2019.
Robin San Roman, Yossi Adi, Antoine Deleforge, Romain Serizel, Gabriel Synnaeve, and Alexandre Défossez. From discrete tokens to high-fidelity audio using multi-band diffusion. In Thirty-seventh Conference on Neural Information Processing Systems, 2023, URL hatps::/openreview
net/forum? ieeeeidd=do 1d=d0ankg jKS
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural us synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4779 4783. IEEE, 2018.
Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yichong Leng, Lei He, Tao Qin, sheng zhao, and Jiang Bian. Naturalspeech Latent diffusion models are natural and zero-shot speech and singing synthesizers. In The Twelfth International Conference on Learning Representations, 2024. URL net/ forum? wcw.dddd--dddooloc
Takashi Shibuya, Yuhta Takida, and Yuki Mitsufuji. Bigvsan: Enhancing gan-based neural vocoders with slicing adversarial network. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10121-10125. IEEE, 2024.
Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. In CVPR, 2024.
Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. In The Twelfth International Conference on Learning Representations, 2024. wtlltttss//ppenzeeeee
Christian J Steinmetz and Joshua D Reiss. auraloss: Audio focused loss functions in pytorch. In Digital music research network one-day workshop (DMRN+ 15), 2020.
Xu Tan, Tao Qin, Jiang Bian, Tie-Yan Liu, and Yoshua Bengio. Regeneration learning: A learning paradigm for data generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 22614 22622, 2024.
Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Kilian Fatras, Guy Wolf, and Yoshua Bengio. Conditional flow matching: Simulation-free dynamic optimal transport. arXiv preprint arXiv:2302.00448, 2(3), 2023.
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111 2023.
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16133-16142, 2023.
Detai Xin, Xu Tan, Shinnosuke Takamichi, and Hiroshi Saruwatari. Bigcodee: Pushing the limits of low-bitrate neural speech codec. arXiv preprint arXiv:2409.05377, 2024.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1199-6205 IEEE, 2020.
15

---

Published as a conference paper at ICLR 2025
Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. Hifi-codec: Group-residus vector quantization for high fidelity audio codec. arXiv preprint arXiv:2305.00275, 2023a.
Dongehao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00704, 2023b.
Dongehao Yang. Jianwei Yu, Helin Wang. Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete diffusion model for text-to-sound generation. /EEE/ACM Transactions Audio, Speech, and Language Processing, 2023c.
Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, Yike Guo, and Wei Xue. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. arXiv preprint arXiv:2408. 17175, 2024.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-500 507, 2021.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: Corpus Derived from LibriSpeech for Text-to-Speech. Proc. Interspeech 2019, pp- 1526-15300 2019. doi: 10.21457/meerpeecch20000 201999e9chh..1922411.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech language models. In The Twelfth International Conference on Learning Representations, 2024.
16

---

Published as a conference paper at ICLR 2025
A FLOW MATCHING WITH OPTIMAL TRANSPORT TANSPORTTE PATH
HITH
In the data space R let us consider an observation x E R sampled from an unknown distribution q(=). Continuous Normalizing Flows (CNFs) transform a simple prior Po into target distribution
Leraalizing Pi = q using a time-dependent vector field ve : [O,, 1] x IR R. The flow Φe [O., 1] x R - RR.
is defined by the ordinary differential equation.
d ग x u2 (x))))));
Φ0(x) x. It Po:
(6)
where (x) denotes the state of the system at time t, driven by the vector field in (+; The
probability density path Pt [0, 1] æ² - Rx0 of this flow can be derived using the change of variables. Specifically, this system transforms the initial probability density Po to Pr. at time t, an variaaaii
and the
re p probability density p. is given by:
OØp' Oy
pvly) = po(
(w)) (w)) det
Given samples from an unknown data distribution ((x), our objective is to transform a simple initial distribution Po (e.g., standard normal) into a target distribution pl q. The challenge lies in the fact that both Pr and the corresponding vector field Uy that generates p., are generally unknown.
To address this, (Lipman et al. 2022) introduce the flow matching objective, which aims to match the vector field u. (2) to an ideal vector field u, (1) that would generate the desired probability path Pt- The flow matching training objective involves minimizing the loss function M. which is defined by regressing the model's vector field (t, x) to a target vector field u, (z) as follows:
LrM (0) = Ee-co.11=-p((=s) |l (o(
- un (x))11
(8)
Here, f U[O, 1], and (x;; f) is a neural network with parameters However, direct access to the and pr is challenging, prompting the introduction of Conditional Flow Matching (CFM):
- (=(+)))11
(f) = Ee--oo.])=--eeet [l æ) -
This expression replaces the impractical marginal probability density and vector field with conditional probability density and conditional vector field, enabling a feasible approach. Crucially, ICFM( and [FM(() share identical gradients with respect 0, ensuring equivalent efficacy in model training. The probability path (I) and the associated vector field (x) can be expressed conditionally as follows:
f pa p (=)=)))(()) and ux(x)= = S Pe (=(())))n((12)) p( (x(
(x)) =
(10)
where p(=) is an arbitrary conditional distribution independent of x and t. Assuming the existence of optimal vector field u, the neural network (2; f) can learn this vector field. Furthermore.
conditien distribbot
(Lipman et al., 2022) indicates that conditional vector field estimation is equivalent to Recondittonn vector
vector field estimation:
min pepn - vz( 0)11 mii (( (x) ll x1) ();1 (1))
where Pa(x x1) Po(x) and pı (z .1) = N2 | x2, 1) assume sufficiently small a. Lastly, generalizing this technique with the noise condition zo ~ N(0, NO,,1) 1), we consider the OT-CFM loss as follows:
LOTCEM (0) = E, (es. 11 OT (o) llu
(xr) I 11) - - ( OT (#r) 14: m;1),
(12)
where is the wrame-wise predicted mean of 21, and represents the flow from zu to TT. The target conditional vector field ue (Ø, ot (xa) xi) = o
Φ: or (zx) = (1 - (1 - ) + tæ OT
I - (1 - mmin) enhances performance due to its inherent linearity. This approach efficiently sademty manages data transformation and significantly enhances fraiiing speed and efficiency by integrating optimal transport paths.
17

---

Published as a conference paper at ICLR 2025
B IMPLEMENTATION DETAILS
For reprodueibility, we will release all source code, checkpoints, and generated samples athttps::/
periodwave. github...... io/ demo/ at Table 12
We also describe the hyperparameter details of our models
Table 12: Hyperparameters of Period
Module
Hyperparameter
PeriodWave
PeriodWave-MB
Downsampling Ratio
[1,4,4.4]
[1,4,4.1]
Upsampling Ratio DBlock Hidden Dim
[4,4,4]
[4,4,1]]
(32,64,188)
[32,128,512]
MBlock Hidden Dim
512
512
Period-aware
UBlock Hidden Dim
[128,64.32]
[512,128,32]
FM Estimator
ResBlock Kernel Size
[3,3]
[3,3]
(UNet)
ResBlock Dilation Size Period
[1,2]
[1.2]]
[1,2,3,5,7]
[1,2,3,5,7]
Activation
SiLU
SiLU
Final ResBlock Kernel Size
[3,3,3]
[3,3,3]
Final ResBlock Dilation Size
[1,2,4]
[1,2,4]
Time Embedding
256
256
Cond. Layer
Period Embedding
256
256
MLP
[512, 2048, 512]
[512, 2048, 512]
Mel Embedding
512
512
First ConvNext V2 Blocks
8
Hidden Dim
1536
1536
Drop Path
0.1
0.1
Mel Encoder
Upsampling Ratio
4
Upsampling Dim
256
Second ConyNext V2 Blocks
4
Second Hidden Dim
1024
1024
Downsampling ratio
[1,2,3,5.7]
[1,2,3,5.7]
Output Dim
512
512
Full-hand Energy Max/Min First-band Energy Max/Min
9. 9.12433660.0316227822
8.75663370.0244998881
First-band Start/End Bin
[0:61]
Energy-based Prior
Second-band Energy Max/Min Second- Start/End Bin
4.242267000044491177
[60:81] 1]
Third-band Energy Max/Min
3.101 1455/.0011401777
Third-band Start/End Bin
[80:93]
Fourth-band Energy Max/Min
2.3407057/0.03
Fourth-band Start/End Bin
[91:100]
FFT Size
1024
1024
Mcl- spectrogram
Hop Size Window Size
256
256
1024
1024
Bins
100
100
FO Min/Max
0/12000
0/12000
Training Step
1M
1M
Learning Rate
2 x 10
2 x 10
Learning Scheduling
Batch Size
128
64
Others
GPUs
2
Noise Scale
0.5
0.5
Segment Size
32,768
32,768
Temperature 7
0.667
0.667
ODE Sampling Steps
16
16
0.9
b..
1.1
18

---

Published as a conference paper at ICLR 2025
Radlis
H N.20xx4p 41p1
Tecapeoal
Flo Malching Estimato
G21 VBlock
NBacktt.112 SpoRktp
Redliek
Redlick
Redtieck
Busthic
Peradly at
Doerlt
SLU
SLU
Pudodp
L,T)
Condibon
(e) Pertec-ooemm Plaw Kanding Estrmtar
dei Dillckk
idi Mitleck
Figure Architecture of PariodWave
Table 13: Objective evaluation results on LISpeech. We utilized the official checkpoints for all
models. BigVGAN models are trained with LJSpeech, VCTK, and LibriTTS datasets.
Training Steps
M-STET PESQ Periodicity
WUV DUVFLIh (()
UTMOS (t)
Ground Truth
4,3804
HiPi-GNN (V1)
2.50M
14.01
1,0441
3.646
0.1064
0,9584
26,839
4,2691
BigVGAN-
5.00M
14.01
1,0046
3.868
0,1054
0,9597
25.142
4.1986
BigVGAN
5.00M
112.4
0,9369
4.210
0,0782
0,9713
19.019
4,2172
PriorGrad (50 steps)
3.00M
2.61
1,2784
3,918
0.0879
0.9661
17.728
3,6282
FreGrad (50 steps)
1.00M
1,78
1,2913
3.275
0.1302
0.9490
27.317
3.1522
4.0662
PeriodWave-MB
0.15M
37.15x2
1,2430
4.141
0.0759
0.9699
16.366
4.2218
steps)
0.30M
37.08x22
1,1574
4,246
0.0722
0,9726
14.426
4,2788
0.50M
37.08x2
1.1722
4.276
0.0701
0.9730
15.143
4.2940
0.0SM
29.73
1,2146
3.821
0.0982
0.9594
19.512
3,9935
PeriodWave (16 steps)
0.15M
0,9644
0.30M
29.73
1,2232
4.211
0.0884
0.9641
18,899
4.2671
0.50M
29.73
1.1574
4.310
0,0782
0,9685
16.104
4,3106
PeriodWave (1 step)
1.0M
29.73
1.5367
2.733
0.1074
0.9524
19.018
3.2725
PeriodWave (2 step)
1.0MM
29.73
1,3033
4,050
0,0853
0.9650
15.980
4.1528
step)
4.2825
Wave (8 step
1.00M
29:73
1,22222
4.269
0.0746
0.9704
14.944
4.3229
Wave (16 step)
1.0MM
29.73
1,1464
4.288
0.0744
0,9704
15.042
4.3243
PeriodWaave-Frees
1.00M
29.73
1,1132
4.293
0.0749
0.9701
15.753
4.3578
C ADDITIONAL RESULTS ON LJSPEECH
We reported additional results on LJSpeech dataset according to the training steps to demonstrate the effectiveness of our proposed methods. 13 showed that models achieved high performance with small training steps. Furthermore, training the model with 0.3M could outperformed the previous powerful GAN-based neural vocoder in all metrics without M-STFT metries. worth noting that our models are only optimized with a single loss while LAN-based methods utilize discriminator loss, feature matching loss, and Mel reconstruction loss to train the model. Furthermore, they require various discriminators to capture the different features to improve the perceptual quality, and this increase the burden to optimize the model, which requires hyper-parameter tuning including weights for each loss, learning rate, and learning rate scheduling methods.
19

---

Published as a conference paper at ICLR 2025
D ODE METHODS
Table 14: Objective evaluation results with different ODE methods and sampling steps.
Methods
ODE
steps
M-STFT (1)
PESQ
Periodicity
V/UV (1)
UTMOS (1)
1
1.8995
1.437
0,1627
0.9114
1.5102
2
L,3598
3.263
0.0929
0,9556
2,9770
4.060
0.0801
8
L1817
4.207
0,0756
0,9654
3.5664
Period MB Buler
16
1.1183
4.256
0.0720
0.9672
3.6209
32
4.265
0.0710
0,9674
3.6482
64
1.0004
4,269
0.7035
0,9678
3,6565
128
0.9601
4,266
0.0700
0.9677
3.6387
256
4.264
0.0697
0,9678
3.6593
1.3342
3,241
0.0954
0,9541
2.9725
2
1,2072
4.132
0,0759
0.9662
3,4662
4
L.0825
4.232
0.0732
0,9670
3,5899
8
LO4577
4.252
0,0703
0,9680
3.6311
PeriodWave-Mer Midpoint
16
0.9729
4.262
0.0704
0,9678
3.6534
32
0.9586
4.263
0.0700
0.9678
3,6583
64
0.9291
4.263
0,0694
0,9679
3.6583
128
0.9094
4.270
0.0694
0,9678
3.6577
256
0.9016
4,270
0.0693
0.9680
3.6066
1.080
0,2749
0.8106
1,2563
2
2.6118
1.482
0,1089
0,9452
1,7786
4
1,9222
2.315
0.0859
0.9591
2,8721
8
L4860
3.150
0.0783
0,9642
3,2758
PeriodWave- MB
RK4
16
1,1990
3.766
010753
0196554
3,4513
32
L,0003
4.084
0.0721
0.9677
3,5204
64
0.9438
4.148
0,0710
0,9683
3,5416
128
0.9238
4.134
0.0712
0,9683
3.5396
256
0.9311
4.123
0.0697
0.9696
3,5350
D.I ANALYSIS ON DIFFERENT ODE SAMPLING METHODS
We explore the different ODE methods analyze the sample quality according the different sampling steps. We utilize three ODE methods including Fuler, Midpoint, and RK4 methods. Table 14 shows that increasing the sampling steps could improve the sample quality in most metrics consistently. We observed that RK4 methods have the lowest performance, resulting in white noise on the generated samples. We can discuss it because we predict the vector field directly including the time point tl for their last order estimation where it is hard to estimate it at the early time steps, resulting in white noise. Meanwhile, Midpoint method show better performance than Euler method, consistently even with half sampling steps which have similar computational cost with Euler method. In this regard, we fixed the Midpoint method for our ODE method. Additionally, using small sampling step could achieve the comparable performance than previous methods.
Table 15: Synthesis speed for baseline models.
Method
HiFi-GAN (V1) BigVGAN-buse
BigVGAN
PriorGrad
FreGrad
Syn.Speed
166.70x 290MB
105.18x
38.28x
8.42x
10.88x
Average Memory
368MB
1,057MB
4,834MB
1,234MB
Table 16: Synthesis speed for Period
Method
PeriodWave
PeriodWave
PeriodWave PariodWave-MB
PeriodWave-MB
PeriodWave-MB
Sampling Steps
16
16
Syn.Speed
56.36s 36x
28.91 453MB
7.48x 462MB
36.55x 424MB
19.01
5.12x
Average Memory
451MB
425MB
432MB
E SYNTHESIS SPEED
We compared the synthesis speed and average memory usages on NVIDIA RTX A6000 GPU for each model. Table 15 indicated the synthesis speed for baseline models. HiFi-GAN shows the highest speed with small memory usage. We reported the synthesis speed our models according to sampling steps at Table 16 Although our model with sampling steps of also has better performance in objective evaluation than other models, our model required more time to generate higher quality samples with iterative generation. Period-wise batch inference could boost the inference speed by about 60%, but this also increases the average memory usage by about two times. For future work, will reduce the inference speed by adopting adversarial learning with distillation methods.
20

---

Published as a conference paper at ICLR 2025
Table 17: Grid Search for FreeU Hyperparameter.
Methods
M STFT (4)
PE)Q
Periodicity (1)
VUUV FI
Pitch
UTMOS
Pave
1.00
1,00
1.2129
4.224
0.0762
0.9652
17.496
3.6495
(16 steps)
0.95
1.05
1,0975
4,253
0.0749
0,9660
17.503
3.7105
0.94
1,06
1,7600
4.256
0.0752
0,9661
17.495
3.7163
0.93
1.07
1,0990
4.255
0.0753
0,9658
17.450
3.7216
0.92
1.08
L.0411
4.258
0.0757
0,9656
17.429
3.7263
0.91
1.09
1.0394
4.254
0.0762
0.9655
17.417
3.7286
0.90
1.10
L,0300
4.245
0.0765
0.9651
17.398
3.7307
4.240
0.0771
17.363
3.7319
0.88
1.12
1,4033
4.230
0.0777
0.9646
17.317
3.7340
0.87
1.0422
4,213
0.0779
0,9643
17.184
3.7336
0.86
1.14
LO065
4.195
0.0784
17.150
3.7330
0.85
1.15
1.0682
4,173
0.0786
0,9640
17.156
3.7307
1.1515
4.033
0,9632
17.197
3.7139
0.50
1,50
1.9477
2.572
0.1074
0.9457
25.241
3.3386
0.95
1.00
L,0836
4.206
0.0764
0.9654
17.892
3.6262
0.95
L.0975
4,253
0.0749
0,9660
17.503
3.7105
0.95
1.10
1.1326
4,243
0.0762
0.9650
17.350
3.7456
0.95
1.15
1.1819
4.164
0.0786
0.9637
17.097
3.7578
1,3371
4.045
0,9575
25,342
3.7477
0.95
1.25
1.2936
3.872
0.0890
0.9555
25.429
3.7252
0.90
1.00
L,3000
4.140
0.0753
0,9665
17.506
3.5755
0.90
1.0124
4,234
0.0757
0.9657
17.522
3.6798
0.90
1.10
1.0360
4,245
0.0765
0.9651
17.398
3.7307
0.90
1.15
L.0892
4.183
0.0782
0.9642
17.079
3.7546
4.070
0,9619
17.196
3,7530
0.90
1.25
1.2222
3.906
0.0856
0.9596
19.787
3.7344
0.90
1,30
1,2915
3,700
0.0921
0.9554
22.346
3.6934
1.1256
3,930
0,9638
20.564
3.4977
0.85
1.05
1.0542
4,136
0.0772
0.9649
17.515
3.6255
0.85
1.10
L.0322
4,196
0.0761
0.9657
174422
3.6948
LO682
4,173
0.0786
0.9640
17.156
3.7307
0.85
1.20
1.1191
4,079
0.0803
0.9636
17.173
3,7424
0.85
1.8116
3,927
0.0850
0.9599
19.675
3.7320
0.85
1.30
1,2502
3.728
0.0906
0.9569
19.905
3.7010
0.80
1.00
1.3068
3,514
0.0815
0.9627
20.699
3.3817
0.80
1.05
1.1911
3,885
0.0792
0,9643
17.507
3.5447
L.1303
4.044
0.0784
17.435
3.6382
0.80
1.15
1.1267
4,086
0.0782
0.9643
17.086
3,6911
0,80
1.1515
4.033
0.0812
0,9632
17.197
3.7139
0.80
1.25
1.1954
3,911
0.848
0,9599
19.626
3.7155
0.80
1.30
1,2518
3.732
0.0898
0.9567
19,809
3.6956
0.75
1.00
1,5238
3,000
0.0846
0,9578
26.287
3.2261
0.75
L.3798
3.447
0.0813
0,9310
17.478
3.4308
0.75
1.10
1,2835
3.738
0.0809
0,9632
17.435
3.5552
0.75
1,2459
3,874
0.0810
0,9632
17.203
3.6325
0.75
1.20
1.2414
3,902
0.0828
0.9626
17.329
3,6688
0.75
1.25
1,2610
3.831
0.0851
0,9600
17.240
3.6840
3.690
19.798
3.6764
0.70
1.00
1.7499
2.559
0.0862
0.9574
26.376
3.0309
0.70
1.05
1.8887
2.949
0.0824
0,9622
17.481
3.2832
1.4682
3,293
0,9614
17.425
3.4427
0.70
1.15
1.4024
3.524
0.0843
0.9616
17.271
3.5477
0.70
1.20
1.374
3.653
0.0852
0.9611
17.236
3.6053
0.70
1.25
L3711
3.662
0.0859
17.262
3.6350
0.70
1.30
1,3885
3,581
0.0912
0.9565
19,774
3.6395
FREEU
We search the hyperparameter of FreeU for single-band model PeviodWave. We found that using balanced weights of backbone feature and skip feature could improve the reconstruction performance and perceptual quality. We utilized a of 0.9 and of 1.1 for our model. Additionally, we found that increasing the weight of backbone feature 3 could further improve the perceptual quality but this would decrease the reproduction performance.
21

---

Published as a conference paper at ICLR 2025
G TRAIN-INFERENCE MISMATCH PROBLEM
Current, two-stage Text-to-Speech (TTS) models consists of acoustic models and neural vocoder. Due noisy Mel-spectrogram, these two-stage TTS models suffer from train-inference mismatch problem. Although one-step GAN-based neural vocoder could generate high-quality Mel-spectrogram, these models might generate the samples with noisy sound due to rain-inferrene mismatch problem.
To reduce this issue, HiFi-GAN (Kong et al., 2020) proposed the fine-tuning methods with the
generated Mel-spectrogrm by teacher-foreing mode. (Lee al. 2022a Jang et al. [021] Kaneko
et al., 2022. Kim et al. 2020 Lancucki 2021 followed this fine-tuning method to improve the perceptual quality of two-stage TTS model.
Meanwhile, end-to-end TTS models (Kim et al. 2021 Lim et al. 2022) outperformed the performance compared to two-stage models in terms of audio quality. They have a limitation of model architecture restriction to align high-resolution waveform signal and text, and they require more training times. Additionally, recent end-to-end TTS models showed lower zero-shot TTS performance than recent two-stage TTS models including VoiceBox (Le et al. 2024), P-Flow (Kim et al. 2024). E2-TTS (Eskimez et al. 2024). ARDiT-TTS (Liu et al. 2024b), and DiTTo-TTS (Lee et al. 2024a).
Although recent TTS models have shown their powerful performance on zero-shot TTS, there still cain-infereene mismatch problem which contains some noise on the generated Mel-spectrogram resulting noisy sound.
To address this issue, we shift our focus from one-step generation to the iterative sampling based waveform generation. Following diffusion- -bassd neural vocoder (Koizumi et al. 2023 Jang et al.) 2023] Huang et al.] [2022b] Koizumi et af.) 2022) Roman et al.] 2023), waveform generation with iterative sampling could refine the waveform signal when the conditioning flawed or imperfect. We also adopt the iterative sampling methods by optimizing flow matching objective to reduce the sampling steps. The results also show that our models have shown better performance even with small sampling steps. Furthermore, our model has shown the best performance on two-stage text-to-speech scenarios by iterative sampling.
G.1 SINGLE SPEAKER TEXI-TO-SPEECH
We conduct two-stage TTS experiments to evaluate the robustness of the proposed models compared previous GAN-based and diffusion
Table 18: Text-to-Speech Results. We utilized Glow-
TTS trained with LJSpeech as TTS model.
Methods
MOS (1)
UTMOS (T)
models. We utilized the official implementation of Glow-TTS which is trained with the LJSpeech dataset. [
HiFi-GAN
3.70=0.03
4.1114
Big BigVGAN-hase
3.71
4.0296
BigVGAN"
3.6920.03
3,9570
that our model has a higher performance on the two-stage TTS in terms of MOS and UTMOS. Although HiFi-GAN shows a lower performance in reconstruction metries, we observed that HiFi-GAN shows high perceptual perfor- mance in terms of UTMOS. BiAN base (14M) has a higher performance than BigVGAN (112M). We that Big VGAN could reconstruct the waveform signal from the generated Mel-spectrogram even with the error that might be the generated Mel-spectrogram. Although our model has a higher reconstruction performance, processes. Additionally, we found that the generated Mel-spectrogram contains a larger scale of energy compared the ground-truth Mel-spectrogram, so we utilized 7 of 0.333 for scaling Zg.
PriorGrad (50 steps)
3.53+00.03
3.3807
FreGrad (50 steps)
3.51
2.8583
PeriodWave (16 steps) PeriodWave FreeU (16 steps)
3.72+0.03
4.2560
3.75+0003
4.3110
models could refine this phenomenon through iterative generative
22

---

Published as a conference paper at ICLR 2025
Table Objective evaluation results with different sampling steps for each band.
Method
steps
M-STFT (4)
PESQ (1)
Periodicity (4)
V/UV FI (1)
UTMOS (T)
[16,16,16,16]
0,9729
4,262
0.0704
0,9678
3.6534
[16,8,4,4]
1.0473
4,259
0.0701
0,9680
3,6506
[16,4,4.4]
1,0500
4.257
0.0703
0.9678
3.6473
PeriodWave-MI
[16.4.2.2]
1.1148
4.255
0.0703
0.9678
3,6482
[16,4,1,1]
1,0883
4.241
0,9677
3.6409
[16,2,1.1]
1.1033
4.224
0,0705
0,9677
3.6370
[16.1.1.]]
1.1133
4.200
0,0710
0.9677
3.6253
[8,2,2.2]
1.1428
4.239
0.0721
0,9670
3.6241
PeriodWave-MB
[8.2,1.1]
1.1152
4.225
0.0723
0,9669
3.6178
[8,1,1,1]
1.1255
4.193
0.0725
0.9670
3.6073
PeriodWave-MB
[4,4,8,16]
1.0609
4.235
0.0732
0.9671
3.5923
[4,4,4,4]
1.0825
4.232
0.0732
0.9670
3.5899
G.2 ANALYSIS ON ADAPTIVE SAMPLING STEPS FOR MULTI -BAN MODELS
We proposed adaptive sampling for multi-band models. We can efficiently reduce the sampling steps for high-frequency bands due to the hierarchical band modeling conditioned on the previously generated DWT components. Table [9 shows that it is important to model the first DWT components. After sampling the first band, we can significantly reduce the sampling steps for the remaining bands, maintaining the performance with only small decrease. The results from the sampling steps of [4,4,8,16 16] demonstrated that is important to model the first band for high-fidelity waveform generation and accurate gighffreuency modeling could improve the M-STFT metrics.
H BROADER IMPACT
Practical Application We first introduce high-fidelity waveform generation model using flow matching. We demonstrated the out-of ciitr---iittibttion robustness of our model, and this means that the conventional neural vocoder can be replaced with our model. Furthermore, we train Codec-based PeriodWave for audio generation and speech language models. We see that our models can be utilized for text-to-speech, voice conversion, audio generation, and speech language models for high-quality waveform decoding.
Social Negative Impact Recently, speech Al technology has shown its practical applicability by synthesizing much more realistic audio. Unfortunately, this also increases the risk of the potential social negative impact including malicious use and ethical issues by deceiving people. It is important to discuss a countermeasure that can address these potential negative impacts such as fake audio detection, anti-spoofing techniques, and audio watermark generation.
23

---

Published as a conference paper at ICLR 2025
PatedWove
PariodWave
Pave
PeriodWave
Mirn
Mini
Mini
(4) Parallel generation
(b) Streaming generation
Figure 4: Neural Audio Codec Decoding. (a) The original PeriodWave is designed for parallel generation. Increasing the sampling steps can significantly improve the audio quality. (b) We propose streaming generation methods using single token delayed streaming generation and 2-step sampling.
Table 20: Streaming audio generation using two steps based on different values of N₂ and Na-
N,
Na
CER (1)
WER (1)
M-STFT (1)
PESQ
Periodicity (1)
VUVV F1 (1)
Fiich (1)
UTMOS (T)
2.11
5.5]
1,4386
2,056
0.1574
0.9121
50,095
3.6/169
1.3364
3.7675
4.50
1,2889
2.211
0.144
47.301
381455
25
2.45
5.53
1.2716
2.233
0.1429
0.9288
40.634
3,8088
I NEURAL AUDIO CODEC
Recently, neural audio codec models (Défossez et 2023 Yang et al. 2023a Zhang al. 2024 Liu et al., 2024a Ji et al. 2024 Xin et al. 2024, Ye et al., 2024) have been investigated for practical applications including speech language models, audio generation, and TTS. Recently, Moshi (Défossez et al.) presented an efficient streaming speech language models and proposed neural audio codec, Mimi, that operates at 12.5 Hz. We train PeriodWave using tokens of Q quantizer from Mimi. We added an additional upsampling layer of Mel-Encoder, and we refer this module as the Token-Encoder. We utilize the same hyperparameter for the Mel-spectrogram version, excluding segment length of 48,000. The model was trained on LibriTTS for 300 epochs (830k steps. 3 days).
Fine-tuning with only adversarial training Following (Lee et al. 2024b), we first fine-tuned the model with reconstruction loss and adversarial feedback, fixing the iteration steps either 2 or 4 to accelerate the inference speed. However, we observed that the fine-tuned model using Mel-spectrogram reconstruction loss generated noisy sound. We can discuss that Mimi compressed the waveform signal at 24,000 Hz into 12.5 Hz tokens, which causes a loss of frequency-relate details in speech, resulting in over-smoothed frequency information with reconstruction loss. Moshi also indicated that removing the reconstruction loss can improve audio quality. Therefore, instead of rec-rrccontrrutiinn loss, we employed multi-scale STFTD with MPD and 2024). With only adversarial training, fine-tuned the model for 5 epochs (50k steps and 9 hours) with a batch size of 32 using four H100 GPUs.
(Gu et al.)
Parallel generation We generated the samples using all sequence of tokens in parallel. We designed the model to enhance the audio quality by increasing the iteration steps. Table 10 indicates that PeriodWave significantly improves the performance compared the original decoder of Mimi Furthermore, see that scaling up model size could further improve the audio quality in parallel generation for high-quality dialogue data collection using speech language models.
Streaming generation We present streaming generation methods for Period as illustrated in Figure Although PeriodWave is trained with parallel generation, we found that PeriodWave can generate samples streaming manner. To achieve this, we utilize N, previous tokens for the past context, and N₂ delayed tokens for the future context. We adopt single frame delayed streaming generation using N. delayed token. We found that using at least one previous and one delayed token is essential for generating waveform signals in streaming manner, as our model consists of non-causal convolutional layer. In our setup, we can use up to N, 25 previous tokens for real-time generation. We compared the performance based on different values of N2 and Na in Table 20
24

---

Published as a conference paper at ICLR 2025
J BASELINE DETAILS
J.1 LJSPEECH
We compared the model with the public-available models which trained with LJSpeech dataset. LISpeech a single speaker dataset consisting of 13,100 high-quality audio samples with a sampling rate of 22,050 Hz. We followed the training and validation lists of HiFi-GAN
HiFi-GAN We first utilize the HiF-GAN, which is the most popular GAN-based neural vocoder. We use the official checkpoint of HiFi-GAN (V1 which was trained for 2.5M steps. They utilize eight number of discriminators including three different scale of multi-scale discriminators and five different periods of multi-period discriminators.
BigVGAN We utilize BigVGAN-bass and BigVGAN which are a novel GAN-based neural vocoder. We utilize the official checkpoints for sampling rate of 22,050 Hz which are trained with large-scale dataset including LISpeech, VCTK, and LibrİTTS. We are sure that they are trained with all LJSpeech dataset without splitting the training and validation dataset. These models are trained with 5M steps.
PriorGrad We utilize PriorGrad which the most popular diffusion-based neural vocoder. We use the official checkpoint of PriorGrad which was trained for 3M steps. We used the same energy-based prior of this models and the default sampling steps of 50.
We utilize FreGrad Nguyen et al. (2024) which is the recent proposed liffusion-basee neural vocoder. They utilize similar approach using discrete wavelet transform so we compare it with ours. We use the official checkpoint of FreGrad which is trained for 1M steps.
FreGrad
@)
J.2 LIBRITTS
We compared the model with the public-available universal vocoder which are trained with LibriTTS dataset. LibriTTS dataset consists of 555 hours of 2,311 speakers with sampling rate of 24,000 Hz. We followed the training processes of BigVGAN including Mel-spectrogram transformation and inference settings. There are diffusion-based models and any implementations which are trained with LibriTTS or other multi-speaker settings. In our preliminary study, diffusion- based models could not generate high-frequency information resulting in low quality audio generation.
UnivNet We utilize the UnivNet-c32 which is large model of UnivNet. UnivNet uses LVCNet which is an efficient generator structure for fast sampling. We use the public-available implementation of UnivNe [0
which is trained with LibriTTS train-clean-3e subset.
Vocos We utilize Vocos wich is a fast time-frequency modeling- based neural vocoder with iSTFT. We utilize the official implementation of Vocos which is trained with LibriTTS dataset for 1M steps. This model shows the fastest inference speed and even has comparable performance with other baselines.
BiANGAN We utilize Big base and BigN which are novel GAN- neural vocoder. We utilize the official checkpoints for sampling rate of 24,000 Hz which are trained with LibriTTS. These models are trained with 5M steps. Specifically, we also utilize the checkpoints which use Snakebeta activation with log- scale parameterization which shows the best quality reported by the official implementation of Big VGAN.
25

---

Published as a conference paper at ICLR 2025
K EVALUATION METRICS
K.1
OBJECTIVE EVALUATION
Following Lee et al. 2023 we utilized four different metrics including multi-resolution STFT (M-STFT), perceptual evaluation of speech quality (PESQ), Periodicity error, and FI score of voice/unvoice classification (V/UV F1). We additionally conduct UTMOS and Pitch distance.
M-STFT utilized open-source implementation of multi-resolution STFT loss of Auraloss (teiineetz & Reiss, 2020). The M-STFT loss was proposed in Parallel WaveGAN (Yamamoto et al. 2020), and we used this distance measure the difference between the ground-truth and generated samples at the multiple resolution STFT domains.
PESQ We utilized the wideband version of perceptual evaluation of speech quality sampled the audio by the sampling rate of 16,000 Hz to calculate PESQ.
We down-
Periodicity and V/UV F1 CarGAN (Morrison et al. 2022 stated the periodicity artifacts perceptu-
ally degrade the audio. We utilized a Periodicity RMSE measure the periodicity error conducted the evaluation on Voice/Unvoice Fl score.
13 We also
UTMOS We utilize the open-source MOS prediction model, UTMOS to evaluate the naturalness of generated samples. The UTMOS reported the consistency results of MOS for neutral English speech dataset.
K.2 SUBJECTIVE EVALUATION
MOS/SMOS We assessed the perceptual quality of synthesized speech using Mean Opinion Score (MOS). Specifically, the naturalness of the synthesized speech was measured with MOS, and its similarity to the ground-truth speech was evaluated using SMOS. utilized crowdsourcing for this evaluation. The details are described in the following Appendix L
26

---

Published as a conference paper at ICLR 2025
Description:
How similar the second recording compared to the first?
Keywords:
audio, similarity, refing
HIT Approval Rate (9% for all Requesters' HITs greater then 88
Qualficcaion Requirementsjj:
Number Approved greater
Location is J
Prijed date with Eaidding Project
Instruction
Figure 5: Detailed information on listeners restrictions and task completion interfaces.
L CROWDSOURCING DETAILS
We conducted MOS and SMOS evaluations using a 5-point scale to measure the naturalness and similarity of the synthesized speech. For this survey, utilized Amazon Mechanical Turk assess the perceptual quality of each model. Specifically, 30 listeners evaluated 150 samples per model, rating them on scale from 1 5. Given that the evaluation data is in English, native English speakers residing in the United States. To ensure the reliability of the evaluators, we implemented strict eligibility criteria: only listeners with an approval rate of 98% or higher for their previous tasks on MTTurk, and with at least 90 approved tasks (HiT's, were allowed to participate in this evaluation. To further enhance the quality of the evaluation, ground-truth samples were included as control measures. We excluded evaluations 1) from listeners who gave a score below 3 to the actual samples and 2) from those who spent less than half the duration of the audio sample on the overall evaluation from the final results. This procedure was intended to filter out inattentive listeners and ensure the integrity and accuracy the evaluation data.
specifically targeted
"https://www.ttrkk...
27

---

Published as a conference paper at ICLR 2025
Table 21: Objective evaluation results from EnCodec tokens. We utilize speech dataset from the test
samples provided in RFWave.
Params
Training
M-STET (1)
PESQQ(1 (1)
Period. (1)
V/UV (1)
Pitch (1)
UTMOS (1)
SSI-MOS (1)
Eneodee
13M
300 epochs (XxA)000)
1.170
2,643
0.112
0:941
26.600
2.542
3.787
Vocas
7M
2M steps
1,074
3.051
0.086
0.957
20.491
3,100
4.130
MBD
41M
4x2day (4x0)000
1,612
2.645
0.108
0.946
26.720
3,300
4,091
RFWave
10 day (4X
1,280
3,020
8.078
8.957
18.
2,988
4.169
Period/aave
31M
IM steps, 4 day (4xAA 100)
0.929
3,644
0.070
0.968
19.538
3.581
4.352
Table 22: Band-wise Comparison of the reconstruction from EnCodec tokens. We utilize speech dataset from the test samples provided in RFWave.
Method
Parums
Training
McI-L
Mel-M
Mel-H (1)
Mel-A (1)
Ficedee Vocos
15M
300 epochs (8xA 100)
0.577
0.517
0.587
0.567
7M
2M steps
0.449
0.406
0.451
0.441
MBD
411M
4x2day (4xV100)
0.837
0,994
1,107
0,921
RFWave
18M
10 day (4xA100
0.603
0.674
0.775
0.651
PeriodWave
steps, 4 day (4xA100)
0.365
0.413
0.387
Table 23: Objective evaluation results from EnCodec tokens. We utilize vocal dataset from the test samples provided in RFWave.
Methoal
Params
Training
M-STPT
PESQ (f)
Parixd: (J)
VITV (1)
Pach
Mc3..(1)
Mr-M (L)
Mct-H(()
Mcl-A (i)
Encedee
spochs (3xX100)
2,416
33.257
0:755
Vices
2M steps
2,849
0.954
24.504
0.546
0,511
4xclday
22.953
1.044
0.957
RFWins
d0
0,962
18840
0.760
0.735
Periallioe
steps,
(4x0A
17.531
0.431
Table 24: Objective evaluation results from EnCodec tokens. We utilize sound effect dataset from the
test samples provided in RFWave.
Method
Params
Training
M-STFT (1)
PESQ
Mel-L
Mel-M (4)
Mel-H (1)
Mel-A (1)
Encodee
15M
300 epechs (8xA100)
1.182
2,409
0.528
0.466
0.456
0.502
Vocos
7M
2M steps
1.294
1.895
0.562
0.474
0.451
0.523
MBD
411M
4x2day (4xV
1.936
1,080
1.054
RFWave
18M
10 day (4xA100)
1,592
2,253
0,701
0.640
0,705
0.690
PeriodWave
31M
1M steps, 4 day (4xA100)
1.101
2.699
0.438
0.398
0.395
0.422
M HIGH-FIDELITY WAVEFORM RECONSTRUCTION FROM THE TOKEN OF ENCODEC
We conducted additional experiments for waveform generation from the token of EneCodec, and we compared the model with EnCodec, Vocos, MBD and RFWave. Following MBD, we used the same Encodec settings to obtain the tokens, using a maximum bandwidth of 6.0 kbps (Q=8, 75Hz for 24,000 Hz waveform).
Following MBD, our model was trained and evaluated exclusively on the same dataset. Also, we utilized the official implementation and official checkpoints of MBD and RFWave, respectively. Specifically. we utilize the same dataset including Common Voice 7.0, DNSChallenge 4 for speech, MTG-Jamendo for music, and FSD50K and AudioSet for environmental sounds by resampling them using Sox resampling to 24,000 Hz. In summary, we utilized the same training dataset and our model was trained with speech, singing, and sound effects together for fair comparison with MBD and RFWave.
We also incorporated the results from the universal test provided by RFWave This testset consists of speech, vocal, and sound effect samples from several different datasets. Specifically, the reported results are based on the single-band PeriodWave model trained for 1M steps. For each model, we follow the recommended inference setup, MBD used 20 steps for each band diffusion,
com/ http:/leee////// htt.://ww.........oooooooocoo/aa htt.:/aoooaa..aaaaooooooooooooo. 635ugpl.vm
28

---

Published as a conference paper at ICLR 2025
used 20 sampling steps for better results of RFWave with CFG of 2. We conducted additional naturalness evaluation by SSL-MOS, 17
The results demonstrated that the single-band PeriodWave significantly outperforms all baselines including MBD and RFWave. With large-scale dataset, our model could increase the capacity to generalize various scenarios including speech, singing, and sound effects.
M.1 BASELINES FOR ENCODEC TOKEN RECONSTRUCTION
EnCodec We utilized the official implementation of EnCodec Défossez et al. (2022), and the official checkpoin to extract the tokens and reconstruct the waveform signal. We extract the tokens compressed by the bandwidth of 6.0 kbps (Q=8, 75Hz).
T8
Vocos We use the official implementation and checkpoint of Vocos Siuzdak 2024 which was
trained with the tokens of EnCodec 19 Vocos is powerful GAN-based baseline model. Although Vocos has fast inference speed, Vocos was trained with multi-period discriminator and multi- resolution discriminator for adversarial feedback optimize the model.
MBD
We utilized the official implementation and checkpoint of multi-band diffusion (MBD)
Roman al. (2023 Previous diffusion-basee models only provide the low-frequency information,
MBD adopted multi band modeling for diffusion models to improve the high-frequency modeling.
RFWave We utilize the official implementation of RFWave Anonymous 2024 which has better performance than MBD. We found that RFWave was submitted to ICLR 2025 and this work also utilize a flow matching for waveform generation like our proposed methods. While model adopt CFM to the weveforn-level modeling with multi-period generation, RFWave leverage spectrogram- based flow matching for efficient iterative waveform generation.
29

---

Published as a conference paper at ICLR 2025
w
ww
Figure 6: The spectrogram of the GT sample
WW
ww
Figure 7: The spectrogram of the sample generated by EnCodec
w
www
www
Figure 8: The spectrogram of the sample generated by Vocos
W
w
www
Figure 9: The spectrogram of the sample generated by MBD
ww
www
www
www
Figure 10: The spectrogram of the sample generated by RFWave
wW
ww
www
Figure 11: The spectrogram of the sample generated by PaviodWave
30

---

Published as a conference paper at ICLR 2025
N
MULTI-BAND TERIODWAVE WITH FREEU
Table 25: Grid Search FreeU Hyperparameter.
Methods
a
M-STFT (1)
PESQ (t)
Periodicity (4)
VUV
Pach
UTMOS
PeriodWave
1.00
1.00
1.2129
4.224
0.0762
0,9652
17.496
3.6495
(16 sleps)
0.95
1.05
1.9955
4.253
0.0749
0.9660
17,503
3.7105
0.90
1.10
1.3600
4.245
0.0765
0.9651
17.398
3.7307
0.85
1.15
1.0682
4.173
0.0786
0.9640
17.156
3.7307
PeriodWave-MM
1.00
4.262
0.0704
0.9678
20.496
3.6534
(16 steps)
0.95
1.05
0.9590
4.291
0.0690
0.9689
20.235
3,7089
0.85
1.15
0.9876
4.224
0.0716
0.9673
20.016
3.7155
When we train the model with only a single CFM objective, the tendency of FreeU is almost same with other settings and dataset.
For validation set, we fixed the alpha and beta values terms reconstruction performance. However, we can decrease the alpha to remove the high-frequency noise, and increase the beta to improve the sharpness and energy of generated samples. In terms generative models, controlling the alpha/beta could improve the diversity of generated samples.
Furthermore, we added the results of PeriodWave-MB with FreeU to demonstrate the consistent results. The results show a similar result of PaveodWave with FreeU,
31


Reviews:
[{"review_id": "kSWbF0vDf6", "reviewer": "Reviewer_TzrS", "summary": "This paper studies the task of speech and audio waveform generation using the flow-matching modeling paradigm. The authors proposed a multi-period approach to generation artifacts in high-frequencies, conditioned on either mel-spectrogram and discrete representation extracted from neural codecs. The authors compared the proposed method to several GAN-based approaches and two diffusion-based methods. The authors empirically demonstrated the proposed method reaches superior performance to the evaluated baselines.", "questions": "I have a few questions and concerns about this submission: \n1. As stated above, it seems the proposed method is heavily connected to the MBD work. What is the reason the authors did not compare their method to MBD? More generally, there are missing experiments to make the contribution of this work clear. At the moment, it is not clear what makes this method better than the prior work. Is it the flow-matching approach? Why not use v-diffusion instead? is it the usage of discrete wavelet transformation over MB? Etc. \n**Specific experiments to add:** Can the authors provide: (1) a direct comparison of the proposed method MBD using the same setup? (2) ablation study on the usage of flow matching vs. diffusion? (3) comparison of discrete wavelet transform to the multi-band approach used in MBD? \n2. There are missing details regarding the evaluation methods. Can the authors define the \"Pitch\" metric mean? How did the authors extract F0 information? What metric did the authors use to estimate F0 quality? Similarly for the other metrics reported in the paper.\n3. Under the training paragraph, the authors mention that training (using 2 A100s) the proposed approach is faster as it requires 3 days, while it takes more than three weeks to train the GAN-based approaches. It seems quite a lot for the GAN-based methods. It makes sense training the proposed method would be faster, however, it seems quite a big difference. Can the authors provide more information on how they trained the baseline methods? More details about the training configurations for both the proposed method and the evaluated baselines? (i.e., batch sizes, learning rates, etc.). Alternatively, in case such training times were reported in prior work, a citation would be fine.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method reaches superior performance to the evaluated baselines.\n2. The authors presented results using both mel-spectrogram and discrete representation extracted from neural codecs.\n3. The authors provide extensive ablation and analysis of the proposed method.", "weaknesses": "1. Limited novelty. It seems the proposed method applies the multi-band (MB) approach (using wavelets instead of MB) over the flow-matching approach rather than diffusion. \n2. Missing comparisons. As the proposed approach is heavily connected to the MBD approach [1], it is not clear why the authors did not compare to this method. \n3. Missing details and inaccurate details (see below).\n\n\n[1] San Roman, Robin, et al. \"From discrete tokens to high-fidelity audio using multi-band diffusion.\" Advances in neural information processing systems 36 (2023): 1526-1538.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sPM1ry2BSz", "reviewer": "Reviewer_HbGY", "summary": "The paper presents PeriodWave, the first flow-matching based vocoder. It relies on a carefully designed multi-period estimator. The authors notably propose to use discrete wavelet transforms and FreeU to better render high frequency information. The model outperforms one-step GANs in TTS tasks. The paper shows that the proposed method not only works as a spectrogram-based vocoder but also for generating from discrete tokens. Source code and checkpoints will be made available.", "questions": "- Section 4.4: How is temperature exactly applied to the gaussian noise? Is this just about scaling the standard deviation?\n- What is the motivation for not comparing with MDB at all in the paper? MBD is conceptually very similar (although only applied to discrete representations and using diffusion). Please motivate this choice somewhere in the paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Extensive experimental work that demonstrates the robustness of the proposed method on both in-domain (mono- and multi-speaker TTS) and out-domain (Music) data. Ablations properly highlight the design choices.", "weaknesses": "I wish there was some comparison with MBD, which is probably the closest model architecture-wise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oknc6gmoI1", "reviewer": "Reviewer_cFmr", "summary": "The paper proposed a flow-matching-based vocoder with improved high-frequency modeling and inference speed.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposed a practical flow-matching-based vocoder by significantly improving the high-frequency modeling issue with the existing flow-matching-based waveform generation method.\n2. The proposed method shows decent performance compared to GAN-based methods. \n3. The proposed method achieves a good balance between quality and synthesis speed compared to conventional diffusion-based methods.", "weaknesses": "1. It is unclear if the hyperparameters from grid searching during FreeU are generalizable to other settings and datasets. Does the alpha and beta need to be re-adjusted under different inference settings and datasets? It seems the FreeU grid search is only conducted on a single-band model of the proposed methods. Did you test it on the multi-band version of Periodwave?\n2. Since high-frequency modeling is the core innovation of the paper, it is recommended to perform analyses focusing on the high-frequency modeling capabilities. For example, measuring the fidelity of the high-frequency bands of the audio signal and comparing it to the baselines. During the out-of-distribution music dataset experiment, only vocal, drum, and bass were emphasized, where the bass is typically the low-frequency part. It would be interesting to analyze the high-frequency components of the music by showing for example, per-band SNR accompanied with detailed spectrograms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "itAtxIxWjG", "reviewer": "Reviewer_nYdJ", "summary": "The paper presents a nove waveform generation model that can capture periodic features of waveforms using a period-aware flow matching estimator and further extend to multi-band estimator. \nExperimental results demonstrate that PeriodWave outperforms existing models in reconstruction tasks from mel and discrete tokens, as well as in TTS applications.", "questions": "1: If α<1, wouldn't it make it harder for the model to capture high-frequency features? \n\n2: The M-STFT metric is relatively low, a well-optimized model in the waveform space shouldn't show significant disadvantages in STFT metrics, is there any explanations?\n\n3:A frequency of 22.05 kHz is not considered very high; it mainly contains harmonics, making it unclear whether the model's performance advantage stems from low-frequency modeling or high-frequency modeling.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "First time applying flow matching to waveform generation, and a method was designed to explicitly extract patterns with different periods to match the spectral characteristics of the waveforms. \nIncorporates the latest effective waveform modeling tricks, significantly enhancing the performance of the vocoder and the extraction of periodic features.", "weaknesses": "1: There is no comparison of the performance differences between flow matching and diffusion under the same network and optimized configurations.\n\n2: The main contribution of this paper lies in the introduction of periodic features; while the choice of periods [1, 2, 3, 5, 7] versus [1, 2, 4, 6, 8] can be understood, the ablation study for [1] may be unfair regarding network parameters (5 layers for [1, 2, 3, 5, 7] and only 1 layer for [1]). It would be more appropriate to compare with configurations like [1, 1, 1, 1, 1], [1, 1, 1, 3, 3], or [1, 1, 1, 3, 9]. Additionally, deriving the reason for selecting period by prime based solely on this comparison may not be sufficient， \n\n3: The experiments emphasize that the network structure primarily optimizes high-frequency information modeling, but this does not adequately demonstrate that the method is specifically related to high-frequency information，It would be better to use band-wise comparison metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "he41NoP2XN", "reviewer": "Reviewer_C9NL", "summary": "Propose PeriodWave, a novel universal waveform generator that can reflect different\nimplicit periodic information when estimating the vector fields.\nThoroughly analyze the limitation of high-frequency modeling, and address this by DWT and FreeU approach for high-frequency noise reduction.\nPresent PeriodWave which outperforms the one-step GAN models in conventional two-stage TTS tasks.\nBased on SOTA neural audio codec Mimi, successfully demonstrate the effectiveness of\nPeriodWave in neural audio codec decoding task both in parallel and streaming generation.", "questions": "Please add the limitations section to the main paper and remove it from the appendix.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Thoroughly analyze the limitation of high-frequency modeling, and address this by DWT and FreeU approach for high-frequency noise reduction.\nPresent PeriodWave which outperforms the one-step GAN models in conventional two-stage TTS tasks.\nPresent thorough experimentation do demonstrate the effectiveness of the proposed method.", "weaknesses": "Although the models can generate the waveform with small sampling steps, Table E shows that the models have a slow synthesis speed compared to GAN-based neural vocoders.\nShows a lack of robustness in terms of high-frequency information because they only train the model by estimating the vector fields on the waveform resolution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)