Generative Pre-training for Speech with Flow Matching
Authors: Alexander H. Liu
Matthew Le
Apoorv Vyas
Bowen Shi
Andros Tjandra
Wei-Ning Hsu
Venue: ICLR 2024 poster (2024)
Keywords: speech pre-training
speech generation
generative pre-training
flow matching
Primary area: generative models
TL;DR: Pre-training generative model for speech generation tasks.

Abstract: Generative models have gained more and more attention in recent years for their remarkable success in tasks that required estimating and sampling data distribution to generate high-fidelity synthetic data. In speech, text-to-speech synthesis and neural vocoder are good examples where generative models have shined. While generative models have been applied to different applications in speech, there exists no general-purpose generative model that models speech directly. In this work, we take a step toward this direction by showing a single pre-trained generative model can be adapted to different downstream tasks with strong performance. Specifically, we pre-trained a generative model, named SpeechFlow, on 60k hours of untranscribed speech with Flow Matching and masked conditions. Experiment results show the pre-trained generative model can be fine-tuned with task-specific data to match or surpass existing expert models on speech enhancement, separation, and synthesis. Our work suggested a foundational model for generation tasks in speech can be built with generative pre-training.

Published as a conference paper at ICLR 2024
GENERATIVE PRE-TRAINING FOR SPEECH WITH FLOW MATCHING
Alexander H. Liu'; Matt Le2, Apoorv Vyas², Bowen Shi?, Andros Tjandra', Wei-Ning Hsu? 'IT CSAIL, "Meta Al
aalexhliummit...
ABSTRACT
Generative models have gained more and more attention in recent years for their remarkable success tasks that required estimating and sampling data distribution to generate high-fidelity synthetic data. In speech, text-to-speech synthesis and neural vocoder are good examples where generative models have shined. While generative models have been applied to different applications in speech, there exists no general-purpont generative model that models speech directly. this work, we take step toward this direction by showing single pre-trained genera- tive model can be adapted to different downstream tasks with strong performance. Specifically, pre- pre-trained generative model, named SpeechFlow, 60k hours of untranscrihed speech with Flow Matching and masked conditions. Experiment results show the pre-trained generative model can be fine-tuned with task-specific data to match or surpass existing expert models on speech enhancement, separa- tion, and synthesis. Our work suggested foundational model for generation tasks in speech can be built with generative pre-training. Audio samples can be found at wllttts://voicebbxx wttpsiow..emaaeeoooaaaoo com/ www.///peec..e... hot.chfooo...mmm html
INTRODUCTION
Discriminative models have long been the mainstream in speech applications since the deep learning era. These models are applied to different types of tasks such speech recognition (Graves al., 2006), enhancement, and separation (Luo & Mesgarani] 2019). Interestingly, even for applications that be naturally formulated generative modeling problems, such as text-to-speech (TTS), we see most popular models remained discriminative (Shen et al., 2018 Ren et al. (02T). Conse- quentially, pre-trained foundation models (Baevski et al. 20200] Hsu al.]1222] that served as the upstream of speech applications focused more on learning useful representation for discriminative tasks rather than modeling the data distribution p(speech). this paper, we seek to answer whether generative models can serve as foundation models for speech applications or not.
Unlike discriminative models, generative models enable sampling of the data distribution. For ex- ample. generative TTS models (Habib et al.] (019) allow different emotions to be sampled given a fixed text as discriminative models produce a fixed output. Up to the present, generative mod- els in speech are usually designed for given purpose via task-specific conditioning distribution mapping. Perhaps the most well-known examples of task- task-specific conditional generative models are neural vocoders (Kong al. 2020 Chen et al.) 2020). These models learn map simple priors (e.g.. normal distribution) to waveform conditioning on acoustic features (e.g., spectrogram). On the other hand, examples for distribution mapping include diffusion models that transform noisy speech to clean speech for denoising (Lu et al. 2021: 2022. Richter et al., 2023), or speech mixture non-overlapping speech for separation (Scheibler et 2023).
In this work, we explore a new direction to pre-train a general-purpose generative model with un- labeled speech. We hypothesize that good generative model on speech without pre-defined ap- plication can be applied to different end tasks that require speech generation. Our model, named SpeechFlow, is generative model that combines masked audio modeling and Flow Matching (Lip- man et al. 2023). ) SpeechFlow is trained with unlabeled speech with the goal of estimating the
Work done during intersship Meta.

---

Published as a conference paper at ICLR 2024
underlying distribution of speech conditioning on masked audio. We show that generative model trained with unlabeled speech data can be adapted to different tasks that require speech generation by fine-tuning with task-specific conditions using labeled data. More specifically, fine-tuned SpeechFlow and compared against expert models in speech enhancement, separation, and synthesis. For each task, fine-tuned SpeechFlow is able to match expert models. Experiment results suggested that pre-trained generative models possess great potential to become foundation models for different speech generation tasks.
RELATED WORK
Generative Speech Models As mentioned earlier, generative models have been applied to dif- ferent tasks in speech. Research in neural vocoders found generative models to be a good suit for ppectrogram-towwavee
prediction. Prevailing generative models are applied to the task with suc- cess, such as generative adversarial model (Kong et al. 2020). flow-based invertible model (Prenger iet al. 2019). and diffusion network (Koizumi et al. 20022). Besides neural vocoders, generative mod- els are also applied to other tasks such as TTS (Valle et al. 2020), speech enhancement (Lu et al.] 2021 2022 Richter et al., 2023) and separation (Scheibler et al. (2023). A fundamental difference between this work and the prior works is that SpeechFlow is not trained for a specific application, but to estimate the underlying distribution of speech itself.
Recent studies also explored speech generation from language modeling perspective. Taking ad- vantage of audio tokenizing techniques (Hsu et al. 2022, Défossez et al. 2022 Zeghidour et al, 2022). Spoken Language Models SLMMs et al. 2022 Kharitonov et al. [202]] Borsos et al,, 2022 have been developed to model language without text. These token-based speech language models are closely related to the proposed method in the sense of training generative models from unlabeled speech. The key difference is the goal of SLMs is to discover the underlying text for textless language processing (Nguyen et al. 2022). In principle, SLMs can also be fine-tuned for different downstream tasks but was not the focus and they are not evaluated on multiple tasks.
Targeting controllable audio generation, VALL-E Wann et al., 2023) extended SLMs by using text and audio prompts to control the audio generated. Voicebox (Le al. 2023) took different ap- proach to tackle the problem by feeding aligned text and partially masked speech to perform speech in-filling non-autoorerressively Despite the different paths VALL-E and Voicebox took, both works discovered a strong zero-shot adaptation ability that emerged when training generative models at scale. While these models are designed for text-conditioned generation, they provided a hint of the great potential of generative models with the superior ability to generate diverse speech. It is worth pointing out that Voicebox is the most related work to this work, sharing the same objective function and model architecture. Voicebox can be viewed as fully supervised text-conditioned SpeechFlow that focused exclusively on TTS task. Later in our experiment, we compare Voicebox to fine-tuned SpeechFlow and reveal the benefit of generative pre-training without text.
Pre-trained Speech Models Conceptually, this work is also related to self-supervised representa- tion learning methods for speech in the sense of learning from unlabeled data for better downstream task performance. One branch of eupervueeeeeeed learning takes the autoregressive approach to learn from predicting the future, such as contrastive predictive coding (Oord et al.. 20188) 8) and autoregre- sive predictive coding (Chung & Glass) 2020). Another branch of works (Ling et al. [020 Ling & Liu 2020) studied masked audio modeling (MAM) instead of future prediction. These models pre- dict masked Spectrogram based on the complementary part of the input that is unmasked. Improving the MAM-based method, similar works replaced the prediction target with latent features such as quantized representation (Baevski et al. 2020) or acoustic units (Hsu et al. 202I). Self-supervised representation learning methods are found to be useful in many different applications such as speech recognition (Yang et al.] 2021). But the success is mostly on discriminative tasks, applying self- supervised models for generation application tasks is often less intuitive (Polyak et al., (021) and under-performing (Tsai et al., 2022). Taking cues from the success of masking-based methods, we incorporate a similar idea into SpeechFlow to make generation conditioned on partially masked speech during pre-tree trainiain Interestingly, we found MAM beneficial to generative pre-training as shown later in Section AA.4.5 Besides self-supervised learning, pre-training have also been stud- ied in the context of supi-supe TTS (Chung et al., 2019) or speech-text alignment (Ao al., 202T), but these works focused on non- gonggeneree models.
2

---

Published as a conference paper at ICLR 2024
3 METHOD
3.1 BACKGROUND: FLOW MATCHING FOR GENERATIVE MODELING
Deep generative models aimed to estimate the unknown distribution q(x) of real world d- dimensional data x E R² with distribution p(z) parameterized by neural networks. To make sam- pling possible, simple prior distribution Pu(x) (e.g.. normal distribution) is naturally a good starting point, and the modeling problem therefore becomes finding neural transport map p1 Fø (Po) such that pi (x) R q(x). Early works such as generative adversarial networks (Goodfellow et al. 2020) and variational audio encoders (Kingma & Welling 2013) showed directly modeling 21 Jo 2o0 where vy ~ Po (1)) ~ ((x), i.e., predicting data from noise using network Je. is feasible. Re- cent studies in diffusion models (Ho et al., 2020; Song et al. 2020) suggested iterative denoising model A+ = fo (.,)( ) that traverses from noise 2i to data Z1 with step size At provides better generation quality (Dhariwal & Nichol, 2021). In this work, we choose to construct the neural trans- port map P1 = Fo (po) using Flow Matching (Lipman et al. 2023) from the Continuous Normalizing Flows (CNFs: Chen et al. 2018)- family.
Formally, CNFs defined a path between simple prior po and target distribution p1 via the time- dependent probability density function Pa. [0, 1J x Rd - Rzo- The flow of along the path, denoted Φ: [0, 1] x Rd + Fd, , is defined using ordinary differential equation (ODE);
xi (1) = vx(d)(z)); do(x)) x;;
(1)
with the time-dependent vector field vi : [o, 1] x Rd + Rd, such that the time-dependent probability density function ps can be derived using the change of variables formula: Pt Oab
Po(o. (z)) det
(z) Under the formulation, a simple objective is to predict the vector field
Or
Ut using neural network paramterized by 0 given the target vector field (z)x) that corresponds to pe(x)) with the Flow Matching objective
LPM (0) = E.. [e(z;) us (=(x))
(2)
However, LF yy (0) is intractable due to the lack knowledge of pt and ut in practice. Interestingly, Lipman al. (2023) showed that conditioning pt and u, real data æ1 results in the Conditional Flow Matching objective CCFM (0) which provided identical gradient w.r.. for training the gen-
erative model. Specifically, we adopt the Optimal Transport conditional path proposed by Lipman et al. (2023) that assumes the mean (x) = fx and standard deviation (x(r) = (1 min)) ) change linearly in time, yielding tractable (e(a))))= N2 pe(x)). ), or 121) and (x(x)=
(2) (21-(1-mm1z) with a sufficiently small Gmin (we use Ic-5) such that p1 (x|x)) is centered around (1- (1-(1-0mm)1)
Tt, In this case, with reparameterization the Conditional Flow Matching objective has the form
(CFM(() Eg.g(x1). 1 (1b) (21 (1 Tmin )=m)
where (x)) = (2(2)) + (x11 and t is sampled uniformly from [0,
3.2 GENERATIVE PRE- TRAINING OF SPEECHFLOW WITH UNLABELED SPEECH
Inspired by the recent success of flow matching model in speech synthesis (Le et al. 2023), we propose to pre-train a generative model with unlabeled speech using flow matching. We consider the problem of modeling q(z) where the acoustic features ø E dx are d-dimensional Mel spec- trogram with L frames. We assume the simple prior Po be the normal distribution. Since genera- tive models are by nature seuppervieed/self-surrriii (no human label required), a flow matching model can be trained with pure speech.
Masked Audio Condition In light of the success of masked prediction in self-supervised speech representation learning (Baevski et al. 2020 Hsu et al.) 2021), we introduce similar concept to SpeechFlow by additionally conditioning v, on partially masked target audio T'aak with a chance of Pewed during training. This can also be interpreted as the model have a chance of Prond to receive
3

---

Published as a conference paper at ICLR 2024
Flow-matching (Eq.4)
Flow-matching (Eq.5)
X1
Ve (Xt)), Amaski B)
(;
Random 0
Pre-trained 0
Task-specific condition
Ymaak
y
X
(xo)
Figure I: An overview of SpeechFlow. (Left) Pre-training with masked audio. (Right) Fine-tuning with task-specific condition such noisy recording, overlapped speech, phone sequence. More details of the model and conditioning are available Section A.3
fully masked T'mas-- Masked condition Fmaak is obtained by randomly selecting T'mask of frames to be masked with a minimum masking span length of /mask.
Note that while this modification results in a conditional generative model, our model is still self- supervised since mask directly derived from unlabeled speech #1 Moreover, vanilla flow match- ing model without any condition is still available after pre- pre-training stage as long as Poond 1. Study on the importance of Peood is provided in Section A.4.5
The rationale behind the auxiliary condition is provide the model more context for predicting it regardless the timestep Moreover, introducing auxiliary condition at the pre-training stage provided an intuitive way fine-tune the model for different tasks as shown later in this section.
With the predicted time-dependent vector field et conditioning masked feature maakk the generative pre-training objective of SpeechFlow can be derived by modifying Equation 3 accord- ingly to obtain
Objective
E of p(ao) oi (1be (20): Fmassk 0)
Omin )
(4)
In practice, use Transformer encoder /Vaswani al. 2017) with learnable parameter @ predict vector field Ut. Masked inputs Tmask are concatenated with the (rr) along the frequency axis, then projected to match the model dimension do, and we append the sinusoidal positional encoding of timestep & to the input, resulting the actual model input with shape dd+x((++1), The output of the model the predicted vector field U2 E 12
3.3 SUPERVISED FINE-TUNING SPEECHFLOW ON DIFFERENT TASKS
Task-specific Condition While the pre-trained SpeechFlow allow to sample new data from Pi (x), most applications speech require certain degree of control over the output. To this end, we introduce the fine-tuning stage for controllable generation using task-specific condition y E X of audio 21, such as noisy speech for speech enhancement and text transcript for text-to-speech generation. We note that this work focused on tasks where y and 2 aligned, i.c., L, L. and leave the unaligned cases for future work. Concrete examples can be found in Section A.3
Objective Following the pre-training stage, the fine-tuning objective can be derived by swapping the masked condition mmakk for pre-training with task-specific condition
E ( p(to)
(v (xo), ys [r1 Omin ).ro
(5)
Note that for fine-tuning, it is critical to reuse from the pre-training stage.
Inference After training. speech generation done by the following steps: (1) sample zy from the simple prior po (z);; (2) an ODE solver to solve &i (Iu) given do,( (xv)/dt u, (x() y: and (n(2)) = In: (3) generated audible speech in time domain from Mel spectrogram æ1. More infer- ence details provided in Section A.2 including conversion from Mel spectrogram waveform.
4

---

Published as a conference paper at ICLR 2024
Table 1: Speech enhancement test results on Voicebank Saicebank-eemadd Valentini-Botinhao et al. 2017 WSJO- CHiME3 (Richter et al. 2023). Best result of each section is bolded. Numbers are taken from prior works unless otherwise specified. For full result that includes more metrics, please refer to Table
Voicehank Demand
WSJO-CHiMES
Method
PESQ
ESTOI
CSIG
COVL
PESQ
ESTOI
CSIG
COVL
Baseline Mixture
1.97
0.79
3.35
2.63
1.69
0.78
3.24
2.42
Models trained Voicehank Demaad
Conv-TasNet' Lao Meseuraii
2019
2.63
0.85
2.40
0.88
MetricGANt et 2021
3.13
0.83
4.10"
3.61
2.13
0.76
3.02'
2.52'
SGMSF+ (Richler etal. 2023
2.93
0.87
4.13"
3.53"
2.48
0.90
3.67"
3.02"
SpeechFFow
3.13
0.87
4,43
3.80
2.70
0.90
4.05
3.36
SpeechFlow wo pre-truin
2.92
0.85
4.22
3.57
2.38
0.86
3.72
3,03
Models trained on Deep Noise Supression Challange 2020
Reddy et al
2020)
DEMUCS
2.55
0.85
3.24
2.88
2.49"
0.92
3.20
SpeechFlow
2.71
0.86
4.07
3.39
2.87
0.91
4.24
3.54
SpeechFlow wio pre-train
2.53
0.84
3.89
3.20
2.56
0.89
3.91
3,22
Topline
Our upper-bound
3.77
0.95
4.97
4.54
3.68
0.96
4.97
4.46
Clean signal
4.50
1.00
5.00
5.00
4.50
1.00
5.00
5.00
Results reproduced by us using the open sourced model released by the authors.
Results reproduced by Richher al. 2023)
Clean Mel spectrogram with error intruduced by pseudo-inversing Mel filter laking phuse from mixture.
EXPERIMENT
4.1
PRE-TRAINING DETAILS
Model & Data We focus on Transformer encoder (Vaswani et al. 2017) with 24 layers, 16 atten- tion heads, dg 1024 dimensional embedding, and feed-forward networks with 4096 dimensions. Convolutional positional embedding (Baevski et al. (0200) and ALiBi attf-aattntioon bias (Press et al.) 2021) are used to encode relative positional information. Following Le al. (2023), skip connec- tions between layers are introduced to mimic U-Net (Ronneberger et al. 2015) architecture. The model has around 330M parameters in total. The model is pre-trained on 60k hours of speech from English audiobook at 16kHz. We consider to be log-scaled Mel spectrogram extracted with 40ms window 100Hz with d = 80, resulting 160/80 /80 dimensional input/output for the model.
Training We pre-train SpeechFlow for 600k steps on 32 V100 GPUs with a batch size of 75 seconds per GPU with FP16. We use Adam optimizer (Kingma & Ba (014)) with the learning rate warming up linearly to 5e-5 for the first 5k steps and linearly decaying to le-5 for the rest of the training. For masking, we set Pirop 10%, 1'mask ~ 4(70%, 100%), and Imak 10. All masked position are filled with zero. In practice, we compute loss at the masked position only.
4.2 FINE-TUNING FOR SPEECH ENHANCCMENT
Task & Metrics Speech enhancement, also known as denoising, aimed to remove unwanted noise from speech recording. We report Perceptual Evaluation of Speech Quality (PSQQ[Ris al. 2001), Extended Short-Time Objective Intelligibility (ESTOI: Jensen & Taal 2016). and Composite Ob- jective Speech Quality and Overall Quality (CSIG/COVL Loizou 2007).
Prior Works Early work Conv-TasNet (Luo & Mesgarani 2019) has been widely used as the baseline system.
convolutional encoder/decoder operating in the time domain maximize scale-invariant source-to-noise ratio. DEMUCS (Dérossez et al. 2020) adopted similar structure with skip-connections and minimized I. r/muttii-rsooltiion STFT loss. MetrieGAN+ (Fu
2021)
proposed to optimize non-differentiable metrics such as PESQ via adversarial training against their approximation using discriminators. SGMSE+ (Richter et al., 2023) reformulated the problem as diffusion process that can be solved with the corresponding generative model (Ho al. 2020
Dataset We fine-tuned and tested SpeechFlow on the benchmark dataset VoiceBank VoiceBank-Demand Demand (VB-DMD: Valentini-Botinhar al. 2017) for fair comparison against most of the prior works
5

---

Published as a conference paper at ICLR 2024
Table Speech separation test results on LibriMix (Cosentino et al. 2020 models are trained on 16kHz
audio without data augmentation. Best model output for each metric is bolded.
Mix
2 Mix
Noise
3 Mix
3 Mix Noise
Method
ST-SDRi
ESTOB
SI-SDRi
ESTOIi
SI-SDRi
ESTOR
SI-SDRi
ESTOB
Conv-TasNet'
15.24
0.22
12.55
0.22
12.30
0.26
10.28
0.21
SepFormer?
14.94
0.31
11.71
0.28
Pscudo-inversed Mel and phase from mixture
Upper-bound w/ clean Spec.
12.43
0.35
11.99
0.46
12.91
0.44
12.62
0.48
SpozchFlow
11.74
0.35
10:46
0.33
11.08
0.35
8.22
0.23
SpeechFlow w/o pre-train
11.24
0.29
10.00
0.31
8.65
0.24
7.39
0.19
Learable inverse-Mer Mel and phase estimation (See Section for more details.)
SpeechFlow
15.85
0.37
0.37
reproduced at 16kHz with official code from SpeechBrain (Ravanelli et al. 2021]. note that this method was originalyy designed for 8kHz audio with data augmentation.
Luo & Mesgarani 2019 reproduced by Cosentino et al. 2020
SSubakan et al. 2021 2023
in the field. Since VB-DMD is a relatively small dataset, we also consider testing on WSJO- CHiMe3 (Richter et al. 2023) to ensure the model is not overfitting. In addition, we also trained our model using 100 hours of noisy speech from Deep Noise Supression Challenge 2020 (DNS2020; Reddy et al. 2020) for extra results to demonstrate the generalizability for SpeechFlow. For train- ing, paired data (æ1, y) is provided where 2 is the target clean signal and is the noisy speech. For testing, only the noisy speech y is provided and the goal is to estimate the clean signal æ1. All datasets are resampled to 16kHz to match pre-training and no data augmentation was applied.
Training As mentioned in Section 3.3 fine-tuning is simply done by replacing the auxiliary masked condition Z'm for pre-training with the acoustic feature of the noisy speech y and minimize 5q.5) Note that, unlike pre-training, has Pdrop = 30% chance to be dropped but never partially masked for fine-tuning. We fine-tuned SpeechFlow on single V100 GPU for 160 75 epochs on VB-DMD / DNS2020 respectively with batch size of 50 seconds. The learning rate is set to peak 2e-5 after 5k updates, then linearly decay to For the control group without pre-training, we searched learning rate between le-4 to le-3 and found 2e-4 the best.
Results Main results are provided in Table T Due to the choice of acoustic feature, our method suffers from the imperfect pseudo-inverse of Mel filters and the lack of phase modeling. In contrast to prior works tailored for enhancement, these restrictions result in a worse bound as shown in the table. Nevertheless, our method still provided comparable or better results against the prior works on both benchmark datasets. Despite using a dataset with different topics and speakers, gen- erative pre-training still improved enhancement results compared to the same model trained on VB- DMD from scratch. Especially on the out-of-domain WSJO-CHiME3 testing, SpeechFlow demon- strated strong generalizability with a clear gap on PESQ, CSIG, and COVL against all other methods. In the case where the larger dataset DNS2020 is used for fine-tuning, similar trend can be found compared to prior work DEMUCS and the testing result on WSJO-CHiMES can be further improved. These results pointed out the great potential of generative pre-training on speech.
4.3 FINE-TUNING FOR SPEECH SEPARATION
Task & Metries The goal of separation is to separate mixture (overlapped) speech into multiple single-speaker speech. In our experiment, we focus on separating 2 to 3 speakers for simplicity. We report the common metric Scale-Invariant Signal-to-Distortiin Le Roux et al.] 2019) that measures the improvement of separated speech over the mixture when comparing against the clean reference in the time domain. In addition, also report the ESTOI improvement (ESTOLi) of the separation result over the mixture to measure the intelligibility.
Ratio improvement (SI-SDRi;
Dataset & Prior Work For separation, SpeechFlow is fine-tuned using a synthetic mixture cre- ated by randomly sampling and mixing 2 or 3 utterances from 360 hours of speech from English audiobook. In addition, noise sampled from WHAM! dataset (Wichern et al. (219)) can be added to the mixture to further increase the difficulty of separation, combining 4 different setups in total. We tested the fine-tuned model on LibriMix (Cosentino et al. (2020) 16khz min. For training, paired data (xt of. y) is provided where ot, of are the target clean signal and y is the mixture. Signals

---

Published as a conference paper at ICLR 2024
Table 3: English zero-shot speaker adaptation TTS results on filtered LS Panayotov et al. 2015) test-clean. Best results are bolded, For cross-sentence reference, the speaker information is provided by 3-second prompt from different utterance sampled randomly. For continuation, the first seconds of the target utterance is used. FT stands for fine-tuning the full model; LoRA stands for fine-tuning with Low-rank Adaptors (Hu et al. 2021) where pre-trained weights are frozen.
labeled data (hr)
cross-sentence reference WER SIM-o
continuation
subjective MOS
Method
SIM-r
WER SIM-o SIM-r
Ground truth
2.2
0.754
3.80
YourTTS
Casanova el al.
2021
475
7.7
0.337
n/a
2.92
VALL-E
Wang ct al. 2025
60k
5.9
0.580
3.8
0.452
0.508
Voicebox Le al. 2023
60k
1.9
0.662
0.681
2.0
0.593
0.616
3,54
Single GPU training
SpeechFlow w/o pre-train
960
2.3
0.526
0.573
2.2
0.467
0.513
SpecchFlow FT
960
2.2
0.678
0.694
2.2
0.613
0.630
SpeechFlow LoRA
960
2.6
0.696
0.711
2.4
0.623
0.640
32 GPU training
SpeechFlow w/o pre-train
960
2.0
0.569
0.598
2.1
0.530
0.557
SpeechFlow FT
960
2.2
0.697
0.703
2.2
0.622
0.629
SpeechFlow LoRA
960
2.1
0.700
0.715
2.1
0.630
0.644
3.43
are randomly cropped into 8-second chunks for training. To ensure the model outputs all speakers, concatenated the clean signals along the time axis (and repeated the condition y accordingly) for both training and testing. The baseline system is Conv-TasNe (Luo & Mesgarami, 2019) from LibriMix We note that while there many other prior works in the field, most of them focused on WSJ2mix dataset (Hershey et al. 2016) with 8kHz audio, which makes fair comparison dif- ficult. To provide more competitive baseline, we reproduce more powerful separation model SepFormer (Subakan et al. 2021 2023) at 16kHz using code provided by the authors
Training The fine-tuning setup follows enhancement with few changes: batch size is reduced to 37.5 seconds; model is fine-tuned for 85 epochs; peak learning rate is set to 3e-5, For Speech- Flow without pre-training, we searched learning rate between le-5 to lc-4 and found 5e-5 the best.
Results Results are provided in Table We found SI-SDRi more sensitive the process of Mel- spectrogram- toarraa-oo-w This can be verified by examining the upper-bound performance using a clean reference Mel spectrogram, which is even worse than the baseline Conv-TasNet. Similarly, we found the more recent transformer-base
model SepFormer (Subakan et al. 2023) struggled in SI-SDRi when training at 16kHz (i.e., 2x longer input). In contrast, we found ESTOTi that reflected the intelligibility of separation result more robust to waveform estimation. Nevertheless, fine-tuned SpeechFlow was able to provide strong separation results. The gap between SpeechFlow and its upper-bound is particularly small in the easy 2 Mix setup. To measure the true quality of the Mel spectrogram generated by SpeechFlow, we also experimented with learnable inverse-Mes and phase estimation (as described in Section A.2) and found the separation result in terms of SI-SDRi. Since optimizing the Mel-spectrogram--e waveform transform is beyond the scope of this paper, apply learnable estimation the best result of 2 Mix and 2 Mix Noise only. The key idea is to show the separation result in the Mel spectrogram is already at high quality, and metrics that are limited by the choice of input/output feature like SI-SDRi can be further improved with extra effort. In conclusion. we found SpeechFlow providing better intelligibility in all cases. It worth noting that the fine-tuning method presented here vanilla solution that might not scale well as the number of speakers increases, more dedicated fine-tuning method is left as future work.
be further boosted
4.4 FINE-TUNING FOR ZERO-SHOT SPEAKER ADAPTATION OF TEXT-TO-SPEECH
Task & Metrics We consider speech generation conditioning on text, i.e., text-to-speech (TTS). In particular, focus the zero-shot speaker adaptation problem (Jia et al., 2018, Casanoval et al. 2021) where the voice of an unseen speaker should be used for synthesis. The problem
https://gitt http://wwwwoooeoeccrrreieei

---

Published as a conference paper at ICLR 2024
setup and the evaluation metrics followed VALL-E (Wang et al. 2023 and Voicebox (Le et al. 2023). Zero-shot adaptation is done by using a 3-second prompt that carries speaker, paralinguistic, and environmental information. To measure the correctness and the intelligibility of the synthetic speech, we measure the recognition word error rate (WER) using HuBERT-L. (Hsu et al. (202T) pre-trained and fine-tuned LibriLight (Kahn et al., 2019) and LibriSpeech (Panayotov et al.) 2015) respectively. Using WaYLM-TDCNN speaker embedding model [hen et al. (2022). speaker similarity is measured by the similarity between the embedding of generated speech and that of the conditioning audio. Similarity to the original conditioning audio (SIM-o) and to the vocoder- resynthesized audio (SIM-r) are reported. In addition to the objective metries, subjective evaluation on cross-sentence reference results using mean opinion score is also provided. See more detail regarding MOS test in Section A.4.6
Prior Works YouTTTS (Casanova et al. 2021) is a flow-based model (Kim et al. (222 trained on multi-lingual data, including VCTK (Yamagishi et al. 2001). TTS TTS-portugess. (Casanova et al. 2022).
M-AILABS French (Munich Artificial Intelligence Laboratories GmbH| 2017). and LibriT''S (Zen eet al. 2019). VALL-E is a decoder-only auto-regressive model trained on LibriLight for zero-shot speaker adaptation TTS. Lastly, the closely related prior work Voicebox combined flow-matching and masked prediction for supervised TTS training. Voicebox can be viewed as a strong baseline using the same amount of data with fully supervised training.
Dataset 960 hours of transcribed speech from English audiobook is used for fine-tuning. The testing protocol follows VALL-- and Voicebox. Montreal Force Aligner (McAuliffe et 2017) is used for phone-speech alignment. Position postfixes are added to each phone following Voicebox. Additional results on fine-tuning with less (100/10 hours) labeled data are provided in Scction[A.4..
Training To enable zero-shot speaker adaptation, fine-tuning condition y includes masked audio Im and the force-aligned phone sequence. We followed the masking strategy of Voicebox dur- ing fine-tuning. We additionally tested fine-tuning with more (32) GPUs and Low-rank Adaptors (LoRA: Hu et al., [021]] we use rank r = 64) to study the impact of computational resource for fine-tuning. Section A.4.2 provided detailed performance analysis based on the number of GPUs used for fine-tuning. The batch size is 75 seconds per GPU in all cases. For standard fine-tuning, the learning rate set to peak le-5 after 5k updates, then linearly decay to 0 for the rest 145k steps. For LoRA fine-tuning, 9.5M new learnable parameters are introduced to the pre-trained model, ac- counting for 2.8% of the full model. All pre-trained weights are frozen. The learning rate set to peak at le-3. Additional results on the impact of the amount of fine-tuning GPU is provided in Section A.4.3.
Results Results provided in Table Comparing fully supervised models Voicebox or VALL-E, a clear advantage in speaker modeling can be found with SpeechFlow despite using much less labeled data. In terms of WER and MOS, SpeechFlow is slightly worse than Voicebox that uses more labeled data. In addition, while single GPU fine-tuning already provided better speaker adaptation than all baselines, Interestingly, LoRA performed the best in terms of both SIM and WER among all fine-tuning setups. This suggested that fine-tuning method for generative model could be worth exploring in the future. Finally, our baseline without pre-training achieved similar WER to that of the pre- are-trained model but a significantly worse SIM. These findings suggested the proposed generative pre- teeiriiiiig improves speaker modeling but not content modeling for speech synthesis.
found fine-tuning with more GPUs provided even stronger results.
4.5 MULTI-TASK FINE-TUNING OF SPRECHFI.OW
Preceding sections showed SpeechFlow can be fine-tuned for different purpose using limited paired data and/or computation. In this section we take one step further to investigate the possibility to build an all-in-one controllable speech generation model via multi-task fine-tuning. Results are car- ried out in Table 4 We simply combined the labeled datasets for enhancement (DNS), separation (oisx+ and TTS for fine-tuning. We upsampled these datasets with factor of 10/4/1 re- spectively to balance the importance each task. Pre-trained SpeechFlow is fine-tuned on single GPU for 700k updates with the same learning rate scheduler peaking at 2e-5.
8

---

Published as a conference paper at ICLR 2024
Table 4: Results for multi-task tunettning. Both single-task and multi-task SpeechFlow are fine-tuned using single GPU. Expert models are the best prior work for each metric of each task from Table [2 For TTS /enbancemenn/separttioo we consider the cross-efferenoeWSSSI0 7SSA is short for zero-shot speaker adaptation.
CHiMEV2Mix+ CCC
scenario respectively.
ZSSA TTS
Enhancement
Separation
Method
WER
SIM-o
PESQ
COVL
SI-SDRi
ESTOIi
Single-task models
Expert prior work
1.9
0.662
2.49
3.32
12.55
0.28
SpeechFlow
2.2
0.678
2.87
3.54
12.41
0.37
Multi-task models
SpeechFlow
2.3
0.651
2.87
3.56
9.73
0.30
For zero-shot speaker adaptation TTS, we observed drop both WER and SIM-o, suggesting multi-task learning can lead to worse performance in specific single task. However, multi-task results are found be better than single-task ones for enhancement. One possible explanation is the separation task trained on mixture+noise can also be viewed as hard enhancement problem the model was additionally trained on. This showcased the benefit of having universal model some tasks might benefit from others. For separation, we found multi-task model deteriorated significantly comparing to the single task model. Preliminary results presented in this section suggested an all-in- one speech generative model can be built from SpeechFlow, but further research and development required to improve the results and cover a more diverse set of tasks.
CONCLUSION
In this paper, we studied the role of generative model as a foundation model instead of a tool for specific task. We show that training SpeechFlow using flow matching with masked condition results in a strong generative model. The model can be deployed to different downstream tasks using simple fine-tuning strategy with single GPU. our experiment, we adapted SpeechFlow to speech enhancement, separation, and zero-shot speaker adaptation TTS with performance comparable to task-specific models. More importantly, SpeechFlow demonstrated the potential to unify generative tasks for speech.
Limitations and Future Works This work focused on developing the mee-rrin-andd-ine-tmm framework for generative speech model. For the selected downstream applications, we assumed a frame-wise condition (c.g.. noisy spectrogram; force-aligned phone label) is available in the fine- tune dataset. Fine-tuning with misaligned data (e.g., raw text, speaker ID) left as an important future work. In addition, SpeechFlow is trained and tested on English-only data. However, since generative model can be trained without label data, we believe the method can easily scaled to more languages in the future. For future works, we would like to point out that the choice of acoustic feature may limit the applications as we discovered in enhancement and separation. Hence finding general acoustic feature would be key step general purpose generative speech model. Finally, we note that some of the expert models compared in different downstream tasks have other focuses besides the reported metrics (e.g., DEMUCS is built to in real-time with fewer pa- rameters). Therefore, we would like to emphasize that this work is mainly to show the potential of pre-trained generative models rather than claiming state-of-the-an different tasks.
ACKNOWLEDGMENTS
The authors would like to thank Gene- Pinee Yang for helpful discussions speech separation and Baishan Guo for setting up human evaluation.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, et al. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. arXiv preprint arXiv:2110.07205, 2021.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0; A frame- work for self-supervised learning of speech representations. Advances in neural information processing systems, 2020.
He Bai, Renjie Zheng, Junkun Chen, Xintong Li, Mingbo Ma, and Liang Huang. A3T: Alignment- aware acoustic and text pretraining for speech synthesis and editing. In International Conference Machine Learning. 2022.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matthew Sharifi, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. AudioLM: language modeling approach to audio generation. ArXiv, abs/2209 03143, 2022.
Edresson Casanova, Julian Weber, Christopher Dane Shulby, Arnaldo Cândido Júnior, Eren Gölge, and Moacir Antonelli Ponti. YourTTS: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning. 2021.
Edresson Casanova, Amaldo Candido Junior, Christopher Shulby, Frederico Santos de Oliveira, João Paulo Teixeira, Moacir Antonelli Ponti, and Sandra Aluísio. Tts-portuguese corpus: corpus for speech synthesis in brazilian portuguese. Language Resources and Evaluation, 56(3): 1043- 1055, 2022.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.007133 2020.
Ricky T. Q. Chen. Tacchiifeeg
torchdiffeq. 2018.
URL attps///iitaub..om//rtitttt
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Kristjanson Duvenaud. Neural ordinary differential equations. In Neural Information Processing Systems, 2018.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505- 1518, 2022.
Yu-An Chung and James Glass. Generative pre-training for speech with autoregressive predictive coding. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3497-3501. IEEE, 2020.
Yu-An Chung, Yuxuan Wang, Wei-Ning Hsu, Yu Zhang, and RJ Skerry-Ryan Semi-supervised training for improving data efficiency in end-to-end speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6940- 6944. IEEE, 2019.
Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vin- cent. Librimix: An open-source dataset for generalizable speech separation. arXiv preprint 1rXiv:2005.11262, 2020.
Alexandre Défossez, Gabriel Synnaeve, and Yossi Adi. Real time speech enhancement in the wave- form domain. ArXiv, abs/2006. 12847, 2020.
Alexandre Défossez, Jade Copet. Gabriel Synnaeve, and Yossi Adi High fidelity neural audio compression. ArXiv, abs/2210.13438. 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Ad- vances in Neural Information Processing Systems, 2021.
10

---

Published as a conference paper at ICLR 2024
Szu-Wei Fu, Cheng Yu, Tsun-An Hsieh, Peter Plantinga, Mirco Ravanelli, Xugang Lu, and Yu Tsao. Metricgan+: An improved version of metricgan for speech enhancement. arXiv preprint arXiv:2104.03538, 2021.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11))) 63(11))1391144 2020.
Alex Graves, Santiago Feránndez, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Raza Habib, Soroosh Mariooryad, Matt Shannon, Eric Battenberg, RJ Skerry-Re Ryan, Daisy Stan- ton, David Kao, and Tom Bagby. Semi-supervised generative modeling for controllable speech synthesis. arXiv preprinl arXiv: 1XX.01/90000109 2019.
John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discrim- inative embeddings for segmentation and separation. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 31-35, IEEE, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
(xXiv:2106.09683 2021.
Yi Hu and Philipos C Loizou. Evaluation of objective quality measures for speech enhancement. IEEE Transactions on audio, speech, and language processing, 16(1):229-238, 2007.
Jesper Jensen and Cees H Taal. An algorithm for predicting the intelligibility of speech masked by modulated noise maskers. IEEE/ACM Transactions on Audio, Speech, and Language Processing.
24(11):2009-2022 2016.
Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, Yonghui Wu, et al. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. Advances in neural information processing systems, 31, 2018.
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Feere-Emmanuel Mazar's e, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, Tatiana Likhomanenko, Gabriel Synnaeve, Armand Joulin, Abdel rahman Mohamed, and Emmanuel Dupoux. Libri-Light: benchmark for asr with limited or no supervision. International Confer- ence on Acoustics, Speech and Signal Processing, 2019.
Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Nguyen, Morgane Rivière, Abdel rahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. Text-free prosody-aware generative spoken language modeling. Annual Meeting of the Association for Computational Linguistics, 2021.
Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
Diederik Kingma and Max Welling, 1312 2013.
Auto-encoding variational bayes. arXiv preprint
11

---

Published as a conference paper at ICLR 2024
Yuma Koizumi, Heiga Zen, Kohei Yatabe, Nanxin Chen, and Michiel Bacchiani. Specgrad: Diffu- sion probabilistic model based neural vocoder with adaptive noise spectral shaping. arXiv preprint arXiv:2203.16749, 2022.
Jungil Kong,
Jaehyeon Kim, and Jackyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 13:17222-17033, 2020.
Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu Nguyen, Jade Copet, Alexei Baevski, Adel Ben Mohamed, and Emmanuel Dupoux. On gen- erative spoken language modeling from raw audio. Transactions of the Association for Computa- tional Linguistics, 9:133 9:1336-1354, 2021.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. arXiv preprint 2.0.::23.0666688.. 2023.
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr-half-baked or well done? In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 626-630. IEEE, 2019.
Shaoshi Ling and Yuzong Liu. Decoar 2.0: Deep contextualized acoustic representations with vector quantization. arXiv preprint arXiv:2012.06659) 2020.
Shaoshi Ling, Yuzong Liu, Julian Salazar, and Katrin Kirchhoff. Deep contextualized acoustic rep- resentations for semi-supervised speech recognition. In ICASSP 2020-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6429 6433. IEEE, 2020.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In International Conference on Learning Representations, 2023.
Yen-Ju Lu, Yu Tsao, and Shinji Watanabe. A study on speech enhancement based on diffusion probabilistic model. In 2021 Asia- Signal and Information Processing Association Anmual Summit and Conference (APSIPA ASC), pp. 659-666. IEEE, 2021.
Yen-Ju Lu, Zhong- Qiu Wang, Shinji Watanabe, Alexander Richard, Cheng Yu, and Yu Tsao. Con- ditional diffusion probabilistic model for speech enhancement. In ICASSP 2022-2022 IEEE In- lernational Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7402-7406. IEEE, 2022.
Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time- frequency magnitude masking for speech separation. IEEE/ACM transactions on audio, speech, and language processing, 27(8): 1256-1266, 2019.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech, 2017.
Munich Artificial Intelligence Laborattoeiee GmbH. The m-ailabs speech dataset caito, 2017. URL m- a labs-
Tu Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Mamdouh Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, and Emmanuel Dupoux. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11 11:250-266, 2022.
Aaron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ast corpus based on public domain audio books. International Conference on Acoustics, Speech and Signal Processing, 2015.
12

---

Published as a conference paper at ICLR 2024
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- deirahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. In Interspeech, 2021.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-20112 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3617-362 IEEE, 2019.
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. ArXiv, abs/2 192422 2021.
Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung- Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na. Yan Gao, Renato De Mori, and Yoshua Bengio. SpeechBrain: A general-purpose speech toolkit, 2021. arXiv:2106.0460 146.4444.
Chandan KA Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et al. The in- terspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results. arXiv preprint arXiv:2005. 13981, 2020.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. International Conference on Learning Representations, 2021.
Flávio Ribeiro, Dinei Florêncio, Cha Zhang, and Michael Seltzer. CrowdMOS: An approach for crowdsourcing mean opinion score studies. In International Conference on Acoustics, Speech and Signal Processing. 2011.
Julius Richter, Simon Welker, Jean-Marie Lemercier, Bunlong Lay, and Timo Gerkmann. Speech enhancement and dereverberation with diffusion- generative models. JEEF/ACM Transac- fions on Audio, Speech, and Language Processing, 2023.
Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE international conference on acoustics, speech, and signal processing. Proceedings ( (Cat. No. OICH37221), volume 2. pp. 752- IEEE, 2001.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assisted Intervention- MICCAÍ 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part III 18, pp. 234-241, Springer, 2015.
Robin Scheibler, Youna Ji, ShonWhan Chung, Jaeuk Byun, Soyeon Choe, and Min-Seok Choi. Diffusion-basee generative speech source separation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1-5. 2023.
Jonathan Shen, Ruoming Pang. Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural uts synthesis by con- ditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 4779-4788 IEEE, 2018.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.133555 2020.
Cem Subakan, Mirco Ravanelli, Samuele Comell, Mirko Bronzi, and Jianyuan Zhong. Attention is all you need in speech separation. In ICASSP 202 202/-202/ IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 21-25. IEEE, 2021.
Cem Subakan, Mirco Ravanelli, Samuele Coreell, François Grondin, and Mirko Bronzi. Exploring self-attention mechanisms for speech separation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
13

---

Published as a conference paper at ICLR 2024
Hsiang-Sheng Tsai, Heng-Jui Chang, Wen-Chin Huang. Zili Huang, Kushal Lakhotia, Shu-wer Yang, Shuyan Dong, Andy Liu, Cheng-I Jeff Lai, Jiatong Shi, et al. Superb-sg: Enhanced speech processing universal performance benchmark for semantic and generative capabilities. arXiv preprint 2203. 06849, 2022.
Cassia Valentini-Botinhaao et al. Noisy speech database for training speech enhancement algorithms and tts models. University of Edinburgh. School of Informatics. Centre for Speech Technology Research (CSTR), 2017.
Rafael Valle, Kevin Shih, Ryan Prenger, and Bryan Catanzaro. Flowtron: autoregressive flow- based generative network for text-to-speech synthesis. arXiv preprint arXIv:2005.05955, 2020.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv, abs/1706.03762. 2017.
Chengyi Wang. Sanyuan Chen, Yu Wu, Zi-Hua Zhang, Long Zhou, Shujic Liu, Zhuo Chen, Yanqing Liu, Huamingg Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers. ArXiv, abs/2301.02111 2023.
Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmetr McQuinn, Dwight Crow, Ethan Manilow. and Jonathan Le Roux. Wham!: Extending speech separation to noisy environments. arXiv preprint
2019.
Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald. CSTR VCTK Corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92). 2019.
Shu-wen Yang, Po-Han Chi, YYng-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan- Ming-Tingg/g Lin, et al. Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2 100.:2005,105,, 2021.
Dong Yu, Morten Kolbæk, Zheng-Hua Tan, and Jesper Jensen. Permutation invariant training of deep models for speaker-indeppendet multi-talker speech separation. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 241-245. IEEE, 2017.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing, 30:495-500 2022.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv: 028802882 2019.
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 AUDIO SAMPLES
Audio samples can be found ltttp::/voiieboxx .ew.ddeolaab. For additional samples, please refer to the supplementary materials at nttps://openreviee net/ hett/EEorummI1-=kpp00sxxxt
wttp:aabbccccmpeeeeeeeeeeee ccm/ cot//speecccoooo..tt
A.2 INFERENCE DETAILS
Generating Mel Spectrogram To generate Mel spectrogram 21, first sample zo from the sim- ple prior (z). The next step is to estimate Φ1 (2o) given Φn = 2y by evaluating ey (oo))), y; 0) multiple Each evaluation required forwarding through the neural network, and larger num- ber of function evaluations (NFEs) leads to more accurate estimation of Φ1 (zo). In addition, we also applied classifier-free guidance (CFG: Dhariwal & Nichol 2021: Le al. (023) prioritize audio quality over diversity. CFG is done by additionally predicting the unconditioned vector field ve (2o); 6) (where the mask-specific condition dropped) obtain a modified prediction
d (1 a.) (x(() y: + a (z)))
CFG allows us improve sample quality by focusing more task-specific conditioned generation with larger a at the cost of doubling NFEs. We use = 0.5 for enhancement and 0.7 for other tasks in practice. For the ODE solver, 2018) to derive d (zo) from do (To) by approximating the integration from = = with step size of 0.0625, resulting 32 NFEs per sample.
midpoint method implemented in torchdiffes (Chen)
Zero-shot Speaker Adaptation TTS To generate audible speech from Mel spectrogram, HiFi- GAN vocoder (Kong et al. 2020) from VoiceBox (Le et al., (2023) is adopted. In addition, phone duration is also needed to determine the output spectrogram length and the frame-wise condition given the input phone sequence. The regression- based duration predictor from VoiceBox is adopted for all TTS-related experiments.
Speech Enhancement Different from TTS, enhancement metrics are more sensitive to the sample- to-sample alignment of the waveform between the hypothesis and reference. This makes Neural vocoder a bad option for the task Alternatively, we found using pseudo-inverse of Mel filter bank to recover linear Spectrogram, adding phase information taken directly from the noisy speech (input condition), and apply inverse Short-Time Fourier Transform (iSTFT) sufficien() As a reference, PESQ on WSJO-CHiME3 dropped from 2.70 to 2.29 when switching the signal processing method to HiFi- GANi vocoder.
Speech Separation For this task, we found both the signal processing method and HiFi-GAN vocoder not enough for the most popular metric SI-SDRi (see discussion in Section 4.3) To this end, we train a 3-layer ResNet for both pseudo-inverse Mel transform and phase estimation using precomputed Mel spectrogram prediction and target waveform the training set. The model takes the separation result (Mel spectrograms from SpeechFlow) and the complex spectrogram of the mixture input, predicting both the linear spectrogram and the phase information to be combined and transformed to the time domain with iSTFT. Since the whole process is differentiable, the model is trained to maximize the permutation- waveform.
-ntittaaiiionnniiianttt invariant YYu al. 2017) SI-SDR loss against the target
"See the last section in demo page for examples.
*See the topline section in Table for the error introduced by the process.
15

---

Published as a conference paper at ICLR 2024
A.3 MODEL ARCHITECTURE AND CONDITION DETAILS
fine-tuning
pre-training
(enfmmeummti (enhmnoummmiseuuattion sppuratioon
Transformer
concat
masked spectrogram
noisy/mmixture spectrogram
Projected feature
embedding
Linear
fine-tuning
(TTS)
condition
(xp)
masked
aligned
spectrogram
phone emb.
concat
Figure 2: Blue blocks are learnable weights. (Left) Model architecture. Time (flow step) is encoded using sinusoidal position embedding with learnable scale. (Right) Condition used for different tasks. For TTS fine-tuning, learnable phone embedding sequence aligned to the spectrogram is element- wise added to the masked spectrogram. Since phone embeddings are randomly initialized and added the masked spectrogram. found ramping up pero-initialized gating value (single scalar be multiplied on phone embedding) yields slightly better results in practice.
Table 5: Detailed configurations for training. ConvPos stands for Convolutional Positional Embed-
ding (Baevski et al.] 2020); Skip Connections introducted in Le et al. 2023); Alibi Bias is introducded in Press et al. 2021)
Fine-tuning
Pre-training
Enhancement
Separation
TTS
Model Parameters
Model Dimension
1024
Number Heads
Number of Layers
24
Feedforward Dimension
4096
Attention Dropout
0.0
Activation Dropout
0.1
ConvPos Width
31
ConvPos Pos Groups
ConyPos Depth
Skip Connections
true
Alihi Bias
Additional weights
No
80-dim. phn. emb.
llyper-Paranne Parameters
Condition drop rate Plup Masking probability Baask Minimum mask span bmuck
10%
30%
30%
20%
41700, 100%]
0%
0%
4417%% 100%]
10 frames
N/A
N/A
10 frames
Training Parameters
Number Updates
600k
(dataset-dopendent, see Section
4.2
150k seconds
Number GPUs
Batchsize per GPU
75 seconds
50 seconds
37.5 seconds
Max length per audio
16 seconds
Learning Rate
Se-5
2e-5
3e-5
le-5
Gradient Clipping Value
0.2 5000
LR Scheduler Warmup Steps
16

---

Published as a conference paper at ICLR 2024
A.4 ADDITIONAL RESULTS
A.4.1 SPEECH EDITING
Following the setup of A3T (Bai et al., 2022). here we additionally consider the task where the center 50% of an recording details and illustration of the task. Similar to A3T and Voicebox (Le et al. (023) that are trained with masked audio and text conditioning, speech editing is another downstream that can be naturally solved with the fine-tuned SpeechFlow. Results presented in Table evaluation metrics and the dataset follows the zero-shot speaker adaptation TTS experiment presented in Section [4.4] Similar to zero-shot speaker adaptation TTS, found SpeechFlow performing close to the state-of-tee titthh--rt model using much less labeled data thanks to pre-training.
be edited. See Figure and Section 4.4 in Bai et al. 2022 for more
Table 6: Speech editing results on filtered LS Panayytov et al. Section for the metries used here.
2015 test-clean. Please refer to
labeled data (hr)
Method
WER
SIM-o
A3T (Bai ct al. Voicebox (Le al.
2022
44
11.5
0.148
2023
60k
2.0
0.613
SpeechFlow LoRA
960
2.2
0.647
A.4.2 FULL RESULT FOR SPEECH ENHANCEMENT
Table 7: Speech enhancement results on the test set of Voicehank-Demant (Valentini-Botinhao et al. 2017) and WSJO-CHiME3 (Richter et al.) 2023). All metrics are the higher the better, best result of each section is bolded. Numbers are taken from prior works unless otherwise specified. PSEQ-nb
the nardow-band version of PSEQ. CBAK refers to composite background intrusiveness (Hu &
Loizou 2007)
Methed
Voicehank Demand
CHIMES
PESQ
PESQQab
ESTOI
CSIG
CBAK
CUVL
PE:SQ
PESC) ab
ESTOI
CSIG
CBAK
COVL
Mixture
1.97
2.58
0.79
3.35
2.44
2.63
1.69
2.29
0.78
3.24
2.25
2.42
Models trained on Voicebank Demand
SUMSSS (Richerr
3.53
Coay Tasher Loo & Masara
2019
2.63
3.42
0.85
2.40
0.88
MetricGAN Pi 201
3.13
3.63
0.8
4.10"
2.90
3.61
2.13
2.67
0.76
3,02
1.88
2,52
DEMUCS
2020
3.07
4.31
3,40
3,63
SprechFlow SprochFlow
3.13
3.74
0.87
4.43
3.41
3,80
2.70
3.36
0.90
4.05
2.97
3.36
Models trained on Deep Noise Supression Challange
2020 (RoMy al. 2020
DEMUCS Defosses 2020
2.55
3.40
0.35
3.24"
3.26'
2.58
2.89
3.20
0.92
3,93
3.24
3.20'
SpeechFlow
2.71
3,68
4.07
2.93
3.39
2.87
0.91
3.54
Topline
Curs upuns-
3.77
4.09
0.95
4.97
4.00
4.54
3.68
3.93
0.96
4.97
3.81
4.46
Clean
5.00
reproduced by ua nsing open squrced released by authoes.
reprsduced by Richter 20233
Obsained from Me) Specnogram ol Lhe cleun signal, emvor introduced by pscudo inversing Mel filter bank and taking phase from the misture.
17

---

Published as a conference paper at ICLR 2024
A.4.3 INCREASING THE NUMBER OF GPUs FOR FINE-TUNING
Unsurprisingly, more GPÜs (larger batch size) results in better performance in general. Given the fact that fine-tuning have smaller gap between the result of using and 32 GPUs, it is worth noting that fine-tuning is more robust than training from scratch terms of speaker similarity.
Table 8: Additional results of English zero-shot speaker adaptation TTS experiment with different number of GPUs. 960 hours of labeled data is used.
cross-sentence reference
continuation WER
# GPUs
WER
SIM-o
SIM-r
SIM-o SIM-r
LoRA (Hu et al.
2021
fine-tuning
(detault)
2.6
0.696
0.711
2.4
0.623
0.640
2.5
0.695
0.710
2.3
0.623
0.640
2,4
0.698
0.713
2.2
0.623
0.639
8
2.3
0.697
0.712
2.2
0.623
0.639
16
2.2
0.697
0.712
2.2
0.625
0.641
32
2.1
0.700
0.715
2.1
0.630
0.644
Training from scratch
2.3
0.526
0.573
2.2
0.467
0.513
32
2.0
0.569
0.598
2.1
0.530
0.557
A.4.4 REDUCING LABELED DATA FOR FINE-TUNING
Interestingly, we found pre-trained model generalized better to unseen speaker comparing against models trained from scratch. However, also harder to overfit the pre-trainiend model on the limited amount of text imput, resulting a worse intelligibility in terms of WER. Nevertheless, with 10 hours of fine-tuning data, SpeechFlow was able to outperform VALL-E (Wang et al. 2023) that was trained 60k hours data.
Table Additional results of English zero-shot speaker adaptation TTS experiment using less beled data. Single GPU is used for fine-tuning the whole pre- pre-trained model.
labeled data (hr)
cross-sentence reference
continuation
Method
WER SIM-0
SIM-r
WER SIM-o SIM-r
Ground truth
2.2
0.754
YourTTS Casanova et
2021
475
7.7
0.337
n/a
VALL-E
Wang et al 2023
60k
5.9
0.580
3.8
0.452
0.508
Voicebox Le al.
1.9
0.662
0.681
2.0
0.593
0.616
960
2.3
0.526
0.573
2.2
0.467
0.513
SpeechFFlw wlo pre-train
0.412
2.2
0.417
2.4
0.360
0.410
2.3
0.330
0.374
960
2.2
0.678
0.694
2.2
0.613
0.630
SpecchFlow
100
2.8
0.613
0.632
2.5
0.555
0.573
10
4.1
0.578
0.600
3.1
0.520
0.541
18

---

Published as a conference paper at ICLR 2024
A.4.5
IMPACT OF PRE-TRAINING SYPER-PARAMETER
(1540
2.0
0.00
2.5
1880
543
0100
5000
D090)
0.58
1003
035
1350
0.660
233
Reluremenn (FESQ)
(ialleyy))
244
Ratasnmuct (PESQ)
1:55
0.25
(PE24))
240
0.500
1.90)
Pre-tesining stepp
learning
Prohability Pood
Figure 3: Impact of different pre-training peper-parameters on zero-shot speaker adaptation (ZSSA) TTS and enhancement. The dashed line stands for the baseline performance without pre- pre-training.
Since the the main focus of our method is on pre-training generative speech model, we provide study on the corresponding hyper-parameters here. To evaluate the pre-trained model in less biased perspective, we consider both speaker similarity of zero-shot speaker adaptation TTS and PESQ of enhancement for multi-tast fine-tuned model. Results are provided in Figure
Learning Rate & Number of Updates. First, we investigate the pre-trained model quality as function of the number pre-training steps or learning rate. We set the total number of updates to 750k, which is about 7 epochs on training set. One caveat is that learning rate decay is applied through out the training, which could also contribute the tapering result. We found most of the gain coming from the early stage before 400 updates and setting the learning rate above 5e-5 is sufficient for stable result.
Conditioning. We also found the model to be stable when setting Peond above 80%. Importantly, we also found unconditioned pre-training i.c., Poon = 0, yielded bad performance on both tasks. The result showcased the helpfulness and the necessity of masked prediction for pre-training. In summary, SpeechFlow is stable as long as masked conditioning is prioritized (over unconditioned pre-training) and the model is trained with sufficient steps and step size.
Masking pareaeearr Table To studied the impact of the proportion for placing mask Temask and the masking span size mmakk In simple terms, found masking a significant proportion is important for SpeechFlow.
Table 10: Additional results of English zero-shot speaker adaptation TTS experiment with different pre-training hyper-parameters. To reduce computation, models this table are only pre-trained for 300k steps.
cross-sentence reference SIM-o
continuation
WER
SIM-r
WER SIM-o
SIM-r
Thmask
(170% 100%]
1mask 10 (default)
2.2
0.655
0.669
0.596
0.610
Ammak
2800 100%
2.2
0.627
0.644
2.1
0,580
0.597
Tmmsk
26060 100%]
2.3
0.581
0.592
2.1
0.562
0.574
Pmuas
24 (60%, 90%]
2.2
0.599
0.612
2.1
0.567
0.577
Amaak 1170%, 90%]
2.1
0.614
0.623
0,589
0.596
Imack
5
2.0
0.661
0.677
2.1
0.600
0.616
Imaa
15
2.2
0.609
0.629
2.2
0.585
0.605
19

---

Published as a conference paper at ICLR 2024
A.4.6 SUBJECTIVE EVALUATION FOR ZERO-SHOT SPEAKER ADAPTATION TTS
Table 11: Subjective test English zero-shot speaker adaptation TTS filtered LS test-clean with cross-sentence reference. Averaged rating along with 95% confidence interval are reported for Mean Opinion Score (MOS).
labeled data (hr)
objective metrics WER SIM-o
subjective MOS
Method
SIM-r
Ground truth
3.80
YourTTS (Casanova et al. 2021
475
7.7
0.337
n/a
2.92.10.10
Voicebox (Le et 2023
60k
1.9
0.662
0.681
3.54
SpeechFlow
960
2.1
0.700
0.715
3.43++0.09
In addition to objective metrics that covers intelligibilityy and similarity measured models, conducted human evaluation to measure the overall quality of audio samples using Mean Opinion Score (MOS) following CrowdMOS (Ribeiro et al. 2011). We randomly selected 50 sentences from the LS test-clean for human evaluation. Each audio sample received 10 ratings in total. Each participant was asked to rate 20 audio samples, including different sentences with audio from different sources ground truth, YourTTS (Casanova et al. 2021). Voicebox (Le et al. 2023), and SpeechFlow. Results are collected through Amazon Mechanical Turk (AMT) with task description provided in Table 12 Annotators are filtered with the following qualifications: (1) They need to be wearing headset; (2) They need to pass an onboarding test (2 simple questions, where in each question people need to pick an audio with higher quality); (3) Post-processing, correlation coef between annotators' answer and the majority answer greater than 0.2.
From the MOS results in Table IT, we confirmed that SpeechFlow is able to generate high quality audio judging by human, falling only slightly behind its fully supervised counterpart Voicebox while using over 62.5x less labeled data.
Table 12: Mean opinion score (MOS) instruction.
Introduction
Hello! We need your help to evaluate the subjective quality and intelligibility of speech. In each task, you will evaluate 2-8s speech segment and rate its overall quality from I to 5. You will be given 20 questions, and it will take you 5-10 minutes to finish.
(1) Please use a headset for listening and adjust your volume level to your comfort during this training, and do not change later during the experiment.
(2) Please consider the following aspects when evaluating the overall quality: (a) clar- ity of speech (b) sound quality (c) naturalness.
For each of the speech audio below, rate the overall speech quality on scale from 1-5. (You need to play the speech audios in order to make selection!)
Score (Quality and Intelligibility of the speech)
5 (Excellent)
(Good)
3 (Fair)
2 (Poor)
1 (Bad)
20


Reviews:
[{"review_id": "1Uj14qVynT", "reviewer": "Reviewer_dpH8", "summary": "This paper proposes to use flow matching for speech generation. Experiments are conducted for speech enhancement, speech separation, and TTS.", "questions": "- Sec 4.1 -- what are. the 60k hours of English speech data for training?\n- Sec 4.3 -- can you give details on the 360 hours of training data?\n- Sec 4.4 -- what's the 960 hours of transcribed English speech used for fine-tuning?\n- Sec 4.4. -- the term \"zero-shot TTS\" is improper because the model is trained with fully supervised data. the reference is improper either as there are earlier and more established works not cited.\n- Sec 3.1 -- typo: \"variational audio encoders\" => \"variational autoencoders\"", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The idea of using flow matching for speech synthesis is sound.", "weaknesses": "1. The novelty is limited. It's basically applying flow matching to the speech synthesis problem.\n2. The evaluation of the experimental results are weak. For speech generation, subjective human evaluation are expected, especially for TTS. Without such evaluation, the results are not persuasive.\n3. There is a disconnection between the main claim and the experimental results. The experimental results show strong performance with using flow matching for speech synthesis in a pretraining and fine-tuning matter (finetuning was done with large datasets, e.g. 360 / 960 hours speech) . However, it's not clear that it's a results of \"a foundational model\". The experiments of SpeechFlow without pretraining is not persuasive because it uses the same model size, which likely leads to overfitting.\n4. The description of the experiments are severely limited. For example, what datasets were used, and the details on the model architecture and hyperparameteres. I have a major concern on reproductivity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ELOvJFqITK", "reviewer": "Reviewer_C8mS", "summary": "Speechflow is a generative model for speech generation and various tasks of it. It is trained with unlabeled speech with the goal of estimating the underlying distribution of speech conditioning on masked audio. Then it is fine-tuned for each specific task using labeled data.", "questions": "-What dataset are you using? for pre-training. You mention 60k hours of English speech.\n-Have you tried different mask instead of filling with zeros?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-Novel idea on making a general-purpose speech generation model that can perform the following tasks outperforming the current SOTA approaches: speech enhancement, speech separation, zero-shot tts.\n-I have listened to the audio samples and the model seems to perform well and produce high quality audio samples for all the tasks.\n-Novelty of modeling speech directly.", "weaknesses": "-No subjective evaluation is presented which could be useful for the users of this model. Most TTS works present both subjective and objective metrics for the evaluation.\n-The work is not a very good match for this venue. It would be more suitable in a speech-related venue like ICASSP or InternSpeech.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iani4Eafnd", "reviewer": "Reviewer_d1Li", "summary": "This paper proposes pre-training a flow-based speech synthesizer using 60kh of untranscribed speech, then fine-tuning it for downstream tasks including speech enhancement, speech separation, and zero-shot TTS.  In any self-supervised paradigm, one must find a way to add labels during the fine-tuning process; here, that problem is solved by using masked spectrograms as pseudo-labels during pre-training, then replacing those with actual labels (noisy speech, mixed speech, or phone sequences) during fine-tuning.  The idea of using masked spectrograms to condition flow was also used in the VoiceBox flow synthesizer, but that paper did not include a self-supervised pre-training stage.", "questions": "The paper should better describe the history of (1) the use of self-supervised pre-training for speech enhancement and speech synthesis, and (2) the use of generative flow in speech synthesis.  I recommend the following references, but I think there may be others that I'm missing:\n\nSelf-supervised training for TTS:  Yu-An Chung, Yuxuan Wang, Wei-Ning Hsu, Yu Zhang and RJ Skerry-Ryan, \"Semi-supervised training for improving data efficiency in end-to-end speech synthesis,\" ICASSP 2019, 6940-6944\n\nSelf-supervised training for speech enhancement: Yang, Shu-wen, Po-Han Chi, Yung-Sung Chuang, Cheng-I. Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu et al. \"Superb: Speech processing universal performance benchmark.\" arXiv preprint arXiv:2105.01051 (2021), and other papers that submitted entries to the Superb challenge.\n\nGenerative flow for speech synthesis:\n\nRyan Prenger, Rafael Valle and Bryan Catanzaro, \"Waveglow: A flow-based generative network for speech synthesis,\" ICASSP 2019, 3617-3621\n\nRafael Valle, Kevin Shih, Ryan Prenger and Bryan Catanzaro, \"Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis,\" arXiV 2020", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Generative pre-training for speech synthesis might have been first proposed in \"Semi-Supervised Training for Improving Data Efficiency in End-to-end Speech Synthesis\" by Chung et al., 2019.  Generative flow was used for vocoding in \"Waveglow: A flow-based generative network for speech synthesis,\" and was used for TTS in \"Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis\" --- neither of those papers used the flow matching paradigm, instead they trained the flow networks using end-to-end training criteria only.  The combination of these two ideas (flow-based TTS and pre-trained TTS) was not proposed in any paper I can find.  The new contribution of this manuscript, the use of generative flow in a self-supervised pre-training stage, is an elegant idea that forms a strong theoretical paradigm, and that is supported by strong experimental results compared to challenging baselines.", "weaknesses": "By omitting key references, this paper seems to be suggesting that nobody has ever thought of using self-supervised pre-training for speech synthesis before, and it seems to be suggesting that nobody has ever used generative flow for speech synthesis before.  The manuscript should include references to key works in both areas, in order to more clearly articulate what is the actual contribution of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q3ZlKwokiF", "reviewer": "Reviewer_STf2", "summary": "The paper proposes to pretrain a flow-based model with unsupervised pre-training and supervised fine-tuning paradigms. The pre-trained generative model can be fine-tuned with task-specific data for speech enhancement, separation, and synthesis. According to the results across several benchmarks, the proposed models match or surpass existing expert models.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper explores a novel direction to pre-train a general-purpose generative model with unlabeled speech using flow-based models. The most similar work is Voicebox, a flow-based model with supervised slot-filling training, and the authors conduct details discussion and experimental comparisons to show the advantages. The pre-trained model can be finetuned to support various tasks such as speech enhancement, separation, and synthesis. The experiments are convincing.", "weaknesses": "One primary limitation of this work is the relatively limited range of supported task types. It would be beneficial for the authors to expand their support to include a wider variety of tasks, such as speech editing tasks, to further demonstrate the capabilities of their pre-trained models. By incorporating additional task types, the authors can provide a more comprehensive evaluation of the model's abilities and showcase its versatility across various domains. This would enhance the overall contribution and applicability of the proposed pre-training approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)