FlashSpeech: Efficient Zero-Shot Speech Synthesis

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-09 07:37:32

Abstract: Recent progress in large-scale zero-shot speech synthesis has been significantly advanced by language models and diffusion models.  However, the generation process of both methods is slow and computationally intensive. Efficient speech synthesis using a lower computing budget to achieve quality on par with previous work remains a significant challenge. In this paper, we present FlashSpeech, a large-scale zero-shot speech synthesis system with approximately 5\% of the inference time compared with previous work. FlashSpeech is built on the latent consistency model and applies a novel adversarial consistency training approach that can train from scratch without the need for a pre-trained diffusion model as the teacher. Furthermore,  a new prosody generator module enhances the diversity of prosody, making the rhythm of the speech sound more natural.
The generation processes of FlashSpeech can be achieved efficiently with one or two sampling steps while maintaining high audio quality and high similarity to the audio prompt for zero-shot speech generation.
Our experimental results demonstrate the superior performance of FlashSpeech. Notably, FlashSpeech can be about 20 times faster than other zero-shot speech synthesis systems while maintaining comparable performance in terms of voice quality and similarity. Furthermore, FlashSpeech demonstrates its versatility by efficiently performing tasks like voice conversion, speech editing, and diverse speech sampling. Audio samples can be found in  https://flashspeech.github.io

Main paper: 
 introduction: 
 In recent years, the landscape of speech synthesis has been transformed by the advent of large-scale generative models. Consequently, the latest research efforts have achieved notable advancements in zero-shot speech synthesis systems by significantly increasing the size of both datasets and models. Zero-shot speech synthesis, such as text-to-speech (TTS), voice conversion (VC) and Editing, aims to generate speech that incorporates unseen speaker characteristics from a reference audio segment during inference, without the need for additional training. Current advanced zero-shot speech synthesis systems typically leverage language models (LMs) [20, 22, 39, 59, 61, 63, 66] and diffusion-style models [18, 23, 26, 52] for in-context speech generation on the large-scale dataset. However, the generation process of these methods needs a long-time iteration. For example, VALL-E [59] builds on the language model to predict 75 audio token sequences for a 1-second speech, in its first-stage autoregressive (AR) token sequence generation. When using a non-autoregressive (NAR) latent diffusion model [47] based framework, NaturalSpeech 2 [52] still requires 150 sampling steps. As a result, although these methods can produce human-like speech, they require significant computational time and cost. Some efforts have been made to accelerate the generation process. Voicebox [26] adopts flow-matching [29] so that fewer sampling steps (NFE1 : 64) can be achieved because of the optimal transport path. ClaM-TTS [22] proposes a mel-codec with a superior compression rate and a latent language model that generates a stack of tokens at once. Although the slow generation speed issue has been somewhat alleviated, the inference speed is still far from satisfactory for practical applications. Moreover, the substantial computational time of these approaches leads to significant computational cost overheads, presenting another challenge.The fundamental limitation of speech generation stems from the intrinsic mechanisms of language models and diffusion models, which require considerable time either auto-regressively or through a large number of denoising steps. Hence, the primary objective of this work is to accelerate inference speed and reduce computational costs while preserving generation quality at levels comparable to the prior research. In this paper, we propose FlashSpeech as the next step towards efficient zero-shot speech synthesis. To address the challenge of slow generation speed, we leverage the latent consistency model (LCM) [32] , a recent advancement in generative models. Building upon the previous non-autoregressive TTS system [52] , we adopt the encoder of a neural audio codec to convert speech waveforms into latent vectors as the training target for our LCM. To train this model, we propose a novel technique called adversarial consistency training, which utilizes the capabilities of pre-trained speech language models [1, 6, 11] as discriminators. This facilitates the transfer of knowledge from large pre-trained speech language models to speech generation tasks, efficiently integrating adversarial and consistency training to improve performance. The LCM is conditioned on prior vectors obtained from a phoneme encoder, a prompt encoder, and a prosody generator. Furthermore, we demonstrate that our proposed prosody generator leads to more diverse expressions and prosody while preserving stability.Our contributions can be summarized as follows:• We propose FlashSpeech, an efficient zero-shot speech synthesis system that generates voice with high audio quality and speaker similarity in zero-shot scenarios.  
related work 2.1 large scale speech synthesis: 
 Motivated by the success of the large language model, the speech research community has recently shown increasing interest in scaling the sizes of model and training data to bolster generalization capabilities, producing natural speech with diverse speaker identities and prosody under zero-shot settings. The pioneering work is VALL-E [59] , which adopts the Encodec [8] to discretize the audio waveform into tokens. Therefore, a language model can be trained via in-context learning that can generate the target utterance where the style is consistent with prompt utterance. However, generating audio in such an autoregressive manner [39, 61] can lead to unstable prosody, word skipping, and repeating issues [44, 52, 57] . To ensure the robustness of the system, non-autoregressive methods such as NaturalSpeech2 [52] and Voicebox [26] utilize diffusion-style model (VP-diffusion [55] or flow-matching [29] ) to learn the distribution of a continuous intermediate vector such as mel-spectrogram or latent vector of codec. Both LM-based methods [67] and diffusion-based methods show superior performance in speech generation tasks. However, their generation is slow due to the iterative computation.Considering that many speech generation scenarios require realtime inference and low computational costs, we employ the latent consistency model for large-scale speech generation that inference with one or two steps while maintaining high audio quality.Since early neural speech generation models [57] use autoregressive models such as Tacotron [60] and TransformerTTS [27] , causing slow inference speed, with O (𝑁 ) computation, where 𝑁 is the sequence length. To address the slow inference speed, Fast-Speech [44, 45] proposes to generate a mel-spectrogram in a nonautoregressive manner. However, these models [46] result in blurred and over-smoothed mel-spectrograms due to the regression loss they used and the capability of modeling methods. To further enhance the speech quality, diffusion models are utilized [13, 40, 41] which increase the computation to O (𝑇 ), where T is the diffusion steps. Therefore, distillation techniques [33] for diffusion-based methods such as CoMoSpeech [64] , CoMoSVC [31] and Reflow-TTS [10] The consistency model is proposed in [53, 54] to generate highquality samples by directly mapping noise to data. Furthermore, many variants [21, 25, 30, 51] are proposed to further increase the generation quality of images. The latent consistency model is proposed by [32] which can directly predict the solution of PF-ODE in latent space. However, the original LCM employs consistency distillation on the pre-trained latent diffusion model (LDM) which leverages large-scale off-the-shelf image diffusion models [47] . Since there are no pre-trained large-scale TTS models in the speech community, and inspired by the techniques [21, 25, 30, 51, 53] , we propose the novel adversarial consistency training method which can directly train the large-scale latent consistency model from scratch utilizing the large pre-trained speech language model [1, 6, 11] such as WavLM for speech generation. 
flashspeech 3.1 overview: 
 Our work is dedicated to advancing the speech synthesis efficiency, achieving O (1) computation cost while maintaining comparable performance to prior studies that require O (𝑇 ) or O (𝑁 ) computations. The framework of the proposed method, FlashSpeech, is illustrated in Fig. 2 . FlashSpeech integrates a neural codec, an encoder for phonemes and prompts, a prosody generator, and an LCM, which are utilized during both the training and inference stages. Exclusively during training, a conditional discriminator is employed. FlashSpeech adopts the in-context learning paradigm [59] , initially segmenting the latent vector z, extracted from the codec, into z 𝑡𝑎𝑟𝑔𝑒𝑡 and z 𝑝𝑟𝑜𝑚𝑝𝑡 . Subsequently, the phoneme and z 𝑝𝑟𝑜𝑚𝑝𝑡 are processed through the encoder to produce the hidden feature. A prosody generator then predicts pitch and duration based on the hidden feature. The pitch and duration embeddings are combined with the hidden feature and inputted into the LCM as the conditional feature. The LCM model is trained from scratch using adversarial consistency training. After training, FlashSpeech can achieve efficient generation within one or two sampling steps.The consistency model [54] is a new family of generative models that enables one-step or few-step generation. Let us denote the data distribution by 𝑝 data (x). The core idea of the consistency model is to learn the function that maps any points on a trajectory of the PF-ODE to that trajectory's origin, which can be formulated as:EQUATIONwhere 𝑓 (•, •) is the consistency function and x 𝜎 represents the data x perturbed by adding zero-mean Gaussian noise with standard deviation 𝜎. 𝜎 𝑚𝑖𝑛 is a fixed small positive number. Then x 𝜎 𝑚𝑖𝑛 can then be viewed as an approximate sample from the data distribution 𝑝 data (x). To satisfy property in equation ( 1), following [54] , we parameterize the consistency model asEQUATIONwhere 𝑓 𝜃 is to estimate consistency function 𝑓 by learning from data, 𝐹 𝜃 is a deep neural network with parameter 𝜃 , 𝑐 skip (𝜎) and 𝑐 out (𝜎) are are differentiable functions with 𝑐 skip (𝜎 𝑚𝑖𝑛 ) = 1 and 𝑐 out (𝜎 𝑚𝑖𝑛 ) = 0 to ensure boundary condition. A valid consistency model should satisfy the self-consistency property [54] 𝑓EQUATIONwhere 𝜎 𝑚𝑎𝑥 = 80 and 𝜎 𝑚𝑖𝑛 = 0.002 following [19, 53, 54] . Then the model can generate samples in one step by evaluatingEQUATIONfrom distribution x 𝜎 𝑚𝑎𝑥 ∼ N (0, 𝜎 2 𝑚𝑎𝑥 I). As we apply a consistency model on the latent space of audio, we use the latent features 𝑧 which are extracted prior to the residual quantization layer of the codec,EQUATION)where y is the speech waveform. Furthermore, we add the feature from the prosody generator and encoder as the conditional feature 𝑐, our objective has changed to achieveEQUATIONDuring inference, the synthesized waveform ŷ is transformed from ẑ via the codec decoder. The predicted ẑ is obtained by one samplingEQUATIONor two sampling stepsEQUATIONwhere ẑinter means the intermediate step, 𝜎 inter is set to 2 empirically. 𝜖 is sampled from a standard Gaussian distribution.A major drawback of the LCM [32] is that it needs to pre-train a diffusion-based teacher model in the first stage, and then perform distillation to produce the final model. This would make the training process complicated, and the performance would be limited as a result of the distillation. To eliminate the reliance on the teacher EQUATIONwhere 𝜎 𝑖 represents the noise level at discrete time step 𝑖, 𝑑 (•, •) is the distance function, 𝑓 𝜃 (z 𝑖+1 , 𝜎 𝑖+1 , 𝑐) and 𝑓 𝜃 -(z 𝑖 , 𝜎 𝑖 , 𝑐) are the student with the higher noise level and the teacher with the lower noise level, respectively. The discrete time steps denoted as𝜎 𝑚𝑖𝑛 = 𝜎 0 < 𝜎 1 < • • • < 𝜎 𝑁 = 𝜎 𝑚𝑎𝑥 are divided from the time inter- val [𝜎 𝑚𝑖𝑛 , 𝜎 𝑚𝑎𝑥 ],where the discretization curriculum 𝑁 increases correspondingly as the number of training steps growsEQUATION)where 𝐾 ′ = 𝐾 log 2 ⌊𝑠 1 /𝑠 0 ⌋+1 , 𝑘 is the current training step and 𝐾 is the total training steps. 𝑠 1 and 𝑠 0 are hyperparameters to control the size of 𝑁 (𝑘). The distance function 𝑑 (•, •) uses the Pseudo-Huber metric [3] EQUATIONwhere 𝑎 is an adjustable constant, making the training more robust to outliers as it imposes a smaller penalty for large errors than ℓ 2 loss. The parameters 𝜃 -of teacher model areEQUATIONwhich are identical to the student parameters 𝜃 . This approach [53] has been demonstrated to improve sample quality of previous strategies that employ varying decay rates [54] . The weighting function refers to EQUATIONEQUATIONIn this way, the error signal from the discriminator guides 𝑓 𝜃 to produce more realistic outputs. For details, we use a frozen pretrained speech language model 𝑆𝐿𝑀 and a trainable lightweight discriminator head 𝐷 ℎ𝑒𝑎𝑑 to build the discriminator. Since the current 𝑆𝐿𝑀 is trained on the speech waveform, we covert both z and ẑ to ground truth waveform and predicted waveform using the codec decoder. To further increase the similarity between prompt audio and generated audio, our discriminator is conditioned on the prompt audio feature. This prompt feature 𝐹 prompt is extracted using 𝑆𝐿𝑀 on prompt audio and applies average pooling on the time axis. Therefore,EQUATIONwhere 𝐹 gt and 𝐹 pred refer to feature extracted through 𝑆𝐿𝑀 for ground truth waveform and predicted waveform. The discriminator head consists of several 1D convolution layers. The input feature of the discriminator is conditioned on 𝐹 prompt via projection [36] .Since there is a large gap on the loss scale between consistency loss and adversarial loss, it can lead to instability and failure in training. Therefore, we follow [9] to compute the adaptive weight withEQUATION)where 𝜃 𝐿 is the last layer of the neural network in LCM. The final loss of training LCM is defined as L 𝑁 ct (𝜃, 𝜃 -) +𝜆 𝑎𝑑𝑣 L adv (𝜃, 𝜂). This adaptive weighting significantly stabilizes the training by balancing the gradient scale of each term.3.4.1 Analysis of the prediction of prosody. Previous regression methods for prosody prediction [44, 52] , due to their deterministic mappings and assumptions of unimodal distribution, often fail to capture the inherent diversity and expressiveness of human speech prosody. This leads to predictions that lack variation and can appear over-smoothed. On the other hand, diffusion methods [26, 28] for prosody prediction offer a promising alternative by providing greater prosody diversity. However, they come with challenges regarding stability, and the potential for unnatural prosody. Additionally, the iterative inference process in DMs requires a significant number of sampling steps that may also hinder real-time application. Meanwhile, LM-based methods [15, 59] also need a long time for inference. To alleviate these issues, our prosody generator consists of a prosody regression module and a prosody refinement module to enhance the diversity of prosody regression results with efficient one-step consistency model sampling.3.4.2 Prosody refinement via consistency model. As shown in 4, our prosody generator consists of two parts which are prosody regression and prosody refinement. We first train the prosody regression module to get a deterministic output. Next, we freeze the parameters of the prosody regression module and use the residual of ground truth prosody and deterministic predicted prosody as the training target for prosody refinement. We adopt a consistency model as a prosody refinement module. The conditional feature of the consistency model is the feature from prosody regression before the final projection layer. Thus, the residual from a stochastic sampler refines the output of a deterministic prosody regression and produces a diverse set of plausible prosody under the same transcription and audio prompt. One option for the final prosody output 𝑝 final can be represented as:EQUATIONwhere 𝑝 final denotes the final prosody output, 𝑝 res represents the residual output from the prosody refinement module, capturing the variations between the ground truth prosody and the deterministic prediction, 𝑝 init is the initial deterministic prosody prediction from the prosody regression module. However, this formulation may negatively affect prosody stability, a similar observation is found in [26, 58] . More specifically, higher diversity may cause less stability and sometimes produce unnatural prosody. To address this, we introduce a control factor 𝛼 that finely tunes the balance between stability and diversity in the prosodic output:EQUATION)where 𝛼 is a scalar value ranging between 0 and 1. This adjustment allows for controlled incorporation of variability into the prosody, mitigating issues related to stability while still benefiting from the diversity offered by the prosody refinement module.This section elaborates on the practical applications of FlashSpeech. We delve into its deployment across various tasks such as zero-shot TTS, speech editing, voice conversion, and diverse speech sampling. All the sample audios of applications are available on the demo page.3.5.1 Zero-Shot TTS. Given a target text and reference audio, we first convert the text to phoneme using g2p (grapheme-to-phoneme conversion). Then we use the codec encoder to convert the reference audio into z 𝑝𝑟𝑜𝑚𝑝𝑡 . Speech can be synthesized efficiently through FlashSpeech with the phoneme input and z 𝑝𝑟𝑜𝑚𝑝𝑡 , achieving highquality text-to-speech results without requiring pre-training on the specific voice.Voice conversion aims to convert the source audio into the target audio using the speaker's voice of the reference audio. Following [43, 52] , we first apply the reverse of ODE to diffuse the source audio into a starting point that still maintains some information in the source audio. After that, we run the sampling process from this starting point with the reference audio as z 𝑝𝑟𝑜𝑚𝑝𝑡 and condition 𝑐. The condition 𝑐 uses the phoneme and duration from the source audio and the pitch is predicted by the prosody generator. This method allows for zero-shot voice conversion while preserving the linguistic content of the source audio, and achieving the same timbre as the reference audio.Given the speech, the original transcription, and the new transcription, we first use MFA (Montreal Forced Aligner) to align the speech and the original transcription to get the duration of each word. Then we remove the part that needs to be edited to construct the reference audio. Next, we use the new transcription and reference to synthesize new speech. Since this task is consistent with the in-context learning, we can concatenate the remaining part of the raw speech and the synthesized part as the final speech, thus enabling precise and seamless speech editing.3.5.4 Diverse Speech Sampling. FlashSpeech leverages its inherent stochasticity to generate a variety of speech outputs under the same conditions. By employing stochastic sampling in its prosody generation and LCM, FlashSpeech can produce diverse variations in pitch, duration, and overall audio characteristics from the same phoneme input and audio prompt. This feature is particularly useful for generating a wide range of speech expressions and styles from a single input, enhancing applications like voice acting, synthetic voice variation for virtual assistants, and more personalized speech synthesis. In addition, the synthetic data via speech sampling can also benefit other tasks such as ASR [48] . 
experiment: 
 In the experimental section, we begin by introducing the datasets and the configurations for training in our experiments. Following this, we show the evaluation metrics and demonstrate the comparative results against various zero-shot TTS models. Subsequently, ablation studies are conducted to test the effectiveness of several design choices. Finally, we also validate the effectiveness of other tasks such as voice conversion. We show our speech editing and diverse speech sampling results on our demo page.4.1.1 Data and Preprocessing. We use the English subset of Multilingual LibriSpeech (MLS) [42] , including 44.5k hours of transcribed audiobook data and it contains 5490 distinct speakers. The audio data is resampled at a frequency of 16kHz. The input text is transformed into a sequence of phonemes through grapheme-tophoneme conversion [56] and then we use our internal alignment tool aligned with speech to obtain the phoneme-level duration. We adopt a hop size of 200 for all frame-level features. The pitch sequence is extracted using PyWorld2 . we adopt Encodec [8] as our audio codec. We use a modified version 3 and train it on MLS. We Table 1 : The evaluation results for FlashSpeech and the baseline methods on LibriSpeech testclean. ★ means the evaluation is conducted with 1 NVIDIA V100 GPU. ♦ means the device is not available. Abbreviation: MLS (Multilingual LibriSpeech [42] ),G (GigaSpeech [4] ), L(LibriTTS-R [24] ), V(VCTK [62] ), LJ(LJSpeech [12] ), W(WenetSpeech [65] ). use the dense features extracted before the residual quantization layer as our latent vector 𝑧.↓ Sim-O ↑ Sim-R ↑ WER ↓ CMOS ↑ SMOS (↑ ) GroundTruth - - 0.Our training consists of two stages, in the first stage we train LCM and the prosody regression part. We use 8 H800 80GB GPUs with a batch size of 20k frames of latent vectors per GPU for 650k steps. We use the AdamW optimizer with a learning rate of 3e-4, warm up the learning rate for the first 30k updates and then linear decay it. We deactivate adversarial training with 𝜆 𝑎𝑑𝑣 = 0 before 600K training iterations. For hyper-parameters, we set 𝑎 in Equation ( 12) to 0.03. In equation (10) ,𝜎 𝑖 = 𝜎 1/𝜌 min + 𝑖 -1 𝑁 (𝑘 ) -1 𝜎 1/𝜌 max -𝜎 1/𝜌 min, where 𝑖 ∈ [1, 𝑁 (𝑘)], 𝜌 = 7, 𝜎 min = 0.002, 𝜎 max = 80. For N(k) in Equation (11) , we set 𝑠 0 = 10, 𝑠 1 = 1280, 𝐾 = 600𝑘. After 600k steps, we activate adversarial loss, and N(k) can be considered as fixed to 1280. We crop the waveform length fed into the discriminator into minimum waveform length in a minibatch. In addition, the weight of the feature extractor WavLM and the codec decoder are frozen.In the second stage, we train 150k steps for the prosody refinement module with consistency training in Equation (10) . Different from the above setting, we empirically set 𝑠 1 = 160, 𝐾 = 150𝑘.During training, only the weight of the prosody refinement part is updated.The model structures of the prompt encoder and phoneme encoder are follow [52] . The neural function part in LCM is almost the same as the [52] . We rescale the sinusoidal position embedding in the neural function part by a factor of 1000. As for the prosody generator, we adopt 30 non-casual wavenet [37] layers for the neural function part in the prosody refinement module and the same configurations for prosody regression parts in [52] . And we set 𝛼 = 0.2 for the prosody refinement module empirically. For the discriminator's head, we stack 5 convolutional layers with weight normalization [50] for binary classification.We use both objective and subjective evaluation metrics, including • RTF: Real-time-factor (RTF) measures the time taken for the system to generate one second of speech. This metric is crucial for evaluating the efficiency of our system, particularly for applications requiring real-time processing. We measure the time of our system end-to-end on an NVIDIA V100 GPU following [52] . • Sim-O and Sim-R: These metrics assess the speaker similarity. Sim-R measures the objective similarity between the synthesized speech and the reconstruction reference speech through the audio codec, using features embedding extracted from the pre-trained speaker verification model [22, 59] 4 . Sim-O is calculated with the original reference speech. Higher scores in Sim-O and Sim-R indicate a higher speaker similarity. • WER (Word Error Rate): To evaluate the accuracy and clarity of synthesized speech from the TTS system, we employ the Automatic Speech Recognition (ASR) model [59] 5 to transcribe generated audio. The discrepancies between these transcriptions and original texts are quantified using the Word Error Rate (WER), a crucial metric indicating intelligibility and robustness. • CMOS, SMOS, UTMOS: we rank the comparative mean option score (CMOS) and similarity mean option score (SMOS) using mturk. The prompt for CMOS refers to 'Please focus on the audio quality and naturalness and ignore other factors.'. The prompt for SMOS refers to 'Please focus on the similarity of the speaker to the reference, and ignore the differences of content, grammar or audio quality. ' Each audio has been listened to by at least 10 listeners. UTMOS [49] is a Speech MOS predictor 6 to measure the naturalness of speech. We use it in ablation studies which reduced the cost for evaluation. • Prosody JS Divergence: To evaluate the diversity and accuracy of the prosody prediction in our TTS system, we include the Prosody JS Divergence metric. This metric employs the Jensen-Shannon (JS) divergence [35] to quantify the divergence between the predicted and ground truth prosody feature distributions. Prosody features, including pitch, and duration, are quantized and their distributions in both synthesized and natural speech are compared. Lower JS divergence values indicate closer similarity between the predicted prosody features and those of the ground truth, suggesting a higher diversity of the synthesized speech.Following [59] , We employ LibriSpeech [38] test-clean for zero-shot TTS evaluation. We adopt the cross-sentence setting in [59] that we randomly select 3-second clips as prompts from the same speaker's speech. The results are summarized in table 1 and figure 5.• VALL-E [59] : VALL-E predicts codec tokens using both AR and NAR models. RTF7 is obtained from [22, 26] . We use our reproduced results for MOS, Sim, and WER. Additionally, we do a preference test with their official demo.• Voicebox [26] : Voicebox uses flow-matching to predict maksed mel-spectrogram. RTF is from the original paper. We use our reproduced results for MOS, Sim, and WER. We also implement a preference test with their official demo. • NaturalSpeech2 [52] : NaturalSpeech2 uses a latent diffusion model to predict latent features of codec. The RTF is from the original paper. the Sim, WER and samples for MOS are obtained through communication with the authors. We also do a preference test with their official demo. • Mega-TTS [17] 8 : Mega-TTS uses both language model and GAN to predict mel-spectrogram. We obtain RTF from mobilespeech [14] and WER from the original paper. We do a preference test with their official demo. • ClaM-TTS [22] : ClaM-TTS uses the AR model to predict mel codec tokens. We obtain the objective evaluation results from the original paper and do a preference test with their official demo. As shown in table 3 , the effect of sampling steps (NFE) on UTMOS and Sim-O revealed that increasing NFE from 1 to 2 marginally improves UTMOS (3.99 to 4.00) and Sim-O (0.51 to 0.52). However, further increasing to 4 sampling steps slightly reduced UTMOS to 3.91 due to the accumulation of score estimation errors [5, 34] . Therefore, we use 2 steps as the default setting for LCM.In this part, we investigated the effects of a control factor, denoted as 𝛼, on the prosodic features of pitch and duration in speech synthesis, by setting another influencing factor to zero. Our study specifically conducted an ablation analysis to assess how 𝛼 influences these features, emphasizing its critical role in balancing stability and diversity within our framework's prosodic outputs.Table 4 elucidates the effects of varying 𝛼 on the pitch component. With 𝛼 set to 0, indicating no inclusion of the residual output from prosody refinement, we observed a Pitch JSD of 0.072 and a WER of 2.8. A slight modification to 𝛼 = 0.2 resulted in a reduced Pitch JSD of 0.067, maintaining the same WER. Notably, setting 𝛼 to 1, fully incorporating the prosody refinement's residual output, further decreased the Pitch JSD to 0.063, albeit at the cost of increased WER to 3.7, suggesting a trade-off between prosody diversity and speech intelligibility.Similar trends in table 5 are observed in the duration component analysis. With 𝛼 = 0, the Duration JSD was 0.0175 with a WER of 2.8. Adjusting 𝛼 to 0.2 slightly improved the Duration JSD to 0.0168, without affecting WER. However, fully embracing the refinement module's output by setting 𝛼 = 1 yielded the most significant improvement in Duration JSD to 0.0153, which, similar to pitch analysis, came with an increased WER of 3.9. The results underline the delicate balance required in tuning 𝛼 to optimize between In this section, we present the evaluation results of our voice conversion system, FlashSpeech, in comparison with state-of-the-art methods, including YourTTS12 [2] and DDDM-VC13 [7] . We conduct the experiments with their official checkpoints in our internal test set.Our system outperforms both YourTTS and DDDM-VC in terms of CMOS, SMOS and Sim-O, demonstrating its capability to produce converted voices with high quality and similarity to the target speaker. These results confirm the effectiveness of our FlashSpeech approach in voice conversion tasks.In this paper, we presented FlashSpeech, a novel speech generation system that significantly reduces computational costs while maintaining high-quality speech output. Utilizing a novel adversarial consistency training method and an LCM, FlashSpeech outperforms existing zero-shot TTS systems in efficiency, achieving speeds about 20 times faster without compromising on voice quality, similarity, and robustness. In the future, we aim to further refine the model to improve the inference speed and reduce computational demands. In addition, we will expand the data scale and enhance the system's ability to convey a broader range of emotions and more nuanced prosody. For future applications, FlashSpeech can be integrated for real-time interactions in applications such as virtual assistants and educational tools.NFE: number of function evaluations.https://github.com/JeremyCCHsu/Python-Wrapper-for-World-Vocoderhttps://github.com/yangdongchao/UniAudio/tree/main/codechttps://github.com/microsoft/UniSpeech/tree/main/downstreams/speaker_verificationhttps://huggingface.co/facebook/hubert-large-ls960-fthttps://github.com/tarepan/SpeechMOSIn CLaM-TTS and Voicebox, they report the inference time for generating 10 seconds of speech. Therefore, we divide by 10 to obtain the time for generating 1 second of speech (RTF).Since we do not find any audio samples for Mega-TTS2[16] under the 3-second cross-sentence setting, we are not able to compare with them.https://huggingface.co/facebook/wav2vec2-largehttps://huggingface.co/facebook/hubert-large-ll60khttps://huggingface.co/microsoft/wavlm-largehttps://github.com/coqui-ai/TTShttps://github.com/hayeong0/DDDM-VC 
 
 Appendix:  


Decision: This paper introduces FlashSpeech, a large-scale zero-shot speech synthesis system with efficient inference and high quality outputs. It is  built on the latent consistency model and uses a new prosody generator module to generate more natural sounding outputs. It has been shown to be effective for a broad range of tasks such as voice conversion, speech editing, and diverse speech sampling.

Strengths:
- The model is quite diverse and does well on a multitude of tasks which is quite impressive.
- The linked audio samples are of great quality and a testament of this work.
- Very well-written paper with a good amount of focus on experimental results. 
- The choice of model components has been explained well and seems to resonate with most reviewers.
- Open sourcing the codebase would be a great way of advancing research in this area of work.

Weaknesses:
- It has been noted in multiple reviews that there isn't much innovation in modeling itself.
- Comparison with similar, recent TTS systems is missing and would be a valuable addition to the discussions in the paper.

Reviews: review: paper_summary: paper_summary: FlashSpeech introduces latent consistency model and adversarial consistency training that significantly reduces inference time, achieving 20 fold speed increase while maintaining high speech quality and speaker similarity. The paper presents a significant advancement in zero-shot speech synthesis.  It supports diverse tasks like voice conversion, speech editing, and diverse speech sampling.
strength_weakness: strength_weakness: (1) The proposed architecture achieves up to 20 times faster inference than previous models, leveraging a non-autoregressive framework, Latent Consistency Model, and adversarial consistency training. This efficiency is complemented by high-quality speech synthesis, espicailly in zero-shot speech synthesis senarios.

(2) High-quality audio output and strong speaker similarity in zero-shot scenarios.

(3)  Broad applicability of the proposed architecuture across various tasks such as TTS, voice conversion, speech editing etc. highlights the model's robustness and potential for real-world deployment in multiple speech synthesis scenarios.

(4) The paper provides extensive experimental results that clearly demonstrate advantages over existing methods.
limitations: limitations: (1) Limited novelty in the architecture.FlashSpeech replicates the core architecture of NS2 but integrates a consistency model to enhance performance.

(2) The paper does not include an evaluation against StyleTTS2 [1], a method known for achieving comparable quality with significantly less data. Including StyleTTS2 in the evaluation would also help assess the generalizability of FlashSpeech.

(3) To improve the reader's understanding, Figure 2 in the paper should be updated to clearly distinguish between trainable and non-trainable parameters.

(4) The Statistical Significance analysis for the subjective evaluation is missing. To ensure the reliability of the subjective evaluation, it is crucial to perform a statistical significance analysis to demonstrate that whether the observed differences results between FlashSpeech and other baselines are statistically significant.


[1] Li, Yinghao Aaron, et al. "Styletts 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models." Advances in Neural Information Processing Systems 36 (2024).

Minor Comments: 

-How the model will perform with noisy or highly variable real-world data? Are there any robustness measures to ensure consistent performance across different acoustic environments and speaker variations?

-Some discussion on how prosody generator balance the contributions of deterministic and stochastic components in the prosody refinement process is required.

Missing Citation:  The previous uses of the Latent Consistency Model (LCM) in Text-to-Speech (TTS) systems might be relevant.
Li, Xiang, et al. "CM-TTS: Enhancing Real Time Text-to-Speech Synthesis Efficiency through Weighted Samplers and Consistency Models." In Findings of NAACL, 2024.
paper_summary: paper_summary: This paper introduces FlashSpeech, a new text-to-speech model that delivers both low latency and high quality, as well as speaker similarity. FlashSpeech, based on the text-to-speech paradigm with duration modeling, utilizes a latent consistency model. This model helps in speech generation based on textual and prosodic features, enabling efficient few-step sampling. With the addition of an adversarial training module, the quality of the generation is improved.

FlashSpeech incorporates a prosody generator augmented with a latent consistency model. This addition of diverse residual signals onto the regressed one increases the diversity of the predicted prosody.

Comprehensive experiments were conducted by the authors to demonstrate the effectiveness of FlashSpeech.
strength_weakness: strength_weakness: - This paper introduces a novel text-to-speech model designed to improve sampling efficiency, while maintaining quality and speaker similarity. It effectively merges a latent consistency model with an adversarial training module to ensure both efficiency and quality. The latent consistency model greatly reduces the sampling steps, thus greatly increase the generation efficiency. The adversarial utilizing speech language model helps improve the generation quality.
- Additionally, the paper presents an innovative method for predicting prosody. This approach uses a regression model for the base and a latent consistency model for residual information, effectively capturing the diversity of prosody. A two-stage training with the first stage training the LCM and the prosody regression part and the second stage training the prosody refinement module is adopted.
- The proposed FlashSpeech proves its ability to perform voice cloning, voice conversion, speech editing, and diverse speech sampling. While the demonstrations for speech editing and diverse speech sampling are somewhat less impressive, the model's versatility is unquestionable.
- Extensive experiments are performed to assess the proposed methods and designs, including a variety of baseline methods for comparison. The effectiveness of the adversarial training module and prosody generator is clearly demonstrated through ablation studies.
- The paper is well-structured and clearly written. It introduces the background with appropriate detail and provides comprehensive explanations of key components, making it an engaging read.
limitations: limitations: - For demo of diverse speech sampling, the samples do not significantly differ in terms of pitch and duration. Specifically, one sample (the second text and sample 2) seems to have an incorrect syllable insertion (sounding like "ah e") before the word "fresh". Providing more details about the settings used to create these samples would give your readers a better understanding of the diversity you are aiming to achieve.
- While FlashSpeech's ability to facilitate speech editing is impressive, the current method appears to synthesize the new text and insert it into the editing segment. This hard concatenation results in unnatural transitions, which are noticeable in the samples from the Speech Editing section on your demo page. Perhaps exploring smoother transition techniques could enhance the naturalness of the edited speech.
- Despite using JSD for both pitch and duration to indicate their similarity to the ground-truth distribution and their diversity, it's hard to determine the accuracy of these two predicted features based solely on the JSD. Therefore, including objective metrics on pitch and duration for reference would be beneficial.
paper_summary: paper_summary: This paper presents FlashSpeech, a novel speech generation system, based on the latent consistency model (LCM). Their proposed adversarial consistency training enables them to train a model from scratch without distillation. FlashSpeech achieves speeds about 20 times faster than existing TTS models, without compromising voice quality, similarity, and robustness.
strength_weakness: strength_weakness: - Well-designed architecture equipped with the latest technologies of deep generative modeling [53,54,32]. Although this model is made by combining existing methods, I believe publishing this paper and model will be beneficial for the community.
- Good experimental results. Competitive performance in quality with SOTA TTS models despite much faster synthesis. In addition, the proposed model is better in a use preference test. Ablation studies are also enough to convince readers.
- Well-written manuscript. I can easily understand their motivation, some details of their training framework, and experimental settings.
limitations: limitations: - Novelty is a little limited. However, as I commented in the previous field, publishing this paper and model will be beneficial for the community. So, I don't think this is a big drawback.
- I wanted to hear synthesized speech.
- Even though the manuscript is well written, it might be hard for some readers to reproduce models. By making the code and pretrained checkpoint publicly available, the authors can contribute more to the community.
paper_summary: paper_summary: Currently, the majority of Zero-shot TTS models predominantly focus on the aspect of audio generation quality. FlashSpeech, building upon the existing latent consistency diffusion model, achieves speech synthesis with RTF 0.02 by employing only one to two sampling steps. Furthermore, it ensures further enhancement in audio synthesis quality through two additional techniques: adversarial loss in the generator and the incorporation of a rhythm predictor introducing noise fluctuations.
strength_weakness: strength_weakness: 1. The RTF (Real-Time Factor) of zero-shot TTS on a single V100 GPU is exceptionally low, measuring only 0.02. This outcome has surprised me.
2. Leveraging latent consistency diffusion models from other domains in zero-shot TTS, which demands high speed, is indeed an appropriate approach. Additionally, the incorporation of two simple engineering tricks, namely adversarial loss and a rhythm generator with perturbations, has led to corresponding improvements.
3. The writing is clear and comprehensible.
limitations: limitations: 1. While the author has effectively packaged the introduction, FlashSpeech does not introduce any groundbreaking structural innovations. The latent consistency diffusion model is merely a direct application, and the utilization of adversarial loss and perturbed rhythm generation are engineering tricks commonly observed in the field of Speech. FlashSpeech offers some insightful implications for the speech community but lacks significant novelty.
2. It would be beneficial for the author to compare FlashSpeech with similar lightweight and fast zero-shot TTS systems, such as P-flow [1] and MobileSpeech[2].
3. I have some concerns regarding the experimental results, such as UTMOS. I have personally tested the corresponding UTMOS on encodec_gt, and the achieved performance did not reach 3.6. I am curious to know why FlashSpeech, based on encodec, surpasses encodec_gt. Could the author provide the UTMOS testing script during the rebuttal period?
4. In regard to the related ablation experiments on the Prosody Generator and Discriminator, I suggest conducting tests on a broader range of metrics, particularly for the Prosody Generator. It is challenging to demonstrate the effectiveness of the Prosody Generator solely based on Pitch JSD and Duration JSD when there is no difference in WER.
5. I find it difficult to comprehend why FlashSpeech achieves higher subjective SMOS scores than NaturalSpeech2, despite having lower Sim-O and Sim-R scores in Table 1. Additionally, there has existed numerous reproductions or open-source models in the zero-shot TTS community, as well as several new works reporting new state-of-the-art results. FlashSpeech should attempt comparisons with more baselines, as the results in Table 1 may be influenced by the gap caused by reproduction.
6. The results in Table 1 may be affected by biases arising from inconsistent training datasets. However, I understand the challenges of reproducing baselines on MLS due to the time-consuming nature of the process. I am curious as to why FlashSpeech did not consider training on librilight, as many zero-shot TTS models are trained on this dataset. Additionally, testing on LibriSpeech may have an in-domain impact on MLS, and I am interested to learn about the out-of-domain performance of FlashSpeech compared to other baseline models.

[1] P-Flow: A Fast and Data-Efficient Zero-Shot TTS through Speech Prompting

[2] MobileSpeech: A Fast and High-Fidelity Framework for Mobile Zero-Shot Text-to-Speech

**I look forward to the author addressing my concerns during the rebuttal, and I will adjust my score (high or low) based on the feedback received in the rebuttal.**