MobileSpeech: A Fast and High-Fidelity Framework for Mobile Zero-Shot Text-to-Speech (2024)

Abstract: Zero-shot text-to-speech (TTS) has gained significant attention due to its powerful voice cloning capabilities, requiring only a few seconds of unseen speaker voice prompts. However, all previous work has been developed for cloud-based systems. Taking autoregressive models as an example, although these approaches achieve high-fidelity voice cloning, they fall short in terms of inference speed, model size, and robustness. Therefore, we propose MobileSpeech, which is a fast, lightweight, and robust zero-shot text-tospeech system based on mobile devices for the first time. Specifically: 1) leveraging discrete codec, we design a parallel speech mask decoder module called SMD, which incorporates hierarchical information from the speech codec and weight mechanisms across different codec layers during the generation process. Moreover, to bridge the gap between text and speech, we introduce a high-level probabilistic mask that simulates the progression of information flow from less to more during speech generation. 2) For speaker prompts, we extract fine-grained prompt duration from the prompt speech and incorporate text, prompt speech by cross attention in SMD. We demonstrate the effectiveness of MobileSpeech on multilingual datasets at different levels, achieving state-ofthe-art results in terms of generating speed and speech quality. MobileSpeech achieves RTF of 0.09 on a single A100 GPU and we have successfully deployed MobileSpeech on mobile devices. Audio samples are available at https://mobilespeech.github.io/ .

heading: 1 Introduction
text: In recent years, remarkable progress has been made in the development of text-to-speech (TTS) technology (Ren et al., 2019; Kim et al., 2021a; Huang et al., 2022a; Li et al., 2023). With advancements in
*Equal contribution. †Corresponding author.
this field, current state-of-the-art TTS systems have demonstrated the ability to generate high-quality voices of unseen speakers while maintaining consistency in tone and rhythm (Chen et al., 2021; Casanova et al., 2022b). However, these systems are often trained on datasets of only a few hundred hours and often require fine-tuning with dozens of sentences during the inference stage, which significantly limits the generative capabilities of the models and is time-consuming and not user-friendly.
With the recent development of large-scale language models (Brown et al., 2020; Touvron et al., 2023) and the increase in training data for speech synthesis from hundreds of hours (Zen et al., 2019) to tens of thousands of hours (Kahn et al., 2020) , current TTS models exhibit powerful zero-shot capabilities, enabling the cloning of unseen speakers’ voices in just a few seconds (Wang et al., 2023; Zhang et al., 2023; Shen et al., 2023; Le et al., 2023; Jiang et al., 2023b; Kharitonov et al., 2023; Borsos et al., 2023b; Huang et al., 2023; Yang et al., 2023b) . However, these models primarily focus on incontext abilities and fail to address issues related to inference speed and model deployment parameters. Additionally, all existing zero-shot TTS models are cloud-based and lack a mobile-based deployment approach. MobileSpeech is the first zero-shot TTS synthesis system that can be deployed on mobile devices. Firstly we think deployable mobile zero-shot TTS models aim to achieve the following goals:
• Fast: through our experiments, we have observed that the real-time factor (RTF) on a single mobile device often exceeds 8-10 times that of an A100 RTF for the same text. Therefore, the inference speed of the zero-shot TTS model must be significantly faster.
• Lightweight: to enable deployment on mobile or edge devices, the model size should be small, and the runtime memory footprint should be minimal.
13588
• High similarity and diversity: zero-shot TTS system should be able to clone timbre and prosody with just a few seconds of prompts, yielding diverse speech even when provided with identical text.
• High quality: to enhance the naturalness of synthesized speech, the model should pay attention to details such as frequency bins between adjacent harmonics and possess powerful duration modeling capabilities.
• Robustness: a highly robust TTS system is essential. A zero-shot TTS system should minimize occurrences of missing or repeated words.
To achieve these objectives, we have made our best efforts to reproduce several state-of-the-art generative models, which will be elaborated in the subsequent section on related work. We found current work is not applicable to mobile devices. Ultimately, based on the Natspeech framework (Ren et al., 2019, 2020) and the Mask generation module (Chang et al., 2022; Borsos et al., 2023b), we additionally design the SMD module and the speaker prompt component, taking into account the discrete codec architecture of speech. In summary, MobileSpeech contributes as follows:
• MobileSpeech is the first zero-shot TTS synthesis system that can be deployed on mobile devices. MobileSpeech achieves a good balance according to the above five evaluation metrics.
• In MobileSpeech, we additionally design the SMD module based on the hierarchical token structure of discrete speech codecs and design the Speaker Prompt module to maintain high similarity, high quality, low latency.
• Training MobileSpeech-english for 580 hours achieved SOTA inference speed and comparable audio quality. Training MobileSpeechchinese for 40,000 hours resulted in SOTA performance in all aspects.
• MobileSpeech has been successfully deployed on mobile phones and is expected to be used by several hundred thousand users.

heading: 2 Related Work

heading: 2.1 Zero-shot TTS
text: Zero-shot speech synthesis refers to the ability to synthesize the voice of an unseen speaker based solely on a few seconds of audio prompt, also known as voice cloning. Due to its impressive performance, it has gained significant attention recently. Previous approaches in the field of voice cloning can be classified into speaker adaptation (Wang et al., 2018; Chen et al., 2021; Huang et al., 2022b) and speaker encoding methods (Casanova et al., 2022b; Arik et al., 2018; Kang et al., 2022). These models are often trained on smaller datasets and require a substantial amount of data for finetuning or employ highly complex encoders. These limitations greatly restrict the audio quality and further usage.
In recent months, with the advancement of generative large-scale models, a plethora of outstanding works have emerged. VALL-E (Wang et al., 2023) leverages discrete codec representations and combines autoregressive and non-autoregressive models in a cascaded manner, preserving the powerful contextual capabilities of language models. It can clone the voice of a target speaker with just a 3-second audio prompt. VALL-E X (Zhang et al., 2023) extends zero-shot TTS to multiple languages based on the cascaded structure of VALL-E. NaturalSpeech 2 (Shen et al., 2023) employs continuous vectors instead of discrete neural codec tokens and introduces in-context learning to a latent diffusion model. SpearTTS (Kharitonov et al., 2023) and Make-a-Voice (Huang et al., 2023) utilize semantic tokens to reduce the gap between text and acoustic features. VoiceBox (Le et al., 2023) is a non-autoregressive flow-matching model trained to infill speech, given audio context and text. Megatts (Jiang et al., 2023b,a) , on the other hand, utilizes traditional mel-spectrograms, decoupling timbre and prosody and further modeling the prosody using an autoregressive approach.
However, none of the aforementioned works take into account the model’s speed and lightweight nature. It is well-known that autoregressive models often require large memory footprints and have slower iteration speeds, and diffusion models often require hundreds of sampling steps. Some work introduce a semantic token cascade structure, which will introduce more time overhead. However, compared to codecs, certain approaches based on melspectrogram exhibit lower generation diversity and
audio quality (Siuzdak, 2023). MobileSpeech, on the other hand, is the first zero-shot TTS system that can be deployed on mobile devices, addressing the fast speed,lightweight, high quality requirements.

heading: 2.2 Generative models
text: Generative models have been extensively utilized in various domains by employing techniques such as language models (Borsos et al., 2023a; Kreuk et al., 2022; Ji et al., 2023; Wang et al., 2023; Zhang et al., 2023), Variational Autoencoders (VAE) (Ren et al., 2021; Lee et al., 2022), Generative Adversarial Networks (GAN) (Kim et al., 2021b; Kong et al., 2020), Normalizing Flow (Kim et al., 2020; Miao et al., 2020), Mask Autoencoders (Chang et al., 2022)and diffusion models (Huang et al., 2022a). Considering the challenge of balancing perceptual quality and inference speed in zero-shot TTS, as well as the recent success of parallel decoding in text (Ghazvininejad et al., 2019), image (Chang et al., 2022), audio (Borsos et al., 2023b), and video (Villegas et al., 2022) generation tasks, MobileSpeech has adopted masked parallel generation as the audio synthesis approach. Furthermore, MobileSpeech has additionally designed the Speech Codec Mask Decoder (SMD) module and Speaker prompt based on the unique characteristics of acoustic tokens and the specific requirements of the zero-shot TTS task, aiming to address the afore-
mentioned trade-off. In Appendix A, we present the detail of discrete acoustic codec and in Appendix B we present the distinctions between maskbased generative models and other generative models in modeling discrete codecs.

heading: 3 MobileSpeech
text: In this section, we will first introduce the overall structure of MobileSpeech, followed by a detailed focus on the Speech Mask Codec Decoder module and the Speaker Prompt module. Furthermore, we will proceed to elaborate on the specific intricacies of MobileSpeech’s training and inference processes, with the explicit details of the model being presented in Appendix C.

heading: 3.1 Overall
text: As shown in Figure 1, MobileSpeech employs FastSpeech 2 as our baseline. The input text is processed to extract phoneme sequence, which are then passed through a text encoder to obtain the corresponding target text representation. Simultaneously, the input consists of speech prompts based on a discrete codec (Défossez et al., 2022). The prompt encoder compresses the channel dimension of the prompt codec acoustic tokens to one dimension. We specifically extract fine-grained prompt duration tokens from the prompt codec acoustic tokens and combine them with the target text, which are then fed into the duration prediction module and
length regularizer to obtain an expanded target text. The Speech Mask Codec Decoder serves as the core module. During the training process, we randomly select a portion of the aligned target text as the prompt to guide the generation of the remaining text. Based on the masking and parallel generation using RVQ features of the speech codec, we design a multi-channel training approach, priority training, and a mechanism that relies solely on text to generate acoustic tokens. Finally, the acoustic tokens are transformed into the final speech output using a pre-trained codec Tokenizer (Siuzdak, 2023).

heading: 3.2 Speech Codec Mask Decoder
text: Based on paired text-speech data {X,Y }, where X = {x1, x2, x3, · · · , xT } represents the aligned text and Y denotes the representation of speech through a pre-trained discrete codec encoder (Défossez et al., 2022), namely Y = C1:T,1:N ∈ RT×N , we consider T as the downsampled utterance length, which is equal to the length of the text. N represents the number of channels for every frame. The row vector of each acoustic code matrix Ct,: represents the eight codes for frame t, and the column vector of each acoustic code matrix C:,j represents the code sequence from the j-th codebook, where j ∈ {1, 2, · · · , N} .
Notably, in contrast to models such as VALL-E (Wang et al., 2023), it is acknowledged that obtaining prompt text in real-world scenarios is challenging, and to mitigate the impact of erroneous prompt text on the model. We have devised a textagnostic speech prompt to guide the generation of target speech in MobileSpeech. Specifically, for a sequence with a time length of T , we randomly extract a segment of length k at the beginning to simulate prompts of arbitrary lengths, where the value of k falls within the range of:
T 3 ≤ k ≤ 2× T 3 (1)
By selecting the split point k, we obtain the corresponding prompt speech Yprompt and the target text Yprompt:
Yprompt = C1:k,1:N , Xtarget = Xk:T (2)
Therefore, disregarding the codec channel count, the optimization objective of the Speech Codec Mask Decoder (SMD) module is to maximize the conditional probability function shown below:
P (Ck:T,1:N | C1:k,1:N , Xk:T ; θ) (3)
It is well known that the use of RVQ-based speech discretization tokens (Défossez et al., 2022; Siuzdak, 2023) induces a hierarchical token structure, where tokens from finer RVQ levels contribute less to the perceptual quality. This allows for efficient factorization and approximation of the joint distribution of the token sequence. We believe that considering this structural aspect holds the most promise for future advancements in long-sequence audio modeling. Therefore, in MobileSpeech, as depicted in Figure 1 (b), we divide the generation process into two parts: the generation of the first channel and the generation of the remaining channels at the same time, denoted as:
P (Ck:T,1:N | C1:k,1:N , Xk:T ; θ) =P (Ck:T,1 | C1:k,1:N , Xk:T ; θ)
× N∏
j=2
P (Ck:T,j | C1:k,1:N , Xk:T ; θ) (4)
For the generation of the first channel P (Ck:T,1 | C1:k,1:N , Xk:T ; θ), we employ a mask-based generative model as our parallel decoder. We sample the mask Mi ∈ {0, 1}T according to a cosine schedule (Chang et al., 2022) for codec level i, specifically sampling the masking ratio p = cos(u) where u ∼ U [ 0, π2 ] . and the mask Mi ∼ Bernoulli(p). Here, Mi represents the portion to be masked in the i-th level, while M̄i denotes the unmasked portion in the i-th level. The prediction of this portion is refined based on the prompt speech and the concat of target text and the unmasked portion of the first channel. Therefore, the prediction for this part can be specified as follows:
P (M1Ck:T,1 | C1:k,1:N , Xk:T , M̄1Ck:T,1; θ) (5)
Due to the fact that in the inference process, we directly obtain the first-layer discrete tokens from the text, it often requires multiple iterations to iteratively select higher-confidence tokens to simulate the masked environment during training. In order to reduce the number of iterations, as shown in Figure 1 (b), MobileSpeech, during the training process, employs a certain probability to completely mask the acoustic tokens of the first channel. Additionally, completely masking the acoustic tokens of the first channel at different stages of model training can guide the model to generate from easier to harder instances, thus increasing the upper limit of the model. We control the two different ways of
generating the first channel through a probability parameter α, as shown below:
P (Ck:T,1 | C1:k,1:N , Xk:T ; θ) = αP (M1Ck:T,1 | C1:k,1:N , Xk:T , M̄1Ck:T,1; θ) +(1− α)P (M1Ck:T,1 | C1:k,1:N , Xk:T ; θ)
(6) For the generation of the remaining channels, similar to other models that randomly select a channel for training during the training process, we incorporate a weighting mechanism in the random selection process. We believe that the importance of channels decreases layer by layer. Therefore, we introduce the Prank mechanism in Appendix D. When predicting the acoustic token of the j-th channel, we utilize the unmasked portion of the j-th layer M̄j and all acoustic tokens preceding the j-th layer for prediction, as shown below:
P (MjCk:T,j | C1:k,1:N , Xk:T , Ck:T,<j , M̄jCk:T,j ; θ) (7)

heading: 3.3 Speaker Prompt
text: MobileSpeech, based on FastSpeech2, adopts an Encoder-Decoder architecture. Different prompt strategies were designed in both stages to encourage mobileSpeech to capture the prosodic and timebre information of the prompt speech. During the experimental process, we observed that MobileSpeech generated speech with relatively uniform prosody and the distribution of durations exhibited excessive uniformity. Therefore, we abandoned the coarse-grained speech prompt guidance before the duration predictor. As illustrated in Figure 2, the Duration Extractor incorporates a Q-K-V attention layer to integrate different information. It first extracts fine-grained prompt duration values from the prompt acoustic token and feeds them into the prompt duration encoder along with positional information. Subsequently, the prompt duration is used to guide the prediction of the target text’s duration. Subsequent ablation experiments have confirmed that this approach can improve the occurrence of disfluent speech phenomena. Both the Duration Extractor and the Duration Predictor modules follow the same structure. The text is first passed through a normalization layer and then mapped to a Query. The Query, along with the prompt acoustic or prompt duration, undergoes a Q-V-K attention layer and a convolutional layer to obtain latent features. The latent features are then
added to the text through residual connections to yield the corresponding output results.
In the Decoder stage of MobileSpeech, as described in Equation 1 and 2, during the training process, we randomly select a sequence of length k from the target text and speech to form a textindependent speech prompt. It is worth noting that, to ensure the robustness of the model, the speaker prompt in the Encoder and the speaker prompt in the Decoder are not identical sequences. In the Speech Mask Codec Decoder (SMD), all tokens of the input prompt acoustic in SMD are unmasked, and inspired by (Du et al., 2023) , we do not include corresponding positional information and prompt encoder in order to preserve the original features of the acoustic tokens as one of the inputs to the SMD module. This is explicitly highlighted with a blue line in Figure 1 (b). Additionally, in the SMD module, we pass the acoustic tokens through the prompt encoder and apply cross-attention to allow the target text, as a weak condition token, to incorporate some information from the prompt acoustic tokens, thereby maintaining contextual capabilities.

heading: 3.4 Train and Inference
text: During the training process, the Duration Predictor and Duration Extractor are optimized using the mean square error (MSE) loss, where the extracted duration serves as the training target. We denote the losses at Duration Predictor and Duration Extractor as Ldur and Lpromptdur, respectively. To improve alignment accuracy and reduce the information gap between the model input and output, we employ the Montreal forced alignment (MFA) (McAuliffe et al., 2017) tool to extract phoneme durations. The duration values are predicted in the logarithmic domain, which promotes a more Gaus-
sian distribution and facilitates training. The SMD module is optimized using the cross-entropy loss function, considering both the first channel and the discrete acoustic tokens randomly selected under the Prank strategy. We use L1smd and L j smd to represent the losses of the first channel and the j-th channel, respectively. Where 2 ≤ j ≤ N . Thus, the overall loss L can be expressed as follows:
L = L1smd + Ljsmd + Lpromptdur + Ldur (8)
During the inference stage, we iteratively generate discrete acoustic tokens by incorporating the prompt speech into the original text state. For the generation of the first channel of acoustic tokens, we employ the confidence-based sampling scheme proposed by (Chang et al., 2022; Borsos et al., 2023b) . Specifically, we perform multiple forward passes, and at each iteration i, we sample candidates for the masked positions. We then retain Pi candidates based on their confidence scores, where Pi follows a cosine schedule. As for the generation of the remaining channels, we directly select the token with the highest probability using a greedy strategy.

heading: 4 Experiments

heading: 4.1 Experiment Setup
text: Datasets We trained MobileSpeech on datasets of varying scales and languages, and evaluated its performance on datasets comprising speech prompts of different lengths and environments. This allowed us to measure the effectiveness of MobileSpeech. For the English section, we trained MobileSpeech exclusively on the libriTTS (Zen et al., 2019) , which comprises 580 hours of data from 2,456 speakers. For the Mandarin Chinese section, MobileSpeech was trained on an internal dataset totaling 40,000 hours. Baselines We consider the latest SOTA models as our baselines. For the English segment, we utilize YourTTS (Casanova et al., 2022b) 1in its official version. We have successfully reproduced results close to the official ones for VALL-E (Wang et al., 2023), surpassing the reproduced results found online 2. As for NaturalSpeech2 (Shen et al., 2023), we employ the reproduced version available online. Additionally, we have achieved results similar to the official ones for Megatts (Jiang et al., 2023b). For the Chinese segment, we compare our
1https://github.com/Edresson/YourTTS 2https://github.com/enhuiz/vall-e
results with the open-source interface 3 of the current state-of-the-art Chinese model Megatts2 (the version of the interface is November 2023) (Jiang et al., 2023a). Training and Inference Settings MobileSpeech was trained on 40,000 hours of Mandarin Chinese data using 32 NVIDIA A100 40G GPUs. The discrete codec was trained end-to-end on the training dataset, with the encoder part of Encodec (Défossez et al., 2022) and the decoder part of Vocos (Siuzdak, 2023) being optimized through engineering modifications. For the 580-hour English training data, the discrete codec employed a pre-trained version. MobileSpeech was trained for 12 epochs on 4 NVIDIA A100 40G GPUs, with each batch accommodating 3500 frames of the discrete codec. We optimized the models using the AdamW optimizer with parameters β1 = 0.9 and β2 = 0.95. The learning rate was warmed up for the first 5k updates, reaching a peak of 5× 10−4, and then linearly decayed. Automatic metrics We have aligned all our objective experiments with VALL-E (Wang et al., 2023). To evaluate speaker similarity (SPK) between the original prompt and synthesized speech, we employ WavLM-TDNN (Chen et al., 2022). However, due to updates in the repository, we have updated the feature extractor, but all our models have been tested by using the same metrics. For assessing automatic speech recognition (ASR) performance, we conduct ASR on the generated audio and calculate the word error rate (WER) compared to the original transcriptions. In this experiment, we utilize the HuBERT-Large (Hsu et al., 2021) model fine-tuned on LibriSpeech 960h as the ASR model. This model is a CTC-based model without language model fusion. To compute the real-time factor, we measure the ratio of the total length of generated speech to the total time taken by the model to generate the corresponding speech features . The inference time in the vocoder component will not be taken into account. The calculation is performed on an empty A100 GPU. Human evaluation We conduct the MOS (mean opinion score) and CMOS (comparative mean opinion score) evaluation on the test set to measure the audio naturalness via crowdsourcing. We keep the text content and prompt speech consistent among different models to exclude other interference fac-
3https://www.volcengine.com/product/ voicecloning
tors. We randomly choose 50 samples from the test set of each dataset for thesubjective evaluation and each audio is listened to by at least 20 testers. We analyze the MOS in three aspects: MOS-Q (Quality: clarity, high-frequency, and original timbre reconstruction), MOS-P (Prosody: naturalness of pitch, energy, and duration), and MOS-S (Speaker similarity). We also analyze the CMOS in terms of audio quality and speech prosody. See Appendix E for details.

heading: 4.2 LibriSpeech Evaluation
text: Regarding the inference phase, to ensure fair comparisons, we followed the experimental protocols outlined in VALL-E (Wang et al., 2023) and employed the LibriSpeech test-clean dataset (Panayotov et al., 2015) , ensuring no overlap with our training data. We specifically utilized samples from the LibriSpeech test-clean dataset with durations ranging from 4 to 10 seconds, resulting in a subset of 2.2 hours. Following VALL-E approach, we use the whole transcription and the first 3 seconds of the utterance as the phoneme and acoustic prompts respectively, and ask the model to generate the continuations, the continue setting of Table 1 is referred to this.
By observing Table 1, the following conclusions can be drawn: (1) Under the same scale of 580 hours training data of libriTTS (where yourtts is trained on a combined dataset of VCTK , LibriTTS , and TTS-Portuguese (Casanova et al., 2022a)), MobileSpeech significantly outperforms YourTTS
in terms of SPK and MOS, surpassing the current SOTA models NaturalSpeech2 and MegaTTS. This indicates that MobileSpeech is capable of generating higher audio quality, better prosody, and more similar audio. The outstanding results are attributed to the utilization of the baseline generated by the MASK parallel codec and the additional design of the SMD module, Speaker Prompt module in MobileSpeech. (2) In terms of audio quality, MobileSpeech also achieves comparable results to VALL-E, which was trained on a large-scale dataset of 60,000 hours, with a speaker similarity metric difference of only 0.046. This difference is attributed to the disparity in the training data volume. (3) MobileSpeech achieves the best results in terms of robustness measured by the Wer metric, even surpassing VALL-E by 0.7. This is because the autoregressive structure of VALL-E often suffers from cumulative errors, resulting in lower robustness. Our experiments reveal that increasing the amount of data can alleviate the issue of high robustness. (4) MobileSpeech exhibits remarkable performance in terms of the Real-Time Factor (RTF) metric, as it can generate 10 seconds of audio within one second. Compared to VALL-E, MobileSpeech achieves an 11-fold speed improvement, and compared to MegaTTS and NaturalSpeech2, it achieves a 4-fold speed improvement. Although YourTTS achieves an RTF of 0.22, its simple structural design leads to unsatisfactory performance in terms of audio quality, speaker similarity, and model robustness.

heading: 4.3 Mandarin Evaluation
text: As our inference dataset for Mandarin Chinese, we compiled a collection of 200 real-world mobile-end recordings with arbitrary durations ranging from 3 to 10 seconds, representing a mixture of clean and noisy scenarios. This dataset serves as our Mandarin Chinese test set. We employed CMOS to measure the performance of MobileSpeech and the
current SOTA model MegaTTS2 on the Mandarin Chinese dataset. Considering that MegaTTS2 is a paid model, we incurred significant costs to conduct this experiment.
The experimental results, as shown in Table 2, demonstrate that MobileSpeech outperforms MegaTTS2 comprehensively in terms of MOSQ, MOS-P, and MOS-S under various conditions, including large-scale training corpora, different language categories, varying prompt lengths, and noisy speech prompts. Particularly, MobileSpeech achieves a significantly higher score of 0.24 in terms of speaker similarity compared to MegaTTS2. By conducting a cross-comparison between MobileSpeech and GT Codec in terms of CMOS-Q, CMOS-P, and CMOS-S metrics, we can observe that MobileSpeech exhibits superior performance in terms of speaker similarity but slightly lags behind in audio prosody.

heading: 4.4 Parameters Evaluation
text: To begin, we present a comprehensive account of the parameters for all state-of-the-art models. In the VALL-E framework, the AR model comprises 229 million parameters, while the NAR model consists of 235 million parameters, resulting in a total of 465 million parameters. In the NaturalSpeech2 model, the overall parameter count is 435 million. Within the MegaTTS system, the VQ-GAN model encompasses 243 million parameters, the LLM model comprises 124 million parameters, and the hifigan vocoder consists of 106 million parameters, resulting in a cumulative parameter count of 473 million. As outlined in the accompanying figure, the Mobilespeech model has a total parameter count of 207 million (detailed parameters can be
found in Appendix C of the paper), demonstrating its status as the most lightweight option.

heading: 4.5 Ablation experiment
text: We conducted ablation experiments on the LibriSpeech test-clean dataset. Firstly, we omitted finegrained prompt duration and directly used prompt acoustic tokens to guide the generation of target text. Additionally, in the SMD module, prompt acoustic tokens were solely used as the input of SMD. This version was referred to as MobileSpeech w/o durprompt. For the MobileSpeech w/o onechannel version, we replaced the SMD module with randomly selecting a layer for mask prediction. The results are presented in Tables 3 and 4. It can be observed that both MobileSpeech w/o durprompt and MobileSpeech w/o onechannel exhibited certain decreases in WER, SPK, and MOS values compared to MobileSpeech. This suggests that each component of MobileSpeech’s design is necessary. During our experimental process, we found that the MobileSpeech w/o durprompt version exhibited a certain amount of disfluency, which affected the overall prosody of the audio. This indicates that fine-grained duration prompts indeed contribute to better modeling of duration information.
We also conducted ablation experiments on the number of iterations in the first channel of MobileSpeech, which has a significant impact on the inference speed. As shown in Table 5, we observed that the number of iterations is inversely proportional to the real-time factor (RTF), and blindly increasing the number of iterations does not necessarily improve the model’s performance. However, when the number of iterations is reduced to 1, the model exhibits significant degradation in its capabilities. Consequently, we selected 8 iterations as the final choice, striking a balance between performance and efficiency.

heading: 5 Conclusion
text: In this paper, we propose MobileSpeech, a Fast and Lightweight Framework for Mobile zero-shot Textto-Speech. MobileSpeech achieves a unique combination of audio naturalness and similarity performance, comparable to other zero-shot TTS systems on English corpora, while significantly enhancing inference speed through its distinctive SMD and Speaker modules. MobileSpeech stands as the first real-time zero-shot TTS system deployable at the edge, demonstrating state-of-the-art performance on Chinese corpora through extensive training on large-scale Chinese language data. We not only validate the significance of each module in MobileSpeech through ablation experiments but also analyze the effects of different configurations and iterations to adapt to various deployment scenarios in mobile environments.

heading: 6 Acknowledgments
text: This work was supported in part by the National Key R&D Program of China under Grant No.2022ZD0162000

heading: 7 Limitations and Future Work
text: In this section, we will analyze the limitations of MobileSpeech and discuss potential future work.
Failure cases During our experiments in realworld scenarios (Figure 2), we observed occasional instances of stuttering and unstable pitch in MobileSpeech. This could be attributed to noisy input prompts or the possibility that the mask mechanism excessively focuses on local positions. In future work, we aim to enhance the model’s noise robustness and design a more robust mask mechanism that considers a broader context.
New Tasks: The current zero-hhot TTS task itself has significant potential for expansion. Most existing models primarily focus on similarity in timbre. However, future research can extend zeroshot TTS to include similarity in rhythm, emotion, and language. It could even involve using multiple audio segments to control different parts or combining with text style control, representing meaningful new tasks.
Broader impacts: Since MobileSpeech could synthesize speech that maintains speaker identity, it may carry potential risks in misuse of the model, such as spoofing voice identification or impersonating a specific speaker. To mitigate such risks, it is possible to build a detection model to discriminate
whether an audio clip was synthesized by MobileSpeech.

heading: A Discrete Acoustic Codec
text: Since audio is typically stored as a sequence of 16- bit integer values, a generative model is required to output 216 = 65536 probabilities per timestep to synthesize the raw audio. In addition, the audio sample rate exceeding ten thousand leads to an extraordinarily long sequence length, making it more intractable for raw audio synthesis. In these days, neural acoustic codec (Zeghidour et al., 2021; Défossez et al., 2022; Yang et al., 2023a; Siuzdak, 2023), with their ability of reconstructing highquality audio at very low bitrates, sub-sequently allowed for extending discrete modeling to audio signals as diverse as multi-speaker speech. To be specific, Soundsteam (Zeghidour et al., 2021) which relies on a model architecture composed by a fully convolutional encoder/decoder network and a residual vector quantizer (RVQ) can efficiently compress speech. Encodec (Défossez et al., 2022) consists in a streaming encoder-decoder architecture with quantized latent space trained in an endto-end fashion.Hificodec (Yang et al., 2023a) propose a group-residual vector quantization (GRVQ)
technique To reduce the number of quantizers. Vocos (Siuzdak, 2023) closing the gap between timedomain and fourier-based neural vocoders for highquality audio synthesis. After our detailed experiments, MobileSpeech finally selected encodec and vocos as the encoder and decoder of acoustic token.

heading: B Codec Modeling
text: In this section, we provide a detailed comparison of different modeling approaches for the RVQ structure based on discrete codecs. As depicted in Figure 3, (a) represents a fully sequential unfolding of each frame’s channels followed by autoregressive modeling, which incurs significant time overhead as the number of channels increases. (b) involves splitting the codec generation into two stages, where the first stage autoregressively generates the codec for the first channel, and the second stage concurrently generates the codecs for the remaining channels. (c) adopts global attention in the temporal dimension and local attention in the channel dimension for modeling. All three methods (a), (b), and (c) are fundamentally autoregressive models, but due to limitations in inference speed and memory consumption, they are not suitable for deployment on mobile devices. In contrast, MobileSpeech adopts a mask-based parallel generation approach, offering a potential deployment solution.

heading: C Model Structure
text: In this section, we introduce the detailed structure of proposed MobileSpeech. To be specific, we will introduce the architecture of text encoder, prompt duration encoder, prompt encoder, cross-attention and ConFormer (Gulati et al., 2020) used in MobileSpeech. Text encoder and prompt duration encoder share the same architecture which is comprised of several FFT blocks as used by FastSpeech2 (Ren et al., 2020) . The value of probability parameter α in MobileSpeech is 0.6 . The detailed configuration information is shown in the table 6.

heading: D Algorithm
text: We provide a detailed description of the Prank algorithm in the Algorithm 1 provided below.

heading: E MOS Evaluation
text: Below, we provide a detailed explanation of the requirements and standards we employed for measuring Mean Opinion Score (MOS).
Algorithm 1 Prank algorithm Require: The channels set {ci}Ni=2 and decreasing corresponding weights {wi}Ni=2; where wj > wj+1, j ∈ {2, 3, · · · , N − 1} Initialize CU = ∅; for each ci do
Repeat ci for wi times; Add repeated ci to CU ;
end for Random sample a channel c from CU ;

heading: Future Work
text: In this section, we will analyze the limitations of MobileSpeech and discuss potential future work. Failure cases During our experiments in realworld scenarios (Figure 2), we observed occasional instances of stuttering and unstable pitch in MobileSpeech. This could be attributed to noisy input prompts or the possibility that the mask mechanism excessively focuses on local positions. In future work, we aim to enhance the model’s noise robustness and design a more robust mask mechanism that considers a broader context. New Tasks: The current zero-hhot TTS task itself has significant potential for expansion. Most existing models primarily focus on similarity in timbre. However, future research can extend zeroshot TTS to include similarity in rhythm, emotion, and language. It could even involve using multiple audio segments to control different parts or combining with text style control, representing meaningful new tasks