Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners (2024)

Abstract: Large language models (LLMs) have successfully served as a general-purpose interface across multiple tasks and languages, while the adaptation of voice LLMs is mostly designed for specific purposes (either single-task or monolingual), where the advantages of LLMs especially for low-resource language processing and zero-shot task generalization are less exploited in the audio community. To bridge the gap, we introduce Make-A-Voice as a multimodal voice LLM and conduct a comprehensive study on its capability to deal with multiple tasks/languages. When trained on ∼200K hours of 6-language data for 4 voice generation applications, Make-A-Voice emerges notable advantages: 1) as scalable learners to improve performance with end-to-end local and global multiscale transformers; and 2) as multitask learners by adjusting prompts to share common knowledge across modalities (speech/singing) and present in-context learning abilities by generalizing to unseen tasks not explicitly train on; 3) as multilingual learners to alleviate data scarcity of low-resource languages by including rich-resource language training data. Experimental results demonstrate that Make-A-Voice exhibits superior audio quality and style similarity compared with competitive baseline models in monolingual/cross-lingual voice generation. 1

heading: 1 Introduction
text: Large language models (LLMs) (Devlin et al., 2018; Raffel et al., 2020; Ouyang et al., 2022; Zhang et al., 2022b) trained on massive corpora of texts have shown their ability to perform new tasks from textual instructions or a few examples. The LLM-based interfaces excel at generating text
* Equal contributions † Corresponding author 1Audio samples are available at https://M-Voice.
github.io
for tasks that require the modeling of complex interactions, and they are trained to predict sequences of discrete tokens, have also been adapted to continuous, audio signals (Borsos et al., 2022; Agostinelli et al., 2023; Kreuk et al., 2022).
Current voice LLMs (Kharitonov et al., 2023; Wang et al., 2023a; Zhang et al., 2023) cast voice synthesis as a language modeling task in a discrete representation space. VALL-E (Wang et al., 2023a) proposes a language model approach for TTS with audio codec codes as intermediate representations. Kharitonov et al. (2023) introduce a hierarchical approach that combines two types of audio tokens. Zhang et al. (2023) train a multilingual conditional codec language model to predict the acoustic token sequences of speech in different languages. Despite the success achieved, most existing voice LLMs are designed for specific purposes (single-task or monolingual), where the advantages of LLMs especially for low-resource language processing and zero-shot task generalization are less exploited in the audio community.
To bridge the gap, we leverage the intuition that GPTs can process multiple tasks or languages as a general-purpose interface and emerge strong incontext learning capabilities. In this work, we conduct a comprehensive study on voice LLMs’ capability to deal with multiple tasks (i.e., speech and singing voice modeling) and multiple languages (i.e., rich and low resource data). We refer to this resulting model as Make-A-Voice, for multimodal LLMs to synthesize and manipulate voice signals. Make-A-Voice is a decoder-only model trained on a mixture of tasks, emerging capabilities of cross-task knowledge sharing, generalization to new tasks, and alleviating data scarcity of lowresource languages.
Make-A-Voice employs self-supervised tokens for the unified voice generation pipeline: 1) Semantic tokens determine the semantic meaning given text or speech; 2) Acoustic tokens provide
10929
acoustic information given various control conditions, which can be learned in a large amount of self-supervised audio-only data. Make-AVoice is trained on ∼200K hours of multilingual data in 6 languages, and we introduce 4 applications: text-to-speech (TTS), voice conversion (VC), singing voice synthesis (SVS), and singing voice conversion (SVC). Experimental results demonstrate that Make-A-Voice achieves SOTA results in monolingual/cross-lingual zero-shot voice generation. Subjective and objective evaluation show that Make-A-Voice exhibits superior audio quality and style similarity compared with baselines. The key takeaways are as follows:
• Make-A-Voice as scalable learners to improve performance. Make-A-Voice inherently leverages the transformer architecture and scales the models’ size in depth and width. Make-A-Voice predicts long sequences with end-to-end differentiable multiscale (local and global) transformers to reduce the extremely long sequences of the acoustic token, where scaling the model size (520M (medium), and 1.2B (large) parameter) from 160M (base) results in 15% and 37% improvement for TTS.
• Make-A-Voice as multilingual learners to alleviate data scarcity of low-resource languages. Combined with more rich-resource language data, model training benefits from a variety of acoustic conditions on top of semantic meanings. Make-A-Voice excels at zero-shot transferring acoustic attributes (speaker identity, emotion, prosody) even in low-resource languages, which has witnessed a 20% SIM gain and 7.9% WER drop in TTS synthesis.
• Make-A-Voice as multitask learners for crosstask knowledge sharing, even generalizable to unseen tasks. Make-A-Voice is trained with a combination of semantic and acoustic modeling tasks across speech/singing voice modalities by adjusting prompts (referring to Figure 1); 1) It shares common knowledge across tasks (e.g., multi-quantization codec modeling across modalities), as evidenced by 27% SIM improvement in SVS with the help of speech tasks; and 2) MakeA-Voice illustrates in-context learning abilities by generalizing to tasks not explicitly trained on, including cross-lingual timbre transferring, generating coherent emotion, and noise continuations.

heading: 2 Related Works

heading: 2.1 Generative Voice Models
text: Text-guided voice synthesis (text-to-speech and singing voice synthesis) typically converts input text into mel-spectrogram (e.g., Tacotron (Wang et al., 2017), FastSpeech (Ren et al., 2019)), which is then transformed to waveform using a separately trained vocoder (Kong et al., 2020; Huang et al., 2021). Recent generative models cast voice synthesis as a language modeling task to perform incontext learning: VALL-E (Wang et al., 2023a) use discrete codes derived from an off-the-shelf neural audio codec model, and regard TTS as a conditional language model. Zhang et al. (2023) leverage back-translation and prompt-guided LLMs for high-quality TTS with limited supervision. Jiang et al. (2023) train a prosody language model with arbitrary-length speech prompts to produce expressive and controlled prosody.
Despite the success achieved, most voice LLMs are designed for single-task or as monolingual. In this work, Make-A-Voice presents emerging capabilities as scalable multilingual and multitask learners, which provides critical takeaways for the.

heading: 2.2 Multitask Learning
text: Building a simple and multitask learning framework has attracted increasing attention in the community: NANSY (Choi et al., 2021a, 2022) is trained in a self-supervised manner that does not require any annotations paired with audio, and efficiently tackles multiple applications after training the backbone network. Lee et al. (2021) design a multitask learning framework with joint speech and text training that enables the model to generate dual mode output (speech and text) simultaneously in the same inference pass. Wang et al. (2023b) combine neural codec language modeling with multitask learning using task-dependent prompting, capable of zero-shot TTS and various speech transformation tasks, dealing with clean and noisy signals. Rubenstein et al. (2023) fuse text-based and speechbased language models into a unified multitask architecture to process and generate text and speech. In this work, we demonstrate voice LLMs’ capabilities as multitask learners across various voice generation tasks. Make-A-Voice demonstrates the improved capability with multitask learning by transferring common knowledge across modalities and illustrates in-context learning abilities by performing tasks Make-A-Voice is not explicitly
trained on.

heading: 2.3 Multilingual Learning
text: Multilinguality has been a very active research area in speech and NLP. Duquenne et al. (2022) introduce a large-scale multilingual speech-to-speech corpus and demonstrate that model pre-training and sparse scaling using a mixture of experts bring significant gains to translation performance. VoiceBox (Le et al., 2023) trains a non-autoregressive flow-matching model on 50K hours of multilingual audiobooks from six languages. Massively Multilingual Speech (Pratap et al., 2023) increases supported languages with rich-resource datasets and effective self-supervised learning. In this work, we present voice LLMs’ capabilities as multilingual learners, where the rich-resource language data is included to learn various acoustic conditions, alleviating the data scarcity of prompt-guided in-context learning for low-resource language.

heading: 3 Voice Large Language Models
text: In this section, we overview the discrete voice representation, namely semantic and acoustic tokens, and then introduce the decoder-only unified voice synthesis model Make-A-Voice. Next, we introduce the designs of the multitask learning in Section 3.3 and multilingual approaches in Section 3.4, as well as the scalable multi-scale transformer architecture in Section 3.5.

heading: 3.1 Voice Representation
text: Semantic tokens. It is crucial to extract rich linguistic information from the speech signal. To this end, we resort to XLSR-53: a wav2vec 2.0 model pre-trained on 56k hours of speech in 53 languages (Conneau et al., 2020). In the following, a k-means algorithm is applied to the learned representations of the unlabelled speech to generate K1 cluster centroids at every 20-ms frame. In the end, a speech utterance y is represented as semantic tokens with [s1, s2, . . . , sT ] , si ∈ {0, 1, . . . ,K1 − 1},∀1 ≤ i ≤ T , where T is the number of frames.
Acoustic tokens. The audio encoder E of codec models (Zeghidour et al., 2021; Défossez et al., 2022) consists of several convolutional blocks with a total downsampling rate of 320 and generates continuous representations at every 20-ms frame in 16kHz. The residual vector-quantizer Q produces discrete representations aq with a codebook size of K2, using a vector quantization layer (Vasuki and Vanathi, 2006). In the end, we flat-
ten all the codebooks and thus a speech utterance y is represented as acoustic tokens with [a1, a2, . . . , aT ] , ai ∈ {0, 1, . . . ,K2 − 1}, ∀1 ≤ i ≤ T , where T is the number of frames.

heading: 3.2 Make-A-Voice: Controllable Voice LLMs
text: As illustrated in Figure 1 and Table 1, Make-AVoice casts voice synthesis as language modeling tasks with self-supervised tokens, where voice synthesis is broken down into more manageable pieces (i.e., semantic modeling or acoustic modeling) and jointly learned in a decoder-only language model, where various conditioning mechanisms are investigated in 1) semantic modeling: semantic tokens s determine the semantic meaning given text or speech; 2) conditional acoustic modeling: acoustic tokens a are guided by control conditions (speaker, emotion, prosody, and style) and learned on top of semantic meanings in a large amount of self-supervised audio-only data. In the end, a unitbased vocoder synthesizes high-fidelity waveforms from compressed acoustic representations.
• Zero-shot TTS / VC. Given a target text y, TTS models first determine semantic tokens s, and then perform in-context learning given acoustic prompt ap derived from a reference utterance. During training, we randomly select two nonoverlapping speech windows from each example and consider one window as a prompt and the other as a target. For VC, we extract the semantic tokens from the Hubert with the K-means model.
• Zero-shot SVS / SVC. Different from speech, singing voice requires accurate rhythm and pitch control guided by MIDI representation, where the fundamental frequency F0 and phone-level duration are given respectively in semantic and acoustic modeling. In practice, F0 could be predicted by a separately-trained neural network provided MIDI score, and thus we directly take the F0 value as condition signals for simplification following (Liu et al., 2022).

heading: 3.3 Multitask Learner
text: As illustrated in Table 1, Make-A-Voice presents strong controllability with flexible conditioning as a unified voice synthesis framework, where MakeA-Voice is trained with a combination of semantic and acoustic modeling tasks across speech/singing voice modalities by adjusting prompts. We signal to the model which task it should perform on a given input by prefixing the information with a
MIDI / TEXT
[ S T AA1 R ]
Singing VoiceAcoustic Features
Fundamental frequency
tag specifying the task, referring to 3.4. Make-AVoice, as a multitask learner, exhibits the following competitive advantages:
• Towards a general-purpose interface across various voice generation tasks.
• Cross-task knowledge sharing: multi-task learning improves overall performance by transferring and sharing common knowledge (e.g., multiquantization codec modeling).
• Generalization to new tasks: it presents incontext learning abilities by performing tasks Make-A-Voice is not explicitly trained on, such as cross-lingual timbre transferring, generating coherent emotion, and noise continuations.
As expected, Make-A-Voice demonstrates the outperformed audio quality and style similarity in zero-shot speech and singing voice synthesis, where a complex task is broken down into more manageable pieces. We refer the reader to Section 5.3 for a detailed analysis of our findings.

heading: 3.4 Multilingual Learner
text: With large-scale training data and powerful models, speech models can now generate high-quality samples with unseen styles (e.g., timbre, emotion, and prosody) derived from an acoustic reference (i.e., custom voice). However, replicating this success is a significant challenge for low-resource languages due to data scarcity.
To alleviate it, Make-A-Voice includes richresource language and trains a model on a mixture of arbitrarily multilingual voice, where 1) limited low-resource language data connects its text to semantic meanings, and 2) a large amount of rich-resource language data contains many speakers with various accents, diverse demographics, and heterogeneous recording conditions, which introduces a variety of acoustic conditions on top of semantic meanings with different conditioning mechanisms. As such, voice LLMs demonstrate capturing acoustic diversity (speaker identity, emotion, prosody) in zero-shot scenarios, even for lowresource languages. We refer the reader to Section 5.3 for a summary of our findings.
We signal the model which language to perform on a given input by prefixing the input with a tag specifying the task and language. For example, to query the model to perform text-to-semantic translation on an utterance in English, the tokenized input would be preceded by the two tags [En] [T2S]. To enable the model to be cross-lingual, we employ multilingual HuBERT and codec models to respectively extract the semantic and acoustic discrete representations, which are pre-trained on human voice in multiple languages.

heading: 3.5 Scalable Architecture
text: Recent research (Agostinelli et al., 2023; Kreuk et al., 2022) leverages the transformer architecture (Vaswani et al., 2017) for improving scalability and proposes to represent audio signals as multiple streams nq of discrete tokens and flatten these codes to the length of T × nq, where T is the number of frames. It comes at a high computational cost for extremely long sequences due to the quadratic cost of self-attention and large feedforward networks per position.
To alleviate it, Make-A-Voice (denoted as θAR) predicts long sequences with end-to-end differen-
tiable multiscale transformers similar to Yu et al. (2023); Yang et al. (2023). This enables subquadratic self-attention, unlocking better performance at reduced cost for both training and generation. As illustrated in Figure 2: 1) the token embedding matrix EG maps integer-valued tokens x0..T to m dimensional embeddings, and concatenate with continuous speech representation in time axis (if any), following which 2) we chunk it into patches of size P of length K = TP , 3) a large global transformer θglobalAR module outputs patch representations G1:Ko = θ global AR (G 0:K−1 i ), and 4) a small local transformer module operates on a single patch containing P elements, each of which is the sum of an output from the global model and an embedding of the previous tokens, and autoregressively predict the next patch L1:Ko = θ local AR (L 0:K−1 i +G 1:K o ).
Make-A-Voice presents the improvements primarily from scaling the models’ size in depth and width without the requirement of scattered modelspecific methodologies. As expected, scaling the model size (160M (base), 520M (medium), and 1.2B (large) parameter) results in better scores. We refer the reader to Section 5.3 for our findings.

heading: 3.6 Reconstructing High-Fidelity Waveforms
text: We train a unit-based neural vocoder from scratch for the acoustic unit to waveform generation. Inspired by BigVGAN (Lee et al., 2022), the synthesizer includes the generator and multi-resolution discriminator (MRD). The generator is built from a set of look-up tables (LUT) that embed the discrete representation and a series of blocks composed of transposed convolution and a residual block with dilated layers. The transposed convolutions upsample the encoded representation to match the input sample rate. Details are included in Appendix C.2.

heading: 4 Training and Evaluation

heading: 4.1 Dataset
text: Table 2 lists the used datasets with six languages, with English (En), Chinese (Zh) (totally ∼200k hours) as rich-resource language settings and four languages French (Fr), German (De), Spanish (Es), Japanese (Ja) (totally ∼80 hours) as lowresource settings. Overall, we have ∼200k hours of 16 kHz audio as training data. For text sequence, we tokenize it into the phoneme sequence with an open-source grapheme-to-phoneme conversion tool (Sun et al., 2019). During the evaluation, we randomly choose sentences to construct the zeroshot testing set for each application task, in which the voice used for prompting is never seen by the model at training, and it has to reproduce the characteristics from a single prompt example. We have attached detailed data configuration in Appendix A.

heading: 4.2 Evaluation Metrics
text: Intelligibility and accuracy. We employ word error rate (WER) to evaluate the intelligibility of the generated speech by transcribing it using a wav2vec ASR system. We transcribe the translated speech for accuracy and then calculate the BLEU score (Papineni et al., 2002) between the generated and the reference text. For English-only setups, we use the large model2 pretrained and fine-tuned
2https://github.com/facebookresearch/ fairseq/blob/main/examples/wav2vec/README.md#
on Libri-Light and Librispeech on 16kHz sampled speech audio. For multilingual settings, we use ASR models publicly released on HuggingFace following (Duquenne et al., 2022).
Style quality and similarity. Speaker similarity score (SIM) assesses the coherence of the generated speech in relation to the speaker’s characteristics, which is calculated as the cosine similarity between the speaker embeddings of the generated speech and the desired speech signals. F0 Frame Error (FFE) measures the timbre and prosody similarity of synthesized and reference audio, respectively.
Subjective evaluation. We also conduct a crowd-sourced human evaluation via Amazon Mechanical Turk, which is reported with 95% confidence intervals (CI), and analyze two aspects: style similarity (speaker, emotion, and prosody) and audio quality (clarity, high-frequency), respectively scoring SMOS and MOS. More information has been attached in Appendix D.

heading: 4.3 Baseline
text: We compare the generated audio samples with other systems, including 1) GT, the ground-truth audio; 2) YourTTS (Casanova et al., 2022), GenerSpeech (Huang et al., 2022b), VALL-E (Wang et al., 2023a) for English zero-shot TTS; 3) YourTTS (Casanova et al., 2022) for zero-shot multilingual TTS; 4) NANSY (Choi et al., 2022) and PPG-VC (Liu et al., 2021) for VC; 5) Diffsinger (Liu et al., 2022) and FFT-Singer for SVS;

heading: 4.4 Model Configurations
text: For semantic representations, we apply XLSR53 pre-trained on 56k hours of speech in 53 languages (Conneau et al., 2020) and use k-means to discretize 12th-layer embeddings into semantic tokens with a codebook of size 1000 and a total downsampling rate of 320. For acoustic representation, we train the SoundStream model with 12 quantization levels, each with a codebook of size 1024 and the same downsampling rate of 320. We take three quantization levels as the acoustic tokens, representing each frame as a flat sequence of tokens from the first, second, and third quantization layers. We trained three sets of Make-A-Voice, with 160M (base), 520M (medium), and 1.2B (large) parameters. As for the unit-based vocoder, we use the modified V1 version of BigVGAN. A comprehensive table of hyperparameters is available in
pre-trained-models
Appendix B. Except explicitly stated, we use our 520M (medium) model for downstream evaluation.
During training, we train Make-A-Voice for 100K steps using 8 NVIDIA V100 GPUs with a batch size of 6000 tokens for each GPU on the publicly-available fairseq framework (Ott et al., 2019). Adam optimizer is used with β1 = 0.9, β2 = 0.98, ϵ = 10
−9. S3 model is optimized with a segment size of 8192 and a learning rate of 1× 10−4 until 500K steps using 4 NVIDIA V100 GPUs. For sampling, we employ top-p (Holtzman et al., 2019) sampling with p = 0.25.

heading: 5 Results And Analysis

heading: 5.1 Make-A-Voice as Multitask Learners
text: Zero-shot Text-to-Speech. 1) For the intelligibility of the generated speech, Make-A-Voice has achieved a WER of 6.7, comparable with other systems, indicating that Make-A-Voice could generate accessible speech of good quality as previous nonautoregressive TTS families. 2) For audio quality, Make-A-Voice has achieved the highest MOS with scores of 4.04 compared with the baseline models, demonstrating the effectiveness of the vocoder in generating high-fidelity waveforms. 3) Regarding style similarity, Make-A-Voice scores the SIM of 0.85, showing that Make-A-Voice surpasses the state-of-the-art models in transferring the style of custom voices. Informally, Make-A-Voice is optimized in a large amount of self-supervised data, which contains many speakers with various accents, diverse demographics, and heterogeneous recording conditions, to improve robustness and generalization in zero-shot scenarios.
Using the examples provided on its demo page, we also compare Make-A-Voice with VALL-E in a small-scale subjective test. We synthesize utterances using the same transcripts and prompts and conduct the objective and subjective test with the same protocol described above. Table 3 shows that, in these examples, Make-A-Voice obtains 1.5 lower in WER and 0.04 higher in SIM than baseline models in zero-shot synthesis.
Singing Voice Synthesis. Table 5 demonstrates that Make-A-Voice (SVS) outperforms the baseline system by a large margin in terms of pitch similarity, showing distinct 70%/30% superiority over FFT-Singer/DiffSinger in terms of FFE objective evaluation. Make-A-Voice can resemble the note prompt and demonstrates its precise pitch reconstruction. Regarding singer similarity, Make-
Model TTS-WER TTS-SIM VC-SIM
Base 9.8 0.84 0.75 Medium 8.3 0.86 0.76 Large 6.1 0.87 0.76
Table 4: LJSpeech results for different model sizes, namely 160M (base), 520M (medium), and 1.2B (large) parameter models. We investigate voice large language models as scalable learners.
Model MOS (↑) SMOS (↑) SIM (↑) FFE (↓) GT 4.08±0.08 / / / FFT-Singer 3.86±0.05 3.91±0.08 0.66 0.12 Diffsinger 3.96±0.07 3.94±0.07 0.67 0.11 Make-A-Voice (Zero-shot) 3.99±0.06 3.96±0.05 0.78 0.08
Table 5: SVS. Note that FFT-Singer and Diffsinger conduct in-domain generation with seen speaker while Make-A-Voice presents zero-shot SVS.
Model MOS (↑) SMOS (↑) SIM (↑) Voice Conversion Prompt 4.26±0.06 / / NANSY 3.89±0.08 3.73±0.10 0.68 PPG-VC 3.97±0.06 3.82±0.05 0.78 Make-A-Voice (Zero-shot) 4.02±0.08 3.78±0.06 0.80 Singing Voice Conversion Prompt 4.21±0.05 / / Make-A-Voice 3.96±0.06 3.72±0.05 0.76
Table 6: Zero-shot VC and SVC.
A-Voice scores the highest SIM of 0.78, surpassing the state-of-the-art models in transferring the style of custom singing voices in zero-shot scenarios even though the voice used for prompting is never seen at training.
Voice Conversion and Singing Voice Conversion. Table 6 shows that Make-A-Voice scores the comparable overall SIM of 0.93 with baseline. It excels at converting speaker identity even in a zero-shot scenario, attributing to the scalable training data covering diverse speakers with various accents. For audio quality, it presents high perceptual quality with outperformed MOS evaluation. To conclude, Make-A-Voice converts the timbre with better naturalness and comparable speaker similarity to baseline models, even though the model is trained without any text transcript paired with audio recordings. For singing voice conversion (SVC), Make-A-Voice also excels at converting singer identity and presents good perceptual quality and naturalness.

heading: 5.2 Make-A-Voice as Multilingual Learners
text: Table 7 presents cross-lingual zero-shot TTS results, where the audio context and the target text are in different languages. For each target text, we sample one 3-second-long audio context from each language, which creates language transfer directions in total. Compared with YourTTS, Make-A-Voice yields better results in most languages, obtaining lower WER and higher SIM averaged across audio contexts. Regarding low-resource language, Make-A-Voice presents potential improvement for
the limited usage of training data at this time.

heading: 5.3 Analysis and Ablation Studies
text: To verify the emerging capabilities of Make-AVoice as scalable multilingual and multitask learners, we conduct ablation studies and discuss the key findings as follows. In this section, we first analyze the model scalability, then investigate the benefits of multilingual and multitask training, and finally explore its generalization to unseen tasks.
Scalability to improve performance. Table 4 reports LJSpeech results for different model sizes, namely 160M (base), 520M (medium), and 1.2B (large) parameter models. As expected, scaling the model size results in better scores. However, this comes at the expense of longer training and inference time. Increasing the model size from 520M to 1.2B leads to additional gains of a further 40% reduction in WER for TTS tasks with a similar style similarity.
Multilingual Learning to alleviate data scarcity of low-resource languages. Data scarcity is a significant challenge to replicating the success of voice LLM for low-resource languages. To verify the effectiveness of multilingualism in alleviating data scarcity, we train Make-A-Voice using 16hrs De data and another one with a combination of six-language data; Make-A-Voice leverages a joint vocabulary, where rich-resource language data contains many speakers with various accents, diverse demographics, and heterogeneous recording conditions, introducing a variety of acoustic conditions on top of semantic meanings, especially
in low-resource languages. Table 8 shows the improved performance with the combination of arbitrarily multilingual voice, leading to the 7.9 reduction in WER and 0.2 SIM gain for text-to-speech synthesis.
Multitask learning for cross-task knowledge sharing. As illustrated in Table 8, we observe that joint training with speech tasks has witnessed the gains of 0.09 improvement in MOS and a 0.17 point improvement in SIM for singing voice synthesis. Since common knowledge can be shared across tasks (such as multi-layer quantization codec modeling), SVS distinctly benefits from large-scale speech tasks.
Multitask learning for generalizing to tasks not explicitly trained on. Besides quantitative results, we present in-context learning abilities by
performing tasks not explicitly trained on, including cross-lingual timbre transferring, generating coherent emotion, and noise continuations. We have attached the information on testing data in Appendix A. As shown in the demo page, we find that 1) Make-A-Voice can preserve the emotion in the prompt at a zero-shot setting, even if the model is not fine-tuned on an emotional TTS dataset; 2) Make-A-Voice effectively reproduces the characteristics from a cross-lingual style prompt, which has not been seen during training; and 3) In a noisy environment, the model also presents the acoustic consistency and maintain the noise conditions from the prompt.

heading: 6 Conclusion
text: To bridge the gap where multiple tasks and languages are less exploited in voice LLMs compared to GPTs, we introduced Make-A-Voice, a multimodal LLM to synthesize and manipulate voice signals. Make-A-Voice took self-supervised tokens first to determine the semantic meaning and then learned acoustic information given condition signals, and we conducted a comprehensive study on its capabilities to deal with multiple tasks and languages. When trained on ∼200K hours of 6- language data for 4 voice generation applications, Make-A-Voice emerged notable advantages as 1) as scalable learners to improve performance with end-to-end differentiable local and global multiscale transformers; and 2) as multitask learners to share common knowledge across modalities (speech/singing) and presented in-context learning abilities by performing tasks not explicitly trained
on; 3) as multilingual learners to alleviate data scarcity of low-resource languages by including rich-resource language training data. Experimental results demonstrated that Make-A-Voice achieved state-of-the-art results in monolingual/cross-lingual zero-shot voice generation. The subjective and objective evaluation showed that Make-A-Voice exhibited superior audio quality and similarity compared with baselines.

heading: 7 Limitation and Potential Risks
text: Although Make-A-Voice as a voice LLM is successfully applied to multilingual zero-shot voice signals at scale, it still suffers from some limitations: 1) Make-A-Voice introduces a strong dependency on the quality of the audio tokenizer. 2) The model only shows in-context learning ability on voice synthesis, rather than all voice recognition and understanding tasks, and 3) a longer sequence length typically requires more computational resources, and degradation could be witnessed with decreased training data.
The low-resource scenario refers to the lack of labeled training data, and we construct a total of around 80 hours of labeled data for low-resource downstream tasks. We leave the study of truly resource-limited language LLMs for future works.

heading: Acknowledgements
text: This work was supported in part by the National Natural Science Foundation of China under Grant No. 62222211.

heading: A Data
text: In this section, we describe details of the data usage in training and evaluating Make-A-Voice.
• Common Voice (Ardila et al., 2019) consists of text paired with recordings where people were asked to read the text aloud.
• Librilight (Kahn et al., 2020) contains 60K hours of unlabelled speech from audiobooks in English, and LibriSpeech (Panayotov et al., 2015), LibriTTS (Zen et al., 2019), Gigaspeech (Chen et al., 2021), AISHELL (Fu et al., 2021), VCTK (Veaux et al., 2017) datasets include transcriptions.
• CSD (Choi et al., 2021b) contains multilingual singing voice. We also use the female-singer OpenCPOP (Wang et al., 2022), multi-singer dataset OpenSinger (Huang et al., 2021), and M4Singer (Zhang et al., 2022a) as the singing voice data.

heading: B Model Configurations
text: We list the model hyper-parameters of Make-AVoice in Table 9.

heading: C Applications
text: C.1 MIDI-to-F0 Converter
Singing voice synthesis (SVS) is a task that generates singing voices from the given music score and lyrics like human singers. Following (Liu et al., 2022; Zhang et al., 2022a), the SVS system typically includes the MIDI-to-F0 converter to predict F0 explicitly. Though the SVS system can be further improved with the direct MIDI condition and implicit F0 prediction, this is beyond our focus.
C.2 Unit-based Vocoder
The generator of the unit-based vocoder is built from a set of look-up tables (LUT) that embed the discrete representation, and a series of blocks composed of transposed convolution and a residual block with dilated layers. We train the enhanced vocoder with the weighted sum of the least-square adversarial loss, the feature matching loss, and the spectral regression loss on mel-spectrogram, where the training objective formulation and hyperparameters follow Kong et al. (2020); Lee et al. (2022).
For speech generation, we train the vocoder with only the discrete unit sequences as input. For singing voice generation, we further include F0-driven source excitation to stabilize longcontinuous waveforms generation following (Liu et al., 2022; Huang et al., 2022a).
As illustrated in Table 10, replacing the unitbased vocoder with a SoundStream decoder for voice synthesis has witnessed a distinct degradation of perceptual quality, proving that the codebook mismatch for the SoundStream decoder between training (12 quantization levels) and inference (3 levels) hurts reconstruction performance. In contrast, a neural vocoder could refine the coarsegrained acoustic tokens and generate waveforms with increasing details.

heading: D Evaluation
text: D.1 Subjective Evaluation
For audio quality evaluation, we conduct the MOS (mean opinion score) tests and explicitly instruct the raters to “(focus on examining the audio quality and naturalness, and ignore the differences of style (timbre, emotion, and prosody).)". The testers present and rate the samples, and each tester is asked to evaluate the subjective naturalness on a 1-5 Likert scale.
For style similarity evaluation, we explicitly instruct the raters to “(focus on the similarity of the
style (timbre, emotion, and prosody) to the reference, and ignore the differences of content, grammar, or audio quality.)". In the SMOS (similarity mean opinion score) tests, we paired each synthesized utterance with a ground truth utterance to evaluate how well the synthesized speech matches that of the target speaker. Each pair is rated by one rater.
Our subjective evaluation tests are crowdsourced and conducted by 20 native speakers via Amazon Mechanical Turk. The screenshots of instructions for testers have been shown in Figure 5. We paid $8 to participants hourly and totally spent about $600 on participant compensation. A small subset of speech samples used in the test is available at https://M-Voice.github.io/.
D.2 Objective Evaluation
Cosine similarity is an objective metric that measures speaker similarity among multi-speaker audio. We compute the average cosine similarity between embeddings extracted from the synthesized and ground truth embeddings to measure the speaker
similarity performance objectively. Word Error Rate (WER) evaluates the faithfulness to the input transcript by transcribing the synthesized utterances using a wav2vec ASR system.
F0 Frame Error (FFE) combines voicing decision error and F0 error metrics to capture F0 information.
Mel-cepstral distortion (MCD) measures the spectral distance between the synthesized and reference mel-spectrum features.
Short-time objective intelligibility (STOI) assesses the denoising quality for speech enhancement.

heading: Future Work
text: We leave the study of truly resource-limited language LLMs for future works