Metis: A Foundation Speech Generation Model with Masked Generative Pre-training
Authors: Yuancheng Wang
Jiachen Zheng
Junan Zhang
Xueyao Zhang
Huan Liao
Zhizheng Wu
Venue: NeurIPS 2025 poster ()
Keywords: Speech Generation
Foundation Model
Masked Generative Model
Primary area: applications
TL;DR: We propose a foundation model for unified speech generation with masked generative pre-training.

Abstract: We introduce ***Metis***, a foundation model for unified speech generation.
Unlike previous task-specific or multi-task models, Metis follows a pre-training and fine-tuning paradigm. It is pre-trained on large-scale unlabeled speech data using masked generative modeling and then fine-tuned to adapt to diverse speech generation tasks.
Specifically, 
(1) Metis utilizes two discrete speech representations: SSL tokens derived from speech self-supervised learning (SSL) features, and acoustic tokens directly quantized from waveforms. 
(2) Metis performs masked generative pre-training on SSL tokens, utilizing 300K hours of diverse speech data, without any additional condition. 
(3) Through fine-tuning with task-specific conditions, Metis achieves efficient adaptation to various speech generation tasks while supporting multimodal input, even when using limited data and trainable parameters.
Experiments demonstrate that Metis can serve as a foundation model for unified speech generation: Metis outperforms state-of-the-art task-specific or multi-task systems across five speech generation tasks, including zero-shot text-to-speech, voice conversion, target speaker extraction, speech enhancement, and lip-to-speech, even with fewer than 20M trainable parameters or 300 times less training data. Audio samples are are available at https://metis-demo.github.io/. We release the code and model checkpoints at https://github.com/open-mmlab/Amphion.

Metis: A Foundation Speech Generation Model with Masked Generative Pre-training
Yuancheng Wang, Jiachen Zheng, Junan Zhang, Xueyao Zhang, Huan Liao, Zhizheng Wu
The Chinese University of Hong Kong, Shenzhen
yuanchengvang@link. cuhk. eduk.edu.cce edu.cn. en, en, wuzhizhengoouhkk.o edw.:cccckkkuc.......... edu.
Abstract
We introduce Metis, foundation model for unified speech generation. Unlike previ- ous task-specific or multi-task models, Metis follows pre-training and fine-tuning paradigm. It is pre-trained on large-scale unlabeled speech data using masked generative modeling and then fine-tuned to adapt to diverse speech generation tasks. Specifically, 1) Metis utilizes discrete speech representations: SSL tokens derived from speech sule-supe learning (SSL) features, and acoustic tokens directly quantized from waveforms. Metis performs masked generative pre- training on SSL tokens, utilizing 300K hours of diverse speech data, without any additional condition. 3) Through fine-tuning with task-specific conditions, Metis achieves efficient adaptation to various speech generation tasks while supporting multimodal input, even when using limited data and trainable parameters. Experi- ments demonstrate that Metis can serve foundation model for unified speech generation: Metis outperforms state-of-the-arr tasc-spec or multi-task systems across five speech generation tasks, including zero-shot text-to-speech, voice con- version, target speaker extraction, speech enhancement, and lip-to-speech, even with fewer than 20M trainable parameters or 300 times less training data. Audio samples are are available at attps://metas demo github. io/. We release the code and model checkpoints https: :// http//ggttttttt//// com/ /open-ee--eee mmlab/ haahh/mmionn...
Introduction
Advancing unified framework capable of addressing diverse tasks is central research objective within the domain of artificial intelligence. In natural language processing [1, 2, 3] and computer vision [4, 5, 6), foundation models leveraging large-scale self-supervised pre-training have demon- strated remarkable adaptability across a wide spectrum of downstream tasks. However, in the domain of speech generation, this potential remains underexplored. A unified speech can integrate various speech generation technologies, such as text-to-speech [7, 8, 9, 10, 11, 12], voice conver- sion [13, 14, 15, 16], and speech enhancement [17, 18, 19]. This integration reduces redundant development, facilitates broader applications across diverse domains, and enhances the efficiency of human-machine interaction.
Previous speech generation models are mostly expert models that require extensive speciice designs [7, 20, 21, 22, 23, 24, 25]. UniAudio [26] and SpeechX [27] are pioneering works that attempt to use autoregressive language models for multiple speech generation tasks. However, they require large amount of paired training data for each task and face challenges in extending to new tasks based on pre-trained models. In addition, the autoregressive approach leads to suboptimal results in certain tasks and is relatively inefficient.
In this paper, we address the research question of how to design a unified speech generation framework that leverages large-scale unlabeled speech data for pre-training and efficiently adapts to diverse speech generation tasks through fine-tuning. Inspired by previous two-stage speech
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

condliss sanglon,
Masked Generative Transfermer
Masked Generative Transformer
O
O O
Fremelew
Masked Gonerativ
Manked Generative Transfernee
101
Exddediigg
Transfornee
Aqiippok....
asdaleeed Spench
(a) Two-stage generation.
(b) Pre-training.
(c) Fine-tuning.
Figure 1: An illustration of Metis. (a) provides an overview of the two-stage speech generation framework, which consists of task-specific (yellow block) and task-independent (light blue block) processes. In this work, we focus on developing a pre-training model for the first stage, illustrated in (b). (c) demonstrates the fine-tuning process, where the pre-trained model is adapted to specific tasks.
generation models [28, 29, 30, 32], we revisit the speech generation process and observe a common structure underlying most tasks: generating intermediate representations from task-specific conditions and synthesizing acoustic representations from these intermediates. The intermediate representations are typically discrete units quantized from self-supervised speech features [33, 34, 35, 36, 37] that encode semantic and prosodic information, while acoustic representations are easily used to reconstruct high-quality waveforms.
Building this observation, we propose unified framework that modularizes speech generation tasks as a two-step process, as illustrated in Figure la: 1) Task-Specific Process: Generating SSL tokens conditioned on task-specific conditions (e.g.. text for text-to-speech, noisy speech for speech enhancement, or visual features for lip-to-speech). 2) Task-Independent Process: Generating acoustic representations from the SSL tokens. This formulation allows diverse tasks to be addressed by varying input conditions while maintaining a consistent generation mechanism. Previous studies [12, 28, 30, 31, 32, 38, 39, 40] have established that the second stage of acoustic refinement can be achieved effectively using fully self-supervised diffusion models [31, 32, 38, 39, 40] or language models [12, 28, 30]. Therefore, our objective is simplified to designing a generative pre-training mechanism for the first stage, enabling the pre-trained model to effectively generalize across various speech generation tasks. this work, we adopt masked generative models [41] for unconditional pre- training. Masked generative models are trained using a straightforward masked token prediction approach and employ an iterative sampling strategy to generate outputs from a fully masked input. We provide more details about masked generative models in Section 3.1. Once pre- trained, the model can be fine-tuned with limited data and parameters by incorporating specifci conditions as additional inputs to adapt to various speech generation tasks.
Based on the above discussion, we propose Metis, a foundation model for unified speech generation. Specifically, Metis has the following key features:
Masked Generative Pre- Pre-Training: Metis performs masked generative pre-training on SSL tokens using large-scale unlabeled speech data without any task-specific condition, as illustrated in Figure 1b. This pre-training phase establishes strong foundation, allowing Metis efficiently adapt to various downstream tasks through fine-tuning with minimal task-specific data.
Efficient A Adaption to Various Speech Generation Tasks: Metis can be efficiently adapted to wide range of speech generation tasks, including zero-shot text-to-speech, voice conversion, speech enhancement, target speaker extraction, and lip-to-speech, illustrated in Figure 1c. The fine-tuned models achieve ofate-ofthh-ee art results, even when using fewer than 20M trainable parameters or significantly less training data.
Metis supports multimodal conditional inputs during the fine-tuning phase, including text, audio, and video. This capability solidifies Metis foundation model for supporting various speech generation tasks. We also explore multi-task fine-tuning, showing that the pre-trained model can be efficiently adapted into powerful multi-task model with minimal modification, enabling novel applications such as text-guided target speaker extraction with task combinations.
Support for Multimodal Conditional Inputs:

---

2 Related Work
Masked Generative Models for Speech Masked generative models (MGMs) are a family of generative models that typically employ -on-autoreeressive transformers [42]. These models have achieved significant success, demonstrating performance comparable to or even surpassing autore- gressive and diffusion models in image [41, 43, 44] and video [45, 46] generation, while offering better balance between quality and speed. In the speech domain, SoundStorm [30] uses the semantic tokens from AudioL. [28] and employs MGMs to generate acoustic tokens from neural audio codec [47], enabling applications like TTS and voice conversion. NaturalSpeech 3 [10] adopts MGMs generate disentangled speech tokens. MaskGGT [12] further leverages MGMs for zero-shot gener- ation, eliminating the need for explicit text-speech alignment or phone-level duration prediction in non-autoregressive TTS models. MaskSR [48] applies MGMs speech enhancement tasks. In this work, we propose a unified speech generation framework based on MGMs.
Unified Speech Generation Developing a unified framework capable of handling various tasks is a key research objective in artificial intelligence. In the field of speech generation, UniAudio [26] em- ploys an LLM for next-token prediction to generate multiple types of audio. Similarly, SpeechX [27] leverages an LLM for unified zero-shot tasks such as TTS, noise suppression, and target speaker ex- traction. Both models achieve this by concatenating the condition and target speech tokens, followed by AR modeling. However, these models require large amounts of paired training data for each task, failing to leverage the vast amount of unlabeled speech data effectively. VoiceBox [11] employs flow matching to unify tasks such as zero-shot TTS, speech editing, and speech enhancement. However, has notable limitations, such as requiring text and clean speech as references for speech enhancement and relying on phone durations during training for zero-shot TTS. Its successor, AudioBox [49], ex- tends VoiceBox to unified audio generation with natural language prompt control. Our work is partly inspired by SpeechFlow [50], which uses flow matching [51] to learn infilling during pre-training and fine-tunes with task-specific conditions for various speech generation tasks, such as zero-shot TTS and speech separation. However, it is limited to frame-level conditions, such as requiring a frame-level phoneme sequence for TTS. Additionally, predicting mel-spectrograms directly during pre-training may be suboptimal due to the need to predict extensive acoustic details.
Speech discrete representation is also highly relevant to our work, and we have included this part in
Appendix F.
3 Method
3.1 Background: Masked Generative Models
In this section, we provide brief introduction to masked generative models [12, 41, 46). Consider a discrete sequence x = [y1.yy...... ....... ynl.
where n denotes the length of the sequence. We define x, m, the operation of masking subset of tokens in x using the corresponding binary mask m, = =[m,,1, m4.2..... n, Specifically, this operation involves replacing x; with a special [MASK] token if m,, = I, and otherwise leaving x; unmasked if m,,i O. Here, each m.,i independently and identically distributed according to Bernoulli distribution with parameter (r), where y(r) E (0. 1] represents mask schedule function (for example, yt) sin( / ), (0. T)). We denote x xo- The masked generative models are trained to predict the complete sequence (masked tokens) based on the observed tokens (unmasked tokens) and the condition c, which can be modeled as (o xx, c). and the model parameters 0 are trained to optimize the sum of the marginal cross-entropy for each unobserved token:
Lmaas -Bx, Σ M log Po | x,,c)
(I)
Note that may be empty, for example, during the unconditional pre-training stage of our model. At the inference stage, masked generative models generate tokens in parallel through iterative decoding. The process begins with a fully masked sequence xy. Assuming the total number of decoding steps is S, for each step j from I to S, we first sample &n from pa(xo x-- c)). Then we sample
[n y(' j - } tokens based on the confidence score to remask, resulting in xy-j.T- where is the sequence length of x. The confidence score for y, in &o is assigned to pe XT-()-1) c.c)
3

---

YoD- is [MASK] token: otherwise, we set the confidence score of 81 to 1, indicating that tokens already unmasked ( will not be remasked. In particular, we choose [n- tokens with the lowest confidence scores to be masked. Note that the method for calculating the confidence score is not unique. For example, Lezama et al. [52] introduces Token-Critic, training critic model to compute confidence scores, aiding the sampling process. Additionally, Xie et al.
[44], Lezama et al. [52] suggest that masked generative modeling can be seen as a simplified version of discrete diffusion.
3.2 Overview of Metis
Figure provides an overview of our system, which adopts two-stage speech generation process for unified speech generation tasks. We first briefly introduce the distinct speech discrete representations used in the two stages in Section 3.3. Then, we present the pre-training (Section 3.4) and fine-tuning (Section 3.5) strategies for the first stage, which form the core of our work. Finally, we provide a brief overview of unified acoustic decoder for all speech generation tasks in Section 3.6.
3.3 Discrete Representations for Two-Stage Generation
Metis employs two discrete speech representations for the two-stage generation, as illustrated in Figure 2. 1) SSL tokens: Derived from SSL features of large-scale speech self-supervised learning models [33, 34, 36, 37], SSL tokens encapsulate both semantic and prosodic information, making them well-suited for conditional generation. To minimize information loss, we employ vector quantization (VQ) model [53, 54] to quantize SSL features into discrete tokens, following Wang et al. [12]. 2) Acoustic tokens: Directly obtained from the waveform via vector quantization. The goal is to preserve all the information from the speech to reconstruct a high-quality waveform. We show more details in Appendix A.
3.4 Masked Generative Pre-training with SSL Tokens
Based on the previous discussion, most speech genera- tion tasks can be generalized into two stages: conditions to SSL tokens and SSL tokens acoustic tokens. The primary distinction among tasks lies in the nature of the conditions. To address this, we propose a unified pre- trained model for the first stage, which can be adapted to various tasks.
feutures
M
Decoder
Decoder
VQ
We use an unconditional masked generative model on SSL tokens for pre-trate graitrainng. Specifically, we randomly mask tokens in the SSL token sequence xss using the strategy outlined in Section 3.1 and predict the masked tokens. We introduce a prompt sequence with the
Encoder
RVQ
features
Encoder
probability p to further enhance the in-context learn- ing ability of the model. With this probability, a pre- es! fix sequence prompt from the SSL token sequence is used as a prompt and remains unmasked. This mech- anism enables the model to leverage prompt informa- tion, thereby improving its adaptability to downstream tasks requiring prompts, such as zero-shot TTS and tar- get speaker extraction. The pre-training objective is to model p |x>>| x sxl .xproopp)
SSL Model
M
Figure 2: Two discrete speech represen- tations for the two-stage speech genera- tion: SSL tokens (left) and acoustic tokens (right).
Our design is motivated by the observation that models trained on extensive data can recover masked SSL tokens from prompts and unmasked tokens, even without task-specife conditions, as shown in other domains [2, 4]. Empirically, the pre-trained model can generate speech that mimics the prosodic style and timbre of prompt. However, the generated speech often lacks intelligibility, producing random word concatenations due to the absence of semantic guidance. This highlights the need for task-specific conditions, which can be efficiently incorporated through tuning adapt the model to various speech generation tasks.

---

3.5 Efficient Adaptation to Various Generation Tasks
Now, we describe how to efficiently adapt the pre-trained model to various speech generation tasks. We first categorize the conditions for different speech generation tasks into two types: non-frame-level conditions and frame-level conditions. For the former, such as TTS, the condition is a phoneme sequence or text token sequence. In this case, the model needs to implicitly learn the alignment between the condition and the SSL token sequence. For the latter, such as voice conversion or speech enhancement, the conditions (c.g., the source speech voice conversion or the noisy speech for speech enhancement) can be aligned with the target SSL token sequence at the frame level. Based on these distinctions, during fine-tuning: for non-frame-level conditions, we simply concatenate the condition with the input sequence along the time dimension. For frame-level conditions, we apply a simple interpolation to align the condition with the input sequence in the time dimension and then pass it through an MLP-based adapter before adding it to the input. Then, the fine-tuned model is trained learn p |x demonstrate that the pre-trained model can efficiently adapt to various tasks and achieve remarkable results with minimal data. Additionally, explored the use of Low-Rank Adaptation (LoRA) [55], which allows fine-tuning with small number of trainable parameters. More details are provided in Section 4.
.xprmppr .x prompr- c), where is the task-specific condition. Our experiments
3.6
Masked Generative Acoustic Decoder
We train -SL-ss-ccoustic model based masked generative modeling, which serves as a unified acoustic decoder for all speech generation tasks. The model is trained to recover masked tokens from masked acoustic token sequence x," ", conditioned on SSL tokens xss/ and prompt acoustic tokens x prompr' This can be formulated as Po( x, prompr- xstt). During training, we randomly select layer for masking in the multi-layered acoustic tokens, while the lower-layer tokens remain unmasked and serve as conditional inputs to the model, following [30]. During inference, we generate acoustic tokens layer by layer. Finally, we generate the speech by first predicting SSL tokens and then generating acoustic tokens.
4
Experiments and Results
4.1 Setup
Model Architecture We adopt the same model architecture as described Wang et al. [12]. except removing the text embedding during the pre-training phase. The model follows the standard Llama-style architecture [42, 56], but replaces causal attention with bidirectional attention.
Dataset We use dataset consisting of 300K hours of speech for pre-training, including 100K hours from the Emilia [57] dataset and an additional 200K hours self-collected through the Emilia pipeline, to train our pre-trained model. The dataset contains diverse range of wilthe-wild multilingual speech data, including 87K hours in Chinese, 185K hours in English, 7K hours in German, 8K hours in French, 2.5K hours in Japanese, and 7.5K hours in Korean. The dataset used for fine-tuning is sampled from the pre-trained dataset.
Training We pre-train our model on GPUs for a total of 1200K steps. We use the AdamW [58] optimizer with learning rate of 1e-4 and 32K warmup steps. We employ a dynamic batch size, where each batch contains 10K tokens (200 seconds) per GPU. During training, we randomly select a prefix of the sequence as prompt that is not masked with probability = 0.8. The length of the prompt is uniformly sampled from the range [0%, 40%] of the total sequence length.
Inference We follow the inference strategy outlined in Wang al. [12], with task-specific step adjustments for the fine-tuned model to generate SSL tokens. Additionally, we use classifier-free guidance [59] for tasks that involve prompts.
We use multiple evaluation metrics to assess different aspects of the generated speech, including similarity (SIM), intelligibilitt (WER), and audio quality (DNSMOS, SIG, BAK, OVRL, NISQA). For the subjective metrics, quality mean opinion score (CMOS) and similarity mean
Evaluation Metries
5

---

opinion score (SMOS) are used to evaluate speech quality and similarity. The results of the subjective evaluation are presented in Appendix D.3. Details about these metrics are provided in Appendix
4.2 Results on Different Speech Generation Tasks
We present the fine-tuning results of Metis on various speech generation tasks. These tasks include zero-shot TTS (without frame-level phoneme condition or duration prediction), voice conversion, speech enhancement, target speaker extraction, and the multimodal task of lip-to-speech.
4.2.1 Zero-Shot TTS
Implementation Details For zero-shot TTS, we fine-tune the pre-trained model on three different datasets: 1K hours and 10K hours randomly sampled from the Emilia dataset, and the LibriTTS [62] dataset, which con- tains 0.58K hours of English speech data, We use two fine-tuning methods: full-scale fine- tuning and LoRA tine-tuning. For LoRA fine- tuning, we the rank r 32, resulting in only 32M trainable parameters, including the newly added text embedding module. All fine-tuned models are trained using 4 GPUs for 200K steps.
Table 1: Results on zero-shot TTS task.
Model
Training Data
WERLD
SINCT)
DNSMOS(")
SeafTTS
Ground
2.14
3.53
VALI-F[1]
FN
[50]
3.55
CosyVoice [32]
Muhi
4.08
XTTS-v2 [61]
37K Molii
3.25
0.46
3.45
MaskGCT [12]
Molii
2.47
0.72
Metis-TIS Letatt
Mulii
4.78
0.72
3.47
Mous-IIS TTS uaA
I0K Multi.
4.55
0.77
3.45
Metis TTS LAA
0.58K EN.
4.63
0.70
3.51
Metis TTS
0,58K EN.
3,04
3.47
Motis-TTS
Mulii
Motis-TTS
Motis-TTS petras
Mulii
4.91
Seed7TS testih
Evaluation and Baseline We evaluate zero-shot TTS models using three test sets:
Ground
Cosy Voice [32]
Muhi.
XTTS.22 [6]]
Multi.
2.88
3.44
1) SeedTTS test-en, a test set introduced in Seed-TTS [31] comprising 1,000 samples ex- tracted from English public corpora, including the Common Voice dataset [63]. 2) SeedTTS test-zh, a test set introduced in Seed-TTS con- sisting of 2,000 samples extracted from Chi-
MaskGCT [12]
100K Mulii
2.18
0.77
3.58
Metis-TIS TTS Leta
Multi
5.21
0.77
3.57
Meus-TIS -TI LaRA22
I0K Multi.
4.48
0.77
3.55
Mens-TTS sursww
IK Multi.
4.23
0.77
3.54
MeT bwiee
10K Mult.
2.30
0.77
3.55
Metis-TTS on pretiin
10K Muli.
4.98
0.73
3.51
This wassun of the mostel is Inited on Libe (62)
nese public corpora, including the DiDiSpeech dataset [64]. 3) LibriSpeech test-clean [65]. widely used test set for TTS. We compare our models with several recent zero-shot TTS models, including AR-based models: VALL-E [8], VoiceCraft [60], XTTS-v2 [61], and Cosy Voice [32]; NARebased models: VoiceBox [11], and MaskGCT [12]. Specifically, MaskGCT can be seen as Metis without pre-training. For VoiceBox and NaturalSpeech 3, we only compare with them on LibriSpeech test-clean since they are not open-source. See more details about these baselines in Appendix B.
Result The results are presented in Table 1, with additional results for LibriSpeech test-clean set provided in Appendix D.I. The table shows that Metis-TTS LoRA 32, fine-tuned with only 1K hours of data and 32M trainable parameters, achieves performance on metrics such as WER, SIM, and DNSMOS that is comparable to or exceeds that of some baselines trained on datasets 10 to 100 times larger. Furthermore, when scaled with full fine-tuning on 10K hours of data, Metis demonstrates an improved WER on SeedTTS test-en, outperforming MaskGCT, which was trained on 100K hours of data, while maintaining comparable WER performance on SeedTTS test-zh. When fine-tuned with only 580 hours of LibriTTS data, Metis achieves competitive WER and SIM performance, surpassing all baselines except MaskGCT on SeedTTS test-en. We also compare with models that are not pre-trained but trained for the same number of steps. The results show that the fine-tuned models converge faster and achieve better performance. In addition, all fine-tuned models exhibit excellent SIM and DNSMOS, indicating high speech similarity and quality.
4.2.2 Voice Conversion
Implementation Details Previous voice conversion systems typically extract timbre-independent features from input signals using methods such as information bottlenecks [14, 66, 67]. timbre perturbation [15, 68, 69], or specialized loss functions [10, 67]. However, these approaches often introduce complexity, risk the loss of semantic information, and may still inadvertently retain timbre-

---

related details [70, 71J. In this work, we use simpler yet effective solution inspired by Anastassiou al. [31], Neekhara al. [72]. Specifically, we leverage lightweight voice conversion model' [73] to perform real-time voice conversion on the target speech using randomly sampled prompt speech, thereby achieving timbre perturbation. The perturbed speech features are then used as input to predict the target speech based on the prompt. Specifically, we employ the w2v-bert-2. 2.20 2.0 features of the perturbed speech as conditioning inputs and randomly extract a prefix of the target speech as the prompt. Our experiments demonstrate that with minimal data and training steps, our pre-trained model can effectively and rapidly adapt to the voice conversion task. We use both full-scale fine- tuning and LoRA fine-tuning. For LoRA fine-tuning, we set the rank 16, resulting in only 9M trainable parameters excluding those of the MLP-based adapter. Additionally, we observe that for the voice conversion task, our pre-traiie pre-trained model converges single A100 GPU after just 10K steps of LoRA fine-tuning and 5K steps of full-scale fine-tuning, using randomly sampled 0.4K hours of training data.
Evaluation and Baseline We evaluate our
models on the VCTK [75] dataset. we ran- domly select 200 samples from the dataset as source speech, and for each sample, we ran- domly select another sample from the same speaker as the prompt speech. We also eval- uate the models on other test sets, the results
Table 2: Results voice conversion task.
Model
Training Dabl
WERLD
SIM(T)
DNSMOSIT)
NESQAIT
V77K
Hirfinecche- [2]]
2.8K
4.87
01:38
3.40
3.79
LM-VC [14]]
1.4K
8.35
01:23
3.46
3.93
176)
are shown in Appendix D.2. We compare our models with several recent voice conversion models, including HireSpeech+++ [21], LM-
Meti: VC 1s 4.3
0.4K 0.4K
4.89
0.50
3.48
4.45
Metis VC cg
730
0.35
3,46
4.22
Be LoRA 16 trin onvire A100 GPC fo: IEK Fe Sec-taingg teatisiioo A100 GPG mape.
VC [68]. UniAudio [26]. and Vevo [74]. More
best iesles and undelfeed.
details about these baselines can be found in Appendix B.2.
Result The results are presented in Table 2. The table shows 1) For similarity, our model out- performs all baseline systems, achieving SIM score of 0.55, significantly higher than the best baseline result of 0.38. 2) For WER, Metis- LoRA 16, <lg 0.0 achieves competitive result of 4.49, second only to Vevo. 3) For audio quality, as measured by DNSMOS and NISQA, our model shows competitive performance, with slight improvements over most baseline systems. In addition, our model requires only a significantly small amount of data to fine-tune for voice conversion.
4.2.3 Target Speaker Extraction
Implementation Details For target speaker extraction, we randomly sample 10K hours of data from the pre-training dataset to cre- ate the fine-tuning training set without any filtering. During training, samples are dynam- ically mixed to simulate multi-speaker speech data. Specifically,
Table 3: Results on target speaker extraction task.
Medel
WERLD
SIGIT
BAR::)
OVRL(T)
NISQA())
SEN(T)
Livildic text
4.27
UmiAndio
20.06
3.64
4:33
random prefix from a sample is extracted as the prompt, and the remaining portion is mixed with another ran- domly selected sample by directly adding their waveforms. The w2v-bert-22 2. features of the mixed speech are then used as conditions. We use two fine-tuning methods: full- scale fine-tuning and LoRA tne-tuning, For LoRA fine-tuning, we explore different LoRA ranks r r= 4, 16, 32, resulting in LoRA modules with trainable parameters of 2M, 9M, and 18M, respectively.
YoleeFilerr [12]
20.10
3.27
3.77
2.91
2.97
0.65
WiSco [23]
6.19
3.56
3.93
3.25
4.04
0.73
TSF [M]]
9.30
4.0%
1.39
4.03
Matis TSE
13.57
Motis-THl
12.52
3.66
3,36
Monts-TRl was
9.66
3.66
4.00
3.35
4.38
8.78
Moti- TSE swras
0.31
3.65
30%
3.34
4:35
0.74
ErstleWW
0.00
AmiAid 176]
$7.51
3.55
4.03
3.30
4.11
0.55
Wiceeitter [10]
24.00
1.7%
2.15
0.71
WeSep [23]
5.58
[in]
Matia- TSE Latc
11,55
3.50
3.34
3.21
4.04
0.77
Ments-TS TSB cas
10:48
3,00
3.85
3.21
4.15
0.77
Metis-Ti NNI uas
8.77
3,60
1.85
3.31
4.15
0.76
Metii- TSE sanara
5.57
0.78
The and sccond best
bod0 undetionl.
LibiMSi,
compuainn WER FisiiMMss
transctied sess of the grouns muab wab..ppockk..o or Will, culouaaion.
Evaluation and Baseline
We evaluate our models on the LibriMix [77] test set. LibriMix is used exclusively for evaluation and not included in the training set, thus highlighting the generalization
lhttps://gitaan http:///wwwaollllaaaaao/////////
7

---

capabilities of our models. In addition, we construct a test set of IK samples from the Emilia dataset, which features greater diversity in both speakers and acoustic environments. We denote by EmiliaMix test. We compare our models with several open-source expert models for target speaker extraction: WeSep [23]. VoiceFilter [22], and TSHLM [76]. We also compare with UniAudio [26]. See more details about these baselines in Appendix B.3.
The Table shows: 1) All fine-tuned models exhibit similar performance on audio quality metrics, while the WER decreases as the number of trainable parameters increases. 2) Our models demonstrate significant improvements across two test sets in audio quality metrics, DNSMOS and NISQA, compared to the basclines. Metis-TSE LoRA 16 achieves state-of-the-art NISQA scores of 4.41 on LibriMix test and 4,46 on EmiliaMix test, surpassing the ground truth scores of 4.11 and 3.83, respectively. 3) Met fine tune achieves a competitive WER, although it is slightly lower than WeSep.
Result The results are presented in Table 3.
4.2.4 Speech Enhancement
Implementation Details For speech enhancement, we simulate noisy speech following previous works [24, 48, 78, 79]. We utilize noise datasets such as WHAM! [80] and DEMAND [81], along with room impulse response (RIR) datasets [82], OpenSLR26 and OpenSLR28. in accordance with For clean speech data, randomly sample 10K hours from our pre- trained dataset without any filtering. Inspired by prior works [48, 78], we simulate speech degradation by probabilistically applying various distortions to the audio signals. These include adding noise within an SNR range -5-5 to 20 dB with probability p = 0.9, introducing reverberation with = 0.35, and limiting the speech signal bandwidth (randomly selected from 2 kHz, 4 kHz, 8 kHz) with probability of with p = 0.25. The w2v-bert-2. O features extracted from the degraded speech serve as input conditions for our models. We fine-tune the models using both full-scale fine-tuning and LoRA fine-tuning. For LoRA fine-tuning, we also experiment with r = 4, 16, 32, corresponding to LoRA modules with trainable parameter counts of 2M, 9M, and 18M, respectively.
the 2020 DNS-Challenge
Evaluation and Baseline We evaluate models using the 2020 DNS-Challenge [83] test sets, which consist of three categories: I) synthetic data with reverb, 2) synthetic data
Table 4: Results on speech enhancement task.
Model
SIG(T:
BAK
OVRL(T
NISQA())
SIM(T)
without reverb, and 3) real recordings. For comparison, we downsample the output of our models to 16 kHz. We compare our mod- els with several recent speech enhancement models, including TF-GridNet [24], Voice- Fixer [78], SELM [79] and MaskSR [48]. Notably, MaskSR is model that also uses masked generative modeling but directly gen- erates acoustic tokens for speech enhancement. See more details about these baselines in Ap- pendix B.4.
DNS2020 with
TF-GridNer [24]
3:23
2.51
VoiceFixer [78]
3.43
4.02
3:13
3.82
0.91
SEI M [79]
3.16
3.58
2.70
MaskSR [1S]
3.53
4.07
3.25
Metis-EE tubs
3.67
4.13
3.43
4.54
0.93
Metis SE LatA
3.67
4.13
3.43
4.57
0.94
Metis-FF
3.43
0.94
3.44
0.94
DNS20220 au reverb
TF-GridNer [24]
3.54
4.05
3:29
4.35
0.68
VoiceFixer [S]
3.50
4.11
3:25
4,27
0.96
SEI M [19]
3.51
4.10
3.36
MaskSR [1S]
3.60
4.15
3.37
Metis-EE tu
4.15
3.43
4.82
0.97
Metis-Sti Lank
4.16
3.43
481
0.97
Metis-E
3,66
3.44
4.77
0.97
Result The results are presented in Table 4. The table shows that our models achieve state- of-the-art performance across all benchmarks. 1) ME achieves state-of-the-ee across all three test sets with the highest SIG, BAK, OVRL, and NISQA scores, showing sig- nificant improvements over previous baselines.
Metis SE mese
3.64
4.17
3.43
4.76
0.97
DNS2200 Real Recording
VoiceFixer [19]
3.31
3.93
3,00
3.66
SEI [?]]
3.59
3.44
3.12
results
MisUSR [48]
3.43
3.14
4.02
3.29
Metii-E toRs
3,30
3.97
Meti-SE ca
3.59
3.99
3.26
3.92
Metis SE mmekae
3.59
4.01
3.27
3.95
2) Results among different fine-tume tnnettuned versions are comparable, even for Metis-SE LoRA 4. 3)
The best and the second best result shown in bold and by underlined
Notably, Metis-SE performs exceptionally well on the real recording dataset, further validating its practical applicability.
8

---

Table 5: Results lip-to-speech task.
Table 6: Results of Metis- Omii target speaker traction.
Model
WER:
DNSMOS(1)
NISQMO
SIM(T)
1.822
Model
LiptSpeech Unit (25)
33.64
3.01
2.70
29.34
Librshiizz frst
Modi 122 tacome
32.25
3.25
3,71
59.73
Ground Trush
4.27
3,62
4.03
3.32
4.11
0.76
LRS3
Welpp (2])
UiptSpeect-Une (15)
2.28
TSELM [76]
3.55
3,29
4.00
Metisel sectee
3100
3
3,75
56.74
3,65
4.02
4.56
Mott-DDmn Ommi
5:00
3.6
400
3.56
Mess- Onani
2.70
3,07
4.03
3.36
4.30
0.75
The lest and the sccred lost roall b-skon bold and sakellaas
4.2.5 Lip-to-Speech
Implementation Details We use combined data set comprising the training and sets of LRW LRS2*, and LRS3 as the training data. We use the same methodology in Ma et al. [84] to pre-process the videos and use the visual speech recognition encoder in Ma al. [84] to extract visual features that served as the conditions and randomly extract prefix of the target speech as the prompt.
Evaluation and Baseline We compare our models with Lip2Speech- speech model. We use the test sets of LRS2 and LRS3 to evaluate our models following previous works [25, 85]. Notably, over 60% of the samples the two test sets have durations less than seconds. Utilizing excessively short audio prompt may degrade the sound quality of the inference results, while directly using part the target audio as a prompt may lead to information leakage. We use speech randomly selected from SeedTTS test-en as prompts for each test case.
[85], an expert lip-to-
The results are presented Table The table shows that Metis-L22 outperforms the baseline Lip2Speech-UND Unit across all metrics. Specifically, the audio quality metries show substantial improvements and the speaker similarity nearly doubles on both datasets (29.34 - 59.73 on LRS2, 32.05 - 56.74 on LRS3), The model also achieves a lower WER than the baseline.
Result
4.3 Multi-Task Fine- TuninT
In addition to fine-tuning the pre-trained model separately for different tasks, we explore the potential of jointly fine-tuning it on multiple tasks, resulting in multi-task model, which we refer to Metis- Omni, For this study, we select four tasks: zero-shot TTS, voice conversion, target speaker extraction, and speech enhancement. Further details and experimental results are provided in Appendix E and Table 11.
We take target speaker extraction as an example to illustrate the effectiveness of Metis- in enabling novel applications through task combinations. As shown in Table 6, Metis-Omni outperforms baseline methods in terms of WER. Furthermore, by integrating text-to-speech and target speaker extraction tasks, the text-guided version of Metis-Omm achieves a remarkable WER reduction to 2.70, demonstrating substantial improvement over all other models. Notably, despite the model being explicitly trained on text-guided target speaker extraction, generalizes well to this novel setting, highlighting the advantage of our system in leveraging multimodal conditional inputs to enable flexible and efficient task adaptation.
Conclusion
In this work, we propose Metis, foundation model for unified speech generation that leverages large-scale unlabeled speech data for pre-training and can be effectively adapted to diverse speech generation tasks through fine-tuning. Our experiments demonstrate that Metis outperforms state- of-the-art task-specific and multi-task systems on zero-shot TTS, voice conversion, target speaker enhancement, speech enhancement, and lip-to- ti-to--speech after fine-tuning, even with fewer than 20M trainable parameters or up to 300 times less training data for certain tasks while supporting multimodal conditional inputs. In addition, we propose Metis-Oma version of our mre-trained model fine-tuned on multiple tasks, which demonstrates further improvements.
hnttps://ww.robotr ox. a..ac. htt.a///gg//ataai/.raaaa//////// html #nttps://www.ron http://wwwwbbbs.ooo/// ac. uk/ io htttps://maa..to/taaaaeee
html

---

Acknowledgment
The authors gratefully acknowledge the funding support from the National Natural Science Foun- dation of China (Grant No. 62376237), the Shenzhen Science and Technology Program (Grant No. ZZDSSS22330620091 1302006), the Shenzhen Research Institute of Big Data (Internal Project Fund, Grant No. T00120230002), and the 2023 Shenzhen Stability Science Program. We would also like to thank the anonymous reviewers and the Area Chair for their insightful comments and valuable suggestions, which helped improve our paper.
10

---

References
[1] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[2] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arviv: / 18 1810008055 2018.
[3] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint urXiv:2303.0655
2023.
[4] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 16000-16009, 2022.
[5] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint 3XXi:21106008254 2021.
[6] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[7] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tin-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006. 200:200000455,, 2020.
[8] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301 021111, 2023.
[9] Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.091 009/23304099//6 16, 2023.
[10] Zeqian Ju. Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang. et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03 100, 2024.
[11] Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36, 2024.
[12] Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Shunsi Zhang, and Zhizheng Wu. Maskget: Zero-shot text-to-speech with masked generative codee transformer. arXiv preprint
2024.
[13] Seyed Hamidreza Mohammadi and Alexander Kain. An overview of voice conversion systems. Speech Communication, 88:65-82, 2017.
[14] Kaizhi Qian, Yang Zhang, Shiyu Chang. Xuesong Yang, and Mark Hasegawa-Johnson. Autove: Zero-shot voice style transfer with only autoencoder loss. In International Conference Machine Learning, pages 5210-5219. PMLR, 2019.
[15] Jingyi Li, Weiping Tu, and Li Xiao. Freeve: Towards high-quality text-free one-shot voice conversion. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[16] Ha-Y Haa-Yeng Choi, Sang- Hoon Lee, and Seong- Lee. Diff-hierve: Diffusion-based hicrar- chical voice conversion with robust pitch generation and masked prior for zero-shot speaker adaptation. International Speech Communication Association, pages 2283-2287, 2023.
[17] Santiago Pascual, Antonio Bonafonte, and Joan Serra. Segan: Speech enhancement generative adversarial network. arXiv preprint 1rX00:1003 2017.

---

[18] Szu- Wei Fu, Cheng Yu, Tsun-An Hsieh, Peter Plantinga, Mirco Ravanelli, Xugang Lu, and Yu Tsao. Metricgan+: An improved version of metricgan for speech enhancement. arXiv 2021.
preprint
[19] Haohe Liu, Qiugiang Kong, Qiao Tian, Yan Zhao, DeLiang Wang, Chuanzeng Huang, and Yuxuan Wang. Voicefixer: Toward general speech restoration with neural vocoder. arXiv preprint Sttv:2209.1333.1 2021.
[20] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Pastspeech: Fast, robust and controllable text to speech. Advances in neural information processing systems, 32, 2019.
[21] Sang-Hoon Lee, Ha-Yeong Choi, Seung-Bin Kim, and Seong- Whan Lee. Hierspeech++: Bridging the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis. arXiv preprint arXiv:2311.12454 2023.
[22] Quan Wang, Hannah Muckenhirn, Kevin Wilson, Prashant Sridhar, Zelin Wu, John Hershey, Rif A Saurous, Ron J Weiss, Ye Jia, and Ignacio Lopez Moreno. Voicefilter: Targeted voice separation by speaker-conditioned spectrogram masking, arXiv preprint arXiv:1810.0450 2018.
[23] Shuai Wang, Ke Zhang, Shaoxiong Lin, Junjie Li, Xuefei Wang. Meng Ge, Jianwei Yu, Yanmin Qian, and Haizhou Li. Wesep: A scalable and flexible toolkit towards generalizable target speaker extraction. arXiv preprint arXiv:2409. 15799, 2024.
[24] Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong- Yeol Kim, and Shinji Watanabe. Tf-gridnet: Integrating full-and sub-band modeling for speech separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
[25] Minsu Kim, Joanna Hong, and Yong Man Ro. Lip-to-speech synthesis in the wild with multi- task learning. ICASSP 2023-202. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1 1-5. IEEE, 2023.
[26] Dongchao Yang. Jinchuan Tian, Xu Tan, Rongje Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:23/0.00704 2023.
[27] Xiaofei Wang, Manthan Thakker, Zhuo Chen, Naoyuki Kanda, Sefik Emre Eskimez, Sanyuan Chen, Min Tang. Shujie Liu, Jinyu Li, and Takuya Yoshioka. Speechx: Neural codec language model as a versatile speech transformer. /EEE/ACM Transactions on Audio, Speech, and Language Processing. 2024.
[28] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: language modeling approach to audio generation. IEEE/ACM transactions audio, speech, and language processing, 31:2523 2533, 2023.
[29] Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin. Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics, 1 11:1033-177 1718, 2023.
[30] Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.096 2023.
[31] Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding. Lu Gao, et al. Seed-tts: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.022 02433, 2024.
[32] Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint arXiv:2407.05400 2024.
12

---

[33] Wei-Ning Hsu, Benjamin Bolte, Yao- Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdi- nov, and AbdeIrahman Mohamed. Hubert: Self-supervieed speech representation learning by masked prediction of hidden units. JEEF/ACM transactions on audio, speech, and language processing, 29:3451-344 2021.
[34] Yu-An Chung, Yu Zhang, Wei Han, Shung-Chemy Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2y-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU),, pages 244-250. IEFE, 2021.
[35] Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, et al. Google usm: Scaling automatic speech recognition beyond 100 languages. arXiv preprint 1.Xiv:233330100333 2023.
[36] Chung-Cheng Chiu, James Qin, Yu Zhang. Jiahui Yu, and Yonghui Wu. Self-supervised learning with candom-projectionn quantizer for speech recognition. International Conference on Machine Learning, pages
PMLR, 2022.
[37] Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, al. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 1606))150--1518. 2022.
[38] Hao-Han Guo, Kun Liu, Fei-Yu Shen, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. Fircredtts: A foundation text-to-speech framework for industry-level generative speech applications. arXiv preprint arXiv:2409.03286 03283, 2024.
[39] Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei. Qiuqiang Kong, Qiao Tian, Yuping Wang. Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. Audioldm 2: Learning holistic audio generation with self-supervised pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
[40] Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark Plumbley. Semanticodec: An ultra low bitrate semantic audio codec for general sound. arXiv preprint arXiv:2405.00233 2024.
[41] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
[42] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[43] Huiwen Chang, Han Zhang. Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming -ssan Yang, Kevin Murphy, William Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint 2xXiv22301.00704, 2023.
[44] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
[45] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang. Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10469, 2023.
[46] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion-fokenizer is key visual generation. arXiv preprint arXiv:2310.057:7. 2023.
[47] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEF/ACM Tronsactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
13

---

[48] Xu Li, Qirui Wang, and Xiaoyu Liu. Masksr: Masked language model for full-band speech restoration. arXiv preprint arXiv:2406.02008 2024.
[49] Apoorv Vyas, Bowen Shi, Matthew Le. Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et Audiobox: Unified audio genera- tion with natural language prompts. arXiv preprint arXiv:2312. 15821, 2023.
[50] Alexander H Liu, Matt Le, Apoorv Vyas, Bowen Shi, Andros Tjandra, and Wei-Ning Hsu. Generative pre-training for speech with flow matching. arXiv preprint arXiv:2310. 1633388 2023.
[51] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
[52] José Lezama, Huiwen Chang, Lu Jiang, and Irfan Essa. Improved masked image generation with token-critic. In European Conference on Computer Vision, pages 70-86. Springer, 2022.
[53] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[54] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high- nigh-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 12873-12883, 2021.
[55] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 20ti::2000.06333 2021.
[56] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.099: 07.0088778, 2023.
[57] Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. arXiv preprint arXiv:2407.05361, 2024.
[58] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.
[59] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12599, 2022.
arXiv preprint
[60] Puyuan Peng, Po- Huang, Daniel Li, Abdelrahman Mohamed, and David Harwath. Voice- craft: Zero-shot speech editing and text-to-speech in the wild. arXiv preprint arXiv:2403.16975 2024.
[61] Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Aljafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, et Xtts: a massively multilingual zero-shot text-to-speech model. arXiv preprint arXiv:2406.04400 2024.
[62] Heiga Zen, Rob Clark, Ron J. Weiss, Viet Dang, Ye Jia, Yonghui Wu, Yu Zhang, and Zhifeng Chen. Libritts: A corpus derived from librispeech for text-to- text-to-speech. In Interspeech, 2019. URL https: //axxii org/ abs/ 19044 02882.
[63] Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders. Francis M Tyers, and Gregor Weber. Common voice: massively-multiinguaal speech corpus. arXiv preprint arXiv: 1912.06670, 2019.
[64] Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, Ruixiong Zhang, Shuaijiang Zhao, Wubo Li, Cheng Gong, Wei Zou, Kun Han, et al. Didispeech: A large scale mandarin speech corpus. ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6968-6977 IEBE, 2021.
14

---

[65] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an ast corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-5210. IEFF, 2015.
[66] Yuanzhe Chen, Ming Tu, Tang Li, Xin Li, Qiuqiang Kong, Jiaxin Li, Zhichao Wang. Qiao Tian, Yuping Wang, and Yuxuan Wang. Streaming voice conversion via intermediate bottleneck features and non-streaming teacher guidance. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[67] Dongya Jia, Qiao Tian, Kainan Peng, Jiaxin Li, Yuanzhe Chen, Mingbo Ma, Yuping Wang, and Yuxuan Wang. Zero-shot accent conversion using pseudo siamese disentanglement network. arXiv preprint arXiv:2212.05777 2022.
[68] Zhichao Wang. Yuanzhe Chen, Lei Xie, Qiao Tian, and Yuping Wang. Lm-ve: Zero-shot voice conversion via speech generation based on language models. IEEE Signal Processing Letters, 2023.
[69] Philip Anastassiou, Zhenyu Tang, Kainan Peng, Dongya Jia, Jiaxin Li, Ming Tu, Yuping Wang, Yuxuan Wang, and Mingho Ma. Voiceshop: A unified speech-to-speech framework for identity zero-shot voice editing. arXiv preprint 4xX1:22404.06674, 2024.
[70] Mateusz Lajszczak, Guillermo Cámbara, Yang Li, Fatih Beyhan, Arent van Korlaar, Fan Yang, Arnaud Joly, Álvaro Martín CortinnCr Ammar Abbas, Adam Michalski, al. Base tts: Lessons from building a billion-pan parameter text-to-speech model on 100k hours of data. arXiv preprint 2xXiv:2402.08093, 2024.
[71] Matthew Baas, Benjamin van Nickerk, and Herman Kamper. Voice conversion with just nearest neighbors. arXiv preprint 2335v 18975, 2023.
[72] Paarth Neekhara, Shehzeen Hussain, Rafael Valle, Boris Ginsburg, Rishabh Ranjan, Shlomo Dubnov, Farinaz Koushanfar, and Julian McAuley. Selfve: Voice conversion with iterative refinement using self transformations. arXiv preprint arXiv:2310.0960 2023.
[73] Zengyi Qin, Wenliang Zhao, Xumin Yu, and Xin Sun. Openvoice: Versatile instant voice cloning. arXiv preprint arXiv:2312.01473 2023.
[74] Xueyao Zhang, Xiaohui Zhang, Kainan Peng. Zhenyu Tang, Vimal Manohar, Yingru Liu, Jeff Hwang, Dangna Li, Yuhao Wang, Julian Chan, Yuan Huang, Zhizheng Wu, and Mingho Ma. Vevo: Controllable zero-shot voice imitation with self-supervised disentanglement. OpenReview, 2024.
[75] Christophe Veaux, Junichi Yamagishi, Kirsten MacDonald, et al. Cstr vetk corpus: English multi-speaker corpus for estr voice cloning toolkit. University of Edinburgh. The Centre for Speech Technology Research (CSTR), 6:15, 2017.
[76] Beilong Tang, Bang Zeng, and Ming Li. Tselm: Target speaker extraction using discrete tokens and language models. arXiv preprint arXiv:2409.07841, 2024.
[77] Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vincent. Librimix: An open-source dataset for generalizable speech separation. arXiv preprint arXiv:2005. / 1262, 2020.
[78] Haohe Liu, Xubo Liu, Qiuqiang Kong, Qiao Tian, Yan Zhao DeLiang Wang, Chuanzeng Huang, and Yuxuan Wang. Voicefixer: A unified framework for high-fidelity speech restoration. arXiv preprint arXiv:2204.0558 055441, 2022.
[79] Ziqian Wang, Xinfa Zhu, Zihan Zhang, YuanJun Lv, Ning Jiang, Guoqing Zhao, and Lei Xie. Selm: Speech enhancement using discrete tokens and language models. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11561-11565. IEEE, 2024.
[80] Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux. Wham!: Extending speech separation to noisy environments. arXiv preprint arXivv1907.011600 2019.
15

---

[81] Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. The diverse environments multi- channel acoustic noise database (demand); A database of multichannel environmental noise recordings. In Proceedings of Meetings on Acoustics, volume 19. AIP Publishing, 2013.
[82] Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael L Seltzer, and Sanjeev Khudanpur. A study on data augmentation of reverberant speech for robust speech recognition. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5220-5224. IEEE, 2017.
[83] Harishchandra Dubey, Ashkan Aazami, Vishak Gopal, Babak Naderi, Sebastian Braun, Ross Cutler, Alex Ju, Mehdi Zohourian, Min Tang, Mehrsa Golestaneh, et Icassp 2023 deep noise suppression challenge. IEEE Open Journal of Signal Processing, 2024.
[84] P. Ma, A. Haliassos, A. Fernandez-Lopezz H. Chen, S. Petridis, and M. Pantic. Auto-avsr: Audio-visual speech recognition with automatic labels. In ICASSP 2023-2023 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[85] Jeongsoo Choi, Minsu Kim, and Yong Man Ro. Intelligible lip-to-speech synthesis with speech units. arXiv preprint arXiv:2305.19603 2023.
[86] James Betker. Better speech synthesis through scaling, arXiv preprint arXiv:2305.07244 2023.
[87] Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High-fidelity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 36, 2024.
[88] Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint arXiv:2306. 00814, 2023.
[89] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210.13: 134338 2022.
[90] Xueyao Zhang, Liumeng Xue, Yuancheng Wang. Yicheng Gu, Xi Chen, Zihao Fang, Haopeng Chen, Lexiao Zou, Chaoren Wang. Jun Han, et al. Amphion: An open-source audio, music and speech generation toolkit. arXiv preprint arXiv:2312.09911 2023.
[91] Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.03411, 2020.
[92] Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv:2106.000 06909, 2021.
[93] Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with ad- versarial learning for end-to-end text-to-speech. In International Conference Machine Learning, pages 5530-5540. PMLR, 2021.
[94] Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre- Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky. Ronan Collobert, Christian Fuegen. et al. Ligri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7669-7673. IEFE, 2020.
[95] Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia. Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:/904.0288 2019.
[96] Wei Kang, Xiaoyu Yang, Zengwei Yao, Fangjun Kuang, Yifan Yang, Liyong Guo, Long Lin, and Daniel Povey. Libriheavy: 50,000 hours asr corpus with punctuation casing and context. In ICASSP 2024-2024 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10991-10999 IEEE, 2024.
16

---

[97] Thai-Binh Nguyen and Alexander Waibel. Convoifiltr: A case study of doing cocktail party speech recognition. In 2024 IEEE International Conference on Acoustics, Speech, and Signal P Processing Workshops (ICASSPW), pages 565-569 IEFF, 2024.
[98] Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: large-scale speaker identification dataset. arXiv preprint arXiv: wr..v11700.086/2,. 2017.
[99] Foivos I Diakogiannis, François Waldner, Peter Caccetta, and Chen Wu. Resunet-a: A deep learning framework for semantic segmentation of remotely sensed data. ISPRS Journal of Photogrammetry and Remote Sensing, 162:94-114, 2020.
[100] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. International conference on machine learning, pages
PMLR, 2023.
[101] Zhifu Gao, Shiliang Zhang, lan Mcl Loughin,, and Zhijie Yan. Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition. arXiv preprint arXiv:2206.0833 2022.
[102] Zhifu Gao. Zerui Li, Jiaming Wang. Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, Zhangyu Xiao, et al. Funasr: A fundamental end-to-end speech recognition toolkit. arXiv preprint arXiv:2305. 1013, 2023.
[103] Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2021-202/ IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6493- 6497. IEEE, 2021.
[104] Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Sebastian Möller. Nisqa: A deep cnn-self- attention model for multidimensionaa speech quality prediction with crowdsourced datasets. arXiv preprint
2021.
17

---

A Details of Two Types of Speech Discrete Representations
Metis employs two discrete speech representations for the two-stage generation, following the approach of Wang et al. [12].
Derived from SSL features of large-scale speech self-supervised learning models [33, 34, (6], SSL tokens encapsulate both semantic and prosodic information, making them well-suited conditional generation.
1) SSL tokens:
minimize information loss, we employ a vector quantization (VQ)
model [53, 54] to quantize SSI. features into discrete tokens, in contrast to the k-means approach used in previous works [28, 29, 86]. SSL features are extracted from the 17th layer w2v Serv-ber-22 [34].
The VQ model has codebook of 8,192 and codebook dimension of 8. The model is trained using only reconstruction loss and VQ loss.
2) Acoustic tokens:
Directly obtained from the waveform via vector quantization. The goal is to preserve all the information from the speech to reconstruct high-quality waveform. We follow the recipe of DAC codec [87] for model architecture, discriminators, and training losses, with modifications to adopt the Vocos [88] decoder for more efficient training and inference. The 24 kHz speech waveform is compressed into discrete tokens using residual vector quantization (RVQ) across 12 layers, each with codebook size of 1,024 and a codebook dimension of
B Baselines
B.1 Zero-Shot TTS
VALL-E [8] It uses an AR transformer to predict codes from the first layer of EnCodec [89] and NAR transformer to predict codes from the remaining layers of EnCodec. We use the released checkpoint Amphion [90]² which is pre-trained on 45K hours the MLS [91] English set.
VoiceCraft [60] A token-filling neural codec language model for text editing and text-to-speech. It predicts multi-layer tokens in a delay pattern. We use the official code and checkpoint which is pre-trained on 9K hours GigaSpeech [92] dataset.
Cosy Voice [32] A two-stage large-scale TTS system. The first stage is an autoregressive model and the second stage is diffusion model. It is trained on 17IK hours of multilingual speech data. We use the official code and checkpoint".
XTTS-v2 [61] An open-source multilingual TTS model that supports 16 languages. It is also based on an autoregressive model. We use the official code and checkpoint
MaskGCT [12] An open-source large-scale NAR TTS system that eliminates the need for explicit alignment information between text and speech supervision, well as phone-level duration prediction. employs masked generative models for two-stage modeling and is trained on 100K hours of multilingual speech data. We use the official code and checkpoint!
NaturalSpeech 3 [10] A large-scale NAR TTS system featuring factorized speech codec for speech decoupling representation and factorized diffusion models for speech generation. It achieves human-level naturalness on the LibriSpeech test set. We report the scores of LibriSpeech test-clean obtained from the original paper.
VoiceBox [11] A large-scale NAR multi-task speech generation model based on flow matching [51]. We report the scores of LibriSpeech test-clean obtained from the original paper.
pth
10 opttppes///agiiideeeeeec////orrp
18

---

B.2 Voice Conversion
HireSpeech++ [21] speech generation system designed based on VITS architecture [93]. is trained on 2.8K hours sourced from Libri-light [94] and LibriTTS [95]. We use the official code and checkpoint
LM-VC [68] It uses an AR hierarchical transformer predict SoundStream [47] codes from soft units similar to HuBERT [33] k-means tokens, trained on the Libri-light dataset [94]. It is not open-source, we report the scores obtained from [74].
UniAudio [26] An AR model system can perform multiple audio generation tasks. uses 500- cluster k-means tokens from HuBERT [33] to predict their proposed acoustic codes for voice conversion. We use the official code and checkpoint". It uses 60K hours of Libri-light [94] for training voice conversion.
Vevo [74] is a versatile zero-shot voice imitation model which can control both timbre and style. It is trained 60K hours of Libri-heavy [96]. We obtain the generated samples from the authors.
B.3 Target Speaker Extraction
UniAudio [26] An AR model system perform multiple audio generation tasks including target speaker extraction. We use the official code and checkpoint.
VoiceFilter [22] A system that isolates a target speaker's voice from multi-speaker audio using a reference signal and neural networks for speaker embedding and spectrogram masking, We use the checkpoint provided from Nguyen and Waibel [97].
WeSep [23]
target speaker extraction model trained on LibriMix [77] and VoxCeleb [98]. We use the official code and checkpoint
TSELM [76] A target speaker extraction model that leverages a NAR transformer to predict discrete speech tokens driven from WavLM [37]. We use the official code and checkpoint
B.4 Speech Enhancement
TF-GridNet [24] A deep neural network for speech separation integrating full- and sub-band modeling in the time-frequency (T-F) domain. It can also used for speech enhancement. We use the checkpoint provided from Interspeech URGENT 2025 Challenge
VoiceFixer [78] A generative framework to address the speech enhancement task. It consists of an analysis stage and synthesis stage and employs a ResUNet [99] to model the analysis stage and a neural vocoder to model the synthesis stage. We use the official code and checkpoint
SELM [79] A speech enhancement model that leverages AR transformer to predict discrete speech tokens driven from WavL.M [37]. We report the scores obtained from [48].
MaskSR [48] It uses masked generative models predict acoustic tokens from DAC [87] codec for speech enhancement. We report the scores obtained from the original paper.
13 Ttttgg///gggthbboooo///gmaaaaaaa
18 bacereerrrzeeoo.callllgegggaatee
19

---

B.5 Lip-to-Speech
Lip2Speech-UnitPP [25] lip-to-speech model trained generate multiple targets, snel-spectograms
and quantized sule--upe speech representations. We use the official code and checkpoint".
C Evaluation Metrics
SIM We evaluate speaker similarity between the prompt speech and the generated speech by computing the cosine similarity of the WavLM TDNN2137 speaker embeddings between the generated sample and the prompt. SIM is reported for tasks involving prompt speech, including zero- shot TTS, voice conversion, target speaker extraction, and lip-to-speech. For speech enhancement, we compute SIM between the generated speech and ground truth using a separate checkpoint
WER Word Error Rate measures the intelligibility of the generated speech. We use whisper-large-i paraformer-zhh [101, 102] the ASR model to calculate WER. WER is reported for zero-shot TTS, voice conversion, target speaker extraction, and lip-to-speech.
[100] for all languages except Chinese, where we use
DNSMOS [103] DNSMOS is neural network- based mean opinion
estimator that correlates
strongly with human quality ratings. It comprises three components: 1) speech quality (SIG), 2) background noise quality (BAK), and 3) overall quality (OVRL). We report DNSMOS scores for all tasks while providing the average scores for zero-shot TTS and VC tasks, and all three metrics for the remaining tasks.
NISQA [104] NISQA is a deep learning framework for speech quality prediction. We use the public checkpoint We report NISQA for voice conversion, speech enhancement, target speaker extraction, and lip-to-speech.
QMOS We the Quality Mean Opinion Score (QMOS) for subjective listening tests evaluate speech quality. QMOS is rated on a 5-point scale: 5 (Excellent), 4 (Good), (Fair), 2 (Poor), and (Bad).
SMOS We use the Speaker Mean Opinion Score (SMOS) for subjective listening tests to assess speaker similarity between the generated speech and the prompt. SMOS rated 5-point scale: (Excellent), 4 (Good), 3 (Fair), (Poor), and (Bad). SMOS is reported for zero-shot TTS.
20 2htee:///ttthhcoooooooo/ggggs/ep
21 http:///hhaacaccaccceeooe///eeea verification hatpee/ssigiiiiiieoccccrrrrrrrrr
23 httege//hhggliiieecoooo/////papp
http://tth..oooeaaaaaaa/l/////te 25 /ctpps://gttbbbccmm/
20

---

Subjective Listening Test (SMOS)
Instractinn:: (ndesecal dier the gar inaid Euanchh sskincc to speaker the picntal at the 27-20 soale
Subjective Listening Test (QMOS)
Intruatees ecc Passa make ean sacio cip carttity ar tinta eventt quilly CVCC
XMA00 coe sh ceneriee
Cloc May butor to bate the aloo the sdent you rore before morinng
Sample
Oawsstad
((beed)
1 Ph
2 Poed
Mhat
Paacket
x pull
(hee)
O. (16e
Sabme
(a) screenshot of the QMOS evaluation interface.
(b) A screenshot of CMOS evaluation interface.
Figure 3: Screenshots of the subjective evaluation webpages.
21

---

D Additional Experimental Results
D.1 Zero-Shot TTS Results LibriSpeech
Table 7 shows zero-shot TTS results on LibriSpeech test-clean. Metis-TTS achieves the highest SIM despite using significantly less training data, while maintaining a competitive WER.
Table 7: Zero-shot TTS results on LibriSpeech.
Training Data
Model
WER()
SIM(T)
LibriSpeech test-clean
VALL-E [10]
45K EN
5.90
0,50
VoiceCraft [60]
9K EN
4.68
0,45
NaturalSpeech [10]
60K EN
1.94
0,67
VoiceBox [10]
60K EN
2.03
0.64
Metis-TTS
10K Multi.
4.33
0.70
D.2 Voice Conversion Results SeedTTS-VC
We also provide voice conversion results SeedTTS- test sets in Table 8. The test sets are provided by [31]2e. Despite being fine-tuned on significantly less training data (0.4K hours compared to 2.8K hours used by HierSpeech+++), Metis-VC achieves higher SIM scores on both SeedTTS- VC-en (0.76 vs. 0,56) and SeedTTS- VC-zh (0.63 vs. 0.39), demonstrating its strong capability in preserving speaker identity. Additionally, Mc achieves higher NISQA score than the baseline model, indicating improved speech quality.
Table 8: Voice conversion results on SeedTTS- test sets.
Training Data
Model
WER())
SIM(1)
DNSMOS(() NISQA(1)
SeedTTS-
HierSpeech++ [21]
2.8K
4.48
0.56
3.57
3.47
Metis-VC fine-tune. efg 0.0
0.4K
7.05
0.76
3.51
4.14
SeedTTS- VC-zh
HierSpeech++ [21]
2.8K
5.45
0.39
3.50
4.14
Metis- VC fine-tune, ofg 0.0
0.4K
6.82
0.63
3.39
3.82
D.3 Results of Subjective Evaluation
We conduct subjective listening tests for two representative tasks: zero-shot TTS and speech en- hancement. For zero-shot TTS, we randomly sampled 20 test samples from the results of SeedTTS test-en. Both QMOS and SMOS are reported for zero-shot TTS. The results shown in Table 9, For speech enhancement, we randomly sampled 20 test samples from the results of DNS2020 with reverb. QMOS is reported for speech enhancement. The results shown in Table 10.
26
22

---

Table 9: Results of subjective evaluation for zero-shot TTS.
Model
QMOS(1)
SMOS(11
Ground Truth
4.32 20.28
4.01 29.30
VALL-E [8]
3.32 20.22
3.58 20.13
VoiceCraft [60]
3.58
3,65 #0.19
Cosy Voice [32]
4.02 20.10
3.97 #0.21
Metis-TTV
4.21 20.31
4.19 10.18
Table 10: Results of subjective evaluation for speech enhancement.
Model
QMOS(T)
Ground Truth
4.57 20.13
TF-GridNet [24]
3.65 20.35
VoiceFixer [78]
3.77 +0.29
Metis-SE
4.29 +0.21
E
Metis-Omni: Multi-Task Fine-Tuning
In addition to fine-tuning the pre-trained model separately for different tasks, we explore the potential of jointly fine-tuning a pre-trained model on multiple tasks resulting multi-task model, which we refer as Metis-Omn
For this study, we select four tasks: zero-shot TTS, voice conversion, target speaker extraction, and speech enhancement. We show more details in Appendix E. A subset of 10K hours of data is randomly sampled from the Emilia dataset as the shared training data for these tasks. During training, the task proportions set conversion, target speaker extraction, and speech enhancement, respectively. As shown in Table 11, Mctis- achieves performance that is either par with or surpasses task-specific fine-tuned models across all tasks. The only exception the WER metric in zero-shot TTS, where performance is slightly lower. A possible reason for this is that, under the same number of training steps, multi-tast fine-tuning allocates fewer steps to zero-shot TTS, leading to less task-specific optimization.
(0.5, 0.2, 0.2) for zero-shot TTS, voice
Table 11: Results of Petis-Omni on four speech generation tasks.
Performance
Tisk
Dataset
Model
Metries
Results
MaskGCT [12]
0.72
SeedTTS-on
Metis: TTS
WER()) SIM(D) DNSMOS(T)
2.41
0.72
3.57
Zero-shot TTS
Metis-Omne
4.78
0.71
13,57
MaskGCT [1]]
SeedTTS-ch
Metis-TTS
WER(D) SIM(D) DNSMOS(T)
2,30
0.77
33.55
Metis.Omm
4.39
0.77
33660
L.-VC [68]
00.29
3.44
VC
VCTK
Mctis-VC
/ SIMI() (DNSMOS(T)
6.65
0.488
33.49
Metis-Ommi
TF-Grident [24]
3.11
3:23
2.51
2.61
DNS2020 witk reverb
Mctis-SE
SIG(T) BAK()) OORR(T) NISQA(()
3.68
[4.14
3.44
4.56
Metis-Ommi
3.68
4..3
3.44
4.53
TF- TF-Grideut [24]
3.54
4,05
3.27
44.35
DNS2020
S(G(T) BAK(T) (VRL(()) NISQA(()
Metis-Omni
3.65
4.17
3.44
4.79
VoicelFixer [78]
3.31
3.93
3.00
3.65
Recording
Metis-SE
SIG(T) RAK()) (VRR(() NISQA(()
4,01
3.27
3.95
Mctis-Omni
3.60
4.00
3.27
3.96
[23]
4.004
LibriMix text
Metis TSF
SIG(T) BAK()) (VRR(()) NISQA())
3.65
4.02
3:34
4.36
TSR
Metis-Ommi
WeSep [29]
3,66
3.50
3.85
4,03
3,36
3.12
3.86
4.40
EmiliaMit test
MSE
S(G(T) BAK(T) (VRL(T) | NISQA(()
3.60
3.85
3.21
4.06
Omni
3,62
3,82
3.24
4.11
23

---

Speech Discrete Representation
Speech representation is a crucial aspect of speech generation. Recently, some speech generation systems [8, 10, 12, 28, 29, 30, 31, 74] have transitioned to using discrete speech representations, which can be broadly categorized into two types: SSL tokens: typically derived by quantizing speech self-supervised learning features [33, 34, 35, 36, 37]. Unlike acoustic tokens, SSL tokens are designed not for directly reconstructing waveform but for encoding essential semantic and prosody information in speech, making them more suitable for prediction in conditional generation models. 2) Acoustic tokens: typically obtained by training VQGAN [53, 54] model for waveform reconstruction, used in speech codecs [47, 87, 89]. Acoustic tokens effective for reconstructing high-quality waveforms. Currently, some speech generation systems [12, 29, 30, 39] utilize both types of representations for speech generation. In this work, also adopt this two-stage paradigm, employing an MGM to generate SSL tokens from any condition and another MGM to generate acoustic tokens from SSL tokens.
G Limitation and Future Work
There are still some limitations that can be studied in the future.
Unified Audio Representation In this work, utilize two distinct discrete speech representations, SSL tokens and acoustic tokens, for two-stage modeling, with SSL tokens specifically designed for the speech domain. Developing a unified discrete representation for all audio types, such as speech, music, and sound effects, that can seamlessly integrate with conditional generation models is an important direction for future research. Additionally, unifying the characteristics of SSL tokens and acoustic tokens to enable both high-quality waveform reconstruction and effective conditional modeling equally significant.
Few-Shot Task Learning In this work, we adapt our pre-trained model to different characters through fine-tuning. In the field NLP, large language models exhibit the remarkable ability to learn new tasks in zero-shot manner without requiring additional training. This capability merits further investigation in future research.
H
Impact Statement
Given that Metis is powerful foundation model capable of generating high-quality speech across multiple tasks, it also presents potential risks of misuse, such as voice spoofing, speaker impersonation, and unauthorized content generation. To mitigate these risks, it is essential to develop robust detection mechanisms for synthetic speech, establish safeguards to prevent malicious use, and implement a responsible reporting system for identifying and addressing misuse cases.
24

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clarify the scope and contribution of our paper in the abstract and the last two paragraphs of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
25

---

Justification:
We discuss the limitations in Appendix G.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the implementation in Section 4 and Appendix D. And we will release our code and model checkpoints.
Guidelines:
26

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release code and model checkpoints. We use open-source datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
Beeiiii/guudes/oodeeuue www.bmmmiiiooooicccy..... for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //n ctt.pss.cc/uublccuuuuuuu gesds//////eeeeeee/e /CodeSubmian ss onoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
27

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide details about experimental setting and evaluation in Section 4 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: For subjective evalution, we provide the confidence interval.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information in Section 4.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
28

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: ttttps://neuue //terr//s.....//eee wwweeeer....puuuuupeeeeeeeeeeeee cc/public/////ccccccccccccccclll httppppliii.tccccccueeeeeeeeeeee
Answer: [Yes]
Justification: We do conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative socictal impacts in Appendix H.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We discuss it in Appendix H.
29

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the original papers for all data and models we used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the wwwrssiithhhdee package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release our code and model checkpoints
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
30

---

Answer: [Yes]
Justification: We provide details about subject evaluation in Appendix C. Figure shows sreenshots of the subjective evaluation webpages.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve potential risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [No]
Justification: We only use LLMs for editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standaaddard components.
Please refer to our LLM policy (https : httpp:://neurrti //euriiis.....// www.ooofenenee/eaeeaeeee /2cicc/ee///////////// /LLM) for what should or should not be described.
31


Reviews:
[{"review_id": "qXZnOjOgfB", "reviewer": "Reviewer_zbfX", "summary": "The paper proposes Metis, a unified speech generation foundation model trained via masked generative pre-training on unlabeled speech data. Metis is trained using two discrete speech representations and can be fine-tuned with minimal labeled data and parameters for multiple downstream tasks. The Key features include: (1) two-stage speech generation using SSL tokens (semantic/prosodic) and acoustic tokens (waveform-level); (2) pre-training on 300K hours of speech with a masked generative model, without any task conditions; (3) efficient adaptation to tasks like zero-shot TTS, voice conversion, speech enhancement, target speaker extraction, and lip-to-speech, using LoRA or full fine-tuning. The model outperforms state-of-the-art systems on multiple tasks and shows strong generalization with limited data.", "questions": "(1) Can the authors further unify inference logic so that new tasks don’t require any architectural changes?\n\n(2) How does Metis handle long-form or streaming generation tasks?\n\n(3) How sensitive is the model to the length or content of prompts used in zero-shot or few-shot generation?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n(1) Unified framework: The paper proposes a clear and modular two-stage pipeline (SSL token prediction → acoustic generation) that enables generalization across tasks.\n\n(2) Efficient adaptation: The model achieves SOTA or near-SOTA performance on five speech generation tasks using <20M parameters and orders-of-magnitude less training data.\n\n(3) Multimodal support: The model is capable of handling text, audio, and video conditions, allowing for novel combinations like text-guided speaker extraction (Metis-Omni).\n\n(4) Extensive experiments: Includes strong baseline comparisons, ablations (e.g., with and without pretraining), LoRA configurations, and results on real-world and synthetic test sets.\n\nWeaknesses:\n\n(1) No unified inference pipeline: While pre-training is unified, fine-tuning still requires task-specific configurations (e.g., frame-level vs. non-frame-level conditioning logic).\n\n(2) Lack of cross-task analysis: While Metis-Omni is introduced for multi-task fine-tuning, detailed cross-task interference analysis is not provided (e.g., does speech enhancement degrade TTS?).", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we want to thank for your carefully reading and providing the constructive comments! Below we address the concerns mentioned in the review.\n\n`Weakness 1:  No unified inference pipeline: While pre-training is unified, fine-tuning still requires task-specific configurations (e.g., frame-level vs. non-frame-level conditioning logic).`\n\nThe unified fine-tuning and inference logic for both frame-level and non-frame-level conditions indeed simplifies and streamlines the overall pipeline. In our work, the non-frame-level conditional task mainly refers to the TTS task, while all remaining tasks (voice conversion, speech enhancement, target speaker extraction, and lip2speech) rely on frame-level conditions.\n\nTo handle non-frame-level conditions, we prepend the condition embeddings to the speech representation sequence along the time axis and apply self-attention over the combined sequence. Recent studies [1,2,3] have shown that for non-autoregressive TTS systems, it is effective to **directly concatenate text embeddings with speech representations at the feature/channel level without requiring upsampling or alignment**. Since text embeddings are much shorter than speech representations in sequence length, **the remaining positions can simply be filled with padding embeddings**.\n\nThis approach allows us to unify both frame-level and non-frame-level conditional tasks: *we concatenate the task-specific condition with the speech representations along the feature/channel dimension, rather than the time dimension, and pad the condition to match the sequence length when necessary*.\n\n`Weakness 2: Lack of cross-task analysis: While Metis-Omni is introduced for multi-task fine-tuning, detailed cross-task interference analysis is not provided (e.g., does speech enhancement degrade TTS?).`\n\nThank you for the suggestion. We will provide more details about the inference process of Metis-Omni.\n\nWe observe that Metis-Omni achieves performance that is on par with or even surpasses task-specific fine-tuned models for speech enhancement and target speaker extraction. One exception is the WER metric in zero-shot TTS, where performance is slightly lower. A possible explanation is that, under the same number of training steps, multi-task fine-tuning allocates fewer effective steps to the zero-shot TTS task, resulting in less task-specific optimization.\n\nAdditionally, regarding the inference process: All frame-level conditional tasks (i.e., voice conversion, speech enhancement, and target speaker extraction) share the same adapter, and no additional task ID is required during inference. For zero-shot TTS, the inference process is identical to that of Metis-TTS, and the adapter used for frame-level conditional tasks is not activated during TTS inference.\nMetis-Omni also supports some generalized inference patterns, even if they were not explicitly encountered during training. For example, we can perform text-guided inference for target speaker extraction and speech enhancement by providing the transcript of the target speech (obtained via an ASR system) as an additional condition. As shown in Table 6, this leads to improved performance in some cases.\n\n`Question 1: Can the authors further unify inference logic so that new tasks don’t require any architectural changes?`\n\nWe propose a unified inference logic for both frame-level and non-frame-level conditional tasks, **as raised in Weakness 1**, by concatenating the task-specific condition with the speech representations along the feature (channel) dimension rather than the time dimension, and padding the condition as needed to match the sequence length.\n\n`Question 2: How does Metis handle long-form or streaming generation tasks?`\n\nIn this work, we primarily focus on non-autoregressive (NAR) speech generation tasks and design our model for non-streaming scenarios. However, recent studies [4] have also demonstrated the potential of masked generative models in streaming inference. Specifically, the speech can be split into chunks, where masked generative modeling is applied within each chunk, and autoregressive generation is performed across chunks. We will explore these directions in future work.\n\n`Question 3: How sensitive is the model to the length or content of prompts used in zero-shot or few-shot generation?`\n\nSince our models are trained on speech segments shorter than 30 seconds (due to limitations of the Emilia dataset), we recommend that the prompt length not exceed 15 to 20 seconds. During training, the maximum prompt length was set to half of the total input length. However, we believe this constraint is largely dependent on the training data, and longer prompts could be supported if trained accordingly.\n\nThanks again for your valuable comments, we would be grateful if we could hear your feedback regarding our answers. We would be happy to answer and discuss if you have further comments.\n\nReference\n\n> [1] Eskimez S E, Wang X, Thakker M, et al. E2 tts: Embarrassingly easy fully non-autoregressive zero-shot tts[C]//2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024: 682-689.\n\n> [2] Chen Y, Niu Z, Ma Z, et al. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching[J]. arXiv preprint arXiv:2410.06885, 2024.\n\n> [3] Ning Z, Chen H, Jiang Y, et al. DiffRhythm: Blazingly fast and embarrassingly simple end-to-end full-length song generation with latent diffusion[J]. arXiv preprint arXiv:2503.01183, 2025.\n\n> [4] Arriola M, Gokaslan A, Chiu J T, et al. Block diffusion: Interpolating between autoregressive and diffusion language models[J]. arXiv preprint arXiv:2503.09573, 2025."}, {"review_id": "CNvvROk6tr", "reviewer": "Reviewer_vbjN", "summary": "This paper proposes a novel pre-training and fine-tuning framework that integrates self-supervised learning (SSL) with acoustic token representations. The core approach is based on masked pre-training and is extended to support a variety of speech generation tasks by conditioning on task-specific prompts implemented as prefix tokens. Fine-tuning is also conducted using a masked generation strategy. The experiments cover a broad range of tasks, including text-to-speech (TTS), voice conversion (VC), speech enhancement (SE), and target speaker extraction (TSE), demonstrating state-of-the-art performance with a relatively small number of trainable parameters.", "questions": "- Abstract: The abstract should be revised for clarity and informativeness. It states that the model uses “fewer than 20M trainable parameters or 300 times less training data,” but fails to specify which baseline models are being compared. Without that context, the claim is vague and less impactful.\n- Lines 30–33: The discussion of limitations in prior work is unclear. The claims about the need for large training data or difficulty in task extension lack supporting evidence or citations. Please elaborate with concrete examples, experimental results, or references that justify these limitations, and explain why the proposed method addresses them more effectively.\n- Section 3.4: Please clarify the properties of the prompt used in the method. Is the prompt length fixed across tasks? If not, how does it vary (e.g., longer or shorter depending on the task)?\n- Section 3.5: The explanation of prompts and conditioning is difficult to follow. Please provide more concrete examples (e.g., what do the prompt and condition look like for TTS, VC, or TSE?) to make the mechanism and design choices clearer.\n- Section 4.1: The source of the “200K hours” of training data is not clearly explained. Please provide more details on the datasets used.\n- Section 4.2: The experimental setup is hard to interpret because each baseline uses different training datasets, strategies, and model sizes. A more standardized comparison or ablation study would help clarify where the proposed method truly excels.\n- Section 4.2.1: I could not find a discussion of how the length parameter $n$ is selected for TTS experiments. Please explain how to set the length.\n- Section 4 – Evaluation Metrics: Please reconsider the use of DNSMOS for tasks like TTS and VC, as it is tailored for speech enhancement. Metrics such as UTMOS or other TTS-specific measures would be more appropriate. Subjective evaluations would also help validate perceptual quality. For SE and TSE tasks, reference-based metrics like PESQ and STOI should be included to assess signal-level fidelity.", "limitations": "While the paper addresses technical limitations in unified audio representation and few-shot learning capabilities, it lacks a discussion on the broader societal impacts of the work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- Simple pre-training techniques based on a prompt, which can be applied to various speech generation tasks\n- Shows SOTA in various speech generation tasks\n\nWeaknesses\n- The idea is too simple and incremental. It is only based on the prompt. Pre-text-based approach is very good, but many speech LLM-based systems are conditioned on a prefix.\n- The paper does not have ablation experiments, and its validation is not obvious. E.g., pre-training w/ and w/o prefix would be very informative to show the effectiveness of the proposed method.", "was_revised": true, "final_justification": "The additional explanations and ablation study discussions address my concerns, and I now recommend this paper for acceptance.", "rebuttal": "Dear reviewer, we want to thank for your carefully reading and providing the constructive comments! Below we address the concerns mentioned in the review.\n\n`Weakness 1: The idea is too simple and incremental... but many speech LLM-based systems are conditioned on a prefix.`\n\nIn this work, **introducing the prefix prompt mechanism during pretraining is not essential, but rather a design choice to enable the model to learn how to utilize prompts for tasks that require them**. This mechanism improves the model's adaptability to downstream tasks involving prompts, such as zero-shot TTS, voice conversion, and target speaker extraction. For tasks that do not require prompts, such as speech enhancement, no prompt is used during fine-tuning.\n\nWe also considered **removing the prefix during pretraining**. 20% of the training steps do not include any prompt, as mentioned on line 216. We emphasize that prefix prompting is not the core focus of our work; it can be introduced solely at the fine-tuning stage for specific tasks if desired. Nonetheless, including it during pretraining does not harm performance, and we adopt the mixed strategy.\n\n`Weakness 2: The paper does not have ablation experiments, and its validation is not obvious. E.g., pre-training w/ and w/o prefix would be very informative to show the effectiveness of the proposed method.`\n\nWe conducted ablation studies to isolate the core contribution of our method and validate the effectiveness of the large-scale pretraining–then–fine-tuning paradigm. Taking zero-shot TTS as an example, Metis-TTS with pretraining significantly outperforms Metis-TTS without pretraining on both SIM and WER metrics using the same training data (WER: 2.28 vs. 4.91; SIM: 0.72 vs. 0.69). We also show that with pretraining, our models can be fine-tuned using significantly fewer trainable parameters via LoRA.\n\nWe add the ablation study on **with or without prompt for pretraining** in the response to **reviewer o6KH's question 2**.\n\n`Question 1: Abstract: The abstract should be revised for clarity and informativeness... the claim is vague and less impactful.`\n\nThank you for your suggestion. The \"300 times less\" refers to the fact that, compared to the 300K hours of pretraining data, our fine-tuning only requires 1K hours or less. We will revise and clarify this in the paper to ensure clearer expression.\n\n`Question 2: Lines 30–33: The discussion of limitations in prior work is unclear...`\n\nThank you for your suggestion. We have provided clearer evidence to support our claims. For task-specific models, state-of-the-art systems in zero-shot TTS typically rely on large-scale training data, ranging from 60K to over 100K hours (e.g., MaskGCT, CosyVoice, F5-TTS, VoiceBox, and NaturalSpeech 3). Similarly, recently proposed state-of-the-art VC models such as Vevo (ICLR 2025) are trained on 60K hours of data. Multi-task models like UniAudio use a total of 165K hours, with 60K/60K/20K/10K hours allocated to TTS, VC, SE, and TSE respectively, and contain up to 1B trainable parameters.\n\nIn contrast, our experiments demonstrate that comparable or even better performance can be achieved with significantly less task-specific fine-tuning data with our masked generative pre-training. For example, in the TTS task, our model achieves performance on par with or better than MaskGCT and CosyVoice (both trained on about 100K hours), using only 10K or even 1K hours of data. In the VC task, we obtain better results than Vevo (SIM 0.50 vs. 0.38), using just 0.4K hours of training data compared to their 60K hours.\n\nOur method addresses this challenge more effectively. It is motivated by the observation that models trained on large-scale data can recover masked SSL tokens from prompts and unmasked tokens, even without task-specific conditions, as shown in other domains [1, 2]. Empirically, our pretrained model is able to generate speech that captures the prosodic style and timbre of the prompt. However, without semantic guidance, the generated speech may lack intelligibility, often resulting in random word concatenations. This highlights the importance of task-specific conditioning, which can be efficiently introduced through fine-tuning, allowing the model to adapt flexibly to various speech generation tasks.\n\n`Question 3: Section 3.4: Please clarify the properties of the prompt used in the method...`\n\nDuring pretraining, we do not incorporate any task-specific assumptions. For each training sample, we randomly select a prefix of the sequence to serve as a prompt, which remains unmasked with a probability of $p = 0.8$. The prompt length is uniformly sampled from the range [0%, 40%] of the total sequence length.\n\nDuring finetuning, prompts are used only for tasks that require them (e.g., zero-shot TTS, voice conversion). For tasks that do not require prompts, such as speech enhancement, no prompt is used. In all cases, the prompt mechanism used during fine-tuning is kept consistent with that of pretraining.\n\n`Question 4: Section 3.5: The explanation of prompts and conditioning is difficult to follow...`\n\nWe are happy to provide more details for clearer explanation. For all tasks that require prompts, the prompts refer to the SSL tokens of the prompt speech. Specifically, for zero-shot TTS, VC, and TSE, the prompt corresponds to the SSL tokens of the reference speaker's speech. The condition varies depending on the task: for **TTS**, it refers to the text embedding derived from the input text; for **VC, TSE, and SE**, it corresponds to the speech features of the input speech, such as the source speech for VC, the mixed speech for TSE, and the noisy speech for SE. These features can be either mel-spectrograms or SSL features. In this work, we use w2v-BERT 2.0 SSL features, which are temporally aligned with the target SSL tokens, that is, they share the same sequence length along the time axis.\n\n`Question 5: Section 4.1: The source of the “200K hours” of training data is not clearly explained...`\n\nThe additional 200K hours of data come from Emilia-Large [3], which primarily consists of YODAS2 [4] data processed through the Emilia pipeline.\n\n`Question 6: Section 4.2: The experimental setup is hard to interpret...`\n\nComparing with open-source models can be challenging due to differences in experimental settings. However, for some tasks such as TTS and VC, certain baselines are trained using the Emilia dataset, and our fine-tuning data is a randomly selected subset of the same dataset, albeit with significantly less data. For the lip-to-speech task, we use the same training data as the baseline models.\n\nFor internal comparisons within our method, such as varying the fine-tuning data size or the number of trainable parameters, we ensure that all other variables are held constant. We appreciate your suggestion and will conduct more rigorous and controlled experiments in future work.\n\n`Question 7: Section 4.2.1: I could not find a discussion of how the length parameter n...`\n\nI would like to clarify your question: does the length parameter n refer to the target speech length during testing? That is also our understanding. Since our method does not rely on explicit alignment between text and speech or a duration predictor, we need to specify a target length at inference time. Following previous NAR TTS models that similarly do not use duration predictors (e.g., MaskGCT, E2TTS, F5-TTS), we adopt a simple rule-based approach to determine the target length:\n\n```\ntarget len. = (prompt len. / prompt phone count) × target phone count.\n```\n\n`Question 8: Section 4 – Evaluation Metrics..`\n\n- Thank you for your suggestion! Due to space limitations, we provide **subjective evaluation results for zero-shot TTS and speech enhancement in Appendix D.3 Results of Subjective Evaluation**, taking these two tasks as representative cases. We evaluate QMOS and SMOS to assess speech quality and speaker similarity, respectively.\n\n- We also provide the results of **UTMOS for TTS and VC in response to Reviewer c6Ub's Question 2**.\n\n- We do not include signal-level metrics such as PESQ in the main paper, as we find these metrics are not well-suited for evaluating generative-model-based methods. This observation is consistent with prior works [5, 6]. We add these results and will include them in the appendix of the paper.\n\nModel | PESQ | ESTOI\n-|-|-\nAnyEnhance | 2.78 | 0.84\nMetis-SE |2.91 | 0.86\n\nThanks again for your valuable comments, we would be grateful if we could hear your feedback regarding our answers. We would be happy to answer and discuss if you have further comments.\n\nReference\n\n> [1] Devlin J, Chang M W, Lee K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 2019: 4171-4186.\n\n> [2] He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 16000-16009.\n\n> [3] He H, Shang Z, Wang C, et al. Emilia: A large-scale, extensive, multilingual, and diverse dataset for speech generation[J]. arXiv preprint arXiv:2501.15907, 2025.\n\n> [4] Li X, Takamichi S, Saeki T, et al. Yodas: Youtube-oriented dataset for audio and speech[C]//2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023: 1-8.\n\n> [5] Li X, Wang Q, Liu X. Masksr: Masked language model for full-band speech restoration[J]. arXiv preprint arXiv:2406.02092, 2024.\n\n> [6] Zhang J, Yang J, Fang Z, et al. Anyenhance: A unified generative model with prompt-guidance and self-critic for voice enhancement[J]. arXiv preprint arXiv:2501.15417, 2025."}, {"review_id": "pHPp7tATOs", "reviewer": "Reviewer_c6Ub", "summary": "This paper proposes a unified speech foundation model based on masked speech modeling and the model is tunable for various speech tasks such as voice conversion, target speaker extraction, TTS, speech enhancement and lip-to-speech. The system uses both self-supervised speech embeddings and audio embeddings. These two sets of input features serve two separate purposes, 1) to map input prompt and 2) to generate high quality audio. Once the masked pre-training is done, there is a fine-tuning stage for the tasks mentioned above. During fine-tuning, the SSL tokens are appended with the conditions (text, audio prompt, lip features). Experiments show results for 5 tasks for both LoRA adapted and finetuned models. The models are compared to the ground truth audio as well as some high-performing models in the literature depending on the task. Outputs are evaluated using scores like WER, speaker similarity, DNSMOS as applicable. A second set of experiments evaluate the case where the finetuning stage involves multiple tasks at once. In that experiment, the authors show that the multitask version of the model can generalize to other tasks such as the text-guided target speaker extraction task. This multitask version slightly outperforms the model tuned only for the TSE task but in terms of WER, it cannot outperform the model trained for text-guided TSE.", "questions": "1) Please comment on whether reference [12] has been already published in a peer-reviewed venue. \n\n2) In some experiments, DNSMOS and NISQA evaluations have been used. However, these scores are discriminative enough to compare the quality of various systems. The values themselves may suggest that the system is working but they do not necessarily imply that the proposed model is significantly outperforming another one (for example, Vevo vs. Metis in Table 2). Could you please comment on this? Have you considered using other speech evaluation metrics such as the UTMOSv2 scores?  \n\n3) Minor mistake: Section 4.2.2. mentions \"HireSpeech++\" in the text which should be HierSpeech++ as mentioned in Table 2.", "limitations": "Limitations have been mentioned in the appendix rather than in the main text.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- Quality: A generic speech model training framework has been proposed that can be tuned for various speech tasks. There are various experiments showing positive results as compared to previously published papers. \n\n- Significance: Speech generative modeling is relevant to the speech community and the ability of the proposed model to handle different speech tasks is an important feature to have as a unified model.\n\nWeaknesses: \n\n- Quality: The paper (the text) might be lacking details in terms of reproducibility (even though the author checklist suggests that the code will be made available). Some of the evaluation criteria may not be well-suited for the individual task and/or they do not show demonstrate the model differences well.  \n\n- Originality: Based on the text, it seems that the current paper follows reference [12] quite closely with some differences on the text embedding component. If the reference [12] has been published in a peer-reviewed venue, it may make the novelty of this paper somewhat limited. Based on the current pdf, reference [12] is an arxiv paper. If that is still the case, this paper might satisfy the novelty criterion. \n\n- Clarity: It is good to have multiple experiments on various tasks but it might have been better to share more details on the core model and its training details.", "was_revised": true, "final_justification": "I would like to thank the authors for responding to my concerns especially regarding the novelty of the paper. If the code is also going to be released, I am willing to increase my score to accept.", "rebuttal": "Dear reviewer, we want to thank for your carefully reading and providing the constructive comments! Below we address the concerns mentioned in the review.\n\n`Weakness 1: Quality: The paper (the text) might be lacking details in terms of reproducibility (even though the author checklist suggests that the code will be made available). Some of the evaluation criteria may not be well-suited for the individual task and/or they do not show demonstrate the model differences well.`\n\nThank you for the comments! We have made every effort to improve the reproducibility of our work. We plan to open-source the code for each model, along with detailed configuration files specifying hyperparameter settings and model architecture.\n\nFor evaluation, all baseline models are open-source and evaluated on public benchmarks. We will also provide scripts for computing evaluation metrics on each test set.\n\n`Weakness 2: Originality: Based on the text, it seems that the current paper follows reference [12] quite closely with some differences on the text embedding component. If the reference [12] has been published in a peer-reviewed venue, it may make the novelty of this paper somewhat limited. Based on the current pdf, reference [12] is an arxiv paper. If that is still the case, this paper might satisfy the novelty criterion.`\n\nReference [12], MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer, has already been published at ICLR 2025. We cite MaskGCT in multiple parts of our paper, as both our work and MaskGCT adopt masked generative modeling as the core generative paradigm.\n\nHowever, our contributions go beyond MaskGCT in several key aspects:\n\n- First, while MaskGCT focuses solely on zero-shot TTS, our work extends masked generative modeling to a broader set of speech generation tasks, including zero-shot TTS, voice conversion (VC), speech enhancement (SE), target speaker extraction (TSE), and a multimodal task, lip-to-speech generation.\n\n- Our goal is not to apply masked generative models to solve a single task, but rather to build a unified foundation model with large-scale pretraining, capable of efficiently adapting to a wide range of speech generation tasks. To this end, we design a unified pretraining mechanism, which enables efficient fine-tuning across diverse speech generation tasks. In the pretraining stage, we adopt unconditional masked generative modeling as the objective, without relying on any task-specific conditioning.\n\nMasked generative modeling has shown effectiveness across various domains, e.g., MaskGIT (image generation), MaskGVIT (video generation) in vision, and MaskGCT (TTS), NaturalSpeech 3 (TTS), AnyEnhance (SE and TSE), MaskSR (SE), and in audio, but all of these are task-specific models. In contrast, our work aims to build a unified framework that aims to generalize masked generative modeling for diverse speech generation tasks.\n\n`Weakness 3: Clarity: It is good to have multiple experiments on various tasks but it might have been better to share more details on the core model and its training details.`\n\nWe provide the details of the pre-training process in Section 4.1, including the model architecture, dataset, and training strategy. For each downstream task, we describe any differences from the pre-training setup in the corresponding experimental sections. Due to space limitations, we are unable to provide more details in the main text. However, we will do our best to include additional information, including detailed model architecture, in the appendix.\n\n`Question 1: Please comment on whether reference [12] has been already published in a peer-reviewed venue.`\n\nWe discuss this question in Weakness 2.\n\n`Question 2: In some experiments, DNSMOS and NISQA evaluations have been used. However, these scores are discriminative enough to compare the quality of various systems. The values themselves may suggest that the system is working but they do not necessarily imply that the proposed model is significantly outperforming another one (for example, Vevo vs. Metis in Table 2). Could you please comment on this? Have you considered using other speech evaluation metrics such as the UTMOSv2 scores?`\n\nYes, since DNSMOS and NISQA are commonly used for evaluating SE and TSE tasks, in our work, we also adopt DNSMOS for evaluating zero-shot TTS and VC, and additionally use NISQA for VC, in order to maintain a relatively streamlined evaluation setup.\nHowever, we note that these metrics are not always reliable. For example, DNSMOS and UTMOS are often inaccurate on non-English speech. We present these metrics mainly for reference, and for zero-shot TTS and VC we believe SIM and WER are more meaningful.\nIn addition, based on our experience, SIM computed with WavLM embeddings also correlates with speech quality to some extent. We also provide subjective evaluation results in Appendix D.3 Results of Subjective Evaluation.\n\nWe are happy to supplement the UTMOS scores below for reference:\n\nZero-shot TTS:\n\n| Model       | UTMOS (SeedTTS test-en) | UTMOS (SeedTTS test-zh) |\n|-------------|--------------------------|--------------------------|\n| Metis-TTS   | 3.82                     | 2.75                     |\n| MaskGCT     | 3.59                     | 2.68                     |\n| CosyVoice   | 3.78                     | 2.81                     |\n| F5-TTS      | 3.51                     | 2.67                     |\n\nVC:\n\n| Model       | UTMOS (SeedTTS-VC-en) | UTMOS (SeedTTS-VC-zh) |\n|-------------|------------------------|------------------------|\n| Metis-VC    | 3.65                   | 2.97                   |\n| SeedVC      | 3.31                   | 2.69                   |\n| Vevo        | 3.51                   | 2.86                   |\n\n`Question 3: Minor mistake: Section 4.2.2. mentions \"HireSpeech++\" in the text which should be HierSpeech++ as mentioned in Table 2.`\n\nThank you for pointing this out!  We will address and fix it.\n\nThanks again for your valuable comments, we would be grateful if we could hear your feedback regarding our answers. We would be happy to answer and discuss if you have further comments."}, {"review_id": "4AjIAjGKJL", "reviewer": "Reviewer_o6KH", "summary": "In this paper, the authors propose a framework that first pre-trains a model using a large amount of unlabeled data, and then fine-tunes it for specific downstream tasks. They introduce different conditioning methods for two categories of downstream tasks: (1) non-frame-level conditioning, and (2) frame-level conditioning. Through evaluation, the authors conduct comprehensive experiments on five different tasks and demonstrate the effectiveness of the pre-trained model and the proposed framework. They also argue that, during the fine-tuning stage, the model can be fine-tuned using a relatively small amount of data and a small number of parameters through LoRA.", "questions": "- The proposed method appears to adopt a non-autoregressive generation approach (with iterative decoding), rather than the more common autoregressive Transformer-based methods typically used in the NLP domain. Is there any specific insight or reasoning behind this architectural choice? One possible advantage could be faster inference speed. It would be helpful to know how the inference speed of the proposed method compares to that of other existing approaches.\n- Why does the model require a prompt during the pre-training stage? Is it because, for non-frame-level conditioning tasks, the model needs to attend to the conditioning information that appears before the generated sequence? If so, does omitting the prompt during pre-training degrade the model’s performance, even after fine-tuning? It would be helpful if the authors clarified whether prompting in pre-training is essential for enabling effective conditioning during downstream tasks.\n- As already mentioned in the weaknesses section, is the choice of LoRA rank related to the difficulty of the tasks? For example, when using LoRA with rank 4 for zero-shot TTS or VC tasks, was the performance significantly degraded or was the model unable to train effectively?\n- Additionally, what happens when the tasks are trained with a smaller amount of data? Using 1,000 hours of data is still a large amount—especially for low-resource languages or personalized systems. If this model can demonstrate effectiveness with significantly less data, it would be highly valuable for the research community.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- This paper demonstrates that the proposed methods can be utilized for various downstream tasks through fine-tuning with a relatively small amount of data and achieve better performance than comparison models across various tasks.\n- The authors conduct a comprehensive evaluation on five different tasks: zero-shot TTS, voice conversion, target speaker extraction, speech enhancement, and lip-to-speech. Additionally, they demonstrate the potential of the proposed model for multi-task learning through experiments. This evaluation clearly demonstrates the effectiveness of the proposed method.\n- I believe this research will benefit the community by showcasing the potential of a general-purpose speech foundation model. The authors also plan to release the model code and checkpoints along with the paper.\n### Weaknesses\n- From an application standpoint, the authors conduct comprehensive experiments and achieve strong results. However, in terms of methodological novelty, it is difficult to identify a clearly novel aspect of the proposed approach. The main novelty appears to lie in using different fine-tuning strategies depending on the nature of the downstream task—specifically, prompting for non-frame-level conditioning and adding the condition to the input for frame-level conditioning. However, I believe both conditioning methods are not novel and are commonly used in various large-scale models.\n- In the evaluation, the explanation or ablation study regarding architectural choices is limited. For example, the authors use different LoRA ranks for different downstream tasks, but there is no insight or explanation provided for this decision. It remains unclear whether the choice of LoRA rank depends on the difficulty or complexity of the downstream task.\n- The purpose of training a general-purpose foundation model is to enable effective and efficient adaptation to a variety of tasks. The authors claim that their proposed method can be fine-tuned for various tasks using a small amount of data. However, in the zero-shot TTS task, they use 1,000 hours of data, which is not a small amount. Therefore, to better demonstrate the effectiveness and generalization ability of their foundation model, the authors should fine-tune it with a smaller dataset and showcase its performance under low-resource conditions.", "was_revised": true, "final_justification": "I believe this paper will contribute to the community by proposing a general-purpose speech foundation model using masked generative modeling (MGM) and demonstrating its adaptability to various tasks. Moreover, the authors plan to release the code publicly.", "rebuttal": "Dear reviewer, we want to thank for your carefully reading and providing the constructive comments! Below we address the concerns mentioned in the review.\n\n`Weakness 1: From an application standpoint, the authors conduct comprehensive experiments and achieve strong results... However, I believe both conditioning methods are not novel and are commonly used in various large-scale models.`\n\nThank you for the thoughtful feedback! While some components of our method (e.g., conditioning strategies) have been explored in prior work, the core novelty of our approach lies in leveraging masked generative modeling for pretraining a unified foundation model for speech generation. This model can be efficiently adapted to a wide range of speech generation tasks using only a small amount of training data and trainable parameters. The fine-tuning strategies across tasks are distinguished only by the type of condition, whether it is frame-level or not. As a result, tasks such as VC, SE, TSE, and lip2speech share the same fine-tuning and model architecture, differing only in the form of input condition, which naturally reflects their task-specific requirements. We also propose a unified fine-tuning and inference mechanism, **as discussed in our response to Reviewer zbxf (weakness 1), to further simplify and standardize the framework across tasks**.\n\nOur goal is not to solve a single task with masked generative models, but to build a general-purpose foundation model that can flexibly adapt to diverse speech generation scenarios through a shared architecture and training pipeline.\n\n`Weakness 2: In the evaluation, the explanation or ablation study regarding architectural choices is limited...`\n\nWe are happy to provide more clarification regarding the choice of LoRA ranks for different tasks. First, we find that LoRA fine-tuning is effective across all tasks, though some tasks can achieve strong performance with significantly fewer trainable parameters.\nIn general, higher LoRA ranks lead to better performance. However, for certain tasks, even a very small LoRA rank (e.g., 4) achieves satisfactory results. For example, in SE and TSE, we observe that a LoRA rank of 4 is sufficient. We hypothesize this is because these tasks use frame-level conditions that are closely aligned with the target speech tokens, and the model's primary goal is to extract the target speech from the input.\n\nIn contrast, for tasks like TTS, the model must implicitly learn the alignment between the condition (text) and the target speech tokens, which requires more modeling capacity. We find that a LoRA rank of 32 provides strong performance in this setting, while lower ranks lead to notable degradation—particularly in WER. For example, using a LoRA rank of 4 for TTS results in a WER of around 10 on the SeedTTS-en test set. Notably, even with a LoRA rank of 32, the number of trainable parameters remains relatively small, only around 32M.\n\nWe will add these details to the paper to provide a clearer explanation of our rank selection strategy.\n\n`Weakness 3: The purpose of training a general-purpose foundation model is to enable effective and efficient adaptation to a variety of tasks... the authors should fine-tune it with a smaller dataset and showcase its performance under low-resource conditions.`\n\nThank you for the great suggestion! In our experiments, we have explored training with even less than 1000 hours of data. For example, in Section 4.2.1, we fine-tune Metis-TTS LoRA-32 on the LibriSpeech dataset using only 0.58K hours of training data. Despite the small scale, the model achieves a WER of 4.55 on the SeedTTS test-en set, outperforming strong baselines such as VALL-E (trained on 45K hours) and VoiceCraft (9K hours), and approaching the performance of CosyVoice (170K hours). It also achieves a SIM score of 0.72, surpassing most baselines.\n\nSimilarly, in the voice conversion (VC) task, we use only 0.4K hours of training data and still outperform the SOTA baseline Vevo (trained on 60K hours) in terms of SIM and quality metrics, with a slightly higher WER (4.49 vs. 3.38).\n\nTo further validate the low-data regime setting, we extend our experiments to even smaller training sets. Due to resource and data constraints, we take TTS and VC as representative tasks and randomly sample 0.1K hours of English data from the Emilia dataset for fine-tuning.\n\nSeedTTS en:\n| Model                     | WER  | SIM  |\n|-|-|-|\n| Metis-TTS  finetune 10K | 2.28 | 0.72 |\n| Metis-TTS  finetune 1K    | 3.86 | 0.71 |\n| Metis-TTS   finetune 0.1K  | 5.97 | 0.68 |\n\nVCTK: \n| Model                     | WER  | SIM  |\n|-|-|-|\n| Metis-VC finetune  0.4K    | 6.65 | 0.48 |\n| Metis-VC finetune 0.1K  | 7.01 | 0.45 |\n\n`Question 1: The proposed method appears to adopt a non-autoregressive generation approach... It would be helpful to know how the inference speed of the proposed method compares to that of other existing approaches.`\n\nWe are happy to provide more justification for choosing masked generative modeling (MGM) as our modeling approach for speech tokens.\n\n- Previous works have demonstrated that MGM can yield better performance under comparable conditions for speech generation tasks. For instance, [1] shows improvements in TTS, and [2,3] report superior results for speech enhancement compared to AR baselines.\n\n- For most tasks with non-frame-level conditions, such as VC, SE, TSE, the conditioning inputs are typically more dense and locally aligned with the target speech tokens. We observe that the attention matrices in these tasks exhibit strong locality, making MGM a more efficient modeling choice. Additionally, as a NAR method, MGM supports faster inference compared to AR models, as detailed below.\n\n- Compared with AR models, MGM shows advantages in both training and inference efficiency, especially in tasks with frame-level conditions. AR models often require concatenating condition tokens with target tokens or using cross-attention, both of which significantly increase training complexity. In contrast, MGM can model targets independently and requires only a few iterative steps during inference. While AR models can benefit from KV caching, we find that MGM offers better inference speed in most scenarios under the same sequence length. For example, in our TTS experiments, MGM achieves lower RTF (0.35 vs. 0.57), and its advantage becomes even more pronounced as the token rate decreases. Our recent exploration shows that at a token rate of 12.5Hz, MGM achieves a lower RTF (0.18 vs. 0.31) than AR for TTS (The RTF is measured on an H100 GPU using 10-second speech samples).\n\n- MGMs represent a promising and under-explored generative paradigm in speech pretraining. While prior works such as MaskGCT (for TTS) and AnyEnhance (for SE) have adopted MGM, they are task-specific models. In contrast, our goal is to explore the potential of MGM in building a unified pretraining framework for various speech generation tasks. Moreover, recent studies have begun exploring MGM-based pretraining in other domains, such as text generation [4] and text-image multimodal modeling [5], further validating its generality and potential.\n\n`Question 2: Why does the model require a prompt during the pre-training stage?...`\n\nIn this work, introducing the prefix prompt mechanism during pretraining is a design choice to enable the model to learn how to utilize prompts for tasks that require them. This mechanism improves the model's adaptability to downstream tasks involving prompts, such as zero-shot TTS, voice conversion, and target speaker extraction. For tasks that do not require prompts, such as speech enhancement, no prompt is used during fine-tuning. In practice, we adopt a mixed strategy during pretraining: prompts are introduced in 80% of training steps, while the remaining 20% are trained without prompts. This design encourages the model to develop in-context learning abilities, specifically to better support prompt-based downstream tasks. \n\nWe are happy to provide an ablation study. We retrained the pre-trained model without using prompts, and then fine-tuned it on the zero-shot TTS task with the prompt mechanism. (Due to time and resource constraints, we will include results for other tasks in future work.) The results show that removing prompts during pretraining leads to a drop in SIM, while WER remains largely unaffected.\n\nSeedTTS en:\n| Model                     | WER  | SIM  |\n|-|-|-|\n| Metis-TTS   | 2.28 | 0.72 |\n| Metis-TTS (pre-training no prompt)     | 2.41 | 0.70 |\n| Metis-TTS (without pre-training)     | 4.91 | 0.69 |\n\nSeedTTS zh:\n| Model                     | WER  | SIM  |\n|-|-|-|\n| Metis-TTS   | 2.30 | 0.77 |\n| Metis-TTS (pre-training no prompt)     | 2.59 | 0.73 |\n| Metis-TTS (without pre-training)     | 4.98 | 0.73 |\n\n`Question 3: ...`\n\nWe discuss the question in weakness 2.\n\n`Question 4: Additionally, what happens when the tasks are trained with a smaller amount of data?...`\n\nThank you for the insightful question! We discuss the question in weakness 3.\n\nThanks again for your valuable comments, we would be grateful if we could hear your feedback regarding our answers. We would be happy to answer and discuss if you have further comments.\n\nReference:\n\n> [1] Wang Y, Zhan H, Liu L, et al. Maskgct: Zero-shot text-to-speech with masked generative codec transformer[J]. arXiv preprint arXiv:2409.00750, 2024.\n\n> [2] Zhang J, Yang J, Fang Z, et al. Anyenhance: A unified generative model with prompt-guidance and self-critic for voice enhancement[J]. arXiv preprint arXiv:2501.15417, 2025.\n\n> [3] Li X, Wang Q, Liu X. Masksr: Masked language model for full-band speech restoration[J]. arXiv preprint arXiv:2406.02092, 2024.\n\n> [4] Nie S, Zhu F, You Z, et al. Large language diffusion models[J]. arXiv preprint arXiv:2502.09992, 2025.\n\n> [5] Yang L, Tian Y, Li B, et al. Mmada: Multimodal large diffusion language models[J]. arXiv preprint arXiv:2505.15809, 2025."}]

Author rebuttal:

Meta-review: This is a paper about building a strong foundation model for various speech generation tasks. There is very little novelty (as pointed out by reviewer vbjN, c6Ub, and o6KH). The prompts during pre-training could potentially be considered one, but still relatively minor. However, the execution is remarkable, and the results are impressive.

Overall, all reviewers recognize the significance of the paper. I'd urge the authors carefully consider the reviewers recommended revision, including ablation (reviewer vbjN) and reproducibility (reviewer c6Ub).
Decision: Accept (poster)