Title: Metis: A Foundation Speech Generation Model with Masked Generative Pre-training
Venue: NeurIPS 2025
Accepted: yes
Keywords: Speech Generation

Foundation Model

Masked Generative Model
Abstract: We introduce ***Metis***, a foundation model for unified speech generation.
Unlike previous task-specific or multi-task models, Metis follows a pre-training and fine-tuning paradigm. It is pre-trained on large-scale unlabeled speech data using masked generative modeling and then fine-tuned to adapt to diverse speech generation tasks.
Specifically, 
(1) Metis utilizes two discrete speech representations: SSL tokens derived from speech self-supervised learning (SSL) features, and acoustic tokens directly quantized from waveforms. 
(2) Metis performs masked generative pre-training on SSL tokens, utilizing 300K hours of diverse speech data, without any additional condition. 
(3) Through fine-tuning with task-specific conditions, Metis achieves efficient adaptation to various speech generation tasks while supporting multimodal input, even when using limited data and trainable parameters.
Experiments demonstrate that Metis can serve as a foundation model for unified speech generation: Metis outperforms state-of-the-art task-specific or multi-task systems across five speech generation tasks, including zero-shot text-to-speech, voice conversion, target speaker extraction, speech enhancement, and lip-to-speech, even with fewer than 20M trainable parameters or 300 times less training data. Audio samples are are available at https://metis-demo.github.io/. We release the code and model checkpoints at https://github.com/open-mmlab/Amphion.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a framework that first pre-trains a model using a large amount of unlabeled data, and then fine-tunes it for specific downstream tasks. They introduce different conditioning methods for two categories of downstream tasks: (1) non-frame-level conditioning, and (2) frame-level conditioning. Through evaluation, the authors conduct comprehensive experiments on five different tasks and demonstrate the effectiveness of the pre-trained model and the proposed framework. They also argue that, during the fine-tuning stage, the model can be fine-tuned using a relatively small amount of data and a small number of parameters through LoRA.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- This paper demonstrates that the proposed methods can be utilized for various downstream tasks through fine-tuning with a relatively small amount of data and achieve better performance than comparison models across various tasks.\n- The authors conduct a comprehensive evaluation on five different tasks: zero-shot TTS, voice conversion, target speaker extraction, speech enhancement, and lip-to-speech. Additionally, they demonstrate the potential of the proposed model for multi-task learning through experiments. This evaluation clearly demonstrates the effectiveness of the proposed method.\n- I believe this research will benefit the community by showcasing the potential of a general-purpose speech foundation model. The authors also plan to release the model code and checkpoints along with the paper.\n### Weaknesses\n- From an application standpoint, the authors conduct comprehensive experiments and achieve strong results. However, in terms of methodological novelty, it is difficult to identify a clearly novel aspect of the proposed approach. The main novelty appears to lie in using different fine-tuning strategies depending on the nature of the downstream task—specifically, prompting for non-frame-level conditioning and adding the condition to the input for frame-level conditioning. However, I believe both conditioning methods are not novel and are commonly used in various large-scale models.\n- In the evaluation, the explanation or ablation study regarding architectural choices is limited. For example, the authors use different LoRA ranks for different downstream tasks, but there is no insight or explanation provided for this decision. It remains unclear whether the choice of LoRA rank depends on the difficulty or complexity of the downstream task.\n- The purpose of training a general-purpose foundation model is to enable effective and efficient adaptation to a variety of tasks. The authors claim that their proposed method can be fine-tuned for various tasks using a small amount of data. However, in the zero-shot TTS task, they use 1,000 hours of data, which is not a small amount. Therefore, to better demonstrate the effectiveness and generalization ability of their foundation model, the authors should fine-tune it with a smaller dataset and showcase its performance under low-resource conditions.\n\nQUESTIONS:\n- The proposed method appears to adopt a non-autoregressive generation approach (with iterative decoding), rather than the more common autoregressive Transformer-based methods typically used in the NLP domain. Is there any specific insight or reasoning behind this architectural choice? One possible advantage could be faster inference speed. It would be helpful to know how the inference speed of the proposed method compares to that of other existing approaches.\n- Why does the model require a prompt during the pre-training stage? Is it because, for non-frame-level conditioning tasks, the model needs to attend to the conditioning information that appears before the generated sequence? If so, does omitting the prompt during pre-training degrade the model’s performance, even after fine-tuning? It would be helpful if the authors clarified whether prompting in pre-training is essential for enabling effective conditioning during downstream tasks.\n- As already mentioned in the weaknesses section, is the choice of LoRA rank related to the difficulty of the tasks? For example, when using LoRA with rank 4 for zero-shot TTS or VC tasks, was the performance significantly degraded or was the model unable to train effectively?\n- Additionally, what happens when the tasks are trained with a smaller amount of data? Using 1,000 hours of data is still a large amount—especially for low-resource languages or personalized systems. If this model can demonstrate effectiveness with significantly less data, it would be highly valuable for the research community.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a unified speech foundation model based on masked speech modeling and the model is tunable for various speech tasks such as voice conversion, target speaker extraction, TTS, speech enhancement and lip-to-speech. The system uses both self-supervised speech embeddings and audio embeddings. These two sets of input features serve two separate purposes, 1) to map input prompt and 2) to generate high quality audio. Once the masked pre-training is done, there is a fine-tuning stage for the tasks mentioned above. During fine-tuning, the SSL tokens are appended with the conditions (text, audio prompt, lip features). Experiments show results for 5 tasks for both LoRA adapted and finetuned models. The models are compared to the ground truth audio as well as some high-performing models in the literature depending on the task. Outputs are evaluated using scores like WER, speaker similarity, DNSMOS as applicable. A second set of experiments evaluate the case where the finetuning stage involves multiple tasks at once. In that experiment, the authors show that the multitask version of the model can generalize to other tasks such as the text-guided target speaker extraction task. This multitask version slightly outperforms the model tuned only for the TSE task but in terms of WER, it cannot outperform the model trained for text-guided TSE.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Quality: A generic speech model training framework has been proposed that can be tuned for various speech tasks. There are various experiments showing positive results as compared to previously published papers. \n\n- Significance: Speech generative modeling is relevant to the speech community and the ability of the proposed model to handle different speech tasks is an important feature to have as a unified model.\n\nWeaknesses: \n\n- Quality: The paper (the text) might be lacking details in terms of reproducibility (even though the author checklist suggests that the code will be made available). Some of the evaluation criteria may not be well-suited for the individual task and/or they do not show demonstrate the model differences well.  \n\n- Originality: Based on the text, it seems that the current paper follows reference [12] quite closely with some differences on the text embedding component. If the reference [12] has been published in a peer-reviewed venue, it may make the novelty of this paper somewhat limited. Based on the current pdf, reference [12] is an arxiv paper. If that is still the case, this paper might satisfy the novelty criterion. \n\n- Clarity: It is good to have multiple experiments on various tasks but it might have been better to share more details on the core model and its training details.\n\nQUESTIONS:\n1) Please comment on whether reference [12] has been already published in a peer-reviewed venue. \n\n2) In some experiments, DNSMOS and NISQA evaluations have been used. However, these scores are discriminative enough to compare the quality of various systems. The values themselves may suggest that the system is working but they do not necessarily imply that the proposed model is significantly outperforming another one (for example, Vevo vs. Metis in Table 2). Could you please comment on this? Have you considered using other speech evaluation metrics such as the UTMOSv2 scores?  \n\n3) Minor mistake: Section 4.2.2. mentions \"HireSpeech++\" in the text which should be HierSpeech++ as mentioned in Table 2.\n\nLIMITATIONS:\nLimitations have been mentioned in the appendix rather than in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel pre-training and fine-tuning framework that integrates self-supervised learning (SSL) with acoustic token representations. The core approach is based on masked pre-training and is extended to support a variety of speech generation tasks by conditioning on task-specific prompts implemented as prefix tokens. Fine-tuning is also conducted using a masked generation strategy. The experiments cover a broad range of tasks, including text-to-speech (TTS), voice conversion (VC), speech enhancement (SE), and target speaker extraction (TSE), demonstrating state-of-the-art performance with a relatively small number of trainable parameters.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Simple pre-training techniques based on a prompt, which can be applied to various speech generation tasks\n- Shows SOTA in various speech generation tasks\n\nWeaknesses\n- The idea is too simple and incremental. It is only based on the prompt. Pre-text-based approach is very good, but many speech LLM-based systems are conditioned on a prefix.\n- The paper does not have ablation experiments, and its validation is not obvious. E.g., pre-training w/ and w/o prefix would be very informative to show the effectiveness of the proposed method.\n\nQUESTIONS:\n- Abstract: The abstract should be revised for clarity and informativeness. It states that the model uses “fewer than 20M trainable parameters or 300 times less training data,” but fails to specify which baseline models are being compared. Without that context, the claim is vague and less impactful.\n- Lines 30–33: The discussion of limitations in prior work is unclear. The claims about the need for large training data or difficulty in task extension lack supporting evidence or citations. Please elaborate with concrete examples, experimental results, or references that justify these limitations, and explain why the proposed method addresses them more effectively.\n- Section 3.4: Please clarify the properties of the prompt used in the method. Is the prompt length fixed across tasks? If not, how does it vary (e.g., longer or shorter depending on the task)?\n- Section 3.5: The explanation of prompts and conditioning is difficult to follow. Please provide more concrete examples (e.g., what do the prompt and condition look like for TTS, VC, or TSE?) to make the mechanism and design choices clearer.\n- Section 4.1: The source of the “200K hours” of training data is not clearly explained. Please provide more details on the datasets used.\n- Section 4.2: The experimental setup is hard to interpret because each baseline uses different training datasets, strategies, and model sizes. A more standardized comparison or ablation study would help clarify where the proposed method truly excels.\n- Section 4.2.1: I could not find a discussion of how the length parameter $n$ is selected for TTS experiments. Please explain how to set the length.\n- Section 4 – Evaluation Metrics: Please reconsider the use of DNSMOS for tasks like TTS and VC, as it is tailored for speech enhancement. Metrics such as UTMOS or other TTS-specific measures would be more appropriate. Subjective evaluations would also help validate perceptual quality. For SE and TSE tasks, reference-based metrics like PESQ and STOI should be included to assess signal-level fidelity.\n\nLIMITATIONS:\nWhile the paper addresses technical limitations in unified audio representation and few-shot learning capabilities, it lacks a discussion on the broader societal impacts of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Metis, a unified speech generation foundation model trained via masked generative pre-training on unlabeled speech data. Metis is trained using two discrete speech representations and can be fine-tuned with minimal labeled data and parameters for multiple downstream tasks. The Key features include: (1) two-stage speech generation using SSL tokens (semantic/prosodic) and acoustic tokens (waveform-level); (2) pre-training on 300K hours of speech with a masked generative model, without any task conditions; (3) efficient adaptation to tasks like zero-shot TTS, voice conversion, speech enhancement, target speaker extraction, and lip-to-speech, using LoRA or full fine-tuning. The model outperforms state-of-the-art systems on multiple tasks and shows strong generalization with limited data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n(1) Unified framework: The paper proposes a clear and modular two-stage pipeline (SSL token prediction → acoustic generation) that enables generalization across tasks.\n\n(2) Efficient adaptation: The model achieves SOTA or near-SOTA performance on five speech generation tasks using <20M parameters and orders-of-magnitude less training data.\n\n(3) Multimodal support: The model is capable of handling text, audio, and video conditions, allowing for novel combinations like text-guided speaker extraction (Metis-Omni).\n\n(4) Extensive experiments: Includes strong baseline comparisons, ablations (e.g., with and without pretraining), LoRA configurations, and results on real-world and synthetic test sets.\n\nWeaknesses:\n\n(1) No unified inference pipeline: While pre-training is unified, fine-tuning still requires task-specific configurations (e.g., frame-level vs. non-frame-level conditioning logic).\n\n(2) Lack of cross-task analysis: While Metis-Omni is introduced for multi-task fine-tuning, detailed cross-task interference analysis is not provided (e.g., does speech enhancement degrade TTS?).\n\nQUESTIONS:\n(1) Can the authors further unify inference logic so that new tasks don’t require any architectural changes?\n\n(2) How does Metis handle long-form or streaming generation tasks?\n\n(3) How sensitive is the model to the length or content of prompts used in zero-shot or few-shot generation?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}