MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer

Authors: Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, Zhizheng Wu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: text-to-speech synthesis, masked generative models, codec language models, voice cloning

Abstract:
The recent large-scale text-to-speech (TTS) systems are usually grouped as autoregressive and non-autoregressive systems. The autoregressive systems implicitly model duration but exhibit certain deficiencies in robustness and lack of duration controllability. Non-autoregressive systems require explicit alignment information between text and speech during training and predict durations for linguistic units (e.g. phone), which may compromise their naturalness.
In this paper, we introduce $\textbf{Mask}$ed $\textbf{G}$enerative $\textbf{C}$odec $\textbf{T}$ransformer (MaskGCT), a fully non-autoregressive TTS model that eliminates the need for explicit alignment information between text and speech supervision, as well as phone-level duration prediction. MaskGCT is a two-stage model: in the first stage, the model uses text to predict semantic tokens extracted from a speech self-supervised learning (SSL) model, and in the second stage, the model predicts acoustic tokens conditioned on these semantic tokens. MaskGCT follows the mask-and-predict learning paradigm. During training, MaskGCT learns to predict masked semantic or acoustic tokens based on given conditions and prompts. During inference, the model generates tokens of a specified length in a parallel manner. 
Experiments with 100K hours of in-the-wild speech demonstrate that MaskGCT outperforms the current state-of-the-art zero-shot TTS systems in terms of quality, similarity, and intelligibility. Audio samples are available at https://maskgct.github.io/. We release our code and model checkpoints at https://github.com/open-mmlab/Amphion/blob/main/models/tts/maskgct.

Decision: accept (poster)
Reviewer scores: 6, 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1729994396504

thread_type:
official_review

<thread_object>
creator: reviewer_ezbh

<content>
summary:
This paper presents MASKGCT, a compact Non-Autoregressive (NAR) text-to-speech model. This model is meticulously designed with a specific emphasis on attaining outstanding audio quality. What sets it apart from prior works is its utilization of semantic tokens extracted from the SSL representation using VQVAE and its avoidance of relying on explicit alignment information.

strengths:
- This paper offers quite interesting and remarkably high-quality demo audio. The audio samples provided within the paper not only demonstrate clear and distinct sound qualities but also exhibit a certain level of innovation in terms of the audio characteristics they present. They are able to effectively capture the attention of the readers and give a practical sense of the research outcomes in the audio domain.
- The proposed pipeline is significantly more compact than previous ones such as NaturalSpeech3. This compactness implies a more streamlined and efficient design. It potentially leads to reduced computational complexity and may offer advantages in terms of implementation and resource utilization.

weaknesses:
- The Word Error Rate (WER) in Table 4 and 5 of all methods is much higher than previously reported. This discrepancy requires more in-depth explanation. It is essential to analyze the factors contributing to this higher WER, such as possible differences in the data sets used, the experimental settings, or any unique characteristics of the methods employed in this study. A detailed exploration and clarification of these aspects would enhance the understanding and validity of the results presented.
- In Section 3.2.1, the authors utilize VQVAE to obtain discrete semantic tokens instead of k-means. However, no corresponding experiment result is provided. It would be beneficial to include the experimental results related to the use of k-means for a more comprehensive understanding. These results could demonstrate the effectiveness and performance of VQVAE in this context, and also facilitate a comparison with the potential outcomes if k-means had been used. This would add more substance to the discussion and support the authors' choice of using VQVAE.
- To my knowledge, using text as a condition without expanding by duration for a Non-Autoregressive (NAR) model was first proposed in e2-tts. The authors should point out this in Section 3.2.2. Acknowledging the prior work in this area is important for providing a complete context and showing the evolution of the research. By highlighting the connection to e2-tts, the authors can better position their work within the existing body of knowledge and give credit to the relevant pioneering research. It also helps readers to better understand the background and significance of the approach taken in this study.

questions:
Address my concerns in the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730035273956

thread_type:
official_review

<thread_object>
creator: reviewer_avmh

<content>
summary:
The authors present a zero-short text-to-speech (TTS) system called Masked Generative Codec Transformer (MaskGCT), which comprises two stages: 1) predicting semantic tokens from text and 2) using these semantic tokens to generate acoustic tokens. Both stages leverage a non-autoregressive masked generative transformer with specific prompt tokens for in-context learning. A VQ-VAE codec and a residual vector quantization (RVQ) codec are separately trained to obtain the semantic tokens and acoustic tokens. Extensive experiments across various tasks demonstrate that MaskGCT, trained on 100k hours of speech data, surpasses existing state-of-the-art zero-shot TTS systems in quality, similarity, and intelligibility.

strengths:
- This work utilizes non-autoregressive masked generative transformers to generate both sematic tokens and acoustic tokens, offering advantages such as flexible length control, better robustness, and higher inference efficiency. This is suggested by the experiments in Section 4.2.2, comparing to the system replacing the first stage with an autoregressive model (i.e., AR + SoundStorm). 
- The authors conduct extensive experiments on a variety of tasks to evaluate the effectiveness and sca of the proposed system, including not only standard zero-shot TTS, but also cross-lingual dubbing, voice conversion, emotion control, and speech content editing, with potential use of post-training and fine-tuning. 
- The selected state-of-the-art baselines are well-chosen.

weaknesses:
- The novelty of this work is somewhat incremental, though I would appreciate its engineering contributions. This work follows a two-stage codec-based sequence-to-sequence strategy, converting text into semantic tokens and then into acoustic tokens, which is in line with SPEAR-TTS [1]. The SoundStorm [2] paper also explores this strategy. Additionally, the idea of applying non-autoregressive masked generative transformer for codec-based TTS is presented in SoundStorm [2] and NaturalSpeech3 [3] (though the NaturalSpeech3 paper describes it from a diffusion perspective.). However, I would like to highlight the distinction that this work employs non-autoregressive masked generative transformers in both stages of the two-stage framework. 
- The authors emphasize that the proposed method does not require text-speech alignment supervision or phone-level duration prediction. However, it appears that the method does require a specified length for the target semantic token sequence, thereby to control the length of the generate speech. The presented experimental results in Table 2, comparing to baseline methods, are based on either ground-truth duration or predicted duration. As descibed in the paper, the authors train a duration predictor to estimate the phone-level durations, which are then summed to determine the total duration of the target speech. This process relies on pre-computed ground-truth phone-level durations. Therefore, it seems that this may not fully align with the claim of not requiring text-speech alignment supervision or phone-level duration prediction. 
- The authors claim that training a VQ-VAE model to quantize the semantic representation can reduce the information loss, compared to using k-means clustering as in previous works. I think it would be better if the authors can provide ablation results or relevant references to support this assertion. 

[1] Kharitonov, E., Vincent, D., Borsos, Z., Marinier, R., Girgin, S., Pietquin, O., ... & Zeghidour, N. (2023). Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics, 11, 1703-1718.

[2] Borsos, Z., Sharifi, M., Vincent, D., Kharitonov, E., Zeghidour, N., & Tagliasacchi, M. (2023). Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.09636.

[3] Ju, Z., Wang, Y., Shen, K., Tan, X., Xin, D., Yang, D., ... & Zhao, S. (2024). Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100.

questions:
- In Table 1, does "ZS TTS" denote "zero-shot TTS", and what's "CL TTS"? For "Imp. Dur.", I also don't fully agree that this work implicitly models duration, as it requires a specified length of the target generated speech.  
- In Section 3.2.3,  why "the number of frames in the semantic token sequence is equal to the sum of the frames in the prompt acoustic sequence and the target acoustic sequence"? 
- Is the output length of the text-to-semantic stage equal to that from the semantic-to-acoustic stage? From Figure 2, the former seems shorter than the latter? 
- In the first paragraph of Section 2, about "SpearTTS utilizes three AR models to predict semantic tokens from text, coarse-grained acoustic tokens from semantic tokens, and fine-grained acoustic tokens from coarse-grained tokens.", the terms "coarse-grained" and "fine-grained" do not seem to be mentioned in the SpearTTS [1] paper, what do the authors mean by them? 
- What's the length of prompt audio? Does it affect the quality of the generated speech?

[1] Kharitonov, E., Vincent, D., Borsos, Z., Marinier, R., Girgin, S., Pietquin, O., ... & Zeghidour, N. (2023). Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics, 11, 1703-1718.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730589448016

thread_type:
official_review

<thread_object>
creator: reviewer_n456

<content>
summary:
This paper introduces MaskGCT, a non-autoregressive (NAR) masked generative modeling method that sequentially generates audio from acoustic tokens, semantic tokens, and text. The semantic tokens are derived from VQ-VAE of W2V-BERT, a self-supervised speech learning model. The acoustic token approach extends DAC with improved training efficiency.

Results demonstrate that MaskGCT either outperforms or matches current autoregressive (AR) based methods and NAR duration-based methods with publicly available implementation details, across multiple benchmarks: LibriSpeech, SeedTTS (Chinese and English), zero-shot TTS, and speech style (timbre, emotion, accent, etc.) imitation learning. Additionally, MaskGCT surpasses AR-based methods in both naturalness and computational efficiency (inference time steps).

strengths:
* This paper experimentally demonstrates that NAR (non-autoregressive) non-duration-based TTS methods outperform both AR (autoregressive) based and NAR duration-based TTS methods with publicly available implementation details. 

* The open-source implementation and comprehensive reproducibility details provided in the appendix represent valuable contributions to the field.

* The paper is well-structured, with clear presentation and thorough technical descriptions.

weaknesses:
Major Concerns:

* The novelty and technical contribution appear limited. The core methodology seems to be primarily a combination of existing work (MaskGIT and SpearTTS). The speech acoustic code is derived from DAC, with no apparent novel methodological contributions.

* Regarding NAR methods, the authors cite phone-level duration prediction as a source of complex model design. However, this claim needs clarification: How is this complexity measured (e.g., training time, loss balancing)? Why are AR-based methods considered less complex? Experience with NaturalSpeech3/Fastspeech2 suggests duration modeling is manageable and straightforward to implement. Does this conclusion apply to traditional small-scale TTS models? The statement appears overly generalized.

* The Related Work section overlooks prominent flow-based methods like VoiceBox and p-flow, which also employ masked generative models. Their exclusion from this category requires explanation.

* Table 1 would be more appropriate in the appendix. Multi-task capability alone is not a significant contribution, as multi-task frameworks are common. The focus should be on concrete improvements in naturalness, prosody, efficiency, or reduced complexity. Placing Table 1 prominently suggests multi-task functionality is the main contribution, which diminishes the paper's actual value.

* In line 210, the claim that k-means based discrete SSL units lead to information loss needs substantiation. Why is VQ-VAE of W2V-BERT considered superior for semantic preservation despite also being discrete SSL units? While Appendix B provides semantic and acoustic comparisons, direct evidence supporting this specific claim is lacking.

Minor Points:

* Figure 2 would benefit from additional notations for clarity. Also, what is the difference between prompt semantic tokens and target semantic tokens? Can you give an example?

* While following the text-semantics-acoustics pipeline established by papers like SpearTTS, the superiority of this approach over alternatives (e.g., direct text2acoustic without semantics) should be justified.

* Section 4.2.3's purpose and motivation need clarification, particularly if addressing speed modification, as specialized tools exist for this purpose.

* Consider whether inference steps alone adequately measure efficiency, or if other metrics common in streaming methods (e.g., latency) would provide more comprehensive evaluation."

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711418491

thread_type:
official_review

<thread_object>
creator: reviewer_qwox

<content>
summary:
This paper proposes a masked generative codec transformer, MaskGCT, which performs speech synthesis using a mask-predict approach. MaskGCT generates semantic tokens from text input and is composed of two components: text-to-semantic MaskGCT, which generates semantic tokens from text, and semantic-to-acoustic MaskGCT, which generates acoustic tokens from semantic tokens. Each MaskGCT operates by filling masked parts based on confidence scores predicted in stages, from a sequence where all discrete speech tokens are masked at the start. Instead of using the commonly used method of obtaining semantic tokens through k-means clustering applied to pre-trained SSL models, MaskGCT employs a separate VQ-VAE model on SSL features to generate semantic tokens, reducing information loss through reconstruction loss. MaskGCT uses a similar mask prediction method for predicting discrete speech tokens as Soundstorm, which is based on MaskGIT. However, unlike Soundstorm, MaskGCT approaches the semantic token generation from text through masked token modeling and includes an additional module for total length prediction to align sequence lengths. MaskGCT demonstrates superior speaker similarity in widely used zero-shot TTS evaluation compared to various zero-shot TTS models. Moreover, a demo page with samples shows the versatility of MaskGCT across various tasks.

strengths:
* The paper shows that a mask-predict approach on discrete tokens can achieve a high level of speaker similarity. 

* Samples on the demo page show that MaskGCT enables zero-shot TTS that effectively captures not only timbre but also emotion and style across a variety of voices, with excellent sample quality. 

* To assess how well the model captures accent and emotion, the paper introduces metrics such as Accent SIM and Emotion SIM, which utilize representations reflecting each attribute. Their results indicate that MaskGCT performs better in imitating accent and emotion than other approaches based on these metrics.

weaknesses:
* In Soundstorm, the mask-predict approach is introduced to acoustic token generation. It seems that the methodological novelty in MaskGCT lies mainly in extending this mask-predict approach to semantic token generation.

* The paper utilizes VQ-VAE to reduce information loss in semantic tokens compared to the k-means clustering approach; however, it does not experimentally demonstrate how this approach improves over k-means clustering.

* Additionally, while using a larger number of semantic tokens may yield good performance in zero-shot TTS, the semantic tokens do not seem disentangled from speaker information, making MaskGCT less suitable for voice conversion tasks. The voice conversion samples on the demo page also appear less similar to the reference speaker.  

* When comparing single-sample generation, MaskGCT's pronunciation accuracy appears lower than that of VoiceBox or NaturalSpeech 3.

* Regarding the seed-TTS evaluation method, although it is not explicitly shown as a baseline in the paper, the objective metrics appear to be worse than those of seed-TTS.

questions:
Comments

* It would be helpful to mention the frame rate of semantic and acoustic tokens.
* I am curious whether only a single sampling step is used from the third layer during acoustic token generation. Is there any difference when using more sampling steps?
* It would be beneficial to also present the overall sampling speed of the model, including the Real-Time Factor.
* In Table 1, it seems that VoiceBox's Imp. Dur. should be marked as "X."
</content>
</thread_object>

</thread>
<thread>
creation date:
1731855576568

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
First of all, we would like to express our gratitude to all the reviewers for their invaluable feedback, which has significantly contributed to enhancing the quality of our paper. Here, we summarize some points raised by multiple reviewers. For each reviewer, we address their specific concerns in the respective comments section.

## The novelty of MaskGCT

We sincerely appreciate multiple reviewers' recognition of our engineering contributions. We have provided detailed model architectures and training specifications in the appendix, and we commit to open-sourcing all checkpoints and code to further benefit the research community.

MaskGCT represents the first NAR TTS system trained on large-scale in-the-wild datasets without phone-level duration prediction. Notably, it achieves state-of-the-art performance across multiple evaluation metrics on various test sets, demonstrating superior capabilities in multilingual generation, accent handling, and emotional expression.

In addition, we also highlight that MaskGCT's novelty lies in two key aspects: (1) MaskGCT explores non-autoregressive TTS without requiring explicit phone-speech alignment supervision or phone-level duration prediction, which significantly simplifies the traditional TTS pipeline while achieving more natural and consistent generation results. (2) MaskGCT introduces VQ-based semantic tokens, which represents a distinct approach compared to previous methods that relied on k-means clustering.

## The choice of semantic codec

All the reviewers have raised concerns about the choice of semantic codec (VQ vs. k-means). We believe a thorough investigation of this matter will significantly strengthen our paper. We will first present our empirical findings (which are already included in our paper), followed by additional experimental results. **All these analyses will be incorporated into the revised version of our paper.**

In our initial experiments, we observed that k-means-based semantic tokens were less effective in predicting acoustic tokens for languages with rich prosodic variations, particularly Chinese, where significant pitch variations were observed. We further support this finding with qualitative analysis.

1. Since k-means can be seen as optimizing the reconstruction loss between input and reconstruction, we present reconstruction loss curves under different k-means and VQ configurations.
We compare four configurations: VQ-8192 (which is the same as in our paper), VQ-2048, k-means-8192, and k-means-2048.
The loss curves can be found at this [link](https://github.com/maskgct/maskgct/raw/refs/heads/main/recon_loss_for_vq.PNG).

1. The information preservation in semantic tokens directly affects the semantic-to-acoustic model's prediction performance. We present the top-10 accuracy (shown in this [link](https://raw.githubusercontent.com/maskgct/maskgct/refs/heads/main/soundstorm_layer1_acc.PNG)) of the semantic-to-acoustic model in predicting the first layer of acoustic tokens. The results demonstrate that VQ-8192 outperforms VQ-2048, which in turn outperforms k-means-8192.

2. We investigate the impact of different semantic representation approaches on acoustic token reconstruction. We train separate semantic-to-acoustic (S2A) models for each configuration and evaluate their performance through speech reconstruction metrics. Across all three test sets, the results reveal a consistent performance ranking in similarity (SIM) scores, with VQ-8192 yielding the highest performance, followed sequentially by VQ-2048, k-means-8192, and k-means-2048. For WER, VQ-based methods also demonstrate superior performance over k-means approaches, though this advantage is less pronounced on LibriSpeech test-clean. Notably, on SeedTTS test-zh (Chinese test set), k-means exhibits a substantial degradation in WER. We attribute this to the stronger coupling between Chinese semantics and prosodic features, where the transition from VQ to k-means results in significant loss of prosodic information in the semantic representations.

| Semantic Codec | LibriSpeech *test-clean* | | SeedTTS *test-en* | | SeedTTS *test-zh* | |
|--------------|----------|---------|----------|---------|----------|---------|
| | SIM-O ↑ | WER ↓ | SIM-O ↑ | WER ↓ | SIM-O ↑ | WER ↓ |
| k-means 2048 | 0.648 | 3.013 | 0.658 | 3.989 | 0.691 | 11.420 |
| k-means 8192 | 0.661 | 2.862 | 0.664 | 3.012 | 0.713 | 8.782 |
| VQ 2048 | *0.671* | **2.177** | *0.692* | *3.187* | *0.744* | *4.913* |
| VQ 8192 | **0.680** | *2.223* | **0.713** | **2.175** | **0.763** | **2.088** |

We believe these additions will strengthen our argument and offer a clearer understanding of the impact of different semantic token approaches on model performance.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces MaskGCT, a two-stage NAR TTS system that predicts semantic tokens from text and then acoustic tokens from semantic tokens using a masked generative transformer. MaskGCT demonstrates superior speaker similarity in widely used zero-shot TTS evaluation compared to various zero-shot TTS models. The main contribution of this paper is to show that it is possible to train a NAR TTS model with a simple pipeline without requiring explicit phone-speech alignment.