MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer
Authors: Yuancheng Wang
Haoyue Zhan
Liwei Liu
Ruihong Zeng
Haotian Guo
Jiachen Zheng
Qiang Zhang
Xueyao Zhang
Shunsi Zhang
Zhizheng Wu
Venue: ICLR 2025 Poster (2025)
Keywords: text-to-speech synthesis
masked generative models
codec language models
voice cloning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a fully non-autoregressive TTS model that eliminates the need for explicit alignment information between text and speech supervision.

Abstract: The recent large-scale text-to-speech (TTS) systems are usually grouped as autoregressive and non-autoregressive systems. The autoregressive systems implicitly model duration but exhibit certain deficiencies in robustness and lack of duration controllability. Non-autoregressive systems require explicit alignment information between text and speech during training and predict durations for linguistic units (e.g. phone), which may compromise their naturalness.
In this paper, we introduce $\textbf{Mask}$ed $\textbf{G}$enerative $\textbf{C}$odec $\textbf{T}$ransformer (MaskGCT), a fully non-autoregressive TTS model that eliminates the need for explicit alignment information between text and speech supervision, as well as phone-level duration prediction. MaskGCT is a two-stage model: in the first stage, the model uses text to predict semantic tokens extracted from a speech self-supervised learning (SSL) model, and in the second stage, the model predicts acoustic tokens conditioned on these semantic tokens. MaskGCT follows the mask-and-predict learning paradigm. During training, MaskGCT learns to predict masked semantic or acoustic tokens based on given conditions and prompts. During inference, the model generates tokens of a specified length in a parallel manner. 
Experiments with 100K hours of in-the-wild speech demonstrate that MaskGCT outperforms the current state-of-the-art zero-shot TTS systems in terms of quality, similarity, and intelligibility. Audio samples are available at https://maskgct.github.io/. We release our code and model checkpoints at https://github.com/open-mmlab/Amphion/blob/main/models/tts/maskgct.

Published as a conference paper at ICLR 2025
MASKGCT: ZERO-SHOT TEXT-TO-SPEECH WITH MASKED GENERATIVE CODEC TRANSFORMER
Yuancheng Wang', Haoyue Zhan?, Liwei Liu', Ruihong Zeng?, Haotian Guo', Jiachen Zheng', Qiang Zhang', Shunsi, Zhang', Xueyao Zhang', Zhizheng Wu 'The Chinese University Hong Kong, Shenzhen GGungzzhou Quwan Network Technology
yuanchengwang link. cuhk. edu. cn, zhi zhizhengwuecuhk.cem edu.. cn
ABSTRACT
The recent large-scale text-to-speech (TTS) systems are usually grouped autore- gressive and non-autoregressive systems. The autoregressive systems implicitly model duration but exhibit certain deficiencies in robustness and lack of duration controllability. Non-autoregressive systems require explicit alignment information between text and speech during training and predict durations for linguistic units (e.g. phone), which may compromise their naturalness. In this paper, we introduce Masked Generative Codec Transformer (MaskGCT), a fully non non-autonn autoregressive TTS model that eliminates the need for explicit alignment information between text and speech supervision, as well as phone-level duration prediction. MaskGCT is two-stage model: in the first stage, the model uses text to predict semantic tokens extracted from a speech self-supervised learning (SSL) model, and in the second stage, the model predicts acoustic tokens conditioned on these semantic tokens. MaskGCT follows the mask-and--rediic learning paradigm. During train- ing, MaskGCT learns predict masked semantic or acoustic tokens based on given conditions and prompts. During inference, the model generates tokens of specified length in parallel manner. Experiments with 100K hours of in-the- wild speech demonstrate that MaskGCT outperforms the current tfate-of-the-art zero-shot TTS systems terms of quality, similarity. and intelligibility. Au- dio samples are available at https: /tt.///skkc..tt..co/ooo//.o.c... github..h...oo. io/ We release our code and model checkpoints at wwtpt:tttt.ttmm.tmmm.tttm /ttp:: htt.//ggtttttuteteg com owwnompnnnmmamammmmmmmammma ab/ hmmph..oor/bb.oo//mmaa//////ooo
INTRODUCTION
In recent years, large-scale zero-shot text-to-speech (TTS) systems achieved significant improvements by scaling data and model sizes, including both autoregressive (AR) T 2 5 6 and non-autoregressive (NAR) models 7 g [O]. However, both AR-based and NAR- systems still exhibit some shortcomings. In particular, AR-based TTS systems typically quantize speech into discrete tokens and then use decoder-only models to autoregressively generate these tokens, which offer diverse prosody but also suffer from problems such as poor robustness and slow inference speed. NAR-based models, typically based on diffusion 7 8) flow matching [9], GAN [10], require explicit text and speech alignment information as well as the prediction of phone-level duration, resulting in complex pipeline and producing more standardized but less diverse speech.
6, 8 9 10 have
Recently, masked generative transformers, a class of generative models, have achieved significant results in the fields of image strating potential comparable to or superior to autoregressive models or diffusion models. These models employ a mask-and-predic training paradigm and utilize iterative parallel decoding during inference. Some previous works have attempted to introduce masked generative models into the field of TTS. SoundStorm [99 was the first attempt to use masked generative transformer to predict multi-layer acoustic tokens extracted from SoundStream, conditioned speech semantic tokens; however, needs to receive the semantic tokens of an AR model as input. Thus, SoundStorm is more of acoustic model that converts semantic tokens into acoustic tokens and does not fully utilize the
video [4 [5], and audio [16 18 generation, demon-

---

Published as a conference paper at ICLR 2025
powerful generative potential of masked generative models. NaturalSpecch 3 into discrete token sequences representing different attributes through special designs and generates tokens representing different attributes through masked generative models. However, it still needs speech-text alignment supervision and phone-level duration prediction.
[ decomposes speech
In this work, we propose MaskGCT, a fully non-autoregressive model for text-to-speech synthesis that uses masked generative transformers without requiring text-speech alignment supervision and phone-level duration prediction. MaskGCT is two-stage system, both stages are trained using the mask-and-pan -nnedddddcdccct learning paradigm. The first stage, the texl-to-semantic (T2S) model, predicts masked semantic tokens with in-context learning, using text token sequences and prompt speech semantic token sequences as the prefix, without explicit duration prediction. The second stage, the semantic-to-acoune tokens extracted from an RVQ-based speech codec with prompt acoustic tokens. During inference, MaskGCT can generate semantic tokens of various specified lengths with a few iteration steps given sequence of text. In addition, we train a VQ-VAE (20] to quantize speech self-supervised learning embedding, rather than using k-means to extract semantic tokens that is common in previous work. This approach minimizes the information loss of semantic features even with a single codebook. We also explore the scalability of our methods beyond the zero-shot TTS task, such as speech translation (cross-lingual dubbing), speech content editing, voice conversion, and emotion control, demonstrating the potential of MaskGCT as a foundational model for speech generation. Appendix A.6 shows a comparison between MaskGCT and some previous works.
(S2A) model, utilizes semantic tokens to predict masked acoustic
Our experiments demonstrate that MaskGCT has achieved performance comparable to or superior to that of existing models in terms of speech quality, similarity, prosody, and inteligiibility. Specifically, (1) It achieves comparable or better quality and naturalness than the ground truth speech across three benchmarks (LibriSpeech, SeedTTS test-en, and SeedTTS test-ch) in terms of CMOS. (2) It achieves human-level similarity between the generated speech and the prompt speech, with improvements of +0.017, -0.002, and +0.027 in SIM-O and +0.28, +0.32 and +0.25 in SMOS for LibriSpeech, SeedTTS test-en, and SeedTTS text-zh, respectively. (3) It achieves comparable intelligibilitt in terms of WER across the three benchmarks and demonstrates stability within a reasonable range of speech duration, which also indicates the diversity and controllability of the generated speech.
In summary, we propose a non-autoregressive zero-shot TTS system based on masked generative transformers and introduce speech discrete semantic representation by training a VQ-VAE on speech self-supervised representations. Our system achieves human-level similarity, naturalness, and intelligibillity by scaling data 100K hours of in-the-wild speech, while also demonstrating high flexibility, diversity, and controllability. We investigate the scalability of our system across various tasks, including cross-lingual dubbing, voice conversion, emotion control, and speech content editing, utilizing zero-shot learning or post-training methods. This showcases the potential of our system as a foundational model for speech generation.
2 RELATED WORK
Large-scale TTS. Traditional TTS systems 21 23 24 25 are trained to generate speech from single speaker or multiple speakers using hours of high-quality transcribed training data. Modern large-scale TTS systems [
2 4 6 aim to achieve zero-shot TTS (synthesizing speech for unseen speakers with speech prompts) by scaling both the model and data size. These systems can be mainly divided into AR-based and NAR-based categories. For AR-based systems: SpearTTS (I] utilizes three AR models to predict semantic tokens from text, coarse-grained acoustic tokens from semantic tokens, and fine- ane-grained acoustic tokens from coarse granine tokens. VALL-E [2 predicts the first layer of acoustic tokens extracted from EnCodec (26) using an AR codec language model, and the final layers with a NAR model. VoiceCraft (5) employs single AR model to predict multi-layer acoustic tokens in a delayed pattern (27]. BASETTS 3 predicts novel speech codes extracted from WavLM features and uses a GAN model for waveform reconstruction. For NAR-based systems: NaturalSpeech 2 7 employs latent diffusion to predict the latent representations from codec model 28 VoiceBox [9) and P-Flow (29] use flow matching and in-context learning to predict mel-spectrograms. MegaTTS [10 utilizes GAN to predict mel-spectrograms, while an AR model predicts phone-level prosody codes. NaturalSpeech g employs a unified framework based on discrete diffusion models to predict discrete representations of different speech attributes. However,
2

---

Published as a conference paper at ICLR 2025
these NAR systems need to predict phoneme-level duration, leading complex pipeline and more standardized generative results. SimpleSpeech [30], DiTTo-TTS [31], and E2 TTS [2] are also NAR-based models that do not require precise alignment information between text and speech, nor do they predict phoneme-level duration. We discuss these concurrent works in Appendix K
Masked Generative Model. Masked generative transformers, class of generative models, achieve significant results and demonstrate potential comparable to or superior to that of autoregressive models
or diffusion models in the fields of image [1] 13 33 video [14] [5], and audio [16 17
18 [9
generation. MaskGIT M is the first work to use masked generative models for both unconditional and conditional image generation. Subsequently, Muse [2] leverages rich text achieve high-
quality and diverse text-to-image generation within the same framewwrr.. MAGVIT-v2 [13] employs masked generative models with novel lookup-free quantization, outperforming diffusion models in image and video generation. Recently, some efforts have been made to adapt masked generative models to the field of audio. SoundStorm [9 takes in the semantic tokens from M and utilizes this generative paradigm to generate tokens for neural audio codec (28). VampNet [16] and MAGNeT [8] apply masked generative models for music and audio generation, while MaskSR 17 extends these models for speech restoration.
Discrete Speech Representation.
Speech representation is a crucial aspect of speech generation. typically utilized mel-speccrograms as the modeling target. Recently, some large-scale TTS systems 2
Early works 22 24
g have shifted to using discrete speech representations. Discrete speech representation be primarily divided into two types: semantic discrete representation and acoustic discrete representation Semantic discrete representations are mainly extracted from various speech
SSL models [34 35| 36 using quantization methods such as k-means. Acoustic discrete representa- tions, on the other hand, are usually obtained by training a VQ-GAN model (20) with the goal of waveform reconstruction, as seen in speech codecs [26 28 37]. Semantic discrete representation typically shows stronger correlation with text, whereas acoustic discrete representation more ef- fectively reconstructs audio. Consequently, some two-stage TTS models predict both semantic and acoustic tokens. FACodec [8] is novel speech codec that disentangles speech into subspaces of
different attributes, including content, prosody, timbre, and acoustic details.
METHOD
3.1
BACKGROUND: NON-AUTOREGRESSIVE MASKED GENERATIVE TRANSFORMER
Given a discrete representation sequence X of some data, we define X, = X & M, as the process of masking a subset of tokens in X with the corresponding binary mask M, -m..) Specifically, this involves replacing 2; with a special [MASK] token if myy 1, and otherwise leaving x; unmasked if m
Here, each my,s is independently and identically distributed according to a Bernoulli distribution with parameter (t), where y(t) E (0, 1] represents mask schedule function (for example, (t) = sin( =t )+), E (0, T)). We denote X,- = X. The non-autoreeressive masked generative transformers are trained to predict the masked tokens based on the unmasked tokens and a condition C. This prediction modeled as ps (Xo)X, C). The parameters 0 are optimized to minimize the negative log- log-likelihood of the masked tokens:
N
Lyask
E XED,IE
Σ
m4s (og(po(2;/X,, C)).
At the inference stage, decode the tokens in parallel through iterative decoding. We start with fully masked sequence Xy. Assuming the total number of decoding steps is S, for each step i from 1 to S, we first sample X from po (X,) XT (1-1) based on the confidence score to remask, resulting in Xr-i g. where N is the total number of tokens
T. C). Then, we sample [N - >( - i. '''l tokens
in X. The confidence score for â; in Xo is assigned to (0 Xx (i-1)) T. C) if 2-- F, is [MASK] token; otherwise, we set the confidence score of à; to 1, indicating that tokens already unmasked X will not be remasked. Particularly, we choose [N . XT - i.) tokens with the lowest confidence scores to be masked.
We give more detailed discussion about the definitions of "semantic" and "acoustic" in Appendix B
3

---

Published as a conference paper at ICLR 2025
Prompt Semards Tolons
Semantic Tokens
Acoustic Todenss
booo
~
Sperch Semaantic Codec
O O
FOo00
Semaati
Acsuttic Codec
~ M
Prompt Speech
MAsAGCT
Generated Speech
Text
0
Figure An overview of the proposed two-stage MaskGCT framework. It consists of four main components: (1) a speech semantic representation codec converts speech to semantic tokens; (2)
a text-lo-semantic model predicts semantic tokens with text and prompt semantic tokens; (3) semantic-to-acousse model predicts acoustic tokens conditioned semantic tokens; (4) a speech acoustic codec reconstructs waveform from acoustic tokens.
The masked generative modeling paradigm was first introduced in [T] and subsequent work such as has further explored under the perspective of discrete diffusion.
3.2 MODEL OVERVIEW
An overview of the MaskGCT framework is presented in Figure Following 19 38 MaskGCT a two-stage TTS system. The first stage uses text to predict speech semantic representation tokens, which contain most information of content and partial information of prosody. The second stage model is trained to learn more acoustic information. Unlike previous works 1 2 38 use an autoregressive model for the first stage, MaskGCT utilizes the non-autoregressive masked generative modeling paradigm for both the two stages without text-speech alignment supervision and phone-level duration prediction: (1) For the first stage model, we trained model learn pe, (S|S,, (SP, P)), where S is the speech semantic representation token sequence obtained from a speech semantic representation codec (we introduce in (.2.II)) S is the prompt semantic token sequence, and the text token sequence. SP and P are the condition for the first stage model. (2) The second stage model trained learn po,, ( (A| A, AA S)), where A the multi-layer acoustic token sequence from speech acoustic codec like [26] 28). Our second stage model is similar to SoundStorm 19]. We give more details about the four parts in the following sections.
3.2.1 SPEECH SEMANTIC REPRESENTATION CODEC
Discrete speech representations can be divided into semantic tokens and acoustic tokens. Generally, semantic tokens are obtained by discretizing features from speech self- supersupee learning (SSL). Previous two-stage, large-scale TTS systems [ 19 38)
typically first use text to predict semantic tokens, and then employ another model to predict acoustic tokens or features. This is because semantic tokens have a stronger correlation with text or phonemes, which makes predicting them more straightforward than directly predicting acoustic tokens. Commonly, previous works have used k-means to discretize semantic features to obtain semantic tokens; however, this method can lead to loss of information. This loss may complicate the accurate reconstruction of high-quality speech or the precise prediction of acoustic tokens, especially for tonally rich languages. For example, our early experiments demonstrate the challenges of accurately predicting acoustic tokens to achieve proper prosody for Chinese using semantic tokens obtained via k-means. We give more need discretize semantic representation features while minimizing information loss. Inspired by [39], we train a VQ- model to learn vector quantization codebook that reconstructs speech semantic representations from a speech SSL model. For a speech semantic representation sequence S R x the vector quantizer quantizes the output of the encoder E(S) to E, and the decoder reconstructs E back to S. We optimize the encoder and the decoder using a reconstruction loss between S and $, employ codebook loss optimize the codebook and use commitment loss to optimize the encoder with the straight-through method (20] The total loss for training the semantic representation codec can be written as:
experimental explorations in Section 4.4 Therefore,
Lintal
Td (Arec |S -S1 + Xxodebookk ||sg(E(S)) - -El|2 + Acommir
where sg means stop-gradien
In detail, we utilize the hidden states from the 17th layer of W2v-BERT 2.0 [4] as the semantic features for speech encoder. The encoder and decoder composed of multiple ConvNext (40
4

---

Published as a conference paper at ICLR 2025
Text-to-Semantic MaskGCT
Semantic-to-Acouste MaskGCT
00000000000000000000000000000000
Selected
Toor
Aonpi Spmoneo Telens
Masking
0
Phetet Acrushi Thewss
Tagget AcOuARY Taken
lagel Sommndd
0 O
Figure 2: An overview of training diagram of the T2S (left) and S2A (right) models. The T2S model trained to predict masked semantic tokens with text and prompt semantic tokens as the prefix. The S2A model is trained to predict masked acoustic tokens of random layer conditioned on prompt acoustic tokens, semantic tokens, and acoustic tokens of the previous layers.
blocks. Following the methods of improved VQ-GAN 41 and DAC 37 we use factorized codes to project the output of the encoder into low-dimensional latent variable space. The codebook contains 8,192 entries, each dimension Further details about the model architecture are provided in A.P4
3.2.2
-SEAANTII MODEL
Based on the previous discussion, we employ non-autoregressive masked generative transformer to train a rext-to-semanter (T2S) model, instead of using an autoregressive model or any text-to-speech alignment information. During training, we randomly extract portion the prefix of the semantic token sequence as the prompt, denoted as SP. We then concatenate the text token sequence P with S to form the condition. We simply add (P, S²) as the prefix sequence to the input masked semantic token sequence S, to leverage the in-context learning ability of language models. We use a Llyma-style [42] transformer as the backbone of our model, incorporating gated linear units with GELU (3] activation, rotation position encoding [44, etc., but replacing causal attention with bidirectional attention. We also use adaptive RMSNorm 45 which accepts the time step as the condition.
During inference, we generate the target semantic token sequence of any specified length conditioned the and the prompt semantic token sequence. In this paper, we also train flow matching [46) based duration prediction model to predict the total duration conditioned on the text and prompt speech duration, leveraging in-context learning. More details can be found in Appendix A.5
3.2.3 SEMANTIC-TO-ACOUSTII MODEL
We also train a comantic-to-acousiie (S2A) model using a masked generative codec transformer conditioned on the semantic tokens. Our semaatic-to-acoussicc model is based SoundStorm [19] which generates multi-layer acoustic token sequences. Given N layers of the acoustic token sequence A 1:N during training, we select one layer j from 1 to N. We denote the jth layer of the acoustic token sequence as A", Following the previous discussion, we mask A at the timestep to get A The model is then trained to predict A' conditioned on the prompt A", the corresponding semantic token sequence S. and all the layers smaller than j of the acoustic tokens. This can be formulated as Po.,.
(A |A, (AP, S, A 1.)- We sample j according For the input of the S2A model, since the number of frames in the semantic token sequence equal the sum of the frames in the prompt acoustic sequence and the target acoustic sequence, simply sum the embeddings of the semantic tokens and the embeddings of the acoustic tokens from layer to j. During inference, we generate tokens for each layer from coarse fine, using iterative parallel decoding within each layer. Figure 2 shows a simplified training diagram of the T2S and S2A models.
2 linear schedule p(j) = +1)+
3.2.4 SPEECH ACOUSTIC CODEC
Speech acoustic codec is trained to quantize speech waveform multi-layer discrete tokens while aiming to preserve all the information of the speech as soon as possible. We follow the residual vector quantization (RVQ) method compress the 24K sampling rate speech waveform into discrete tokens
5

---

Published as a conference paper at ICLR 2025
of 12 layers. The codebook size of each layer is 1,024 and the codebook dimension is 8. The model architectures, discriminators, and training losses follow DAC [37], except that we use the Vocos [47] architecture the decoder for more efficient training and inference. Figure S shows the comparison between the semantic codec and acoustic codec.
3.3 OTHER APPLICATIONS
MaskGCT can accomplish tasks beyond zero-shot TTS, such as duration-controllabee speech trans- lation (cross- pross-linguul dubbing), emotion control, speech content editing, and voice conversion with simple modifications the assistance of external tools, demonstrating the potential of MaskGCT as a foundational model for speech generation. We provide more details in Appendix F
H
4 EXPERIMENTS AND RESULTS
4.1 EXPERIMENTAL. SETTINGS
Datasets. We use the Emilia 48 dataset to train our models. Emilia is a multilingual and diverse in-the-wild speech dataset designed for large-scale speech generation. In this work, we use English and Chinese data from Emilia, each with 50K hours of speech (totaling 100K hours). We evaluate our zero-shot TTS models with three benchmarks: (1) LibriSpeech [49 test-clean, a widely used test set for English zero-shot TTS. (2) SeedTTS test-en, test set introduced in Seed-TTS [6] of samples extracted from English public corpora, includes 1,000 samples from the Common Voice dataset (50). (3) SeedTTS test-zh, test set introduced in Seed-TTS of samples extracted from Chinese public corpora, includes 2,000 samples from the DiDiSpeech dataset (51]. We also scale the training dataset to six languages to support multilingual zero-shot TTS. We provide additional experimental details and evaluation results about multilingual zero-shot TTS in Appendix
Evaluation Metrics. We use both objective and subjective metrics evaluate our models. For the objective metrics, we evaluate speaker similarity (SIM-O), robustness (WER), and speech quality (FSD). Specifically, for speaker similarity, we compute the cosine similarity between the WavLM TDNN (36] speaker embedding of generated samples and the prompt. For Word Error Rate (WER), we use a HuBERRTbaaedd ASR model for LibriSpeech text-clean, Whisper-lagge-v3 for Seed-TTS test-en, and Paraformer-zi zz for Seed-TTS test-zh, following previous works. For speech quality, use Fréchet Speech Distance (FSD) with self-supervised wav2vec 2.0 [52) features, following [9]. For the subjective metrics, comparative mean option score (CMOS) and similarity mean option score (SMOS) are used to evaluate naturalness and similarity, respectively. CMOS is on a scale of -3 to 3, and SMOS is on scale of to 5.
Baseline. We compare our models with state-of-the- art zero-shot TTS systems, including Natural- Speech 3 ( VALL-E [2) VoiceBox 9. VoiceCraft (5), XTTS-v2 [53] and Vosye [54). More details of each model can be found in Appendix [ We also train an AR-based T2S model to replace
the T2S part of MaskGCT, we term it as AR + SoundStorm.
Training, We train all models on 8 NVIDIA A100 80GB GPUs We train two T2S models of different sizes (denoted T2S-Bass Base and T2S-large). For more details about the model architecture, please refer to Appendix A We report the metrics of T2S-larg by default, and you can find a comparison of model sizes in Section 4.5 We also compare two different methods of text tokenization: Grapheme-to- toooneemeeeee (G2P) (55) and Byte Pair Encoding (BPF) [56). See more details of the two methods in Appendix A.7 We report the metrics of G2P by default. We optimize these models with the Adam W (57] optimizer with learning rate of le-4 and 32K warmup steps, following the inverse square root learning schedule. We use the classifier-free guidance (58), during training for both the T2S and S2A models, we drop the prompt with a probability of 0.15. See more details about classifier- free guidance and classifier- free guidance rescale in Appendix C
Inference. For the T2S model, we use 50 steps the default total inference steps. The classifier-fre free guidance scale and the classifier-free guidance rescale factor 59 are set to 2.5 and 0.75, respectively.
veri fication

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation results for MaskGCT and the baseline methods on LibriSpeech test-clean, SeedTTS test-en, SeedTTS test-zh. The boldface denotes the best result, the underline denotes the second best. gr length denotes the result obtained by using ground truth total speech length. The
results in '0" means the result is the best one selected from five random samples (rerank 5).
System
SIM-O f
WER
FSD +
SMOS t
CMOS 1
LibriSpeech test-clean
Ground Truth
0.68
1.94
4.05++a.2
0.00
VALL-E [2
0.50
5.90
3.47 10:0
-0.52
VoiceBox 9
0.64
2.03
0.762
3.80
-0.41
NaturalSpeech 3
0.67
1.94
0.786
4.26
0.16
VoiceCraft 5
0,45
4.68
0.981
3.52
-0.33
XTTS-v2 [53
0.51
4.20
0.945
3.02
0,.98
MaskGCT
0.687 (0.73)
2.634(1.976)
0.886
4.27
0.10
MaskGCT (rule-based)
0.686
2.976
0.797
MaskGCT (gt length)
0.697
2.012
0.746
4.33 20.1
0.13
Seed test-en
Ground Truth
0.730
2.143
3,92+015
0,00
Voice [54]
0.643
4.079
0.316
3.52
-0.41 2a1s
XTTS-v2 (53]
0.463
3.248
0.484
3.15
-0866
VoiceCraft [S]
0,470
7,556
0.226
3.18+020
-1.08+aus
MaskGCT
0.717(0.76)) 7660)
2.623(1.283)
0.188
4.24
0.03
MaskGCT (rule-based)
0.719
2.712
0.167
MaskGCT (gt length)
0.728
2.466
0.159
4.13
0.12 2011
Seed ITS test-zh
Ground Truth
0.750
1.254
3.86+aur
0.00
Cosy Voice [54]
0.750
4.089
0.276
3,54
-0.45
XT [33]
0.635
2.876
0.413
2.95
-0.81
MaskGCT
0.774(0.805)
(.273(0.843)
0.106
4.09
0.05
MaskGCT (rule-based)
0.771
2.409
0.106
MaskGCT (gt length)
0.777
2.183
0.101
4.11
0.08
For sampling. we use top-k of 20, with the sampling temperature annealing from 1.5 to O. We add Gumbel noise to token confidences when determining the remasking process, following [.]. For the S2A model, we use [40, 16, 1, 1, 1, 1, 1, 1, steps for acoustic RVQ layers by default, we find the S2A model can also perform well with fewer inference steps of [10, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, (see Appendix [..3). We use the same sampling strategy as the T2S model, except that we use greedy sampling instead of top-k sampling if the inference step is 1.
4.2 ZERO-SHOT TTS
In this section, we show the main results of zero-shot TTS: we show comparison results with SOTA baselines in Section 4.2.T] we compare MaskGCT with replacing T2S model to an AR model in Section 4.2.2 We present the performance of MaskGCT across varying speech tempos in Section 4.2.3 Additionally, we present the results of zero-shot TTS for speech style imitation in Section 4.3 multilingual zero-shot TTS in Appendix E and cross-lingual speech translation (dubbing)
in Appendix
4.2.1 COMPARISON WITH BASELINES
We compare MaskGCT with baselines in terms of similarity, robustness, and generation quality. The main results are shown in Table MaskGCT demonstrates excellent performance on all metrics and achieves human-level similarity. naturalness, and intelligibility. In similarity, MaskGCT's SIM-O and SMOS both outperform the best baseline, whether assessed using the total length of ground truth or the predicted total duration (0.67->0.687 in LibriSpeech, 0.643->0.71 in SeedTTS test- en, 0.75 7055-00774 in SeedTTS test-zh for SIM-O; +0,01 in LibriSpeech, +0.72 in SeedTTS test-en, +0.55 in SeedTTS test-zh for SMOS). When compared with human recordings. MaskGCT achieves human-level similarity across all three test sets (+0.017, -0.002, and +0.027 for SIM-O respectively in the three test sets, and +0.28, +0.32, and +0.25 for SMOS respectively in the three test sets). In robustness, MaskGCT likewise results nearly on par with ground truth (with 2.634, 2.623. 2.273 WER on LibriSpeech, SeeTTTS fest-en, and SeedTTS test-zh, respectively). exhibiting enhanced
7

---

Published as a conference paper at ICLR 2025
robustness compared AR-based models and performing on par better than such as VoiceBox and NaturalSpeech 3, without relying on phone-level duration predictions. In generation quality, MaskGCT achieves +0.10, +0.03, and +0.05 CMOS across the three test sets when compared with human recordings, indicating that MaskGCT attains human-level naturalness on these test sets. We also observe that MaskGCT exhibits excellent performance when using both ground truth total duration and predicted total duration, indicating the robustness of MaskGCT within a reasonable range of total speech duration.
models
Table 2:
Comparison results the evaluation of MaskGCT and AR+SoundStorm. AR+SoundStorm can be regarded as replacing the T2S MaskGCT with the AR T2S model.
System
SIM-O
WER
FSD..
SMOST
CMOS
LibriSpeech test-clean
AR + SoundStorm 0.672
3.267
0,998
4.20 g0m
-0.02
MaskGCT
0.687
2.634
0,886
4.27 1014
0.10 1aM
TT'' test-en
AR SoundStorm
0.683
2.846
0,323
4.03 +a2s
-0.05 +012
MaskGCT 0.717
2.623
0.188
4.24 :012
0.03
Se test-zh
AR SoundStorm 0.747
3.865
0.238
3.78 +a.s
-0.32
MaskGCT 0.774
2.273
0.106
4.09
0.05
4.2.2 AUTOREGRESSIVG VS. MASKED GENERATIVE MODELS
We compare MaskGCT replacing T2S MaskGCT with an AR T2S model (which we call AR + SoundStorm). Table shows the performance of these two models on all three test sets. MaskGCT demonstrates improved similarity, robustness, and CMOS (+0.. on LibriSpeech test-clean, +0.08 SeedTTS test-en, and +0.37 on SeedTTS test-zh) across all three test sets. We also conduct comparisons on more challenging hard cases (such as repeating words, and tongue twisters, which often considered as samples where TTS systems are prone to hallucinations). MaskGCT exhibits a more pronounced robustness advantage in these scenarios. See details in Appendix In addition, compared AR-based models, MaskGCT offers the capability control the total duration of the generated speech, along with fewer inference steps, requiring only 25 to 50 steps for T2S models achieve optimal results for speeches of any length. Conversely, the inference steps for AR-based models increase linearly with the length of the speech.
4.2.3 DURATION LENGTH ANALYSIS
We analyze the robustness the generated re- sults of MaskGCT under different changes in to-
SeedTTSS rest
SeadTTS rest oh
WER
tal duration length (which can also be regarded as changes in speech tempo). The results are shown in Figure 3 We explore the results of multiplying the ground truth total duration by 0.7 to 1.3. The results show that the lowest WER is achieved at a total duration multiplier 1.0, indicating that the models perform best when the speech is played at its natural speed. When the multiplier is 0.9 1.1, the model is still able to achieve a WER very close the best. When the multiplier is 0.7 or 1.3, the WER slightly higher but still within a reasonable range. This shows that our model can generate reasonable and accurate content at different speech tempos.
0.7
0.8
0.9
1.1
1.2
Total Duration Multiplier
Figure 3: WER vs. Total Duration Multiplier.
4.3 SPEECH STYLE IMITATION
Zero-shot TTS endeavors to learn how to speak, including voice timbre and style, from prompt speech. Previous works utilized SIM-O to measure the similarity between generated speech and reference speech; however, SIM-O primarily assesses the similarity in voice timbre. In addition to evaluating the model's zero-shot cloning ability through timbre similarity metrics, we also explored MaskGCT's capability clone overall style from
expressive and stylized dimensions: accent and
8

---

Published as a conference paper at ICLR 2025
emotion. We randomly sampled portion of data from the L2-ARCTIC [60] accent corpus and the ESD [6]] emotion corpus to construct our accent and emotion evaluation datasets. Additionally, we introduce supplementary metrics to assess the model's performance. For accent imitation, employ SIM-Accent, to measure the similarity in accent between the generated speech and reference speech. The calculation process is analogous SIM-O, but utilize Common the accent representation features of the speech. We also incorporate a subjective evaluation metric, Accent SMOS, which is similar to SMOS but focuses on accent rather than timbre. For emotion, we introduce Emotion SIM (with emotion2ved
(2 63) to derive
[4
to extract features) and Emotion SMOS.
Our experiments demonstrate that MaskGCT exhibits powerful style cloning capabilities. For accent imitation, MaskGCT achieves the highest SIM-O of 0.717, close to the ground truth of 0.747. It also maintains a competitive WER 6.382 and the best Accent SIM of 0.645. Additionally, MaskGCT leads in CMOS of 0.23, SMOS of 4.24, and Accent SMOS of 4.38. For emotion imitation, MaskGCT achieves the highest SIM-O of 0.600. It also attains a competitive WER of 12.502 and strong Emotion SIM of 0.822. Furthermore, MaskGCT leads in all subjective metrics with CMOS of -0.31, SMOS of 4.07, and Emotion SMOS of 3.76, indicating natural and pleasant emotion imitation.
Table 3: Evaluation results for MaskGCT and the baseline methods on accent imitation.
System
S
WER
Accent SIM
CMOS T SMOS 1
Accent SMOS 1
Accent Corpus L2-Arctit
Ground Truth
0.747
10.903
0.633
0.00
VALL-E
0.403
10.721
0,485
-1.04+as0
3.12+041
2.77 +84s
Cosy Voice
0.653
6.660
0.640
0,10+am
4.23
VoiceBox
0.475
6,181
0,575
-0.55
3.23
3,49+a29
VoiceCraft
0.438
10.072
0,517
-0.39
3.51
3.29 1..38
MaskGCT
0.717
6.382
0.645
0.23+aor
4.24
4.38 :0.5
Table 4: Evaluation results for MaskGCT and the baseline methods on emotion imitation.
System
SIM-O f
WER
Emotion SIM + CMOS +
SMOS
Emotion SMOS f
Emotion Corpus ESD
Ground Truth
0.673
11.792
0.936
0.00
VALL-E
0.396
15.731
0.735
21.43
2.52
2.63 20:6
CosyVoice Voice
0.575
10.139
0.839
-0,45
3.98
3,66
VoiceBox
0.451
12.647
0.811
-0.65
3.81
3.61 +a.9
VoiceCraft
0.345
16.042
0.788
-0.60+m2z
3.42+a31
3.52
MaskGCT 0.600
12,502
0.822
-0,11
4.07 10:7
3.76
4.4
CHOICE OF SEMANTIC REPRESENTATION CODEC
In this section, investigate the impact of different semantic representation approaches acoustic token reconstruction. We primarily evaluate two types of semantic codecs: VQ-based and k-means- based. For VQ-based approaches, 2048, denoted as "VQ 8192" and "VQ 2048", respectively. Similarly, for k-means-based approaches, train models with 8192 and 2048 clusters, denoted "k-means 8192" and "k-means 2048", To assess how different semantic representations affect acoustic prediction, we train separate semantic- to-acoustic (S2A) models for each configuration and evaluate their performance through speech reconstruction metrics. The comparative results are presented in Table
implement two configurations with codebook sizes of 8192 and
Table 5: Evaluation results for S2A models with different semantic codecs. In this experiment, we use the ground-trutt -ruth semantic tokens to predict acoustic tokens.
Semantic Codec
SIM-O f
WER +
SIM-O
WER +
SIM-O f
WER
LibriSpeech test-clean
SeedTTS test-en
SeedTTS test-zh
k-means 2048
0.648
3.013
0.658
3.989
0.691
11.420
k-means 8192
0.661
2.862
0.664
3.012
0.713
8.782
VQ 2048
0.671
2.177
0.692
3.187
0.744
4.913
VQ 8192
0.680
2.223
0.713
2.175
0.763
2.088
commonaccent_eccap
https://github..oo httmm//@lkoaacc//emttiioooo

---

Published as a conference paper at ICLR 2025
The experimental results demonstrate consistent patterns across all three test sets. For SIM-O, the results reveal a consistent performance ranking across all test sets, with VQ 8192 yielding the highest scores, followed sequentially by VQ 2048, k-means 8192, and k-means 2048. Regarding Word Error Rate (WER), VQ-based methods consistently outperform k-means approaches, though this advantage is less pronounced in the LibriSpeech fest-clean set. Notably, on the SeedTTS test-zh (Chinese test set), k-means exhibits a substantial degradation in the performance of the WER. We attribute this phenomenon to the stronger coupling between semantic and prosodic features in Chinese, where the transition from VQ to k-means results in a significant loss of prosodic information in the semantic representations.
4.5 ABLATION STUDY
Inference Timesteps. We explore the impact of inference steps of the T2S model on the results, ranging from 5 steps to 75 steps. Initially, SIM increases significantly and stabilizes after 25 steps. For test-zh, it rises from 0.761 at 5 steps to 0.771 at 75 steps, and for test-en, from 0.696 to 0.715. SIM peaks around 25 steps. WER improves more dramatically, especially up 25 steps. For test-zh, drops from 10.19 at steps to 2.507 at 25 steps, and for rest-en, from 8.096 to 2.346. Both SIM and WER show minimal changes beyond 25 steps. These findings suggest that SIM can be optimized with around 10 steps, while achieving the lowest WER requires approximately 25 steps. Beyond this, both metrics show minimal changes, indicating that further increases in steps do not yield substantial improvements, For practical applications, 25 inference steps may be optimal for balancing SIM and WER, ensuring efficient and effective performance. See more details in Appendix A.2
Model Size. We compare the performance differences of T2S models with varying model sizes. The result is shown in Table We observe that the large model outperforms the base model across all metrics, albeit not significantly. We suggest that our system can achieve good performance with just the setting of the base model when using 100K hours of data. the future, we will explore more comprehensive scaling laws for both model size and data scaling.
Table 6: Comparison results between T2S-Large and T2S-Bas
System
SIM-O 1 WER +
FSD
Parraeters
Te test-en
T2S-Base
0.714
2.514
0.189
315M
T2S-Large
0.728
2.466
0.159
695M
Seed I'S test-zh
T2S-Base
0.769
2.216
0.123
315M
T2S-Large
0.777
2.183
0.101
695M
Text Tokenizer. We compare two text tokenization methods: Grapheme-to-Phonemee (G2P) and Byte Pair Encoding (BPE). See more details Appendix(A.7
CONCLUSION
In this paper. we present MaskGCT, a large-scale zero-shot TTS system that leverages fully non- autoregressive masked generative codec transformers while not requiring text-speech alignment supervision and phone-level duration prediction. MaskGCT achieves high-quality text-to-speech synthesis using text to predict semantic tokens extracted from speech self-supervised learning (SSL) model, and then predicting acoustic tokens conditioned on these semantic tokens. Our experiments demonstrate that MaskGCT outperforms the state-of-the-art TTS system on speech quality, similarity, and intelligibility with scaled model size and training data, and MaskGCT can control the total duration of generated speech. We also explore the scalability of MaskGCT in tasks such speech translation, voice conversion, emotion control, and speech content editing, demonstrating the potential of MaskGCT foundational model for speech generation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work is partially supported by the NSFC under Grant 62376237, Shenzhen Science and Technol- ogy Program ZDSYS22020022200000000 We thank all anonymous reviewers for their insightful comments and suggestions.
REFERENCES
[1] Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics, 11:1703-1718, 2023.
[2] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301 200:233010211... 2023.
[3] Mateusz Lajszczak, Guillermo Cámbara, Yang Li, Fatih Beyhan, Arent van Korlaar, Fan Yang, Arnaud Joly, Álvaro Martín-Cortine Ammar Abbas, Adam Michalski, et al. Base tts: Lessons from building billion-parameter text-to-speech model 100k hours of data. arXiv preprint wrXiv:2402.08093 2024.
[4] Jaehyeon Kim, Keon Lee, Seungjun Chung, and Jaewoong Cho. Clam-tts: Improving neural codec language model for zero-shot text-to- -s-eee.. arXiv preprint Stt1:2204.00788./ 2024.
[5] Puyuan Peng, Yao Huang, Daniel Li, Abdelrahman Mohamed, and David Harwath. Voice- craft: Zero-shot speech editing and text-to-speech in the wild. arXiv preprint 1677: 2024.
[6] Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, et al. Seed-us: A family of high-quality versatile speech generation models, arXiv preprint arXiv:2406.0244
2024.
[7] Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng. Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.09110 16, 2023.
[8] Zeqian Ju. Yuancheng Wang. Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.0m 10000, 2024.
[9] Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilin- gual universal speech generation scale. Advances in neural information processing systems, 36, 2024.
[10] Ziyue Jiang, Jinglin Liu, Yi Ren, Jinzheng He, Chen Zhang, Zhenhui Ye, Pengfei Wei, Chunfeng Wang, Xiang Yin, Zejun Ma. et al. Mega-tts 2: Zero-shot text-to-speech with arbitrary length speech prompts. arXiv preprint 230.0:2077.072188 2023.
[11] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325 2022.
[12] Huiwen Chang. Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming -suan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint arXiv:2301.0077
2023.
[13] Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2142-2152, 2023.
11

---

Published as a conference paper at ICLR 2025
[14] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsun Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10459-10469, 2023.
[15] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander Hauptmann, et Language model beats diffusion-tokenizzrr is key to visual generation. arXiv preprint arXiv:2310.057777 2023.
[16] Hugo Flores Garcia, Prem Seetharaman, Rithesh Kumar, and Bryan Pardo. Vampnet: Music generation via masked acoustic token modeling. arXiv preprint arXiv:2307.04686 2023.
[17] Xu Li, Qirui Wang, and Xiaoyu Liu. Masksr: Masked language model for full-band speech restoration. arXiv preprint arXiv:2406.02092, 2024.
[18] Alon Ziv, Itai Gat, Gael Le Lan, Tal Remez, Felix Kreuk, Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. Masked audio generation using a single con-autoregressve transformer. arXiv preprint arXiv:2401.04577 2024.
[19] Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint XXXi:23355.06636 2023.
[20] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[21] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tic-Yan Liu. Fastspeech Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006. 0455: 2020.
[22] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Fastspeech: Fast, robust and controllable text to speech. Advances in neural information processing systems, 32, 2019.
[23] Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang. Yichong Leng, Yuanhao Yi, Lei He, et al. Naturalspeech: End-to-end text-to-speech synthesis with human-level quality. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[24] Yuxuan Wang, RJ Skerry-Ryan Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, al. Tacotron: Towards end-to-end speech synthesis. arXiv preprint
2017.
[25] Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversar- ial learning for end-to-end text-to-speech. In International Conference on Machine Learning. pages 5530-5540. PMLR, 2021.
[26] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv preprint arXiv:2210. /3438, 2022.
[27] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défosse7. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 2024.
[28] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
[29] Sungwon Kim, Kevin Shih, Joao Felipe Santos, Evelina Bakhturina, Mikyas Desta, Rafael Valle, Sungroh Yoon, Bryan Catanzaro, et al. P-flow: fast and data-efficient zero-shot tts through speech prompting. Advances in Neural Information Processing Systems, 36, 2024.
[30] Dongchao Yang, Dingdong Wang, Haohan Guo, Xueyuan Chen, Xixin Wu, and Helen Meng. Simplespeech: Towards simple and efficient text-to-speech with scalar latent transformer diffusion models. arXiv preprint arXiv:2406.02328 2024.
12

---

Published as a conference paper at ICLR 2025
[31] Keon Lee, Dong Won Kim, Jachyeon Kim, and Jaewoong Cho. Ditto-tts: Efficient and scalable zero-shot text-to-speech with diffusion transformer. arXiv preprint arXiv:2406.1 2024.
[32] Sefik Emre Eskimez, Xiaofei Wang, Manthan Thakker, Canrun Li, Chung- -isien Tsai, Zhen Xiao, Hemin Yang, Zirun Zhu, Min Tang, Xu Tan, et al. E2 us: Embarrassingly easy fully non-autoregressive zero-shot us. arXiv preprint arXiv:2406. 18009, 2024.
[33] José Lezama, Huiwen Chang, Lu Jiang, and Irfan Essa. Improved masked image generation with token-critic. In European Conference on Computer Vision, pages 70-86. Springer, 2022.
[34] Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244 250. IEEE, 2021.
[35] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451-3460, 2021.
[36] Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavim: Large-scale self-supervised pre- training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 1666):1505-1518, 2022.
[37] Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 36, 2024.
[38] James Betker. Better speech synthesis through scaling. arXiv preprint arXiv:2305.07245 2023.
[39] Zhichao Huang, Chutong Meng, and Tom Ko. Repcodec: A speech representation codec for speech tokenization. arXiv preprint arXiv:2309. 00169, 2023.
[40] Zhuang Liu, Hanzi Mao, Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11976-1 1986, 2022.
[41] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint arXiv:2110.04627, 2021.
[42] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 23X7::3307.09888 2023.
[43] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.088/ 08415, 2016.
[44] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Nearoompputing. 568:127063 2024.
[45] Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
[46] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
[47] Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint arXiv:2306.000 000844 2023.
[48] Haorui He, Zenggiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang. Jiaqi Li, Peiyang Shi, et al. Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation. arXiv preprint arXiv:2407.05361 2024.
13

---

Published as a conference paper at ICLR 2025
[49] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an ast corpus based on public domain audio books. In 2015 IEEE international conference on acousties, speech and signal processing (ICASSP), pages 5206-5210. IEEE, 2015.
[50] Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A 2019.
speech corpus. arXiv preprint urXiv:
[51] Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, Ruixiong Zhang, Shuaijiang Zhao, Wubo Li, Cheng Gong, Wei Zou, Kun Han, al. Didispeech: A large scale mandarin speech corpus. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6968-6972. IEFE, 2021.
[52] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:1444--22460, 2020.
wav2vec 2.0:
[53] Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Aljafari. Joshua Meyer. Reuben Morais, Samuel Olayemi, et al. Xtts: a massively multilingual zero-shot text-to-speech model. arXiv preprint 24X1v:2406.09904 2024.
[54] Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. arXiv preprint vrXiv:2407.05000 2024.
[55] Mathieu Bernard and Hadrien Titeux. Phonemizer: Text to phones transcription for multiple languages in python. Journal of Open Source Software, 6(68):3958. 2021.
[56] Philip Gage. A new algorithm for data compression. The C Users Journal, 12(2):23-38, 1994.
[57] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv=1711.05101, 2017.
[58] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12598 2022.
arXiv preprint
(59) Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps flawed. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pages 5404-5411, 2024.
[60] Guanlong Zhao, Evgeny Chukharev-Hudilainee Sinem Sonsaat, Alif Silpachai, Ivana Lucic, Ricardo Gutierrez-Osuna, and John Levis. L2-arctic: A non-native english speech corpus. 2018
[61] Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset. In ICASSP 202/-202 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 920-924, IEEE, 2021.
[62] Huaying Xue, Xiulian Peng, Yan Lu, al. Convert and speak: Zero-shot accent conversion with minimum supervision. In ACM Multimedia 2024.
[63] Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson. Autove: Zero-shot voice style transfer with only autoencoder loss. In International Conference on Machine Learning, pages 2210-5219 PMLR, 2019.
(64] Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. emotion2vec: Self-supervised pre-training for speech emotion representation. arXiv preprint arXiv:2312.15185 2023.
[65] Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. arXiv preprint arXiv:2206. 04658, 2022.
[66] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 03000, 2022.
14

---

Published as a conference paper at ICLR 2025
[67] Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. Advances in Neural Information Processing Systems, 33:8067-8077, 2020.
[68] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM transactions on audio, speech, and language processing, 11:2223-25333 2023.
[69] Hao-Han Guo, Kun Liu, Fei- Yu Shen, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. Fireredtts: foundation text-to-speech framework for industry-level generative speech applications. arXiv preprint
2024.
[70] Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models. arXiv preprint arXiv:2308. 16692, 2023.
[71] Xueyao Zhang, Liumeng Xue, Yuancheng Wang, Yicheng Gu, Xi Chen, Zihao Fang, Haopeng Chen, Lexiao Zou, Chaoren Wang. Jun Han, et al. Amphion: An open-source audio, music and speech generation toolkit. arXiv preprint arXiv:2312.099) 2023.
[72] Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre- Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, al. Libri-light: A benchmark for with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7669-7673. IEEE, 2020.
[73] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087744 2023.
[74] Chenpeng Du, Yiwei Guo, Feiyu Shen, Zhijun Liu, Zheng Liang, Xie Chen, Shuai Wang. Hui Zhang. and Kai Yu. Unicats: A unified context-aware text-to-speech framework with contextual vq-diffusion and vocoding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17924 17932, 2024.
[75] Jingyi Li, Weiping Tu, and Li Xiao. Freeve: Towards high-quality text-free one-shot voice conversion. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
(76) Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao Kai Yu, and Xie Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:2410.06885, 2024.
[77] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple. arXiv preprint arXiv:2309. /5505, 2023.
[78] Anonymous. AS* AS=2S-flo: Alignment- pre-training for speech synthesis with flow match- ing. In Submitted to The Thirteenth International Conference on Learning Representations, 2024. /ttpa:://peeeeeeeeeeeeeeeeeeeeee wttp://wwwwwwweeee.eeeeeeee
under review.
[79] Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. arXiv preprint arXiv:1904.02880 022.9900.0882,2000 2019.
15

---

Published as a conference paper at ICLR 2025
A DETAILS OF MASKGCT
A.1 MODEL. ARCHITECTURE
We use Llama Transformer architecture as the backbone of our model, incorporating gated linear units with GELU [43 activation (SwiGLU), rotation position encoding 44 etc.. but replacing causal attention with bidirectional attention. We also use adaptive R.MSNorm [45 which accepts the time step the condition. Table presents the key hyperparameters of the models.
Table 7: Overview of the key hyperparameters of MaskGCT.
T2S-Base
T2S-Large
S2A
Layers
16
16
16
Model Dimension
1,024
1,536
1,024
FFN Dimension
4,096
6,144
4,096
Attention Heads
16
16
16
Attention Type
Bidirectional SwiGLU
Bidirectional
Bidirectional
Activation Function
Positional Embeddings
RoPE (0 = 10,000) 315M
Number of Parameters
695M
353M
A.2 INFERENCE STEPS FOR THE T2S MODEL
Figure 4 shows the relationship between inference steps and metrics SIM and WER for SeedTTS test-zh (left) and test-en (right). Initially, SIM increases significantly, stabilizing after 25 steps. For test-zh, SIM rises from 0.761 at 5 steps 0.771 75 steps, and for test-en, from 0.696 to 0.715. SIM reaches high values with just 10 steps but peaks around 25 steps. WER improves more dramatically, especially up 25 steps. For test-zh, WER drops from 10.19 at 5 steps to 2.507 25 steps, and for test-en, from 8.096 to 2.346. Both SIM and WER show minimal changes beyond 25 steps. These findings indicate that while SIM metrics can be sufficiently optimized with around 10 inference steps, achieving the lowest WER values requires approximately 25 inference steps. Beyond this threshold, both SIM and WER metrics exhibit minimal changes, implying that further increases in inference steps do not yield substantial improvements in these performance metrics. Therefore, for practical applications, 25 inference steps may be considered optimal for balancing SIM and WER, ensuring efficient and effective performance.
0.8
12
0.74
SIM
WER
SIM
WER
0.79
10
0.72
0.78
SIM
WER
SIM
WER
0.77
0.76
0.75
0.68
10
25
50
75
5 10
25
50
75
Inference Steps
Inference Steps
Figure 4: Inference Steps SIM and WER. The results on the left are for SeedTTS test-zh, and the results on the right are for SeedTTS test-en. In this ablation study, we utilize the ground truth speech length.
16

---

Published as a conference paper at ICLR 2025
A.3 INFERENCE STEPS FOR THE S2A MODEL
The S2A model generates tokens layer by layer during inference. Since the acoustic codec follows an RVQ structure, we can view the S2A inference as process from coarse to fine. We also use more iterations in the initial layers,
the first few layers earry more information. By default, use inference steps of [40, 16, 1, 1,1, 1. 1, 1, 1, 1, 1, 1, 1] for each layer, however, we find that the S2A model can also perform well with fewer steps, such as [10, 1, 1, very slight performance loss.
1, with only
Table 8: Evaluation results of different inference steps for the S2A model.
Inference Steps
SIM-O
WER
FSD J
Seed TTS test-en
10. 1.
1.
1.
1
1.
0.709
2.796 2.466
0.164
40, 16,
0.728
0.159
Seed T test-zh
10.
1,
1.
1,
0.766
2.268
0.111
40,
16,
1,
0.777
2.183
0.101
We present the real-time factor (RTF) of MaskGCT an A100 GPU for generating 20-second speech across various inference steps in Table Across all configurations presented, there is no also present the RTF of AR SoundStorm. For
significant performance difference. Additionally, AR + SoundStorm, generating 20-second speech requires 1000 steps for text-to-semantic inference. However, can leverage kv-cache accelerate the process.
Table 9:
Real-time factor (RTF) comparison of MaskGCT and AR SoundStorm A100 GPU for generating a 20-second speech.
Model
T2S steps
S2A steps
RTF
MaskGCT
40, 16, 1,
0.52
MaskGCT
50
110 1,
1, 1, 1, 1,
1,1
0.44
MaskGCT
25
10,
1.
1, 1.
1,1,1 1,1
0.31
AR SoundStorm
1000
[40, 16,
1.
0.98
A.4 DETAILS OF SEMANTIC AND ACOUSTIC CODEC
For semantic codec, we train a VQ-VAE model using the hidden features from the 17th layer of W2v-BERT 2.0, incorporating factorized codec [33] technology. The original hidden dimension of 1,024 is projected into a diverddimesiionll space for quantization. The codebook size set to 8,192, with codebook dimension of 8. We employ only the L1 loss the reconstruction target, optimizing the codebook with codebook loss and commitment loss. The input features are normalized to have a mean of 0 and a variance of I, based on the statistics of the training dataset. The encoder and the decoder are each composed of 12 mirrored ConvNext blocks, featuring a kernel size of 7 and a hidden size of 384.
For acoustic codec, the basic architecture of the encoder follows 377 and the decoder follows [47] The Vocos-based decoder can model amplitude and phase, enabling waveform generation through inverse STFT transformation without requiring upsampling. The number of RVQ layers. codebook size, and codebook dimension are set to 12, 8, 192, and 8, respectively. We utilize the multi-scale mel-recoonstruction loss [37] Lrc. for the adversarial loss Lub., we employ both the multi-period discriminator (MPD) and the multi-band multi-scale STFT discriminator, as proposed by [37 65). Additionally, we incorporate the relative feature matching loss Lteat For codebook learning, the codebook loss Leodebook and the commitment loss Lcommit from VQ-VAE We set A = 10.0,
Audv = 2.0, Apcat 2.0, Acodebook 1.0, Agommit 0.25 as coefficients for balancing each loss terms. Figure shows the overview of the semantic codec and acoustic codec, Table I0 presents the detailed model configurations of semantic codec and acoustic codec.
17

---

Published as a conference paper at ICLR 2025
Semantic features
Time/Spec Discriminator
Speech Semantic
Codec
Decoder
Speech Acoustic
Codec
VQ
Decoder
Encoder
RVQ
Semantic features
Encoder
W2Y-BERT 2.0
m
Figure 5: An overview of the semantic codec (left) and acoustic codec (right). The semantic codec is trained quantize semantic features with single codebook and reconstruct semantic features. The acoustic codec is trained to quantize and reconstruct the speech waveform using RVQ, with time and spectral discriminators to enhance the reconstruction quality further.
Table 10: The detailed model configurations of semantic codec and acoustic codec.
Semantic Codec
Acoustic Codec
Input
W2v-BERT 2.0 hidden
Waveform
Sample Rate
16K
24K
Hopsize
320
480
Number of (R)VQ Blocks
1
12
Codebook size
8, 192
1,024
Codebook Dimension
8
8
Decoder Hidden Dimension
384
512
Decoder Kernel Size
7
7
Number of Decoder Blocks
12
30
Number of Parameters
44M
170M
A.5 DETAILS OF DURATION PREDICTOR
MaskGCT requires specifying the target speech duration during inference, so train a flow match- ing [46] 66) based duration predictor to obtain the total duration of the target audio by summing the phone-level duration. Note that do not need to actually the phone-level durations but only use them to make a reasonable estimate of the total duration, leaving other total duration predictor methods for future works explore. The duration predictor has similar Transformer architecture to MaskGCT, with 12 layers, 12 attention heads, and a hidden size of 768. We also adapt in-context learning and classifier-free guidance for the duration predictor. During training, randomly select prefix segment of the phoneme sequence and its corresponding duration as prompt, which is not added with noise. At the same time, we use a probability 1. to drop the prompt. We model the du- ration in the log domain using flow matching. We denote
a random variable of log dduration + 1), ar as randomly sampled Gaussian noise, then (2 1) = 2, = (1 1)=0 + Lz1, where the timestep t [0, 1]. The loss function of the duration predictor is E,,,,, (v,(2,, (x1 )2 In the inference stage, we use a midpoint ODE solver to generate the target from randomly sampled Gaussian noise with total of 4 steps. We pretrain duration aligner (between phoneme and W2v-BERT'S 2.0 semantic feature) based on monotonic alignment search (MAS) [67] to get the ground truth duration for each phoneme.
is noteworthy that there are several methods to determine the total duration length. Our trained dura- tion predictor solely for providing rough estimate to facilitate inference and comparison. Altern-
18

---

prompt phone numbber Published as a conference paper at ICLR 2025
tively, simpler approach could be: target total duration rarget phone number prompt total duration
Table 11 illustrates the comparative results of MaskGCT under three different total duration calcula- tion methods. The results indicate that our model, using simple rules to predict total duration, can generate speech with SIM and WER that essentially comparable those of the ground truth. The results indicate that our model, using simple rules to predict total duration, can generate speech with SIM and WER that are essentially comparable those of the ground truth.
Table 11: Comparison of results for MaskGCT under three different total duration calculation methods.
Method
SIM-O t
WER 4
LibriSpeech test-clean
rule-based
0.686
2.976
duration predictor
0.687
2.634
length
0.697
2.012
STedTTS test-en
rule-based
0.719
2.712
duration predictor
0.717
2.623
gt length
0.728
2.466
SeedTTT test-zh
rule-based duration predictor
0.771
2.409
0.774
2.273
gt length
0.777
2.183
A.6 COMPARISON BETWEEN MASKGCT AND OTHER SYSTEMS
Table 12: A comparison between MaskGCT and existing systems. "Model" stands for modeling method and "Rep." stands for the representation used. MaskGCT uses masked generative modeling for acoustic and semantic tokens (*A." stands for acoustic, "S." stands semantic, "F" stands for factorized tokens used in NaturalSpeech 3). MaskGCT implicitly models duration ("Imp. Dur.") allows flexible control over total length of generated speech ("Len. Ctrl"). MaskGCT supports various speech generation tasks. "ZS TTS" denotes zero-shot TTS and "CL TTS" denotes cross-lingual TTS.
System
Model
Rep.
Dur.
Len. Ctrl.
TTS
CL TTS
Dubbing
Edit
VALL-E
Autoregressive Diffusion
Tokens
NaturalSpeech2 2
A. Features
X
VoiceBox
Dillusion
A. Features
X
VoiceCraft
Autoregressive
Tokens
/
NaroralSpecch
Masked Generative
Tokens
X
MaskGCT
Masked Gencrative
S.&.A. Tokens
V
A.7 TEXT TOKENIZER
We consider two text tokenization methods: Hyypheme-to-Phonemee (G2P) and Byte Pair Encoding (BPE). For G2P, we employ phone- mize for English and combination of jieba and pypinyin 8 for
Table 13: G2P vs. BPE.
SIM-O 1
WER
Chinese, For BPE, we utilize the BPE method and vocabulary from
TTS fest-en
Whispe with vocabulary size exceeding 30,000. Table 13 shows the comparison results of MaskGCT using the two different text tokenization methods. The results indicate that G2P outperforms BPE in English with a higher SIM-O of 0.728 compared to 0.711 and lower WER of 2.466 versus 4.036. Conversely, Chinese,
G2P
0.728
2,466
BPE
0.711
4,036
Seed test-zh
G2P
0.777
2.183
BPF
0.769
1.921
G2P maintains a slightly higher SIM-O (0.777 vs. 0.769) but BPE achieves a lower WER (1.921 vs.
http:e//gitthbb.co///ss////1
19

---

Published as a conference paper at ICLR 2025
2.338). These findings suggest that while G2P is superior in preserving text similarity and reducing errors in English, BPE is more effective in minimizing WER in Chinese. We hypothesize that the reason might be that the Chinese G2P system we used still has deficiencies in handling polyphonic characters. In contrast, BPE can learn different pronunciations for the same character based on context.
B DISCUSSION ABOUT SEMANTIC AND ACOUSTIC DEFINITIONS
In this paper, we refer to the speech representation extracted from the speech self-supervised learning (SSL) model as the semantic feature. The discrete tokens obtained through the discretization of these semantic features (using k-means or vector quantization are termed semantic tokens. Similarly, we define the representations from melspectrogram, neural speech codecs, speech VAE acoustic features, and their discrete counterparts are called acoustic tokens. This terminology was first introduced in [68] and has since been adopted by many subsequent works (8 is important to note that this is not strictly rigorous definition. Generally, we consider semantic features or tokens to contain more prominent linguistic information and exhibit stronger correlations with phonemes or text. One measure of this is the phonetic discriminability in terms of the ABX emrr rate. this paper, the W2v-BERT 2.0 features we use have phonetic discriminabilityy within less than 5 on the LibriSpeech dev-clean dataset, whereas acoustic features, for example, Encodec latent features, score above 20 on this metric. However, it worth noting that semantic features or tokens not only contain semantic information but also include prosodic and timbre aspects. In fact, we suggest that for certain two-stage zero-shot TTS systems, excessive loss information in semantic tokens degrade the performance of the second stage, where semantic- -oa.utic--t--cuustii conversion occurs. Therefore, finding a speech representation that is more suitable for speech generation remains challenging problem.
19 39
69 70)
C CLASSIFIER- FREE GUIDANCE
We adopt the classifier-fres free guidance [58] technique for both the T2S model and the S2A model. We also introduce classifier-free guidance with rescaling. following [59]. In the training stage, we randomly drop the prompt with a probability of 0.15 to model the probability distribution
pa X) without the prompt. During inference, we compute the output embedding Do _ (X{)))
go X XP) + Wefe (go(X XP) (X)) of the last layer of the model, where Wefz the classifier-
free guidance scale, then we compute the rescale embedding (X(x))
9o cly (X x
std (x) s)/dd((g (X(X))) the final output embedding is computed as Wrescale 90 rescale (X[X))
(1 Wrescate) 9ø [X] XP). In our paper, Wefg and Wrescnle are set as 2.5 and 0.75 by default.
D EVALUATION BASELINES
A large-scale TTS system uses an autoregressive and an additional non-autoregressive model to predict discrete tokens from neural speech codec (26 We reproduce VALL-E with Amphion toolkit [1] and Librilight [72 dataset.
VALL-E 2
non-autoregressive model large-scale TTS systems with factorized speech codec for speech decoupling representation and factorized diffusion Models for speech generation. It achieves human-level naturalness on the LibriSpeech test set. We report the scores of LibriSpeech test-clean obtained from 8 and ask for the generated samples for subjective evaluation.
NaturalSpeech
VoiceBox D. non-autoregressive model large-scale multi-task speech generation model based on flow matching [46). We report the scores of LibriSpeech test-clean obtained from [8] and ask for the generated samples for subjective evaluation.
X2 (53 An open-source multilingual TTS model that supports 16 languages. It is also based on an autoregressive model. We use the official code and pre-trained checkpoin 10
/
20

---

Published as a conference paper at ICLR 2025
VoiceCraft [3], A token-infilling neural codec language model for text editing and text-to- text-to-speech, predicts multi-layer tokens in a delay pattern. We use the official code and pre-trained checkpoin
CosyVoice [54]. A two-stage large-scale TTS system. The first stage is autoregressive model and the second stage is a diffusion model. It is trained on 170,000 hours of multilingual speech data. We use the official code and pre-trained checkpoin
E MULTILINGUAL ZERO-SHOT TTS
We validate the effectiveness of MaskGCT across four additional languages beyond Chinese and English, specifically Japanese, Korean, German, and French. On the foundation of our existing training data, we expand by 2,500 hours of Japanese, 7,400 hours of Korean, 6,900 hours of German, and 8,200 hours of French. We collect these data using the data collection pipeline proposed by [48] For evaluation, we use the test sets provided in (88) We still employ SIM-O and WER as evaluation metrics, with Whisper-medium serving as the ASR model for WER assessment. We utilize XTTS- v2 and the two models proposed in [48]: Emilia-AR and Emmlia-NAR as comparative baselines. Table shows the results. MaskGCT demonstrates significant improvements over the baselines, with the exception of WER in Japanese. It is noteworthy that we only retrained our text-to-semantic model using the expanded data, without retraining the tokenizers and semantie-toaccoustic models. We believe that further enhancements in our model's performance can be achieved if all components are retrained on the expanded data.
Table 14: Evaluation results for MaskGCT and baseline methods on the test sets for Japanese, Korean, German, and French.
Ja
Ko
Fr
De
System
WER
SIM-O
WER
SIM-O
WER
SIM-O
WER
SIM-O
Emilia-AR
3.6
0.625
10.9
0.681
8.2
0.589
6.8
0.680
Emilia-NAR
10.8
0.562
15.2
0.608
17.5
0,550
13.3
0.633
XTTS-v2
2.981
0.579
12.45
0.617
6.898
0.531
9.168
0.569
MaskGCT
3.903
0.678
9.417
0.732
5,598
0.667
5.126
0.745
F DURATION- SPEECH TRANSLATION
The goal of the speech translation task is to translate speech from one language to another while preserving the original semantic, timbre, and prosody. In some scenarios, we also need to ensure that the total duration remains relatively unchanged, such as in cross-lingual dubbing. Our model can achieve this seamlessly, with the ability to control the total duration and, through in-context learning, use the pre-translation speech the capabilities of our model, we randomly select 200 samples from SeedTTS test-zh and 200 samples from SeedTTS test-en. Additionally, we sample 200 examples for each language of Japanese, Korean, German, and French from each of the test sets provided in [48). Subsequently, we utilize GPT4o- mini 73 to translate each sample into one of the other five languages. using the translated text as the target text. We use the duration of prompt speech as the duration of target speech. This process yields 30 sets of test data. Table shows the results of the 30 sets of experiments. We observe that MaskGCT maintains good level of speaker similarity across translations between the six languages. Both "X to En" and "En to X" generally perform well, characterized by relatively low WER values and moderate SIM-O scores. "X to Ja" also achieve low WER values. However, for languages other than English, "X to Zh", "X to De", and "X to Fr" exhibit higher WER values. We hypothesize that the primary reasons for this include the difficulty in maintaining accurate pronunciation while preserving the same duration before and after translation, as well as the limited training data for Fr and De. Achieving more robust cross-lingual translation remains focus for future work. We also show some examples of speech translation in our demo page.
prompt maintain the timbre and prosody. To quantify
21

---

Published as a conference paper at ICLR 2025
Table 15: Evaluation results in cross cross- linsull speech translation with consistent total duration.
Zh
Em
Ja
Ko
De
Fr
WER
SIM-O
WER 7.466
SIM-O 0.678
WER
SIM-O
WER
SIM-O
WER
SIM-O
WER
SIM-O
Zh
7.864
0.720
9.751
0.736
25.54
0.724
16.21
0.687
7.411
0.535
5,870
0.544
12.18
0.543
12.43
0.579
17.48
0.590
Ja
13.93
0.647
7.387
0.642
10.98
0.703
12.85
0.649
14.61
0.645
26.58
0.722
De 19.54
0.714
5.148
0.740
6.072
0.678
12.02
0.667
14.53
0.672
Fr
32.84
0.672
12.17
0.682
6.076
0.640
12.07
0,582
21.65
0.682
G POST-TRAINING FOR EMOTION CONTROL
MaskGCT can unlock more extensive capabilities with post-training. We take emotion control as an example. After being pretrained large-scale dataset, fine-tune the T2S model by adding an additional emotion label as prefix to the original input sequence. We use an emotion dataset, ESD [6]], which consists of 350 parallel utterances with average duration of 2.9 seconds spoken by 10 native English and 10 native Mandarin speakers, to fine-tune our model. The experimental results show that MaskGCT can unlock emotion control capabilities for zero-shot in-context learning scenarios. For the construction of the train and test datasets, we selected one male and one female speaker each from native English and native Mandarin backgrounds, resulting in total of four speakers for the test dataset. The remaining 16 speakers were allocated to the training dataset. For the 350 parallel Chinese utterances, utterances designated for training. Similarly, for the 350 parallel English utterances, we randomly selected 21 utterances for the test set, with the rest used for training. To assess the consistency between the generated audio and the target emotion label, we trained an emotion classification model using the constructed train dataset. This model achieved a classification accuracy 72% on the test dataset. We show some examples in our demo page.
randomly chose 22 utterances for the test set, with the remaining
H SPEECH CONTENT EDITING
Based on the mask-and- pedddppeecct mechanism, our text-to-semantic model supports zero-shot speech content editing with the assistance of text-speech aligner. By using the aligner, we can identify the editing boundary of the original semantic token sequence, mask the portion that needs to be edited, and then predict the masked semantic tokens using the edited text and the unmasked semantic tokens. However, we have observed that our system is not very robust in editing tasks. A possible conjecture that need to adopt training paradigm better suited for editing tasks, such as fill-in-mask [9] 74. We show some examples in our demo page.
I VOICE CONVERSION
MaskGCT supports zero-shot voice conversion by fine-tuning the S2A with modified training strategy. The zero-shot voice conversion task aims to alter the source speech to sound like that of target speaker using reference speech from the target speaker, without changing the semantic content. We can directly use the semantic tokens Sge extracted from the source speech and the prompt acoustic tokens Age extracted from the reference speech to predict the target acoustic tokens Agt- Since Sare may retain some timbre information, we perform timbral perturbation on the semantic features input to the semantic codec encoder. Specifically, we apply timbral perturbation to the input mee-sppctrogram features the W2y-BERT' 2.0 model, following the method outlined in VC (75.. We fine-tune our S2A model using this training strategy. We show some examples in our demo page.
J HARD CASES EVALUATION
We evaluate the performance of MaskGCT some hard cases (SeedTTS test-hard), which refer to instances where large-scale TTS models, particularly those AR-based models, often exhibit hallucinations. These cases include phrases with repeating words, tongue twisters, and other complex
22

---

Published as a conference paper at ICLR 2025
linguistic structures. Examples of such cases include: "the great greek grape growers grow great greek grapes", "How many cookies could a good cook cook If a good cook could cook cookies? good cook could cook much cookies as good cook who could cook cookies", and ttouugh thought. But the thought thought wasn't the thought thought thought. If the thought thought thought had been the thought thought, I wouldn's i have thought much".
Table 16: The evaluation results of MaskGCT and AR SoundStorm SeedTTS test-hard.
System
SIM-0
WER
Seed test-hard
AR SoundStorm
0.692
34.16
AR SoundStorm (rank 5)
0.739
17.05
MaskGCT
0.748
10.27
MaskGCT (rank 5)
0.776
6.258
K DISCUSSION ABOUT CONCURRENT WORKS
DiTTo-TTS E2 TTS 32) and F5-TTS 76 are also NAR-based models that do not necessitate precise alignment information between text and speech, nor do they forecast phoneme-level duration. These are concurrent works with MaskGCT. The three models all employ diffusion modeling speech representations within continuous spaces. SimpleSpeech models the latent representation of a wav codee based on finite scalar quantization (FSQ) 77]. DiTTo-TTS utilizes the latent representation of way codec based on residual vector quantization (RVQ), and E2 TTS and F5-TTS directly model the mel-spectrogram with flow matching. However, Both F5-TTS and A2Flow (78 mention that direct modeling mel-spectrogran is characterized by slow convergence and difficulty in achieving convergence on small datasets. To further investigate this issue, we train the T2S models small datasets (LibriTTS [79] and 1K hours subset of Emilia) while reusing the S2A model (which is entirely self-supervised and does not require text). The results in Table [7 demonstrate that our model performs well even when trained on small datasets. This is likely due to the case of predicting semantic tokens and the powerful modeling capabilities of masked generative models.
SimpleSpeech 300
Table 17: The evaluation results of MaskGCT on small training datasets.
Model
SIM-O f
WER |
SeedTTS test-en
MaskGCT (LibriTTS 0.58K hours)
0.677
3.043
MaskGCT (Emilia 1K hours)
0.696
3.378
MaskGCT (Emilia 100K hours)
0.728
2.466
STedT'SS test-zh
MaskGCT (Emilia 1K hours) MaskGCT (Emilia 100K hours)
0.754
3,012
0.777
2.183
We also compare MaskGCT with direct text-lo-acoustic approach using masked generative models (which can be seen as removing semantic tokens as a condition and adding text as a condition based on the S2A model) acoustic tokens from text challenging to converge, resulting in lower SIM and significantly higher WER, demonstrating that the two-stage model reduces the overall modeling difficulty.
10K hours subset. The results in Table I8 indicate that directly predicting
23

---

Published as a conference paper at ICLR 2025
Table 18: Comparison of MaskGCT and Text-to-Acoustic.
Model
SIM-O
WER
TTs test-en
Text-to-Acoustic (Emilia 10K hours)
0.651
12.75
MaskGCT (Emilia 10K hours)
0.719
2.872
TTS test-zh
Text-to-Acoustic (Emilia 10K hours)
0.727
17.08
MaskGCT (Emilia 10K hours)
0.762
3,302
L BOARDER IMPACT
Given that our model can synthesize speech with high speaker similarity, it carries potential risks of misuse, including spoofing voice identification or impersonating specific speakers. Our experiments were conducted under the assumption that the user consents to be the target speaker for speech synthesis. To mitigate misuse, essential to develop robust model for detecting synthesized speech and to establish system for reporting suspected misuse.
24


Reviews:
[{"review_id": "LjC1DtFCve", "reviewer": "Reviewer_QWox", "summary": "This paper proposes a masked generative codec transformer, MaskGCT, which performs speech synthesis using a mask-predict approach. MaskGCT generates semantic tokens from text input and is composed of two components: text-to-semantic MaskGCT, which generates semantic tokens from text, and semantic-to-acoustic MaskGCT, which generates acoustic tokens from semantic tokens. Each MaskGCT operates by filling masked parts based on confidence scores predicted in stages, from a sequence where all discrete speech tokens are masked at the start. Instead of using the commonly used method of obtaining semantic tokens through k-means clustering applied to pre-trained SSL models, MaskGCT employs a separate VQ-VAE model on SSL features to generate semantic tokens, reducing information loss through reconstruction loss. MaskGCT uses a similar mask prediction method for predicting discrete speech tokens as Soundstorm, which is based on MaskGIT. However, unlike Soundstorm, MaskGCT approaches the semantic token generation from text through masked token modeling and includes an additional module for total length prediction to align sequence lengths. MaskGCT demonstrates superior speaker similarity in widely used zero-shot TTS evaluation compared to various zero-shot TTS models. Moreover, a demo page with samples shows the versatility of MaskGCT across various tasks.", "questions": "Comments\n\n* It would be helpful to mention the frame rate of semantic and acoustic tokens.\n* I am curious whether only a single sampling step is used from the third layer during acoustic token generation. Is there any difference when using more sampling steps?\n* It would be beneficial to also present the overall sampling speed of the model, including the Real-Time Factor.\n* In Table 1, it seems that VoiceBox's Imp. Dur. should be marked as \"X.\"", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper shows that a mask-predict approach on discrete tokens can achieve a high level of speaker similarity. \n\n* Samples on the demo page show that MaskGCT enables zero-shot TTS that effectively captures not only timbre but also emotion and style across a variety of voices, with excellent sample quality. \n\n* To assess how well the model captures accent and emotion, the paper introduces metrics such as Accent SIM and Emotion SIM, which utilize representations reflecting each attribute. Their results indicate that MaskGCT performs better in imitating accent and emotion than other approaches based on these metrics.", "weaknesses": "* In Soundstorm, the mask-predict approach is introduced to acoustic token generation. It seems that the methodological novelty in MaskGCT lies mainly in extending this mask-predict approach to semantic token generation.\n\n* The paper utilizes VQ-VAE to reduce information loss in semantic tokens compared to the k-means clustering approach; however, it does not experimentally demonstrate how this approach improves over k-means clustering.\n\n* Additionally, while using a larger number of semantic tokens may yield good performance in zero-shot TTS, the semantic tokens do not seem disentangled from speaker information, making MaskGCT less suitable for voice conversion tasks. The voice conversion samples on the demo page also appear less similar to the reference speaker.  \n\n* When comparing single-sample generation, MaskGCT's pronunciation accuracy appears lower than that of VoiceBox or NaturalSpeech 3.\n\n* Regarding the seed-TTS evaluation method, although it is not explicitly shown as a baseline in the paper, the objective metrics appear to be worse than those of seed-TTS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TnoPChPCYx", "reviewer": "Reviewer_n456", "summary": "This paper introduces MaskGCT, a non-autoregressive (NAR) masked generative modeling method that sequentially generates audio from acoustic tokens, semantic tokens, and text. The semantic tokens are derived from VQ-VAE of W2V-BERT, a self-supervised speech learning model. The acoustic token approach extends DAC with improved training efficiency.\n\nResults demonstrate that MaskGCT either outperforms or matches current autoregressive (AR) based methods and NAR duration-based methods with publicly available implementation details, across multiple benchmarks: LibriSpeech, SeedTTS (Chinese and English), zero-shot TTS, and speech style (timbre, emotion, accent, etc.) imitation learning. Additionally, MaskGCT surpasses AR-based methods in both naturalness and computational efficiency (inference time steps).", "questions": "See weaknesses.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* This paper experimentally demonstrates that NAR (non-autoregressive) non-duration-based TTS methods outperform both AR (autoregressive) based and NAR duration-based TTS methods with publicly available implementation details. \n\n* The open-source implementation and comprehensive reproducibility details provided in the appendix represent valuable contributions to the field.\n\n* The paper is well-structured, with clear presentation and thorough technical descriptions.", "weaknesses": "Major Concerns:\n\n* The novelty and technical contribution appear limited. The core methodology seems to be primarily a combination of existing work (MaskGIT and SpearTTS). The speech acoustic code is derived from DAC, with no apparent novel methodological contributions.\n\n* Regarding NAR methods, the authors cite phone-level duration prediction as a source of complex model design. However, this claim needs clarification: How is this complexity measured (e.g., training time, loss balancing)? Why are AR-based methods considered less complex? Experience with NaturalSpeech3/Fastspeech2 suggests duration modeling is manageable and straightforward to implement. Does this conclusion apply to traditional small-scale TTS models? The statement appears overly generalized.\n\n* The Related Work section overlooks prominent flow-based methods like VoiceBox and p-flow, which also employ masked generative models. Their exclusion from this category requires explanation.\n\n* Table 1 would be more appropriate in the appendix. Multi-task capability alone is not a significant contribution, as multi-task frameworks are common. The focus should be on concrete improvements in naturalness, prosody, efficiency, or reduced complexity. Placing Table 1 prominently suggests multi-task functionality is the main contribution, which diminishes the paper's actual value.\n\n* In line 210, the claim that k-means based discrete SSL units lead to information loss needs substantiation. Why is VQ-VAE of W2V-BERT considered superior for semantic preservation despite also being discrete SSL units? While Appendix B provides semantic and acoustic comparisons, direct evidence supporting this specific claim is lacking.\n\nMinor Points:\n\n* Figure 2 would benefit from additional notations for clarity. Also, what is the difference between prompt semantic tokens and target semantic tokens? Can you give an example?\n\n* While following the text-semantics-acoustics pipeline established by papers like SpearTTS, the superiority of this approach over alternatives (e.g., direct text2acoustic without semantics) should be justified.\n\n* Section 4.2.3's purpose and motivation need clarification, particularly if addressing speed modification, as specialized tools exist for this purpose.\n\n* Consider whether inference steps alone adequately measure efficiency, or if other metrics common in streaming methods (e.g., latency) would provide more comprehensive evaluation.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qcPzwlOflS", "reviewer": "Reviewer_AvMh", "summary": "The authors present a zero-short text-to-speech (TTS) system called Masked Generative Codec Transformer (MaskGCT), which comprises two stages: 1) predicting semantic tokens from text and 2) using these semantic tokens to generate acoustic tokens. Both stages leverage a non-autoregressive masked generative transformer with specific prompt tokens for in-context learning. A VQ-VAE codec and a residual vector quantization (RVQ) codec are separately trained to obtain the semantic tokens and acoustic tokens. Extensive experiments across various tasks demonstrate that MaskGCT, trained on 100k hours of speech data, surpasses existing state-of-the-art zero-shot TTS systems in quality, similarity, and intelligibility.", "questions": "- In Table 1, does \"ZS TTS\" denote \"zero-shot TTS\", and what's \"CL TTS\"? For \"Imp. Dur.\", I also don't fully agree that this work implicitly models duration, as it requires a specified length of the target generated speech.  \n- In Section 3.2.3,  why \"the number of frames in the semantic token sequence is equal to the sum of the frames in the prompt acoustic sequence and the target acoustic sequence\"? \n- Is the output length of the text-to-semantic stage equal to that from the semantic-to-acoustic stage? From Figure 2, the former seems shorter than the latter? \n- In the first paragraph of Section 2, about \"SpearTTS utilizes three AR models to predict semantic tokens from text, coarse-grained acoustic tokens from semantic tokens, and fine-grained acoustic tokens from coarse-grained tokens.\", the terms \"coarse-grained\" and \"fine-grained\" do not seem to be mentioned in the SpearTTS [1] paper, what do the authors mean by them? \n- What's the length of prompt audio? Does it affect the quality of the generated speech?\n\n[1] Kharitonov, E., Vincent, D., Borsos, Z., Marinier, R., Girgin, S., Pietquin, O., ... & Zeghidour, N. (2023). Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics, 11, 1703-1718.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work utilizes non-autoregressive masked generative transformers to generate both sematic tokens and acoustic tokens, offering advantages such as flexible length control, better robustness, and higher inference efficiency. This is suggested by the experiments in Section 4.2.2, comparing to the system replacing the first stage with an autoregressive model (i.e., AR + SoundStorm). \n- The authors conduct extensive experiments on a variety of tasks to evaluate the effectiveness and sca of the proposed system, including not only standard zero-shot TTS, but also cross-lingual dubbing, voice conversion, emotion control, and speech content editing, with potential use of post-training and fine-tuning. \n- The selected state-of-the-art baselines are well-chosen.", "weaknesses": "- The novelty of this work is somewhat incremental, though I would appreciate its engineering contributions. This work follows a two-stage codec-based sequence-to-sequence strategy, converting text into semantic tokens and then into acoustic tokens, which is in line with SPEAR-TTS [1]. The SoundStorm [2] paper also explores this strategy. Additionally, the idea of applying non-autoregressive masked generative transformer for codec-based TTS is presented in SoundStorm [2] and NaturalSpeech3 [3] (though the NaturalSpeech3 paper describes it from a diffusion perspective.). However, I would like to highlight the distinction that this work employs non-autoregressive masked generative transformers in both stages of the two-stage framework. \n- The authors emphasize that the proposed method does not require text-speech alignment supervision or phone-level duration prediction. However, it appears that the method does require a specified length for the target semantic token sequence, thereby to control the length of the generate speech. The presented experimental results in Table 2, comparing to baseline methods, are based on either ground-truth duration or predicted duration. As descibed in the paper, the authors train a duration predictor to estimate the phone-level durations, which are then summed to determine the total duration of the target speech. This process relies on pre-computed ground-truth phone-level durations. Therefore, it seems that this may not fully align with the claim of not requiring text-speech alignment supervision or phone-level duration prediction. \n- The authors claim that training a VQ-VAE model to quantize the semantic representation can reduce the information loss, compared to using k-means clustering as in previous works. I think it would be better if the authors can provide ablation results or relevant references to support this assertion. \n\n[1] Kharitonov, E., Vincent, D., Borsos, Z., Marinier, R., Girgin, S., Pietquin, O., ... & Zeghidour, N. (2023). Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics, 11, 1703-1718.\n\n[2] Borsos, Z., Sharifi, M., Vincent, D., Kharitonov, E., Zeghidour, N., & Tagliasacchi, M. (2023). Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.09636.\n\n[3] Ju, Z., Wang, Y., Shen, K., Tan, X., Xin, D., Yang, D., ... & Zhao, S. (2024). Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. arXiv preprint arXiv:2403.03100.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rrIEoxRvGQ", "reviewer": "Reviewer_EzBH", "summary": "This paper presents MASKGCT, a compact Non-Autoregressive (NAR) text-to-speech model. This model is meticulously designed with a specific emphasis on attaining outstanding audio quality. What sets it apart from prior works is its utilization of semantic tokens extracted from the SSL representation using VQVAE and its avoidance of relying on explicit alignment information.", "questions": "Address my concerns in the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper offers quite interesting and remarkably high-quality demo audio. The audio samples provided within the paper not only demonstrate clear and distinct sound qualities but also exhibit a certain level of innovation in terms of the audio characteristics they present. They are able to effectively capture the attention of the readers and give a practical sense of the research outcomes in the audio domain.\n- The proposed pipeline is significantly more compact than previous ones such as NaturalSpeech3. This compactness implies a more streamlined and efficient design. It potentially leads to reduced computational complexity and may offer advantages in terms of implementation and resource utilization.", "weaknesses": "- The Word Error Rate (WER) in Table 4 and 5 of all methods is much higher than previously reported. This discrepancy requires more in-depth explanation. It is essential to analyze the factors contributing to this higher WER, such as possible differences in the data sets used, the experimental settings, or any unique characteristics of the methods employed in this study. A detailed exploration and clarification of these aspects would enhance the understanding and validity of the results presented.\n- In Section 3.2.1, the authors utilize VQVAE to obtain discrete semantic tokens instead of k-means. However, no corresponding experiment result is provided. It would be beneficial to include the experimental results related to the use of k-means for a more comprehensive understanding. These results could demonstrate the effectiveness and performance of VQVAE in this context, and also facilitate a comparison with the potential outcomes if k-means had been used. This would add more substance to the discussion and support the authors' choice of using VQVAE.\n- To my knowledge, using text as a condition without expanding by duration for a Non-Autoregressive (NAR) model was first proposed in e2-tts. The authors should point out this in Section 3.2.2. Acknowledging the prior work in this area is important for providing a complete context and showing the evolution of the research. By highlighting the connection to e2-tts, the authors can better position their work within the existing body of knowledge and give credit to the relevant pioneering research. It also helps readers to better understand the background and significance of the approach taken in this study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)