Continuous Autoregressive Modeling with Stochastic Monotonic Alignment for Speech Synthesis
Authors: Weiwei Lin
Chenhang HE
Venue: ICLR 2025 Poster (2025)
Keywords: Speech Synthesis;text-to-speech;
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: an autoregressive speech language model without vector quantization

Abstract: We propose a novel autoregressive modeling approach for speech synthesis, combining a variational autoencoder (VAE) with a multi-modal latent space and an autoregressive model that uses Gaussian Mixture Models (GMM) as the conditional probability distribution. Unlike previous methods that rely on residual vector quantization, our model leverages continuous speech representations from the VAE's latent space, greatly simplifying the training and inference pipelines. We also introduce a stochastic monotonic alignment mechanism to enforce strict monotonic alignments. Our approach significantly outperforms the state-of-the-art autoregressive model VALL-E in both subjective and objective evaluations, achieving these results with only 10.3\% of VALL-E's parameters. This demonstrates the potential of continuous speech language models as a more efficient alternative to existing quantization-based speech language models. Sample audio can be found at \url{https://tinyurl.com/gmm-lm-tts}.

Published as a conference paper at ICLR 2025
CONTINUOUS AUTOREGRESSIVE MODELING WITH STOCHASTIC MONOTONIC ALIGNMENT FOR SPEECH SYNTHESIS
Weiwei Lin
Chenghan He
Department of Electrical and Electronic Engineering The Hong Kong Polytechnic University Hong Kong SAR, China weiwei iinnoonnett.ppo pol yully..k.
Department of Computing The Hong Kong Polytechnic University Hong Kong SAR, China
ABSTRACT
We propose a novel autoregressive modeling approach for speech synthesis, com- bining a variational autoencoder (VAE) with multi-modal latent space and an autoregressive model that uses Gaussian Mixture Models (GMM) as the con- ditional probability distribution. Unlike previous methods that rely residual vector quantization, our model leverages continuous speech representations from the VAE's latent space, greatly simplifying the training and inference pipelines. We also introduce a stochastic monotonic alignment mechanism to enforce strict monotonic alignments. Our approach significantly outperforms the state-of of-the- art autoregressive model VALL-E in both subjective and objective evaluations, achieving these results with only 10.3% of VALL-E's parameters. This demon- strates the potential of continuous speech language models as more efficient alternative to existing quantization based speech language models. Sample audio can be found at https:: /ttps::///iiiuuttt/ com./ gttp//mmm--mmm 1m- tts
INTRODUCTION
Transformers trained with autoregressive (AR) objectives have become the dominant approach in natural language processing (NLP) (Radford et al. 2019; Brown et al. 2020 Vaswani (017)). The vast successes NLP have inspired researchers to apply Transformers and autoregressive objectives to image and speech domains as well (Ramesh et al. 2021 Wang et al. Betker 2023). Since speech and images are continuous signals, discretization is critical first step before applying discrete au- toregressive training. As result, autoregressive modeling of images and audio typically involves two stages of training. In the first stage, a VQ-VAL (Van Den Oord al., 2017) or a variant of VQ-VAE (Zeghidour et al.) | Défossez et al. 2022 Kumar al. 2024) trained to encode input data into discrete latent representations using vector quantization bottleneck. After train- ing the VQ-VAE, encoder (Ramesh et al. 2021). The AR model captures the sequential dependencies in the latent space, learning to predict the next latent code based on previous ones, which enables high-fidelity generation. One of the most attractive features of AR modeling in speech is its ability to model both the distribution of acoustic vectors and the duration simultaneously. This is particularly beneficial compared to non-autoregressive models, which often require separate duration modules and exter- aligmments (Ren et al.) 2020 2019 Li al.) 2022). Another advantage of AR models their capability for in-context learning of speaker style and voice through prompting, which has proven to be more powerful than traditional speaker embedding- -haen voice cloning (Wang et al. Casanova ct al. 2024).
autoregressive model is trained on the discrete latent codes produced by the
has been found that to faithfully reconstruct audio signals, the required codebook size can be prohibitively large (Zeghidour et al. 2021), and the use of discrete codes often introduces artifacts and mispronunciation during reconstruction (Casanova et al. 2024; Betker 2023). Additionally, codebooks are often under-utilized, and training stability Straight-hhrough (ST) estimator (Lancucki et al. 22020) Mentzer et al. 2023). While using multiple codebooks with residual vector quantization (RVQ) mitigates some of these issues (Zeghidour al.
be compromised due to the use of the

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of VALL-E and Our Method.
Method
VALL-E
Ours
Codec Model
RVQ
GMM- MAM
TTS Model
Decoder-only- Transooorrrssssserr
GMM-LM-
Codec Related Param.
16.7M
0
TTS Model Param.
496.5M
51.5M
Dominant Computation
2DL
DI,²
All Tokens AR
No
Yes
Strict Mono. Alignment
No
Yes
WER L Q-MOS
6.04
2.83
3.54
4.11
2021
Défossez et al. 2022). it comes at the cost of requiring a specialized second-stage model to
handle the additional codebooks (Borsos al. 2023b Wang et al.)). To address the computational overhead introduced by these extra codebooks, some approaches avoid using all tokens in autore-
gressive modeling (Wang et al., Lyth & King. 2024 Copet et al. 2024)..
All of the aforementioned issues stem from the use of quantization during VAE training. In this paper, demonstrate that vector quantization is not necessary prerequisite for learning toregressive model. By constraining the latent representation to be continuous multimodal distri- bution during VAE training and recovering it in the second stage with a continuous autoregressive model, we are able to build an autoregressive model that retains all the advantages of AR modeling (prompting, integrated duration modeling, and sampling) without many of the challenges associated with VQ. Table [ compares our proposed continuous autoregressive model with VALL-E (Wang) jet al.), a state-of-the-art autoregressive text-to-speech (TTS) model. Without the need for multi- ple codebooks from residual vector quantization, we are able to use single Conformer model to perform autoregressive modeling on all tokens, cutting theoretical computation in half compared to VALL-E's AR and NAR mixed approach. Furthermore, the removal of quantization allows to build an ultra-compact TTS model with only 51. 31.5M parameters, which is 10.3% of VALL-E's size. Despite its smaller size, model achieves lower WER and higher MOS than VALL-E, thanks to the continuous autoregressive modeling approach.
2 RELATED WORK
Neural Speech Codecs and Representations. Unsupervised learning speech representation have long line of research including learn representating by reconstruction from a bottleneck using au- toencoder (Chorowski et al. 20199] Qian et al. 2019) and mutli-taks learning (Ravanelli et al., 2020). More recently, the focus is shift to learning discrete representation, which often refer to as neural speech codecs. Neural speech codecs can be divided into two categories: semantic codecs and acoustic codecs. Semantic codecs, typically learned by clustering features from self-supervised
models such as Wav2vec2, HuBERT, and WavLM (Baevski et al.; 2020 Hsu et al. 2021 Chen
et al. 2022), primarily preserve the phonetic information of speech (Choi et al. 2024 Due to the disentangied nature of these features, semantic codecs are often coupled with speaker embeddings
TTS (Polyak al. 22021 Chen & Duan 2022). Acoustic codecs, on the other hand, are designed to reconstruct the speech waveform, preserving all information from the speech, including phonetic, speaker, and acoustic environment details. Acoustic codecs are typically trained using a VQ-GAN model (Van Den Oord et al. 2017 Esser et al.) (02T). which learns to reconstruct the input through a convolutional encoder-decoder architecture with GAN-based training and quantization layers. The discrete representation significantly reduces storage requirements and enhances I/O efficiency, mak- ing it an appealing alternative to traditional speech codecs (Zeghidour et al.) 2021 Défossez et al. 2022) Furthermore, it enables the direct application of language models, such BERT and GPT Devlin 2018 Brown et al. 2020), to speech processing tasks. However, vector quantization can lead to mispronunciations during reconstruction (Casanova et al. 2024 Betker 2023). , and building a speech codec in this way often requires a very large codebook (Zeghidour et al.. 202T). To address this, (Zeghidour et al. 2021) proposed using multiple codebooks during the vector quantization pro-
2

---

Published as a conference paper at ICLR 2025
cess, where each codebook quantizes the residuals of the previous one. This approach is referred to as residual vector quantization (RVQ) in the literature. RVQ was further extended in Encodec (Défossez al. 2022). where the authors employed multiscale spectrogram discriminators, loss balancer, and lightweight transformers to improve both the speech quality and efficiency of RVQ. By introducing several vector quantization techniques from the image domain, along with improved loss functions, RVQ was further refined, resulting in very low-bit-rate codecs in (Kumar et al. 2024).
Autoregressive Models in Speech Synthesis. Tacotron (Shen et al., 2018 Wang et al. 2017) pioneered autoregressive modeling in TTS using an RNN trained with regression loss on Mel- spectrograms. Unlike most autoregressive models, Tacotron cannot perform sampling because relies on a regression loss. In contrast, the use of discrete speech codecs is appealing, as they can be directly applied to Transformers using standard cross-entropy loss. However, using multiple code- books from RVQ complicates the design of downstream models. Flattening all the codes leads to a quadratic increase in computational complexity with the number of codebooks. Significant ef- forts have been made to reduce computation for RVQ in downstream models, including strategies where not all tokens are used during autoregressive modeling or where different codebook codecs modeled in separate stages (Copet al. 2024 Wang et al.). In (Borsos et al., 2023a), the authors proposed a three-stage audio generation process with a semantic codec, a coarse acoustic codec, and fine acoustic codec. However, using three autoregressive models results in slower inference. In (Borsos et al., 2023b), the authors proposed generating acoustic vectors in parallel across multiple codebooks by conditioning on semantic codes using a MaskGit decoding scheme. VALL-E (Wang iet al.)) performs autoregressive modeling on acoustic codecs by first generating the initial acoustic codecs autoregressively, then using a non-autoregressive model to predict the remaining codecs.
Non-Autoregreesive Models in Speech Synthesis. Non-autoregressive models, utilizing adversar- ial learning (Kong et al., 2020 Kim et al.] 2021: Lim et al.) 2022), diffusion (Jeong et al. 2021 |022]Liu et al. 2022 Li et al. [2024), and flow matching (Le et al.,
Popov et al., 2021] Huang et al. 2023 Mehta et al. 2024b), have become strong competitors to autoregressive models due to their faster inference speed and more stable generation. The primary challenge for non-AR models is how to align phonemes with acoustic vectors. FastSpeech (Ren et al. 2219 2020) pioneered non- AR modeling by introducing duration predictor and extracting phoneme durations from either an autoregressive model or an external aligner as ground truth. The duration predictor is trained with regression loss. However, using regression loss instead of probabilistic modeling for duration has limitations. (Mehta et al.] 2024a) found that probabilistic modeling produces better results than a regression-based approach, especially when modeling spontaneous speech. VITS (Kim et al. 2021) is a non-autoregressive model that supports probabilistic duration modeling using stochastic duration modeling. However, monotonic alignment search requires nested loops over the acoustic vectors and text sequences. which cannot be vectorized, severely impacting large-scale training. More recently, VoiceBox (Le et al., 2023) proposed separating the acoustic model and duration modeling, using conditional flow matching (CFM) as the objective, which leads to improved speech generation and better duration modeling.
VAES with Learned Prior. While the prior distribution in VAEs is typically fixed standard Gaussian, numerous studies have demonstrated that learning a prior can enhance the latent space structure, thereby facilitating better representation learning. For instance, (Dilokthanakul et al.) 2016) proposed using a Gaussian mixture model (GMM) as the prior for VAËs to enable unsuper- vised clustering, resulting in interpretable clusters and state-of-the-art performance in unsupervised tasks. Similarly, (Tomezak & Welling 2018) introduced a method where the VAE learns a mixture of posteriors conditioned on pseudo-data as its prior. They demonstrated that this VAMP prior con- sistently outperformed the standard VAE across six image datasets. Additionally, (Makhzani et al.) 2015) showed that instead of relying on traditional variational inference, it is possible to align the aggregated posterior with an arbitrary prior distribution using adversarial learning techniques.
3 AUTOREGRESSIVE MODELING WITH CONTINUOUS NEURAL SPEECH CODEC
In this section, we introduce the three major components of our model: (1) A Gaussian Mixture Models VAE (GMM-VAL that compresses speech with a modti- latent constraint. (2) A Gaussian Mixture Models language model (GMM-LM) that autoregressively models the compressed
3

---

Published as a conference paper at ICLR 2025
Mel. Loors
Meil Loos
Codebook Loss
Commit Loss
sample
Divergeries
Loss
Leoereed moture parama.
Diseriminstor
Encodar
Cons:
VQT
VQ2
Come Decoder
Disc: Lovs
Errooder
Con.
Decouer
Cany,
Disciiiinaor
Figure 1: Training procedure and architecture difference of typical residual vector quantitation codec model (left) and our proposed GMM-VAL speech codec model (right).
acoustic vectors conditioned on text. (3) An attention mechanism that aligns the encoder and decoder strictly monotonic fashion.
3.1 LEARNING CONTINUOUS SPEECH CODECS WITH GAUSSIAN MIXTURE MODELS VAE
Learning compact representations of nigh-dimensional signals has been a long-standing area of research. (Hinton & Salakhutdinov, 2006; Vincent et al. 2008 Kingma & Welling 2014 Van Den Oord et al.] 2017). Variational autoencoders (VAEs) (Kingma & Welling] 2014) provide gen- eral framework for learning generative models and performing data compression. Let x denote the input the VAE and the latent code. The objective is maximize the Evidence Lower Bound
(ELBO):
Ex(z)x) (oggtl/)lllog/////pl [[og]( (L(
where (x())) [logglopxxxii p(x[z)] the log- likelihood of the reconstructed data, and (L(g(z/))]LV(OO,) is the KL divergence term that encourages the latent distribution to be close to a standard Gaus- sian. The KI. divergence acts as critical regularization mechanism ensure a meaningful latent space. In VQ- VAE (Van Den Oord et al., 2017), the continuous latent variable z is replaced by discrete latent code 2, corresponding to an index learned codebook with K possible entries. Each codebook entry is a D-dimensional vector Instead of sampling from latent distribution during decoding, VQ- VAEV quantizes the input, mapping the continuous encoder output to the near- est discrete code. Denote the output of the VAE encoder h E(x), and the posterior is defined as:
(1.
if = arg min |h voll2
q(= = k
0. otherwise
where E.)) denotes the encoder, typically composed of several convolutional layers. Since quan- tization is a non-differentiable operation that prevents gradient backpropagation, a straight estimator is used pass gradients from the decoder to the encoder (Van Den Oord al. 2017). A vector quantization objective (codebook loss) and a commitment loss are added:
C = log (x kg) + 3 =e sg (1, + ||sg (ze) 1
(3)
where 2, (x) Vk: k == argmin, [x) - voll2
where sg represents the stop-gradien operator. Unlike traditional VAEs, the prior p(2) is assumed to be uniform, so the KI. divergence is constant and has not effect on learning. Nonetheless, regu- larization achieved through vector quantization. Quantization also implicitly defines a categorical distribution during training, allowing for the second stage where autoregressive model is trained to recover the prior/latent distribution (Van Den Oord et al.) 2017). However, it has been observed that the codebook size can be prohibitively large for compact discrete audio codes (Zeghidour al. 2021. and mispronunciations can occur during decoding (Casanova et al. 2024 Betker 2023).
We propose VAE compression model that supports multi-moda. latent space, but without relying on quantization. Our key finding is that as long as the VAE stage defines a valid latent distri- bution that can be recovered in the second stage, quantization is not the only option. Since

---

Published as a conference paper at ICLR 2025
Cross Entropy
Crosa Entropy
Negativel Log- Likelitood
mn
MA
SoftMax Layer
SoftMex Loyer
Conformer Decoder
Transformer
Decoder
NAR Transformee
Monatanic Alighment
Acoustic Token Embedding
Acoustic Tokan
Embedding
Embedeing
Conformer Encoder
MAP00 D8l mD
Conv. Encoder
Text Embedding
Eneader
1DA
abA
Transcript
Figure 2: Training procedure and architecture difference of VALL-E with residual vector quantita- tion codec model (left) and our proposed GMM-LM with GMM-VAE speech codec model (right).
goal is not build a generative model in the VAE stage, use deterministic mapping but skip the quantization step in Eq. 2 The posterior can be written as q(hxx x) 8(h Exx)), where is the Dirac delta function. The appeal of the categorical distribution defined by VQ its ability to model multi-modal distributions, which is essential for speech. To achieve similar effect, we use
learned mixture of Gaussians prior with parameters { A4, ) allowing the continuous latent distribution to be multi-modal The ELBO for this setup can be written as:
C Eg [ogg(x(bh))))) ( Σ MW (mi, Ei)),
(4)
where À controls the strength of the regularization, is the mixture index, and L is the total number of mixtures. Since our posterior is a deterministic function and the prior a mixture of Gaussians, we no longer have analytical solution for the KL divergence. Instead, to compute the KL divergence. Standard deep learning libraries, such as Py Pyrorc,, already support this estimation, and backpropagation through mixture sampling is derived in (Graves) (016). We refer to a VAE trained with the proposed GMM constraint as GMM-VA -AE. When training GMM- VAE for speech signals, we replace the reconstruction term with Mel-spectrogram reconstruction loss and add discriminators, as done in RVQ, Encodec, and DAC (Zeghidour et al. [202]] Défossez (t 2022 Kumar et al. 2024). Figure[sillustrates the differences training procedures and model architecture between the residual vector quantization model and the proposed GMM-VAE.
Monte Carlo estimation
3.2 AUTOREGRESSIVE SPEECH MODELING WITH GMM-LM
Autoregressive models are typically associated with Transformer models that use discrete inputs and outputs /Vaswani 2017 Radford et al., 2019). The conditional probabilities time step can be
expressed as:
P ( 2- .1, Y) Categorical
((
(5)
where f() is neural network that takes in the previous predictions 2t-1, 2, ....... 2y and the con- ditioning information V, to predict the next token. The network typically trained with standard cross-entropy loss. However, the conditional probability of autoregressive model does not neces- sarily need to follow a categorical distribution. as long as it is conditioned on the previous token and produces valid distribution (Salimans al. 2017).
5

---

Published as a conference paper at ICLR 2025
To this end, we define autoregressive model for the continuous random variable h, IR where the conditional probability is represented as a mixture of Gaussians (Tschannen et al. 2024
p (he h h1 Y) = Σ W (be: Vin (T+)²²
(6)
n=1
where w/, u and T represent the n-th mixture's weight, mean, and diagonal variances for frame t. N is the number of Gaussian components. The mixture parameters are produced by neural network f() that takes in the previous imputs and conditioning information:
[w [w],...,w':: Vi...., TI.........
<>] = f(h h1 Y).
To ensure the mixture weights and variances are valid, we apply softmax to normalize the w values and softplus to the T/ values in the network output. In the case of Gaussian mixture autoregressive modeling, we no longer need an embedding layer or a softmax layer. Since there is no stop token in the continuous casc, the model requires additional linear classifier to predict the stop token, in (Shen et al. 2018). The autoregressive model is implemented using a Conformer encoder-decoder architecture. In this paper, refer to the proposed Gaussian Mixture Models based autoregressive model as GMM-LM. Figure 2 illustrates the differences in training procedures and architecture between VALL-E with residual vector quantization codec model (left) and our proposed GMM- LM with GMM- VAE speech codec model (right).
3.3 STOCHASTIC HARD MONOTONIC ALIGNMENT LEARNING
In speech synthesis, monotonic alignments often result lower word error rates and improved naturalness in generated Sreech(Chen et al., 2023). Our monotonic alignment mechanism is based on the work of (Raffel et al., 2017). Given the energy Eiy between encoder state j and decoder state i, we represent the alignment probability using sigmoid function:
Pi. Sigmoid (ex.))
(8)
If use dot product compute the energy. pi,j can be computed in parallel but does not enforce monotonic constraint. To enforce monotonic attention, an iterative process (Raffel et al. 2017 He
al. 2019) is used compute the attention score from Pi.j
ai = 04-1.1-1 -1-
Pt.- + a.-1.PP
(9)
Here, Oi, represents the attention weight used sum up encoder states. However, while ass is monotonic in expectation, it does not guarantee that every alignment step will strictly follow the monotonic constraint for each sample. In (Raffel et al.) 2017), the authors propose adding Gaussian noise to the energy term to encourage binary pt,j during training. However, this approach introduces bias during optimization, and discrepancy between training and test time can lead to performance issues (Chiu & Raffel 2017). We propose replacing the soft Pi, with binary value u forwand sampled from a Bernoulli distriuttion during the forward pass:
forward Bernoulli (pr.g) ii
(10)
This ensures that attention weights always binary. To enable backpropagation. approximate the gradient using a Gumbell-Softaa relaxation of the Bernoulli distribution (Jang et al. 2016):
exp ((log (Pt,)) + 91)/s)
backward W
(11)
exp ((log (px.)) + g/)/s) + exp ((log (1 Pc.) + g2)/s)
where 91. 92 GO, 1), and is temperature parameter that controls the degree of relaxation.
We schedule s to decrease gradually toward the end of training. Algorithm | provides a detailed
explanation of how encoder and decoder features monotonically aligned and fed into the decoder to compute the negative log- log-likelihood.
4 EXPERIMENT SETUP
Training Procedure All models were trained on the LibriLight corpus (Kahn et al. 2020). which contains 60k hours of unlabeled 16kHz audiobook speech. We transcribed the audio using Whisper

---

Published as a conference paper at ICLR 2025
V2 (Radford et al. 2022). Since the number of utterances per speaker vary significantly, used a class-balanced sampler to ensure each speaker was sampled the same number of times during training. For GMM-VAE.
used the same model architecture (Kumar et al. 2024). with modifications that removed all quantization-relate layers. Although possible learn all the mixture parameters for GMM- MAM,, in this paper, only learned the means of each mixture. We assumed the mixture weights were equal, and the covariance matrices were to identity. For GMM-LM, we used the Conformer encoder and decoder from SpeechBrain [Ravanelli et al., (021) with monotonic attention modification proposed in Section 3.3 We trained models with 51.5M (Mini), 101M, 170M, and 315M (Large) parameters. The details of the GMM- LM configurations can be found in the Table A.2 in Appendix. For GMM-VAE training, GPUs, with total batch size of 680, and each segment length was 8960. We used scheduler-free AdamW W (Defazio et al. 2024 with learning rate of 0.001, training for total of 1, 1,000k steps. For GMM-LM training, we used GPUs with total batch size of 240. We used the same scheduler- AdamW with learning rate of 0.01 and gradient norm clip of 0.005, training for a total of 200k steps. We did not observe any impact of random seeds on the runs.
described in the DAC paper
used 8
Evaluation Protocol We used the LibriSpeech test set for evaluation (Panayotov al. 2013). For zero-shot TTS, we used standard objective measures such as word error rate (WER) and speaker sim- ilarity (SIM) to assess the content and speaker fidelity of the generated speech. We used wav2vec2- large (Baevski et al., (020). fine-tuned on LibriSpeech 960h, to transcribe the generated speech and then computed the WER against the ground truth. For SIM, we used the model released by We- Speaker (Wang et al. 2023). trained on the VoxCeleb datasets (Nagrani et al. 2017 Chung et al., 2018). The objective metries were computed over the entire test set. For subjective metrics, we con- ducted mean opinion score (MOS) evaluations on the generated audio to assess the naturalness and speaker cloning fidelity. For each model, 40 samples were generated and evaluated by 20 listeners. The listeners were asked to evaluate the speech quality and naturalness (Q-MOS) and the speaker similarity between prompt speech and the generated speech (S-MOS). In addition to zero-shot also interested in the quality of GMM- VAE and GMM-LM reconstructions. We used standard reconstruction metrics such as Mel Distance, ViSQOL, and SI-SNR, as in (Kumar et al.) 2024). For GMM-LM, used teacher forcing to generate the reconstructed codec, which then used by the CMMMVAE. decoder to reconstruct the waveforms.
TTS,
Baseline Models. We evaluated model against state-of-the-eee TTS models such VALL-E (Wagg et al.), HierSpeech++ (Lee et al.) 2023) (97M Parameters), and StyleTTS2 (Li et al. 2024)
(142M Parameters). For HierSpeech++ and StyleTTS2, we used the official implementations to
train the models.
Prompt Duration
Model
WER( (
SIM
S-MOSt
Q-MOS 1
GroundTruth
2.26
4.24 2 0o6
4.24
StyleTTS-2
3.32
0.70
3.44 0.10
3.82
HierSpeech++
3.45
0.71
3.52 0.13
3.71 0.07
Seconds
VALL-E
6.04
0.73
3,65 :0111
3.54 0.08
Ours-Mini
2.83
0.80
3.84 2
4.11 0.08
Ours-Large
2.81
0.85
4.04
+ 0.07
4.06 0.11
StyleTTS-2
3.34
0.77
3.57 = 000
3.92 0.10
HierSpeech++
3.22
0.75
3.66
3.80
8 Seconds
VALL-E
7.54
0.73
3.71 + 0.11
3.58 0.08
Ours-Mini
3,02
0.81
3,82
3.87
Ours-Large
2.75
0.88
3.92 0.13
4.02 0.10
StyleTTS-2
3.02
0.76
3.52 0.13
4.02 0.08
HierSpeech++
3.12
0.77
3.73 1 0.111
3.84 0.07
15 Seconds
VALL-E
9.68
0.63
3.62
0.12
3.32
Ours-Mini
2.92
0.80
3.78 0.08
4.16 0.09
Ours-Large
2.72
0.91
4.17
3.92
Table 2: Comparison of models' zero-shot TTS across different prompt durations.
performance on WER, SIM, S-MOS, and P-MOS
7

---

Published as a conference paper at ICLR 2025
5 EXPERIMENT RESULTS
5.1 ZERO-SHOT TEXT-TO-SPEECH
We evaluated our model's zero-shot text-to-speech ability using prompts against both state-of-the- art autoregressive model VALL-E, and non-autoregressive models, StyleTTS-2 and HierSpeech++. To study the effect of prompt length on speech synthesis, we divided the prompts into three groups: 3 seconds, 8 seconds, and 15 seconds. If an utterance exceeded the specified duration, we trun- cated it; otherwise, we sampled another utterance from the same speaker and concatenated until reaching the desired duration. We used two versions of our model for evaluation: mini version with 51 .1.2M parameters and a large version with 315M parameters, both of which output 6 Gaussian mixture parameters with diagonal covariances. Both models were trained encoder features from a GMM- VAE model trained with 3 mixture components, constrained by À = 50. The results are shown in Table 2 The first observation from Table 2 is that non-AR models such as StyleTTS2 and HierSpeech++ achieved much better WER than VALL-E. We found that VALL-E was prone to skipping words or failing the middle of sentences, suggesting that simple cross-attention mech- anisms may not be sufficient for autoregressive models. Contrary to our expectations, VALL-E's WER worsened as the prompt length increased. Another observation is that non-AR models do not significantly benefit from longer prompts, as there was no performance improvement from 8-second 15-second prompts in SIM and S-MOS for HierSpeech++ and StyleTTS2. By contrast, our pro- posed GMM-LM model consistently outperformed both VALL-- and the state-of-the-an non-AR models in content fidelity and speaker cloning, demonstrating clear advantage in speaker-related metrics with longer prompts. We also found that scaling our model from mini to large primarily benefited speaker-related metrics (SIM and S-MOS), while WER and general speech quality did not improve much.
Zero-Shot TTS Content Performance (WER 4)
Zero-Shot TTS Speaker Performance (SIM 1)
Mo Wi/WER
-8: Ref SM
CHR--AA WIR
GVN/WA SM
13
0.75
0.70
50
120
130
203
220
500
30
100
150
200
230
300
Modic:
sue
Figure 3: Comparison on GMM-LM models' zero-shot TTS performance trained on Mel- Spectrogram features and GMM-VAE features.
5.2 MEL-SPECTROGRAM VS. GMM- VAE FEATURES
In this section, we evaluate GMM-LM models trained on Mel-spectrograms against GMM-LM mod- els trained on GMM-VA encoder features. We trained models with 51.5M, 101M, 170M, and 315M parameters, each outputting 6 Gaussian mixture parameters with diagonal covariances. For models using GMM-VAE features, the GMM-VAE model was trained with 3 mixture components, con- strained by À = 50. For the Mel-spectrogram model, we used a 120-dimensional Mel-spectrogram with a hop length of 240. We evaluated the models' zero-shot TTS performance using objective metrics on the LibriSpeech test set. The results are presented in Figure 3 From Figure 3 it can be seen that GMM-LM models trained on GMM-VAR VAE features significantly outperform those trained on Mel-spectrogram features. The discrepancy is especially pronounced with smaller models. Al- though increasing model size reduces the performance gap for Mel-based models, this comes at the cost of increased computation and model size. Thus, we conclude that GMM-LM with GMM-VAE features is a more efficient approach that delivers better performance for speech synthesis. Ad- ditionally, found that GMM-LM models using Mel-spectrograms take much longer to achieve proper alignment compared to GMM-LM models using GMM- VAE features. We hypothesize that the GMM-VAE step helps extract semantic information, which greatly aids the alignment steps.
8

---

Published as a conference paper at ICLR 2025
Method
Model
Mel Distance + ViSQOL
SI-SDR
Encodec
1.47
3.41
5.66
macoding-Decodiigg
GMM-VAE
1.13
3.96
8.89
VALL-E
2.26
2.25
-0.15
Teacher Forcing
GMM-LM
1.82
2.86
4.45
Table 3: Comparison of reconstruction quality for codec models (Encodec and GMM-VAE) and downstream models with teacher forcing (VALL-E and GMM-LM).
No. Gaussian
Mode
X
0.1
1
10
50
100
1
GMM- VAE Recon
0.76
0.89
1.14
1.67
2.06
GMM-LM Teacher Forcing
4.62
3.82
3.16
4.48
3
GMM-VAL MAME-AE Recon. 0.71
0.77
0.98
1.13
1.77
GMM-LM Teacher Forcing
6.72
3.04
1.87
1.82
4.27
GMM- VAE Recon.
0.73
0.86
0.95
1.37
1.83
6
GMM-LM Teacher Forcing
4.63
4.07
2.54
2.44
5.26
Table 4: Reconstruction Quality (Mel Distance J) of GMM-VAE and GMM-LM (with teacher forc- ing) across different number of Gaussians and values.
5.3 RECONSTRUCTION QUALITY OF DISCRETE AND CONTINUOUS CODECS
In this section, we evaluate the reconstruction quality of GMM-VAE model in comparison with the RVQ codec model, Encodec. The GMM- VAE model was trained with 3 mixture components, constrained by 50. Our primary focus on how well these models can reconstruct the input, as this greatly impacts tasks like zero-shot TTS. Even if codec models can perfectly reconstruct the input, the downstream models are unable to predict the correct codec, the effort futile. Therefore, we also evaluate the reconstruction quality with downstream models such VALL-E and our GMM-LM. Specifically, we encode the input speech to generate speech codecs, then use VALL-E or GMM-LM in teacher forcing mode predict the codec. Finally, we reconstruct the speech by decoding the reacher-forced predicted codec. We use common speech reconstruction quality metrics, including Mel Distance, ViSQOL (Hines al. 2015). and SI-SDR. The results are presented in Table 3 It is clear from the table that our LMM--AB significantly outperforms Encodec in speech reconstruction. The performance gap is even larger when the downstream models are included.
5.4 ABLATION STUDY
Divergence constraint. Since we no longer perform quantization when training the speech codec, the divergence constraint in Eq. |ecomes the only factor that directly influences the latent distribu- tion, aside from the reconstruction loss. Because our goal is to use the codec for downstream models rather than compress speech per se, evaluate both the reconstruction quality of GMM-VAE and the downstream GMM-LM reconstruction with teacher forcing, using Mel-spectrogram distance. The GMM-LM model used the large version with 6 Gaussian mixtures and diagonal covariances. Specifically, we investigate how the value of À and the number of Gaussian means in the divergence constraint affect reconstruction. The results are presented in Table 4
note that without any divergence constraint, we were unable to successfully train the downstream GMM-LM models. As expected, the VAE reconstruction Mel distance increases with larger A. However, the effect of À on downstream model reconstruction is less straightforward. With À 0.1, although we observed very low Mel distance, we were unable to train downstream models (the model could not produce intelligible speech conditioned
important
text). When increasing À from I to 50, although the GMM- VAE reconstruction quality decreases. the downstream model's teacher forcing recon- struction quality improves. In terms of the number of Gaussians in the constraint term, found

---

Published as a conference paper at ICLR 2025
that using three Gaussians yields better results than using single Gaussian, with both RMM-VAE and GMM-LM showing smaller Mel distances. However, increasing the number of Gaussians to led to general decrease in reconstruction quality. Since our focus is improving the GMM-LM generation quality, we selected three Gaussians with = 50 as the optimal hyper-parameters.
GMM-LM Parameterization. The GMM parameterization in GMM-LM directly affects the output distribution. We aim to investigate how the number of Gaussian mixtures and the choice between full diagonal covariance matrices impact model performance. We assess the generated speech using objective metrics such as WER and SIM. Furthermore, since GMM parameterization also influences the sampling process, we measure the diversity of the generated samples under differ- ent parameterization schemes. Specifically, given the target text and speech prompt, we randomly sample from the parameterized GMM to generate speech three times. We ask human evaluators to rate the diversity of the three samples on scale from 0 to 5. These results are compared with VALL-E, StyleTTS2, and HierSpeech++, as presented in Table For comparison, we also included Tacotron's ' approach, shown in the first row of Table 5 which trains autoregressive model using only the mean and a regression loss. This approach does not allow for sampling. We found that us- ing single Gaussian with an L1 loss led to weaker performance compared to models with multiple mixtures. As we increased the mixture count to 3, and then to 6, we observed improved diversity in both diagonal and full covariance models. However, the WER improvement was mixed: gener- ally the full covariance model worsened with more mixtures, while the diagonal covariance model maintained approximately the same WER. Finally, we did not observe better results with 10 mix- tures. In conclusion, there is no clear advantage in using full covariance matrices or increasing the number of mixtures beyond When comparing the speech diversity of GMM-LM with VALL-E, StyleTTS2, and HierSpeech++, our model consistently produced significantly more diverse sam- ples. For StyleTTS2 and HierSpeech++, the multiple generated samples showed little variation. For VALL-E, while some degree of diversity were observed, the model frequently failed during random sampling. When excluding failed samples, the diversity remained limited. contrast, our model consistently generated samples with diverse speaking styles.
No. Mixture
Variance Type
Loss
WER 4
SIM f
Diversity ?
None
L1 Mel-Dist
4.29
0.75
Diagonal
NLL
3.34
0,81
2.17
Full
NLL
3.45
0.82
2.32
Diagonal
NLL
2.89
0.85
3.14
3
Full
NLL
3.74
0.72
2.85
Diagonal
NLL
2.72
0.91
3,42
Full
NLL
4.82
0.69
3.01
10 Diagonal
NLL
5,21
0.71
3.08
VALL-E
9.68
0.63
1.47
StyleTTS2
3.02
0.76
0.6
HierSpeech++
3.12
0.77
0.3
Table Zero-shot TTS performance GMM-LM with different mixture counts and variance types, evaluated on WER, SIM, and sample diversity.
6 CONCLUSION
We presented new autoregressive modeling approach for speech synthesis, utilizing Gaus- sian Mixture VAE (GMM-VAE) for compression and Gaussian Mixture Model language model (GMM-LM) for autoregressive generation. By eliminating the need for quantization and introduc- ing stochastic monotonic alignment, our method ensures stable, efficient, and high-quality speech synthesis. Our experiments showed that this approach outperforms traditional VQ-based methods, achieving lower WER and higher MOS scores with reduced model complexity. These results demon- strate the potential of continuous latent spaces for autoregressive modeling in speech synthesis, of- fering simpler and more effective alternative to VQ models.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENT
This work was supported by the RGC Hong Kong SAR, Grant No. PolyU 15228223. We would like thank Prof. Man-Wai Mak for the leadership in this project.
REPRODUCIBILITY STATEMENT
We have provided detailed descriptions of the hyperparameters and training procedures in Section Our implementations are based on publicly available codebases (descript-audio-ccoded and Speech- Brain) and datasets. With the detailed explanations of the training process and hyperparameters, readers should be able to easily reproduce our models. Additionally, we plan to release the code and pre-trained models soon.
REFERENCES
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, wav2vec 2.0: frame- work for self-supervised learning of speech representations. In Proc. Advances in Neural Infor- mation Processing Systems, 2020.
James Betker. Better speech synthesis through scaling. arXiv preprint arXiv:2305.07243 2023.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: language modeling approach to audio generation. JEEE/ACM transactions on audio, speech, and language processing. 31:2523-2533: 2023a.
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint arXiv:2305.096:3 2023b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Edresson Casanova, Kelly Davis, Eren Gölge, Görkem Göknar, Iulian Gulea, Logan Hart, Aya Al- jafari, Joshua Meyer, Reuben Morais, Samuel Olayemi, et al. Xtts: a massively multilingual zero-shot text-to-speech model. arXiv preprint arXiv:2406.04490 2024.
Li-Wei Chen, Shinji Watanabe, and Alexander Rudnicky. A vector quantized approach for text to speech synthesis on real-world spontaneous speech. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 12644 12652, 2023.
Meiying Chen and Zhiyao Duan. Controlve: Zero-shot voice conversion with time-varying controls on pitch and rhythm. arXiv preprint arXiv:2209. 11866, 2022.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. WavLM: Large-scale self-supervised pre- training for full stack speech processing. JEEE Sel. Top. Signal Process., 105-)11555 2022.
Chung-Cheng Chiu and Colin Raffel. arXiv:1712.05382, 2017.
Monotonic chunkwise attention.
arXiv preprint
Kwanghee Choi, Ankita Pasad, Tomohiko Nakamura, Satoru Fukayama, Karen Livescu, and Shinji Watanabe. Self-supervised speech representations are more phonetic than semantic. arXiv preprint 1-Xii:2406.086/11 19, 2024.
Jan Chorowski, Ron Weiss, Samy Bengio, and Aäron Van Den Oord. Unsupervised speech rep- resentation learning using wavenet autoencoders. JEEE/ACM transactions on audio, speech, and language processing, 27(12):04412205 2019.

---

Published as a conference paper at ICLR 2025
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. arXiv preprint 1806. 2018.
Jade Copet, Felix Kreuk, Itai Gat. Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défossez. Simple and controllable music generation. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Aaron Defazio, Harsh Mehta, Konstantin Mishchenko, Ahmed Khaled, Ashok Cutkosky, et al. The road less scheduled. arXiv preprint 1.xXxx:2405.566335 2024.
Alexandre Défossez, Jade Copet. Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arviv preprint arXiv:2210.13333, 2022.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1XX0v:881.044550 2018
Nat Dilokthanakul, Pedro AM Mediano, Marta Gamelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture varia- tional autoencoders. arXiv preprint arXiv:161 1.02648, 2016.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 12873-12883, 2021.
Alex Graves. Stochastic backpropagation through mixture density distributions. arXiv:1607.05690, 2016.
arXiv preprint
Mutian He, Yan Deng, and Lei He. Robust sequence-to-sequenee acoustic modeling with stepwise monotonic attention for neural tts. arXiv preprint arXiv: 1906. 00672, 2019.
Andrew Hines, Jan Skoglund, Anil C Kokaram, and Naomi Harte. Visqol: objective speech quality model. EURASIP Journal on Audio, Speech, and Music Processing, 2015:1-18, 2015.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-507) 2006.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE ACM Trans. Audio Speech Lang. Process., 29:3451-3460, 2021.
Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. Prodiff: Progressive fast diffusion model for high-quality text-to-speech. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 2595-2605, 2022.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztion with gumbel-softmax, arXiv preprint arXiv:1611.00 0.00 144, 2016.
Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-tts: A denoising diffusion model for text-to-speech. arXiv preprint arXiv:2104001409 2021.
Jacob Kahn, Morgane Riviere, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Fierre-Emmanus Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Libri- light: benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673. IEEE, 2020.
Jaehyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference Machine Learning, pp- 5530-5540. PMLR, 2021.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
12

---

Published as a conference paper at ICLR 2025
Jungil Kong, Jachyeon Kim, and Jackyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022-17053. 2020.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar. and Kundan Kumar. High- fidelity audio compression with improved rvqgan. Advances in Neural Information Processing Systems, 36, 2024.
Adrian Lańcucki, Jan Chorowski, Guillaume Sanchez, Ricard Marxer, Nanxin Chen, Hans JGA Dolfing, Sameer Khurana, Tanel Alumäe, and Antoine Laurent. Robust training of vector quan- tized bottleneck models. 2020 International Joint Conference on Neural Networks (IJCNN),
pp. 1-7. IEEE, 2020.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, al. Voicebox: Text-guided multilingual universal speech generation at scale. arXiv preprint 1.X.x:2336..6883 2023.
Sang-Hoon Lee, Ha-Yeong Choi, Seung-Bin Bin Kim, and Seong-Whan Lee. Hierspeech++:: Bridg- ing the gap between semantic and acoustic representation of speech by hierarchical variational inference for zero-shot speech synthesis. arXiv preprint arXiv:2311.12454 2023.
Yinghao Aaron Li, Cong Han, and Nima Mesgarani. Styletts: A style-based generative model for natural and diverse text-to-speech synthesis. arXiv preprint arXiv:2205. 000222055.5433 15439, 2022.
Yinghao Aaron Li, Cong Han, Vinay Raghavan, Gavin Mischler, and Nima Mesgarani. Styletts 2:
Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. Advances Neural Information Processing Systems, 36, 2024.
Dan Lim, Sunghee Jung, and Eesung Kim. Jets: Jointly training fastspeech2 and hifi-gan for end to end text to speech. arXiv preprint arXiv:2203. 16852, 2022.
Songxiang Liu, Dan Su, and Dong Yu. Diffgan-tts: High-fidelity and efficient text-to-speech with denoising diffusion gans, arXiv preprint arXin:2201.11972, 2022.
Dan Lyth and Simon King. Natural language guidance of high-fidelity text-to-speech with synthetic annotations. arXiv preprint arXiv:2402.01992, 2024.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv:1571.05644 2015.
Shivam Mehta, Harm Lameris, Rajiv Punmiya, Jonas Beskow, Éva Székely, and Gustav Eje Hen- ter. Should you use a probabilistic duration model in us? probably! especially for spontaneous speech. arXiv preprint w:Xiv:240605540. 2024a.
Shivam Mehta, Ruibo Tu, Jonas Beskow, Éva Székely, and Gustav Eje Henter. Matcha-tts: A fast Its architecture with conditional flow matching. In ICASSP 2024-2024 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pp- 11341-11345. IEEE, 2024b.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: Vq-vae made simple. arXiv preprint arXiv:2309. 2023.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: large-scale speaker identifi- cation dataset. arXiv preprint arXiv: 1706 170.008612...... 2017.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In Proc. International Conference on Acousties, Speech and Signal Processing, ICASSP, pp. 5206-5210, 2015.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- deirahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2104.00035,, 2021.
13

---

Published as a conference paper at ICLR 2025
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad- its: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning. pp.
PMLR, 2021.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnse Autove: Zero-shot voice style transfer with only autoencoder loss. In International Conference on Machine Learning, pp. 5210-5219. PMLR, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. CoRR, abs/2212.04355 2022.
Colin Raffel, Kinh-Thang Luong, Peter Liu, Ron J Weiss, and Douglas Eck. Online and linear-time attention by enforcing monotonic alignments. In International conference on machine learning, pp. 2837-2846. PMLR, 2017.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021.
Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Joao Monteiro, Jan Tr- mal, and Yoshua Bengio. Multi-task self-supervised learning for robust speech recognition. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6989-6993. IEEE, 2020.
Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju- Ju-Chieh Chou, Sung- Lin Yeh, Szu- Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio. SpeechBrain: general-purpose speech toolkit, 2021. arXiv:2106.04622
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. Advances in neural information processing systems, 32, 2019.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. Fastspeech 2: Fast and high-quality end-to-end text speech. arXiv preprint arXiv:2006.045: 2020.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517. 2017.
Jonathan Shen, Ruoming Pang. Ron Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, al. Natural TTS synthesis by con- ditioning WaveNet on mel spectrogram predictions. In International Conference on Acoustics, Speech and Signal Processing, pp. 4779-4783, 2018.
Jakub Tomezak and Max Welling. Vae with a vampprior. In International conference on artificial intelligence and statistics, pp. 1214-1223, PMLR, 2018.
Michael Tschannen, Cian Eastwood, and Fabian Mentzer. Givt: Generative infinit-vocaabulary transformers. In European Conference on Computer Vision, pp. 292-309, Springer, 2024.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. (096-1103, 2008
14

---

Published as a conference paper at ICLR 2025
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers, 2023. URL: htps://arxiv. 0rg/abs/2331.02211.. doi: doi, 10.
Hongji Wang, Chengdong Liang, Shuai Wang. Zhengyang Chen, Binbin Zhang, Xu Xiang, Yanlei Deng, and Yanmin Qian. Wespeaker: research and production oriented speaker embedding learning toolkit. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEER, 2023.
Yuxuan Wang, RJ Enerry-Ryam Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotrom: Towards end-to-end speech synthesis. arXiv preprint arXiv: 703. 10135, 2017.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing, 30:495-507, 2021.
A
APPENDIX
Algorithm GMM-LM Monotonic Alignment and Decoder Training Procedure
Require: Acoustic features (h;)/= extracted from the GMM-VAF encoder. Phoneme features
Y 1 from the GMM-LM text encoder, and the GMM-LM Decoder D.
1: Compute energy terms in parallel for all i,j using the dot product:
eij = h
2: Sample initial alignments for all i, from a Bernoulli distribution:
u [
forward
Bernoulli (errouuli(Sgmoii(ee... (iigmoid(e:.))
Save the Gumbel-Softman relaxation w.
buckward (Eq. 11 for gradient computation.
4: Initialize au = [1, 0, .......) 0] E RR,, LNLL = and C = (an empty sequence for accumu-
lating contexts).
5: for each time step = 1 to / do
6: Let u, [44,1; 2, [.
7: Refine alignments to enforce monotonicity:
a: = Q;-1 . u, + shift (a;-1 (1 - (1-u)),1)
> shift(, 1) shifts the vector by one position to the right, filling the leftmost entry with zero.
8:
Compute the attention context using the refined alignments:
c Y az h;
9:
Update the accumulated attention contexts:
C = concat(C.-1..c)
10:
Pass the accumulated context C, to the decoder network D to obtain mixtures parameters:
D(C,) [+] .....
w 4+1 N
1 [+] 1
wit are normalized using the softmax and 7 are passed through the softplus to ensure they are positive.
11:
Compute the negative log- ligeliiil for the next frame:
LNL. += - -log (ba41 {wot1 1+41 ++1 Jn=1)
12: end for
15

---

Published as a conference paper at ICLR 2025
A.1 ABLATION STUDY ON ALIGNMENT MODULE
Even with powerful neural codecs, we found that monotonic alignment forcing remains crucial for the stability of autoregressive models. We compared our monotonic alignment approach with using direct cross-attention, location-sensitive attention (Shen et al. 2018), and various monotonic atten- tion variants. It is worth noting that location-sensitte attention is not strictly monotonic, meaning can still result in word repetition and lookahead errors. We also found that it was slow to train be- cause energy terms are computed iteratively. Since the alignment module primarily affects phonetic content, we used WER to measure the performance of different alignment methods. For monotonic attention, we explored the use of pre-sigmoid noise, Gumbel Softmax, and the proposed Straight- Through (ST) Gumbel in Eq. 10 and Eq. II The results are presented in Table 6 As seen from the table, using cross-ates attention alone led to poor results-sone some samples even failed to produce intelli- gible speech. Location-sensitio attention performed better, but it still suffered from frequent word repetition errors, and both training and inference were slow. Our proposed monotonic alignment with ST Gumbel outperformed all other methods, including the original monotonic attention with pre-sigmoid noise.
Alignment
Cross Att.
Loc. Att.
Monotonic Att.
W. Noise
w. Gumbel
w. ST Gumbel
WER( %)
6.6
6.02
3.77
3.34
2.72
Table 6: Word Error Rate (WER) comparison of various alignment methods
A.2 DETAILS OF MODEL ARCHITECTURE
For GMM-LM, we used the Conformer encoder and decoder from SpeechBrain (Ravanelli et al., 2021) with monotonic attention modification proposed in Section 3.3 We trained models with 51,5M (Mini), 101M, 170M, and 315M (Large) parameters. The details of the GMM-LM configu-
rations can be found in the Table A.2
Name
Mini
101M
170M
Large
Encoder
No. Layers
3
3
Encoder Dim.
512
512
512
512
Feedforward Dim.
1024
1024
1024
1024
Decoder
No. Layers
6
6
6
12
Decoder Dim.
512
768
1024
1024
Feedforward Dim.
2048
3144
4096
4096
Total Params. (M)
51.5
101
170
315
Table 7: Model Architecture and Parameters
A.3 RECONSTRUCTION QUALITY ON TRAINING SET
Table 4 shows that the GMM-VAE with 3 mixtures and X = 50 achieves the best reconstruction quality on the evaluation set. To further investigate the underperformance of the 6-mixture model, we selected a subset of the training data (randomly sampled 200 speakers with 5 utterances each) and computed the reconstruction loss. The results, along with the evaluation set reconstruction loss, are presented in T From Table we observed that the reconstruction loss on the training set is consistently lower than that on the evaluation set across all configurations. This discrepancy may be attributed to slight domain difference between the LibriSpeech evaluation set and the LibriSpeech- Light training set, with the latter containing noisier recordings. Another key observation is that the 6 6-mixture model achieves lower reconstruction loss the training set compared to the 3-mixture model, yet it performs slightly worse in terms of reconstruction quality on the evaluation set. This suggests that the 6-mixture model may suffer from overfitting.
16

---

Published as a conference paper at ICLR 2025
A.4 ZERO-SHOT VOICE CLONE NOISE ROBUSTNESS
In this section, we evaluate the robustness of TTS zero-shot voice cloning against noise. Specifically, we added varying levels of noise the LibriSpeech clean test set and used the noisy audio as prompt input for the TTS models. The generated speech was assessed using WER and SIM metrics under different noise levels, with the results presented Table g From the table, we observe that, with the exception of VALL-E, most models demonstrate robustness to noisy prompts in terms of WER. However, speaker similarity (SIM) significantly decreases for models like StyleTTS-2 and HierSpeech+++. In contrast, the proposed method shows strong robustness to noise for both WER and SIM, particularly its large version, where WER and SIM degrade only slightly across different
noise levels.
WER (%)
SIM
Model
-20 dB
-15 dB
-10 dB
-20 dB
-15 dB
-10 dB
StyleTTS-2
3.45
3.49
3.54
0.72
0.71
0.68
HierSpeech+++
3.61
3.72
3.91
0.69
0.64
0.58
VALL-E 9.84
10.86
12.61
0.61
0.54
0.51
Ours-Mini 2.85
3.01
3.04
0.77
0.74
0.74
Ours-Large 2.77
2.82
2.97
0.88
0.87
0.85
Table 8: WER and SIM comparison across models under different prompt noise levels. Lower WER and higher SIM indicate better performance.
A.5 GMM- VAE MODEL DETAILS
The GMM-VAE architecture is inspired by the DAC model Kumar et al. (024) and comprises convolutional encoder and decoder. Both the encoder and decoder feature a convolutional layer, which performs upsampling downsampling based the stride, followed by residual block. Each residual block combines convolutional layers with non-linear Snake activations. The encoder progressively downsamples the input audio waveform with strides of [2, 8, 8]. while the decoder mirrors this by upsampling at corresponding rates of [8, 8, 4, 2]. The dimensionality of the encoder and decoder is configured to 64 and 1536, respectively. The GMM-VAE contains a total of 76.5M parameters, distributed as 22.4M in the encoder and 54. 1M in the decoder.
A.6 COMPARISON OF GMM-LM AND VQ-BASED LM WITH MONOTONIC ALIGNMENT
As shown in Table 6 the proposed monotonic alignment significantly improves the WER of the GMM-LM model. This raises natural question: can discrete AR models benefit from the pro- posed monotonic alignment in a similar way? To investigate this, conducted comparison using discrete autoregressive (AR) models with the proposed monotonic alignment method, implemented using the same architecture as the GMM-LM large version (3 (315m), except for the discrete codec em- bedding layer and softmax layers. Specifically, we implemented two discrete AR encoder-decoder models:
Discrete AR. The first model used discrete codes extracted from a VQ-VAE codec model, implemented with a single codebook containing 8192 entries and using the same architec- ture the DAC model (Kumar al. 2024). This model was trained to predict the next tokens using standard cross-entropy with a single softmax layer.
Discrete AR with Delay Pred. The second model used discrete codes extracted from the DAC model with 8 codebooks, each containing 1024 entries. The model adopted delayed codebook prediction with multiple softmax layers, as proposed in MusicGen (Copet et al. 2024 and (Lyth & King 2024).
The results of these experiments presented in Table 10 of the revised manuscript and summa- rized below. The results clearly show that monotonic alignment contributes to WER improvement. However, with monotonic alignment, the discrete models do not achieve the same performance
17

---

Published as a conference paper at ICLR 2025
No. Gaussian
Mode
0.1
50
100
Training Set
0.72
0.78
0.98
1.54
1.85
Evaluation Set
0.76
0.89
1.14
1.67
2.06
Training Set
0.68
0.70
0.93
1.11
1.54
3
Evaluation Set
0.71
0.77
0.98
1.13
1.77
6
Training Set
0.67
0.73
0.86
1.02
1.32
Evaluation Set
0.73
0.86
0.95
1.37
1.83
Table Reconstruction Quality (Mel Distance )) of GMM-VAE and GMM-LM (with teacher fore- ing) across different number of Gaussians and values.
Model
Output
Codec Model
Cross Att.
Mono. Align.
Discrete AR
Logits Group
VQ-VAE
8.02
5.35
Discrete AR with Delay Pred.
Multiple Logits Groups
DAC
7.83
5.87
GMM-LM
GMM Parameters
GMM-VAE
6.60
2.72
Table 10: WER (%) of the models using different codec models, with and without the proposed monotonic alignment on LibriSpecch-test-llaa..
the proposed GMM-hased version. We believe this performance gap can be attributed to two factors:
Quantization and discrete representations introduce limitations: Discrete representations can result in mispronunciations and artifacts. This is why some researchers use embed- dings before the AR model's softmax layer as input to waveform decoders as workaround
(Casanova al. 2024
Betker 2023).
2. Challenges of applying monotonic aligmment to RVQ- models. RVQ models increase the complexity of TTS systems, they require multiple softmax heads to predict codes from different codebooks. While it is possible to predict all codebooks in parallel at each timestep, this approach ignores dependencies between codebooks and yields suboptimal re- sults. A better approach, used in VALL-E and the delayed prediction model in Table I0, is to predict "coarse" codes first, followed by "fine" codes. However, this approach com- plicates monotonic alignment because alignment must be performed using only the coarse codes at each timestep. This limitation likely contributes to the higher WER observed with RVQ-based models.
18


Reviews:
[{"review_id": "xF3ZG4DhWp", "reviewer": "Reviewer_8JGM", "summary": "This paper presents a novel approach to autoregressive speech modeling using continuous speech features, contrast to recent trends that rely on discrete units. The method consists of two key components: (1) A feature extraction model based on VAE, which replaces quantized codebooks (as in RVQ) with a learned mixture of Gaussian priors (GMM-VAE), and (2) A text-to-speech model that employs a Gaussian Mixture Model Language Model (GMM-LM) to model these continuous features in autoregressive manner, which also incorporating a new monotonic alignment constraint. Experimental results demonstrate that this continuous speech modeling consistently outperforms previous methods using discrete codec representations like Residual Vector Quantization (RVQ) in TTS tasks.", "questions": "- Does the frame rate of GMM-VAE features align with the mel spectrogram hop length (240ms) described in 5.2?\n- Why does the model require relatively few Gaussian mixtures compared to VQ codes in RVQ, and any insights on what the mixture components capture?\n- Could you clarify how the monotonic alignment mechanism in Figure 2 (right) works? It seems to align the encoded text and speech prompts prior to decoding. Additionally, a more comprehensive description of the GMM-LM would be nice, including how speech and text features are fed into each decoder step, and the formulation of $e_{i,j}$​.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear comparison to previous approaches and introduces novel continuous variants for both VAE training and TTS stages\n- The introduction of GMM-LM is novel, and the formulation is clear and simple. It also enables probabilistic sampling which is a plus for TTS applications\n- Nice results with much less model parameters", "weaknesses": "- Limited discussion of prior Gaussian mixture VAE work, e.g., \"Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders\". (Minor: The notation, either GMM-VAE or VAE-GMM, should be consistent.)\n- The counterintuitive result where increasing Gaussian mixtures in GMM-VAE leads to worse reconstruction, where a 6-mixture GMM should subsume the modeling capacity of a 3-mixture GMM\n- Some modeling details are missing, e.g., GMM-VAE frame rate, which is crucial as it could affect the type of information captured", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ofS3EmKyJ1", "reviewer": "Reviewer_dDJ8", "summary": "Authors propose a novel means of auto-regressive TTS modeling that eschews quantization units in favor of Gaussian mixtures. Model performance consistently outperforms other standard TTS models, demonstrating high quality TTS is able without traditional VQ-VAE setup.", "questions": "Given the reliance of the model architecture on monte-carlo estimation, how sensitive are results to random seeding during expeirmentation?\n\nWhat is the performance on more noisy datasets than LibriSpeech? Is the Gaussian approach suitably robust across evaluation sets?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Authors present provide a thorough discussion of related work and their motivation for their approach. Description of architecture is clear and easy to follow, along with pointers for reproducibility. High performance of model is significant enough for comparison with other approaches.", "weaknesses": "There is a minor question of motivation in the author's approach: they take the stance that the community views vector quantization approaches as a necessity, but there are a fair amount of approaches in the speech modeling community that have used straight reconstruction approaches. While their gaussian mixture approach is still suitably novel, this position seems to ignore other considerations that go into VQ approach. Notably that the use of discrete tokens is relatively easy to implement in parallel with text encoding, all while minimizing storage an I/O limitations from audio/image processing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X9W24xnCu9", "reviewer": "Reviewer_UrKc", "summary": "This paper applies a VAE with GMM before extracting latent representations, and then trains an autoregressive model on the extracted continuous latent. The approach models the autoregressive conditional distribution also by GMM.", "questions": "- Section 3.3 is a little bit unclear. For instance, how was this energy function $e_{ij}$ calculated? Does it have something to do with the cross-attention weights acquired by the transformer?\n- What are the specs for the baseline methods: StyleTTS-2 and HierSpeech++? Are they of similar parameter size?\n- I am wondering the validity of modeling the autoregressive distribution as a GMM. In your case, even if the KL regularization loss of GMM-VAE makes $q(h|x)$ a mixture of Gaussians, does it in any sense implies that the autoregressive conditional distribution $p(h_t|h_{t-1}, \\cdots)$ is also close to a Gaussian mixture?\n\nThe paper is well-written and interesting, but I think the experiment issue mentioned in Weaknesses should be addressed before the paper is ready to be published.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of using GMM-VAE to regularize the latent distribution, serving a similar role as discretization is novel and interesting. The method is also easy to understand and straightforward. The paper can also inspire research on the use of continuous variational approaches in the speech synthesis domain, which has been recently dominated by discrete-based approaches.", "weaknesses": "To me, the main issue of the paper is that the contribution of monotonic alignment and GMM-VAE are not separated. Specifically, the paper claims that \"Despite its smaller size, our model achieves lower WER and higher MOS than VALL-E, thanks to the continuous autoregressive modeling approach.\" in lines 82-83. However, in experiments of Section 5, you are comparing your method with monotonic alignment v.s. existing methods that do not enforce monotonic alignment. Some studies [1] have shown that enforcing monotonic attention patterns can lead to much lower WER and even better naturalness (it is probably also the reason that it is proposed). This makes me question if the lower WER and higher MOS come primarily from the use of monotonic alignment, which is not the main novelty of the paper, rather than from the use of GMM-VAE and GMM-LM. Furthermore, while the authors do provide a comparison of alignment methods in Appendix A.1, the one without monotonic alignment (Cross Att.) does result in higher WER than all the baselines. This further substantiates that the performance increase may not come from GMM-VAE and GMM-LM. I would suggest the author do an ablation study on the monotonic alignment and add it to Table 2 to interleave the contributions of the two components.\n\n[1] L. Chen, A. Rudnicky, S. Watanabe, ”A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech”, in Proceedings of AAAI 2023, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)