CLaM-TTS: Improving Neural Codec Language Model for Zero-Shot Text-to-Speech
Authors: Jaehyeon Kim
Keon Lee
Seungjun Chung
Jaewoong Cho
Venue: ICLR 2024 poster (2024)
Keywords: text-to-speech
speech synthesis
neural audio codec
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: With the emergence of neural audio codecs, which encode multiple streams of discrete tokens from audio, large language models have recently gained attention as a promising approach for zero-shot Text-to-Speech (TTS) synthesis. Despite the ongoing rush towards scaling paradigms, audio tokenization ironically amplifies the scalability challenge, stemming from its long sequence length and the complexity of modelling the multiple sequences. To mitigate these issues, we present CLaM-TTS that employs a probabilistic residual vector quantization to (1) achieve superior compression in the token length, and (2) allow a language model to generate multiple tokens at once, thereby eliminating the need for cascaded modeling to handle the number of token streams. Our experimental results demonstrate that CLaM-TTS is better than or comparable to state-of-the-art neural codec-based TTS models regarding naturalness, intelligibility, speaker similarity, and inference speed. In addition, we examine the impact of the pretraining extent of the language models and their text tokenization strategies on performances.

Published as a conference paper at ICLR 2024
CLAM-TTS: IMPROVING NEURAL CODEC LANGUAGE MODELING FOR ZERO-SHOT TE-T-EO-
Jaehyeon Kim, Keon Lee, Seungjun Chung, Jaewoong Cho
KRAFTON (jay 310, keonlee, s. 1. chung, jucch))krafftom
ABSTRACT
With the emergence of neural audio codecs, which encode multiple streams of discrete tokens from audio, large language models have recently gained attention promising approach for zero-shot Tex--t-Speech (TTS) synthesis. Despite the ongoing rush towards scaling paradigms, audio tokenization ironically am- plifies the scalability challenge, stemming from its long sequence length and the complexity of modelling the multiple sequences. To mitigate these issues, we present CLaM-TTS that employs probabilistic residual vector quantization (1) achieve superior compression in the token length, and (2) allow language model to generate multiple tokens at once, thereby eliminating the need for cas- caded modeling to handle the number of token streams. Our experimental results demonstrate that CLaM-TTS is better than or comparable to state-of satte-of-the--ee the-arrrtttt neural codec-based TTS models regarding naturalness, intelligibility, speaker similarity, and inference speed. In addition, we examine the impact of the pretraining extent of the language models and their text tokenization strategies on performances.
INTRODUCTION
Large language models (LLMs), characterized by considerable number of model parameters and trained on massive text data, have demonstrated remarkable zero-shot learning capabilities (Brown et al. 2020 Chung et al.) 2022 Kaplan et al., 2020). While scaling paradigm affects not only natural language processing domain but also other fields such as image generation (Ramesh) et al., 2021 Saharia et al. 2022). image recognition (Radford et al.) 202T). and speech recogni- tion (Baevski et al. 2020b Radford et al. 2023). significant challenges their efficient training and inference simul lancously arise. In the realm of image processing, discretizing image represen- tation (Razavi et al. 2019: Ramesh et al.) 2021 Esser et al. 2021) has been shown to mitigate these manageable size.
issues by effectively reducing the input length
Language modeling in the speech domain has become feasible with the emergence of neural audio codecs (Zeghidour et al. 2021 Défossez et al. 2023) that enable high-fidelity audio tokenization. For Text-to-Speecch (TTS) synthesis, there have been several attempts to adopt the LLMs for zero- shot TTS, which namely synthesize the diverse speech of any human voice (Zhang al., [023]] Wang et al. 2023 Kharitonov et al., [0033] Rubenstein et al., 2023). These attempts move away from the previous research direction train models on curated high-quality recording datasets and produce human- like voices on benchmark datasets (Li et al.) 2019 Kim et al. 2021] Tan et al., 2024 Casanova et al., 2022). demonstrated that, by training LLMs on tens of thousands hours of diverse audio data, zero-shot adaptation can be accomplished with just few seconds of audio input.
Despite the significant advancements in TTS at scale, it still poses challenges to further scale up the models. Neural audio codecs typically generate multiple sequences of audio tokens. For instance, Encodec (Défossez et al.) 2023) encodes a 5-second speech into 8 sequences of 375 audio tokens. Several work (Kharitonov et al. 2023 Borsos et al., 2023b) employ the semantic tokens from self- supervised speech representation learning (Chung et 2021 as an intermediary between text and audio tokens. Although semantic tokens compress information
concisely than audio tokens, 5-second speech segment still demands 125 semantic tokens, presenting a hurdle even setting aside the further complexities of audio token modeling from them.

---

Published as a conference paper at ICLR 2024
Mel Decoder
Mel Decoder
0010
do
Residual codes
Residual Vector Quantizer
Residual Vector Quantizer
Sampler (rule)
representations
Weight Predictor
Mean Predictor
EOS Classifier
Mel Encoder
GM-based Latent Decoder
Encoder
Mel-spectrogram
1 Text
BoS
Transformer
Decoder
(a) Mel-VAE
(b) Latent Language Model
Figure I: An overview of CLaM-TTS. Training of CLaM-TTS unfolds in two stages: (a) we train Mel- VAE that encodes a mee-sppetroogam to the discrete latent representation from using probabilis- tic RVQ; (b) using the pre-trained residual vector quantizer from the first-stage, latent language model, a Gaussian mixture (GM) based latent transformer decoder is trained; The decoder aims to predict latent variables that, when quantized, match with the ground-truth audio tokens.
In this work, we aim to bring the capability of efficient training and inference of large-language models within the TTS domain. To this end, we propose an improved Codec Language Model-based TTS (CLaM-TTS) system that encodes speech into multiple token sequences similar to existing methods but in more concise way. With CLaM-TTS, all multiple tokens at each timestep in these sequences are generated through a single autoregressive step of a language model, eliminating the need for iterative generative modeling along the number of sequences. The core of method lies in the probabilistic discrete representation learning, ensuring that all discrete latent codes participate in the training process, resulting in high-quality autoencoder for speech. Furthermore, we provide a principled framework enabling a latent language model to efficiently generate a stack of tokens once; the latent language model produces a continuous latent audio representation and converts it to a discrete representation with the proposed probabilistic quantization method. We scale up the training dataset to 100K hours. Our experimental findings indicate that CLaM-TT TTSS either surpasses or is on par with leading zero-shot neural codec-based TTS models in aspects such as naturalness, intelligibility, speaker similarity, and inference speed. Furthermore, we investigate how the depth of pretraining in the language models and their methods of text tokenization influence TTS outcomes. Our generated samples are available on our demo page
2 RELATED WORK
Neural audio codec
The neural discrete representation learning within a variational autoencoder (VAE) framework, called the vector-quantized VAE (VQ-VAE), has been proven effective in encod-
ing raw-waveforms into discrete tokens (Baevski et al. 2020a), employed as a speech codec (Oord)
al. 2017 Gårbacea al. 2019). Similar VQ-VAE, the neural audio codec methods usually
use a framework that jointly trains an encoder, a decoder, and quantizer (Li et al.) Zeghidour et al.
2021
Jiang al. [022 Jayashankar et al. 2022 Défossez et al. 2023 Kumar et al.
[2023] Wu al.
2023
Zeghidour et al. (2021) pioneers using residual vector quantization (RVQ) (Gry 1984 Va-
suki & Vanathi, 2006 Lee et al., (022) in a neural audio codec model. It operates efficienlly on
clean and noisy speech and music, even at low bitrates. EnCodec (Défossez et al.) (023) employs
similar model structure with improved training efficiency and stability to achieve a downsampling
rate of 320 for input waveforms. Kumar et al. (2023) identify the issue of codebook under-utilization
in EnCodec and improve the codebook utilization with the techniques introduced in Yu et al. (2021) resulting in state-of-the-arr performance a neural audio codec.
httpa: h/wl:///llatttte wtw.aaattttttt..t.tttt..
2

---

Published as a conference paper at ICLR 2024
Building these advancements, focus more the discrete representation learning of speech rather than general audio and optimize the compression level to be suitable for the TTS task. In other words, we compress spe-sppectssgs rather than raw waveforms, delegating the task of converting mel-spectrograms back into raw waveforms to standard vocoders.
Large-scale TTS A (Borsos et al. 2023a is a language model directly trained on audio to- kens. In AudioLM, semantic tokens are first generated. These tokens originate from self-supervised discrete speech representation methods (Hsu al. 202T; Chung al.) 202I) that have previously
been utilized for speech resynthesis or generation without text (Lakhotia et al., 2021 Polyak et al.) 2021 Nguyen et al.]) Following this, the model produces acoustic tokens of neural audio codes given the semantic tokens. Wang et al. (2023) propose the first neural codec language model, Vall-E, for text-to-speech that utilizes a pre-trained neural audio codec, EnCodec (Défossez et al. 2023).
In different approach, following Audiol MM, text-to-speech has been realized by applying language modeling to generate the semantic tokens from text, as demonstrated by Kharitonov et al. (2023).
A shared characteristic among these neural codec language models is their two-stage pipeline; they autoregressively generate coarse-grained audio tokens and decode them into fine-grained represen- tations. Recent work in music generation hints an efficient way to eliminate the second-stage modeling by interleaving audio tokens in delayed pattern (Copet et al., 2023), but its application in TTS remains unexplored.
Given the complexities in modeling long audio sequences, several studies have incorporated phonemes and durations to alleviate the need for speech synthesizers to predict speech rates (Shen et al. 2023 Le et al. [202)] Jiang et al. (003)) Some work shows that non non-autoregreesivee generative models, such as a diflusion model and flow-matching (Ho et al.) 2020: Lipman et al., 2023), can pro- duce diverse and natural-sounding speech with large-scale training. A hybrid method is utilized in another approach, employing non-aut autorerresseeve architecture except prosody modeling (Jiang al.)
2023). This method aligns with previous work that applies VQ-VAEs to capture fine-grained speech features so that the prosody is controllable by them (Sun et al.
2020
Ren et al. 2022).
To address the challenges associated with neural codec language models while not relying on the phoneme and its duration that requires significant domain expertise, we design a language model that generates from coarse to ene-grained tokens without needing a two-stage pipeline. Our approach is similar to recent work that utilizes pre-trained language models, Spectron (Nachmani et al., 2023) and SoundStorm (Borsos et al. 2023b). While Spectron employs pre- pre-trained transformer decoders to directly model the mel-spectrogram and then fine-tunes it, our method preserves the pre-trained text encoder and decodes speech tokens that shorter than the mel-spectrogram using latent transformer decoder. SoundStorm freezes a pre-trained text encoder similar to ours, but it generates semantic tokens and subsequently decodes acoustic tokens using an iterative generative model.
PRELIMINARIES
Building upon the approach proposed by Wang et al. (2023), we explore zero-shot TTS through the lens of neural codec language modeling task. We consider a setting that includes two types of data: (i) text data and (ii) mel-spectrogram representation of its corresponding speech data, denoted by æ and y, respectively. We model a sequence of T discrete codes Cir := (cI....,cy } from latent representations 21:7 of mel-spectrogram y. using the framework of a Variational Autoencoder (VAE) with Residual Vector Quantization (RVQ). Here, c; represents the D-depth of quantized, discrete codes. We interchangeably use Ct,1-D with Subsequently, neural language model pa(Cur((z) is employed, aiming to predict C1.T from the text transcript æ. During the inference phase, the language model generates C1.7 for given text æ. which is subsequently transformed into speech through the VAE decoder and a pre-trained vocoder.
3.1 RESIDUAL-QUANTIZZN VARIATIONAL AUTOENCODER (RQ-VAE)
2022 is a neural network architecture representing data as discrete codes An RQ- VAE (ee et al. using residual vector quantization. It comprises of three components: (1) an encoder parameterized by o that maps data y into a sequence of latent representations æ1:7; (2) a residual vector quantizer RQ+ (.), converting the latent vector z, at each time into the discrete code representation G,
3

---

Published as a conference paper at ICLR 2024
(Z,( the corresponding quantized embedding 2,; and (3) decoder parameterized by that reconstructs the data & from a sequence of the quantized latent representations &1:T-
Here Ct,1-D represents the set {cr,1....... Ct,D with D indicating the total depth of the quantizer. The latent representation from the encoder is quantized through the multi-stage nearest- wearest-ncighbburr lookup over the codebook embeddings, of which the vocab size is V. The process is defined as finding the optimal code from the codebook, which minimizes the residual error at each depth d:
Ct = arg min ||redd-1 epp dd)' Ft,d rt,d-1 Cp(Cr,, for all d D1, (1) <'((.....)
where 7r,0 = 2 and (x; d) corresponds to the e-th embedding vector in the codebook at depth d. The sum of embeddings Ed=1 Eyp (Ct,dd d) becomes the quantized latent representation ê, which is converted back to the input space through the decoder. The codebook embeddings are updated with the clustered latents by the exponential moving average updates (Razavi et al. 2019).
3.2 MEAN-FIELD VARIATIONAL INFERENCE
Consider latent variable model characterized by a joint distribution Pyl (2,, Ct.:D)) parameterized by vb. Here, 2₁ denotes an observed random variable, Ct,1:D indicates set of latent random vari- ables (c,,1.-... Cr In this model, variational inference is method approximate the intractable distribution (c.1..D x2) by solving an optimization problem with respect to parameters of approx- imate distribution g(Cr,I.D (z))). We can derive lower bound the marginal log-likelihood pol (Zz), known as the evidence lower bound (ELBO) (Blei et al. 2017))
log (z,) = log Σ Po pu(zx(e,LD))(e,.. 2 Be(e))pll))) log p.((e.e....)pic..... ((Ct,
Ct.1DD
Mean-field variational inference (Koller & Friedman 2009 Blei et al. 2017) is specific approach of variational inference that assumes the independence among the latent variables conditioned on the observed variable: g(Ct,1:D|z)) = I g(c,,a) We can show that each optimal variational posterior distribution q" (Ct,a 2z), which maximizes the ELBO, satisfies:
q' (Ct,alze) x exp (Ey(c.. 2.) Po
Zt, 21,-a)p(C,,d ct,d)))
where Ct denotes the latent variables of all depth Ct except Ct.d. An iterative coordinate ascent algorithm based on Eq. can be used to update the distribution q (Bishop & Nasrabadi] (006). and the complexity of the algorithm mainly lies the computation of the expectation over q(e, 2.).
METHOD
4.1 MEL--VAE
We aim to develop neural codec that can generate discrete speech codes within short sequence length to make speech audios suitable for language model utilization. To achieve this, we em- ploy a RQ-VAE that compresses mel-spectrograms of speech audios (see Fig, [T). We introduce a variational inference based method for learning residual codewords address the codeword col- lapse issue found in conventional vector quantization methods (Kaiser et al.) 2018 Roy et al.) 2018 Zeghidour et al. 2021 Kumar et al., 2023).
We illustrate Mel-VAE similar to RQ- VAE following most of notations from Sec. 3.1 The encoder maps a mee-specto y into sequence of latent representations 21:T, and a residual vector quantizer RQ o converting the latent vector 2; at each time t into the discrete code representation Cr or its corresponding quantized embedding é, = Σd exp (Cr,aa ai d). The decoder reconstructs the mel-spectrogram ÿ from sequence of quantized latent representations êT.
With the assumptions that q(C,,1:/zz) I ((cr,a)2.) and p(Cr.a- ct,4) is uniformly dis- tributed, mean-field variational inference yields the condition of such distribution as the following (see Eq. -
(ct,dlz)) x exp(E k.) Moggy(zelee... Ct, ct,-)))),
(3)
4

---

Published as a conference paper at ICLR 2024
where the latents follows normal distribution: (r(zz))e) = (2(z Ed
However, the mutual interdependence of codes at every depth in the latter equation makes it difficult to solve without an iterative approach. Instead of using an iterative coordinate update approach, we approximate _g(c,,_/(z.) [log Pei (zz (c,,d cr,4a) pointwisely as log Pu (2. Ci.-a) for all d, where IiI.D & (z)). The posterior then has form: q (cr,alz) Pe (z Cr,ds Ci,-))-
We independently optimize the codebook embeddings at each depth d, in variational inference framework:
[(paa Zt, ai (er. #.) [- log ci ci
(4)
[(v; zt. Ci,).)) = Ed= C(va; ze, c;,-a)-
(5)
The other modules of Mel-VAE, including the encoder parameters e and the decoder parameters w. are trained with commitment loss, reconstruction loss, and adversarial losses.
[(w, o; y, Ct, A - >el Ex ey(ce,; + AaCade,
where xy, Aes and Åa corresponds to coefficients of the reconstruction loss, commitment loss, and ad- versarial losses, respectively. For adversarial training, we adopt the multi-length discriminator (Chen et al. (2020) that distinguishes mee-spectro different lengths and a modified
spectrogram discriminator (Lee et al. 2023) that accepts mel-spectrogram rather than linear spectro- gram. We combine the least squares GAN objective (Mao et al. 2017) and the 7.1 feature matching
loss (Kumar et al.) 2023) into a loss function denoted by Ladv-
4.2 LATENT LANGUAGE MODELING
We propose conditional speech code language model given text aimed enhancing the efficiency of the model. This improvement stems from the insight that speech codes are generated through vector quantization. Our approach leverages this by predicting the vector itself, which can then be converted into multiple tokens at each layer via residual vector quantization. This is departure from previous methods that predict speech code tokens sequentially.
Specifically, rather than directly predicting tokens from text, we consider continuous latent repre- sentation z, of speech that can be converted into a speech code using residual vector quantization:
Po(Ci: (x) = II t=1
T
po po (,, c2i)
II t=1
T
/
Po czi)dz
= II pe (ze (x, cct) (Celz)))))) )dz t=1
T
where Cut indicate C1:--1, and we employ p.: (c zz), the probabilistic quantizer distribution learned
together with the Mel-VAE model (see Sec. 4.1 as replacement of po(ca 2,, T, cet). Here, define the conditional distribution Po(z;, (22 xx, Cet) as Gaussian mixture model:
po (z (ac, C&t) Σk (k|k, c.e) (c)))(( no (o x, c,r)), oB T).
In this model, we can derive the following variational lower bound the log-likelihood:
log (clx))
E Eg(H)oang))) ( cx.. + log po (k|a c2)) + B(b, c:)]
LvB (0) + B(v, ct).
for any g(k(z,cç)) The derivation of the lower bound and the definition of B(b, e.) are provided Appendix A Here, we set q(k)a cgt) x exp(-Drr. (z.lc) (po [z. Cce, k))).
With the second loss LEOS (f), which is associated with training binary classifier to identify the end of speech (EOS), the total loss for training the latent language model is the sum of the two losses above: L(0) Lve (0) (P).
As shown in Fig. we implement an autoregressive latent model that yields three distinctive outputs: the mixture weights and the means of the Gaussian mixture distribution as well as the probability of concluding the generation. Specifically, it incorporates a transformer decoder followed three parallel modules, comprising (1) prediction layer with sofumax activation for pa (A) 2z, cct); (2) prediction layer for Po(k, x, cce); (3) a binary classifier layer for EOS prediction. Additionally, we use the pre-trained quantizer RQ of Mel-VAE.
5

---

Published as a conference paper at ICLR 2024
4.3 MODEL ARCHITECTURE AND INFERENCE
Model Architecture For the MeI-VAE, we adopt causal 1d convolutional U-Net, a variant of the model used in Ho et al. (2020). We remove the skip connections and attention layers and append 1-d ConvNeXt (Liu et al. 2022) blocks used in Siuzdak (2023) to the final layer of the decoder. We employ 32-stage residual vector quantization with codebook size of 1,024 for each depth. For the text-to-code latent language model, we adopt a transformer-base
encoder-decoder LM, especially a pre-trained Byrge Xue et al.) 2021a) similar to Borsos al. (2033b). We keep the text encoder frozen. Please refer to Appendix C for more detailed model configuration.
Inference The text-to-code generation unfolds in three steps: (1) at time step 4, we randomly se- lect & from the distribution Pa c<t); (2) following this, randomly sample the latent vector 2 from (z æ Cct. k).. Consequently, at time step f. the discrete code is obtained through the learned quantizer, e (Z,(2z) (3) if the probability of EOS exceeeds 0.5, conclude the generation, or proceed to step, otherwise. Subsequently, the generated codes are decoded to mel-spectrograms us- ing the decoder of MeI-VAE, then converted to raw-waveforms through an off-the-shelf pre- traiere vocoder, BigVGAN (Lee et al. 2023).
5 EXPERIMENTAL SETUP
We employ 100K hours of over 12K distinct speakers' speech- dataset spanning 11 languages: English, Korean, Chinese, Japanese, German, Dutch, French, Spanish, Ital- ian, Portuguese, and Polish. We train two models: (1) CLaM-en: an English-only model on 55K- hour English dataset and (2) CLaM-multi: a multilingual model trained on II-language dataset. We provide details of dataset for each language in Appendix B.I and data pre-processing in Ap- pendix B.2 and B.3
Training Dataset
Training (1) Mcl-VAE: We train the model on 4 NVIDIA A100 40GB GPUs for around 2M steps. Each GPU processes a size one minibatch containing concatenated mel-spectrograms of several audios. We trim the trailing end to have it precisely 32,768 frames long. We use Adam optimizer (Kingma & Ba 2015) with a constant learning rate of 0.0002 throughout the training. (2) Text-to-code: We train only the decoder and use learned codebook from Mel-VAE. The model is trained on 4 NVIDIA A 100 40GB GPUs for around 4M steps with dynamic batching while keeping maximum code size of 2,560. We use AdamW optimizer (Loshehilov & Hutter 20T9), and the learning rate is fixed to 0.0002 throughout the training. Throughout all our experiments, during the model inference, we sample using top-p sampling (Holtzman al. 2020) with 0.5 and is sam- pled with temperature (Kingma & Dhariwal 2018 of 2.6, which matches the empirical standard deviation in our validation dataset.
Baselines We compare the proposed model with the following four baselines: (1)
YouTTTS (Casanova et al., 20022. ) a zero-shot TTS built on VITS (Kim et al. (021) which is flow-
based end-to-end TTS (representing Conventional TTS), (2) Vall-E (Wang et al., (023) and (3)
SPEAR-TTS (Kharitonov et al. 2023) (representing Neural Codec LM), and (4) VoiceBox (Le
et al. 2023), a flow-matching- based TTS model trained large-scale training data (representing
Non- Aun--utoreegessvvee Model with Phoneme Input and Duration).
Metries (1) Intelligibility and Robustness: We measure these attributes by character error rate (CER) and word eтror rate (WER) of the synthesized transcription from generated speech concerning the input text. We follow the procedure in Wang et al. (2023). In English-only Evaluation, we syn- thesize the transcription by using the automatic speech recognition (ASR) model, the CTC--ased HuBERT-Largd Fl (Hsu et al. 2021) model pre-trained on LibriLight (Kahn et al. (2020) and then fine-tuned on LibriSpeech (Panayotov al. 2015). In the Multilingual Evaluation, we use Ope- nAl's Whisper (Radford et al. 2023) mode We adopt NVIDIA's NeMo-text- -trooeec----ooessinngg (Zhang 2021 Bakhturina et al. 2022) for text normalization; (2) Speak Similarity: We assess the
et al.
speaker similarity of two separate speech audio clips by following the same procedure outlined
Wang et al.
(023). We employ WaVLM-TDCNN Chen et al. 2022) which outputs the embed-
htrpa:
httpa////oagnnrssssoc.o//eeeeeer arge- 5960-ft
attps: https////ehttiaa..oooaaaaaaaaaaa hawpaaa/ddddldeddddd... "large-v2"
attps: https://ittaalccc////b.bbeeeeeet
httpa:// http:///hhh...coooo/ooooo/o///// htwp:///eeeeeeeoo.eeeeeeeaaaoaaa fiat

---

Published as a conference paper at ICLR 2024
ding vector representing the speaker's voice attribute. We measure the cosine similarity between the two embedding vectors to get score in [-1,1], where a higher score indicates a higher speaker similarity of the audios. We borrow the definition of SIM-o and SIM-r from|Le et al. (2023). SIM-o measures the similarity between the generated and the original target speeches, while SIM-r mea- sures the similarity concerning the target speech reconstructed from the original speech by Mel- VAE and the pre-trained vocoder; (3) Subjective Speech Quality: We measure the quality of the generated speech from human evaluations via three types of Mean Opinion Score (MOS) (Ribeiro) et al. 2: i) Quality MOS (QMOS) for an overall audio assessment, ii) Similarity MOS (SMOS) to measure speaker similarity between the prompt and the generated speech, and iii) Comparative MOS (CMOS) to compare our model with available baselines. Detailed settings of subjective tests are described Sec. B.5
Tasks We measure the performances of the proposed model under two different tasks: 1) continu- ation: Given a text and corresponding initial 3 seconds of the Ground Truth speech as a prompt, the task is to seamlessly synthesize the subsequent portion of the speech, and 2) cross-sentence: The model is given a text, a 3-second speech segment, and its corresponding transcript (the transcript is different from the text). The task is to synthesize speech reading the text in the style of the pro- vided 3-second speech. We include our samples across the tasks discussed above, covering speaker diversity, text prompting, and other aspects, on our demo page.
6 EXPERIMENTAL RESULTS
6.1 ENGLISH-ONLY EVALUATIONS
Evaluation Methodology We evaluate performances of CLaM-en across continuation and cross- sentence tasks. Following the evaluation setting in Wang et al. (023), we employ a subset of the LibriSpeech test-clean dataset. This subset comprises speech clips ranging from 4 to 10 seconds, each with a corresponding transcript. Note that YouTTSS has official checkpoints, Vall-E has an unofficial c and others do not have checkpoints. We use checkpoints of YourTTS and Vall-E for evaluations. We compare the other baselines with ours via the performances reported in their papers (Wang et al., 2023 Kharitonov et al.) 2023 Le et al. 2023). Since SPEAR-TH ATRS and VoiceBox also evaluate using the same approach with Vall-E, they can be directly compared with our model as well. Details of evaluation are provided in Appendix B.4
Tab. aand the results of continuation and cross-sentence task, respectively. Ours offers great performances for all measures, ranking either first or second, except SIM-r in cross- sentence task.
Analysis
is worth noting that VoiceBox (Le et al. 2023), a phoneme- duration model, shows better per- formances than ours. However, it requires both phoneme and duration for speech synthesis, whereas our model directly employs a pretrained language model. This allows for the seamless integration of LMs, which are trained across a broad spectrum of texts and tasks, enabling plug-and-pl and-play methodology. Experimental results of training several T5 variants is shown in Appendix D.2 illus- trating the trade-off between leveraging the inherent capacity of LMs and ensuring robustness. We also compare the end-to-end inference time for a 10-second utterance. Our method is faster than the generation speed of Vall-E reported Le et al. (203)). While ours is faster than VoiceBox with 64 decoding steps, VoiceBox can use fewer iterations of decoding steps. Tab. B presents the subjective audio evaluations. CLaM-en significantly outperforms the baseline, YouTTTS, in quality and intel- ligibility, as indicated by QMOS. Our adherence to the prompt audio surpasses that of the baseline, as measured by the SMOS. The comparative scores (CMOS) highlight CLaM-en's 's proximity to the Ground Truth regarding naturalness, clarity, and comprehensibility. Overall, CLaM-en's 's generated speech naturalness, quality, intelligibility, and similarity exceed the baseline.
6.2 MUL.TI. EVALUATIONS
We evaluate our model, CLaM-multi trained on the multilingual dataset. On the test set, we measure WER, CER, and SIM-o which are defined in Sec. 5 Here, we only consider continuation task in this experiment since we cannot get full alignmnets between audio and text for all languages and datasets. Tab, 4shows the partial results of the multilingual continuation task. We sample hundred
tttpa://gtthbb.co telteng/val 1-8
7

---

Published as a conference paper at ICLR 2024
Performances for the English-only continuation task. The boldface indicates the best result, the underline denotes the second best, and the asterisk denotes the score reported in the baseline paper. Ours offers great performances for all measures, ranking either first or second. The inference time indicates the generation time of 10s speech.
Table 1:
Model
WER +
CER J.
SIM-o f
SIM-r
Inference Time +
Ground Truth
2.2*
0.61
0.754
0.754
n/a
YourTTS Casanova et al
2022)
7.57
3.06
0.3928
Vall-E Wang et al. 2023
3.8*
0,452*
0,508
-6.2s
Vall-E (unothcial)
3.81
1.58
0.2875
0.3433
Voicehox (Le et al.
2023
2.0*
0.593*
0.616*
~6.45 (64 NFE)
CLaM-en
2.36
0.79
0.4767
0.5128
4.15s
Table 2: Performances for the English-only cross-sentence task.
Model
WERL
CER,
SIM-oT
SIM-r
YourTTS (Caaanova et al.
2022
7.92 (7.7*)
3.18
0.3755 (0.337*)
Vall-E Ball-E et al. 2023
5.9*
0.580*
Vall-E (unofficia))
7.63
3.65
0.3031
0,3700
SPEAR-TTS (Kharitonov et al. 2023
1,92
0.560*
Voicebox (I. et al. 2023
1.9*
0.662*
0.681
CLaM-en
5.11
2.87
0.4951
0,5382
random samples from the test set of each dataset, ranging from 4 to 20 seconds, and average the scores of three trials. Refer Tab. for evaluation on other languages and datasets.
6.3 ABLATION STUDY
Effectiveness of Proposed RVQ To demonstrate the effect of the proposed RVQ on Mel-VAE, conduct an ablation study by assessing speech reconstruction capability. We train Mel-
VAEs: one with the proposed RVQ and the other with the baseline RVQ Dééosssez et al. 2023). We
train both for 500k steps on the same training dataset described in Sec. [S] The generated speech
compared to the Ground Truth speech using two metrics: Perceptual Evaluation of Speech Quality
(PESQ) (Rix et al. 200T) and Virtual Speech Quality Objective Listener (ViSQOL) (Chinen al. 2020) in speech mode. For evaluation, we randomly select 1,800 samples from the test set of each dataset, proportional to the size of each dataset, each being at least seconds long. The scores of these samples are then averaged for comparison. Tab. 5h shows that ours is more effective than the baseline RVQ. See Fig. 2 to verify the superior codebook usage of our approach. We also compare the fully trained Mel-VAL with Encodec at 6K bitrates (Défossez
al. 2023), which is widely
employed in neural codec language models (Wang et al. 2023 Zhang et al. 2023). ). Tab.5p confirms that ours outperforms Encodec in speech reconstruction performance across both measures.
Comparision of Pre-trained LM and Input Variants Our language model is based on T5 (Raffel et al. 2020). We conduct an ablation studty to compare T5, its variants and phoneme encoder of comparable size. The results indicate that ByT5 surpasses other T5 variants with the sole exception of the phoneme model. This suggests that: 1) the more the pretraining phase is leveraged, the greater the potential increase in TTS performance, and 2) moderate-sized language modeling, phonemes remain an effective input representation. For the experimental results and comprehensive analysis, refer to Appendix [.22
In addition to the ablation studies presented, we have conducted further experiments detailed in the appendix, which explore the effects of codeword emitting rate on speech codec quality and language modeling as well the scale of training data model efficacy. For comprehensive results and discussions, refer to Appendix DD3 and D.4
7 DISCUSSION
Choice of Codeword Rate Our approach enjoys 10Hz codeword rate for efficient modeling. We set the codeword rate following the average phoneme rate in English speech (Roach) 2009) since phoneme is the minimum spoken unit. Nevertheless, conjecture this may have to be ad-
8

---

Published as a conference paper at ICLR 2024
Human evaluations with 40 LibriSpeech test-clean speakers show ClaaMen''s speech gen- eration surpasses the baseline in quality. intelligibility, similarity, and naturalness, nearing Ground Truth. QMOS and SMOS scores include 95% confidence interval.
Table 3:
Model
QMOS
SMOS
CMOS (vs. CLaM-en)
YourTTS (Casanova et al. 2022)
2.19
2.32.1.21
-1.68
CLaM-en
3.87
3.4910.14
0.00
Ground Truth
4.45=0.09
4.18+0.15
+0.63
Table 4: Performances of CLaM-multi for the multilingual continuation task.
Language Dataset
WER,
CER.
SIM-o
English / MLS English
8.71
5.19
0.4000
English (HuBERT) MLS English
7.71
3.19
0,4000
German / MLS German
9.63
4.11
0.4219
Dutch / MLS Dutch
12.25
4.97
0.5983
French MLS French
10.29
4.08
0.5671
Spanish / MLS Spanish
4.02
1.91
0.5292
Italian MLS Italian
19.70
5.19
0,5459
Portuguese / MISS Portuguese
9.66
3.72
0.5658
Polish MLS Polish
14.70
5.34
0,5519
Table 5: Effectiveness of our proposed RVQ. The results show that ours outperforms the conven- tional RVQ in (a) and Encodec in (b) across both measures, even with higher compression rate.
(a)
(b)
Model
PESQ
ViSQOL
Model
PESQ
ViSQOL
ours + VGAN
2.63
4.48
ours + VGAN
2.95
4.66
RVQ + BiAVGAN
2.54
4.44
Encodec
2.59
4.26
justed depending on the language speaker. A more compressed codeword rate, for example, 5Hz, might lead to more significant information loss than their efficiency. There exists an efficiency- performance tradeoff for rates above 10Hz, which can be optimized needed.
Robustness We have noticed some words can be muddled, omitted, or repeated, which predom- inantly stems from autoregressive modeling. We will address it by employing non-autoregressive architecture or improving the attention mechanism in future work.
Expressiveness 100K hours of training data may not ensure a complete representation all voice types, especially accentuated ones. Our datasets predominantly capture audiobook reading styles, leading to limited diversity in speaking styles. We believe that increasing the model and data size can significantly tackle the expressiveness challenges in zero-shot TTS.
Instruction Prompting We suggest various ways to the full knowledge of the language model. One can incorporate speaker metadata into each transcript to perform various intriguing tasks. Such tasks might include synthesizing speech even conversations characterized by specific genders, voice ages, or accents. We leave the other tasks for future work.
8 CONCLUSION
We introduce CLaM-TTS, which leverages mean-field variational inference based probabilistic residual vector quantization (1) achieving significant compression in token length, and (2) allow- ing latent language model to generate multiple tokens at once, thereby eliminating the need for cascaded modeling to handle the number of token streams. We scale up the training dataset to 100K hours. We empirically show that CLaM-TTS is better than comparable to state-of-the-an neu- ral codec-based TTS models regarding naturalness, intelligibility. speaker similarity, and inference speed.

---

Published as a conference paper at ICLR 2024
9 ACKNOWLEDGMENTS
The authors would like to thank Kangwook Lee for helpful discussions, as well as Beomsoo Kim, Gibum Seo, and Dongwon Kim for their essential support throughout the processes of data handling and evaluation of the implementation.
10 ETHICS STATEMENTS
CLaM-TTS is zero-shot TTS model that leverages a pre-trained large language model, offering efficient learning and inference vast scale. The model's capability to produce any voice and mimic with only minimal voice input presents potential dangers, including spoofing misuse. Given the escalating risks associated with such models, it should be imperative to develop detection model to identify audio outputs from the model and to establish rigorous protocol for evaluating its effectiveness.
11 REPRODUCIBILITY STATEMENTS
For the implementation of our model, we provide Fig. and description of the model architecture in Sec. [.3 along with the hyperparameters of the model configuration in Tab. 7 To ensure the reproducibility our experiments, we also share details, including a list and statistics our training data in Sec. 5 and Appx. [B.I] data preprocessing procedures in Appx. B.2 and Appx. B.3 training configuration and the evaluation methodology in Sec. [5] If our potential legal concerns can be addressed,
prepared to progressively disclose, for research purposes, the inference code, pre-trained weights, and ultimately, the full training implementation.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In International Conference on Learning Representations, 2020a.
Alexei Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 13.1449-11246, 2020b.
Evelina Bakhturina, Yang Zhang, and Boris Ginsburg. Shallow Fusion of Weighted Finite-State Transducer and Language Model for Text Normalization. In Proc. Interspeech 2022, 2022.
Jeong-Uk Bang, Seung Yun, Seung-Hi Kim, Mu-Yeal Choi, Min-Kyu Lee, Yeo-Jeong Kim, Dong- Hyun Kim, Jun Park, Young-Jik Jik Lee, and Sang-Hun Kim. Ksponspeech: Korean spontaneous speech corpus for automatic speech recognition. Applied Sciences, 10(19):6936, 2020.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
David M Blei, Alp Kueukelbir, and Jon D McAuliffe. Variational inference: review for statisti- cians. Journal of the American statistical Association, 1 112(518):859- 877, 2017.
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pictquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. Audiolm: language modeling approach to audio generation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023a.
Zalán Borsos, Matt Sharifi, Damien Vincent, Eugene Kharitonov, Neil Zeghidour, and Marco Tagliasacchi. Soundstorm: Efficient parallel audio generation. arXiv preprint urXiv:2305. 09636, 2023b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Edresson Casanova, Julian Weber, Christopher Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir A Ponti. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. In International Conference on Machine Learning, pp. 2709-2720 PMLR, 2022.
Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuai- jiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang. Zhao You, and Zhiyong Yan. GigaSpeech: An Evolving, Multi-Domain ASR Corpus with 10,000 Hours of Transcribed Audio. In Proc. Interspeech 2021, pp. 3670-3674, 2021. doi: 10.21457/ntersspecc.2000 202tleechhh.2119966 1965.
Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, and Tie- Yan Liu. Hifisinger: Towards high-fidelity neural singing voice synthesis. arXiv preprint 2xXiv:2009001788 2020.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Wavim: Large-scale self- pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505- 1518, 2022.
Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O' Gorrman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric. In 2020 twelfth international conference on quality of multimedia experience (QoMEX), pp. 1-6. IEEE, 2020.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction fineetunddd language mod- els. arXiv preprint arXiv:2210. 14116 2022.
11

---

Published as a conference paper at ICLR 2024
Yu-An Chung, Yu Zhang, Wei Han, Chung-Chemn Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self- supervised speech pre-trale rre-training. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 244-250. IEFE, 2021.
Jade Copet, Felix Kreuk, Itai Gat. Tal Remez, David Kant, Gabriel Synnaeve. Yossi Adi, and Alexan- dre Defossez. Simple and controllable music generation. A. Oh, T. Neumann, A. Globerson, K. SSenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 17700447722 Curran Associates, Inc., 2023.
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. Featured Certification, Reproducibility Certification.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE conference on computer vision and puttern recogni- tion, pp. 188831 2021.
Yue Yinl Daijiro Moril Seiji Fujimoto. Reazonspeech: A free and massive corpus for japanese asr.
Cristina Gårbacea, Aäron van den Oord, Yazhe Li, Felicia SC Lim, Alejandro Luebs, Oriol Vinyals, and Thomas C Walters. Low bit-rate speech coding with vq-vae and wavenet decoder. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 735-739. IEEE, 2019.
Robert Gray, Vector quantization. IEEE Assp Magazine, 1(2):4-9,, 1984.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6651, 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICIR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 29:3451-3460, 2021.
KKatsuya lida.
Kokoro speech dataset.
htttps://gittubb.cmm/kiiis
Kokoro- Speech- -Daaaset 2021.
Keith Ito and Linda Johnson. The lj speech dataset.
https://keithito..c
LJ- Swe-ppeeh--Daasstttte 2017.
Tejas Jayashankar, Thilo Koehler, Kaustubh Kalgaonkar, Zhiping Xiu, Jilong Wu, Ju Lin, Prabhav Agrawal, and Qing He. Architecture for variable bitrate neural speech codec with configurable computation complexity. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 861-865. IEEE, 2022.
Xue Jiang, Xiulian Peng, Chengyu Zheng, Huaying Xue, Yuan Zhang, and Yan Lu. End-to-end neural speech coding for real-time communications. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 866-870. IEEE, 2022.
Ziyue Jiang, Yi Ren, Zhenhui Ye, Jinglin Liu, Chen Zhang, Qian Yang, Shengpeng Ji, Rongjie Huang, Chunfeng Wang, Xiang Yin, et al. Mega-Its: Zero-shot text-to-speech at scale with intrin- sic inductive bias. arXiv preprint arXiv:2306.03500 2023.
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu,
Mazaré, Julien Karadayi, Vitaliy Liptchinsky. Ronan Collobert, Christian Fuegen, et al. Libri- light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2020.
IEEE,
12

---

Published as a conference paper at ICLR 2024
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. In International Con- ference Machine Learning, pp. 2390- 2399. PMLR, 2018.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arViv preprint (xXiv:2001.0836/ 2020.
Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, Read and Prompt: High- Fidelity Speeeh with Minimal Supervision. Transactions of the Association for Compu- tational Linguistics, 1 11:170301111 1718, 12 2023. ISSN 2307-387X. doi: 10.1162/tal.a.0006
Chanwoo Kim and Richard M Stern. Robust signal-to-noise ratio estimation based on waveform amplitude distribution analysis. In Ninth Annual Conference of the International Speech Commu- nication Association. Citeseer, 2008.
Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference Machine Learning, pp. 5530-5540. PMLR, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. Advances in neural information processing systems, 31, 2018.
Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Michiel Bacchiani, Yu Zhang, Wei Han, and Ankur Bapna. LibriTTS-R: A Restored Multi- Speaker Text-to- Text-to-Speech Corpus. In Proc. INTERSPEECH 2023, pp. 5496-5500, 2023. doi:
10.21457/Inerrppecc.2233 20tprrpppecch.221555-..
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvqgan. A. Oh, Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 77980-27993 Curran Associates, Inc., 2023.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 1-143-11354 2021.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. Voicebox: Text-guided multi- lingual universal speech generation at scale. A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt. and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 14005-14034. Curran Associates, Inc., 2023.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11523-11532, 2022.
Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 6706-6713, 2019.
13

---

Published as a conference paper at ICLR 2024
Shuyang Li, Huanru Henry Mao, and Julian McAuley. Variable bitrate discrete neural representa- tions via causal atteentnn In 2nd Pre-registration workshop (NeurIPS 2021). Remote.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Repre- sentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-1 1986, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, IA, USA, May 6-9, 2019. OpenReview.net, 2019.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international confer- ence on computer vision, pp. 2794-2802, 2017.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi. In Proc. Interspeech 2017, pp. 498-502, 2017. doi:
http://nterspeecc2017---1
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Eliya Nachmani, Alon Levkovitch, Julian Salazar, Chulayutsh Asawaroengchai, Soroosh Mari- ooryad, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich, Lms with a voice: Spoken language modeling beyond speech tokens. arXiv preprint arXiv:2305.15255 2023.
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello. Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, et al. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11: 250-266, 2023.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. Advances in neural information processing systems, 30, 2017.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-520111 5210. IEEE, 2015.
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- delrahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. In INTERSPEECH 2021-Anmual Conference of the International Speech Communication Association, 2021.
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. MLS: A Large-Scale Multilingual Dataset for Speech Research. In Proc. Interspeech 2020, pp. 2757- 2761, 2020. doi: 10.21437//meerseccch220000
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Ma- chine Learning, pp.
PMLR, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(555548--551, 2020.
14

---

Published as a conference paper at ICLR 2024
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning. pp. 8821-8831 PMLR, 2021.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Yi Ren, Ming Lei, Zhiying Huang, Shiliang Zhang, Qian Chen, Zhijie Yan, and Zhou Zhao. Prosospeech: Enhancing prosody with quantized vector pre-training in text-to-speech. In ICASSP 2022-2022 IEEE/ International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 7577-7581. IEEE, 2022.
Flávio Ribeiro, Dinei Florêncio, Cha Zhang, and Michael Seltzer. Crowdmos: An approach for crowdsourcing mean opinion score studies. In 2011 IEEE international conference on acousties, speech and signal processing (ICASSP), pp. 2416-2419. IEEE, 2011.
Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a a new method for speech quality assessment telephone networks and codecs. In 2001 IEEE international conference on acoustics, speech, and signal processing. Proceedings (Cat. No. 01CH37221), I,, volume 2, pp. 749-752. IEEE, 2001.
Peter Roach. English phonetics and phonology paperback with audio CDs (2): A practical course. Cambridge university press, 2009.
Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. Theory and experiments on vector quantized autoencoders. arXiv preprint 1rX.v:1805.1100 2018.
Paul Rubenstein, Chulayuth Asawaroengchai, Due Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12999 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479-366 -64944, 2022.
Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint 2rXiv:2304.091 16, 2023.
Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier- based neural vocoders for high-quality audio synthesis. arXiv preprint 2rXiv:2306.00814, 2023.
Guangzhi Sun, Yu Zhang, Ron J Weiss, Yuan Cao, Heiga Zen, Andrew Rosenberg. Bhuvana Ram- abhadran, and Yonghui Wu. Generating diverse and natural text-to-speech samples using a quan- tized fine-grained vae and autoregressive prosody prior. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6699-6703. IEEE, 2020.
Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu. Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, Sheng Zhao, Tao Qin, Frank Soong, and Tie- Yan Liu. Naturalspeech: End- to-end text-to-speech synthesis with human-leve quality. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1- 2024. doi: 10. 19110PAA...2220.3352222.
Linh The Nguyen, Thinh Pham, and Dat Quoe Nguyen. XPhoneBERT:: A Pre-trained Multilingual Model for Phoneme Representations for Text-to-Speech. In Proc. INTERSPEECH 2023, pp- 5506-5510, 2023, doi: 10.24433/nnesppecchh22-444
A Vasuki and PT Vanathi. A review of vector quantization techniques. IEFE Potentials, 25(4): 39-47, 2006.
Christophe Veaux. Junichi Yamagishi, Kirsten MacDonald, et al. Superseded-estr vetk corpus: En- glish multi-speaker corpus for estr voice cloning toolkit. 2016.
15

---

Published as a conference paper at ICLR 2024
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huamingg Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXtv:2301.02111, 20233.0.
Yi-Chiao Wu, Israel D Gebru, Dejan Marković, and Alexander Richard. Audiodec: An open-source streaming high-fidelity neural audio codec. In /CASSP 2023-2023 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards token-free future with pre-trained byte-to-byte mod- els, 2021a.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tuu Tur, Iz Beltagy. Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 483-498, Online, June 2021b. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022 naacl-main.411
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang. James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. In International Conference on Learning Representations, 2021.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 30:495-507, 2021.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech. In Proc. Interspeech 2019, pp. 1526-1530, 2019. doi: 10224457/mmerrpeehh..0094444
Binbin Zhang, Hang Lv, Pengcheng Guo, Qijie Shao, Chao Yang. Lei Xie, Xin Xu, Hui Bu, Xiaoyu Chen, Chenchen Zeng, al. Wenetspeech: A 10000+ hours multi-domain mandarin corpus for speech recognition. ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6182-6186. IEEE, 2022.
Yang Zhang, Evelina Bakhturina, and Boris Ginsburg. NeMo (Inverse) Text Normalization: From Development to Production. In Proc. Interspeech 2021, pp. 4857-4859, 2021.
Ziqiang Zhang, Long Zhou, Chengyi Wang. Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Speak foreign languages with your own voice: Cross-lingual 2023.
neural codec language modeling. arXiv preprint
16

---

Published as a conference paper at ICLR 2024
A
VARIATIONAL LOWER BOUND
We have
log po (Ce (c cct) = log
pa(c,, 2elo0 cet) dz,
(a)
log
dz
(zl cx)p((r)
log
po(zolz, c<i)
dz
Ee; Pul (Zi (1((()))(c))
b log / polziloe, cat)
(2(
dz
Ee; Pv (z(((c))
(c)
pulzoles)
logro(z)1or,com cet) - log Σ (e(l)))))
dzn,
(7)
where (a) follows from the modeling assumption that c₁ is independent of x and Cct given zy (b) follows from the assumption that p((²) is uniformly distributed for all t; and (c) follows by Jensen's inequality.
We assume that an unobserved discrete random variable & involved in generating the latent vector 2, by some random process.
log po [e, Cee) log E po (z,, cet)
log Σ po (kl cco)po(zi ccr,
(k Ces) Cea, k) q(k x c(t)
log Σ g(kk/, cgt)
pe ( c2i) g(ki x cgt)
Σ q(kl,,,.) cst) logpo Ccr, k) log
(8)
By incorporating this relation into Eq. 7 we have:
log po x cct)
log po(zlzz, cct) - log Σ po (zilct)
dzi
c/
Σ q(k|z 2, csu)
Is poolzlla) log po(z)lax, kc
dzy + log polk|o
Eo; Pul
q(k)x,, ((t) cst)
log po(z Cet; (2(
po (k|k cec)
=k Epo(eclo))
+ log q(klz,, czt)
Ec; (2((z)))))
=By(k)aees)) DKL (2 (2 ar, Gets k))] DKz(g(klaa egt (po) cct)) + ((bb, ct),
where B(b, ct) (xp(o)))))
log po(2))a) Eej (o(()))))
Epo Boop $o. 0.
B ADDITIONAL DETAILS OF EXPERIMENT
B.1 DATASET STATISTICS
The training datasets for each language follows:
17

---

Published as a conference paper at ICLR 2024
English: 1) Multilingual LibriSpeech (MLS) (Pratap al. 2020), multi-speaker and multilingual
transcribed speech dataset sourced from Libri Vox audiobooks. 2) GigaSpeech (Chen et al. 202I)
consists of multi-domain speeches, such audiobooks, podcasts and YouTube along with human transcriptions. The audio dataset includes multiple speakers but lacks associated speaker informa-
tion. 3) LibriTTS-R (Koizumi et al., (003) is a restored version of the LibriTTS (Zen et al. 2019)
corpus which shares the identical metadata with LibriTTS, 4) VCTK (Veaux et al.) 2016) and 5) LJSpeech (Ito & Johnson (2077) multi-speaker and single-speaker English datasets, respectively, widely used in speech synthesis community.
Korean: AlHub 14th features recordings of everyday people reading provided script sentences. 2) AIHub 150 has recordings of 50 professional voice actors for seven emotions (joy, surprised, sad, angry, scared, hate, neutral), five speaking styles (narrating, reading, news-like, dialogie, broad- casting), and three vocal ages (kid, young, old). 3) KsponSpeech (Bang et al. (0220) consists of 2,000 speakers and each recording has an individual freely talking about various topies in a quiet environment. The transcription follows specific guidelines regarding laughing, breathing, and few more.
Chinese: WeNetSpeech (Zhang et al. 2022) is similar to GigaSpeech, comprising multi-domain speeches without speaker information.
Japanese: 1) ReazonSpeech (Fujimoto) is labelled dataset made up of roughly 19,000 hours of broadcasted speech. It involves multiple speakers without speaker information. 2) Koko- roSpeech (Tia 2021) contains recordings of single speaker reading 14 novel books.
Others: The datasets feature seven language subsets from MLS: German, Dutch, French, Spanish, Italian, Portuguese, and Polish.
Tab. 6 shows detailed statistics of each dataset. For the LJSpeech, VCTK, KokoroSpeech, and ReazonSpeech datasets, 99% of the entire dataset is used for training. If a specific training is predefined, we use it as-is.
B.2 DATA PRE-PROCESSING
Note that Gigaspeech, WeNetSpeech, and ReazonSpeech do not provide speaker labels. For the datasets, we exclude audio instances that contain two or more speakers using an open-source speaker diarization model [0] We also compute the SNR of the audios in three datasets using waveform amplitude distribution analysis (WADA) (Kim & Stern 2008). Audios with WADA-SNR > 20dB
are only included in our training set.
We preprocess the large datasets to efficiently store and iterate over them. Audio metadata is first constructed the parque [ rates and reduces I/O overhead, making it suitable for metadata storage. Parquet contains speaker attributes (name, gender, accent, emotion, and age), audio path, length, sample rate, and text. The constructed metadata is combined with audio data read byte streams to form datasets using web- dataset:
format. Parquet, storing data column-wise, offers high compression
[2] Specifically, audio byte streams are paired with JSON data from parquet, and every 10k pairs stored single TAR file. Storing data in this manner facilitates the addition of metadata items or text forms (e.g., phoneme, normalized text) in the future.
We utilize the stored speaker metadata part of the text prompt. The speaker attributes appended in front of each text. One example would look like the following.
man, old, neutral:
We have to reduce the number of plastic bags.
Please check our demo page for text prompting applications.
httpss//paccet..ppppeee.oo/////
htt
https///clan-tts.gitbbb.
18

---

Published as a conference paper at ICLR 2024
Table 6: Statistics of datasets for different languages. We blank the number of speakers when the original dataset doesn't t provide speaker information.
Lang
Dataset
Train Total (hrs)
# Speakers (Train)
Rate (Hz)
MLS
44,659.747 44,6/9445
3,490
16,000
GigaSpeech
9,997.82 10,050.65
16,000
English
LibriTTS-R
730.00 / 769.97
2,456
24,000
VCTK
40.63 / 41.44..0 .04
109
48,000
LJSpeech
23.68 / 23.92
1
22,050
AlHub 14
8,086.04 / 9. 1,110.33
3,495
48,000
Korean
AlHub 15
836.78 / 951.98
50
48,000
Ksponspeech
965.15 / 975.54
2,000
16,000
Chinese
WeNetSppech
10,005,41 10.063.67
16,000
ReazonSpeech
18,846.81 19,037.18
16,000
Japanese
KokoroSpeech
58.11 / 58.69
22,050
German
MLS German
1,966.51 / 1,995.08
176
16,000
Dutch
MLS Dutch
1 1,579.76 76
40
16,000
French
MLS French
1,076.58 / 1,096.72
142
16,000
Spanish
MLS Spanish
917.68 / 937.68
86
16,000
Italian
MLS Italian
247.38 / 257.83
65
16,000
Portuguese
MLS Portuguese
160.96 / 168.35
42
16,000
Polish
MLS Polish
103.65 107.87
11
16,000
B.3 APPROXIMATION FOR AUDIO RESAMPLING
We pre-process the audio dataset to create mel-spectrograas with the same resolution. We first revisit the audio resampling process before describing our method. Let us denote the resampled audio, its sample rate, and its corresponding mel-spectrogram as Atarget, Starget, and Mtarget Asoures and S Ssource- Please
respectively. The original audio and its sample rate can be labelled note that our proposed model uses the mel-spectrogram as both input and output, making Mtarget final goal for the data processing. First, apply the STFT to Asource to obtain frequency domain components. We then perform linear interpolation in the frequency domain to adjust the number of samples per second match Starget- Due the occurrence of aliasing, Low-pass filter applied. Next, we use the inverse STFT (ISTFT) to acquire the upsampled Atarget- Finally, we apply the STFT again derive Mtarget.
Our approach, in contrast, is as follows: To determine Mtarget, we adjust the pre-set FFT size (which also the same window size) and hop size according the ratio of Ssource to Starget- Using the value, we produce a linear spectrogram and then apply a mel-filter bank to generate a mel-spectrogram. We refer to it as Marget and regard it as an approximation for Mzorget. Unlike Mtarget, the process of obtaining Mtarget does not actually change the number of audio samples, which can lead inaccuracies at high frequencies. However, due to the nature mel-spectrograms, it be disregarded in regions of important low-frequency features. The advantage of deriving Mtarget is that we only need to apply the STFT once, and there's no need to store any audio in between. It allowed us to use datasets with different sample rates without time and storage con- sumption due to audio resampling.
In the maximum audio length per batch from the train set and sort them by sample rate. Then we determine the ratio of each data sample rate over the target to modify FFT size (window size) and hop size, respectively. The mel-spectrogram calculated by these STFT parameters input to model.
training scheme, we first randomly sample the amount of data up to the pre-defined
B.4 DETAILS OF ENGLISH-ONI EVALUATION
We average the result over three repetitions of each experiment with randomly selected set of prompts per trial. In WER and CER evaluation on contimuation task, we include prompt reconstruc- tion at the beginning of the generated speech. Meanwhile, we exclude reconstructed prompts in SIM
19

---

Published as a conference paper at ICLR 2024
evaluation the same task. Recall that model takes the speech prompt tögether with the responding transcript. pross-sentence task, we use Montreal Forced Aligner (MFA) (McAulifie et al., 2017) to align transcript and audio. We randomly select starting point at the beginning of a word audio and use subsequent 3-second audio for baselines. Otherwise, we use the audio and text that include many words possible within the subsequent seconnd duration. This yields audio clips whose average length is around 2.7 seconds. We name it as word-based prompting.
B.5 SUBJECTIVE EVALUATION
14 In QMOS, direct eval- We carry evaluations using Amazon Mechanical Turk (MTurk uators to assess the quality and clarity of each recording, considering sound quality and clarity of speech. For SMOS, evaluators gauge likeness of samples to the provided speech prompts, taking into account the speaker similarity, style, acoustics, and background disturbances. In CMOS, evalu- ators compare overall quality of synthesized sample to that of reference. Using the given scale, they judge whether the synthesized version was superior or inferior to the reference.
QMOS and SMOS employ scale from -3 (indicating the synthesized speech is much worse than the reference) to 3 (indicating it's much better), with I-unit intervals. For QMOS, SMOS, and CMOS, samples garner 10, and 12 ratings respectively. We omit evaluators whose average scores deviate by two standard deviations or more from the mean over every evaluator. All evaluators are US-based
scale of integer, where 5 signifies top quality. CMOS uses
Table 7: The detailed model configurations of Mel-VAB.
Module
Configuration
Value
hidden size
256
Encoder
channel multiplication
[1, 1, 2, 2]
dropout
0.0
hidden size
256
Decoder
channel multiplication
[1, 1, 2, 2]
dropout
0.0
ConvNeXt hidden layer
80
Depth
32
Probabilistic RVQ
Vocab Size
1024
Channel Size
512
Table 8: The detailed model configurations of Text-to-Code.
Module
Configuration
Value
hidden size
1536
Encoder
Number of Heads
16
Number of Layers
36
Feedforward Dimension
3840
dropout
0.1
hidden size
1536
Decoder
Number of Heads
16
Number Layers
12
Feedforward Dimension
3840
dropout
0.1
Weight Predictor Dimension (k)
2048
Latent MoG
label smoothing
0.01
20

---

Published as a conference paper at ICLR 2024
C MODEL CONHIGURATIONS AND IMPLEMENTATION DETAILS
C.I MODEL CONFIGURATIONS
We provide detailed model configurations in Tab. and
C.2
IMPLEMENTATION DETAILS
MelVAE
Our embedding vector of residual vector quantization is structured to decrease in length with each increasing depth d, denoted as evi d). This configuration ensures that embeddings at higher depths capture more detailed features. The formulation for this parameterization is given by:
ev(cr,di d)
evp (c d) =ad =adTes (cr,
where ad = sccplmar..call.oggle
softmax(ssale.logitss))
Here, ag dynamically adjusts the scale of embedding vectors at the depth d, with the trainable parameters of the residual vector quantizer being mar_scale logil,, scale logilsg, and ac
Latent MoG We present an efficient approach to managing the number of mixtures, k in the Mixture of Gaussians (MoG) of our latent language model. The dimensionality of the MoG output is directly proportional to both l and the dimension of the mean vector ot (k, x, Cet) R". This dimensionality can become unwieldy as & increases. To address the challenge of managing the excessive size of the output weight, we employ strategy to compress it. This approach yields a low- rank prediction já (k: xi Cet) E R and a projection matrix M Rmxnn where n < m, effectively enabling us to represent soo (k; Cet) as M (k, x, Cet). This compression facilitates more effi- cient computation of the training loss, which involves calculating the L2 distance between the mean vector se (k, Czt) and the quantized latent representation of speech &, as || (k, x, Czt) The equation is expanded as follows:
& pe(k, I, cct))
|| - I,,
2 2, + juo x, czt) " (M' 2(M (k,, Cet).
(10)
Precomputing M M and M 2, in Eq. 10 significantly reduces memory usage during L2 distance computation. For numerical stability, we apply spectral normalization (Miyato et al. 2018 to M. In practice, we use all these techniques when k >> 512.
D ADDITIONAL RESULTS
D.1 DURATION-BASED PROMPTING
In duration-based prompting, we randomly pick one utterance from the target speaker and choose a 3-second segment. We then use the same model from Sec. 5] which is OpenAl's Whisper, to transcribe this segment, and input this audio-text pair as prompt. However, this method introduces transcription errors from Whisper, causing our model to include incorrect text at the beginning of generated speech and hence to have poor WER and CER performances.
Tab. shows the results that word-based prompting is more effective than daration-based in our Character-basee Text-to-Code model.
D.2 COMPARISON OF T5 VARIANTS
We compare the performance of the available T5 variants: 1) T5, which approaches every NLP task as a text-to-text conversion, regardless whether it's translation, summarization, or question- answering. m'T5 (Xue et al. 2021b), multilingual variant of T5 handling multiple languages
21

---

Published as a conference paper at ICLR 2024
Table 9: Results of comparison between (word-based) cross-sentence and cross-
sentence task.
Model
WER
CER
SIM-o
CLaM-en (word-based) from Tab. 2
5.11
2.87
0,4951
CLaM-en (duration-base
8.68
5.69
0.5026
Table 10: Results of Multilingual continuation task. The scores of WeNetSpeech are absent because
our ASR tool doesn't recognize the generated Chinese speech. Since Japanese lacks spacing, WER measurement is not feasible.
Lang
Dataset
WER,
CER
SIM-o
GigaSpeecc
9.75
2.86
0.3738
LibriTTS-R
3.12
0.96
0.5112
English
VCTK
1.48
0.73
0,3849
LJSpeech
6.55
4.75
0.5879
GigaSpeech
16.90
4.74
0.3738
English
LibriTTS-R
4.33
0.74
0.5112
(HuBERT)
VCTK
5.26
1.98
0.3849
LJSpeech
7.65
3.10
0,5879
AiHub 14
20.21
1.80
0,5423
Korean
AlHub
13.08
2.35
0.5280
Ksponspeech
30.24
20.02
0.4488
Chinese
WeNetSpeech
0,2600
ReazonSpeech
49.37
0.2699
Japanese
KokoroSpeech
11.46
0.5653
within one model. 3) ByTs, which trained byte sequences rather than subword tokens. 4) Flan-T5 (Chung et al. 2022). which employs prompting for pre-training. 5) T5-Im-adapt, another T5 model pre-trained on denoising and fine-tuned as prefix language model. And to compare the case where the Text-to-Code model receives phoneme sequence as an input, we employ 6) phoneme encoder XPhoneBert (The Nguyen al. (002)) that the same size with an encoder of t5-base. We train only the decoders with identical decoder structures across both settings. For each variant, we implement Text-to- to-Co with each base architecture and train it on the same Mel-VAE code following Sec. 4.2] All models initialized using pre-trained weights. Then,
evaluate the models in a continuation task, measuring WER, CER, and SIM-o. We use the MLS English subset for training and follow the training procedures described in Sec. Evaluations conducted on the LibriSpeech test-clean set as outlined Sec. 6.1
Tab. 11 shows that ByT5 outperforms the other variants of the T5 model except that the phoneme model. Notably, ByT5 significantly surpasses T5 by merely changing the input format. Flan-T5 and T5-Im-adapt also demonstrate better performance than T5 by pre-training and fine-tuning prefix language model. The results imply that fine-tuning ByT5 as a prefix language model might yield even better results if then trained Text-to-Code. The superior results of phoneme variant demonstrate why many TTS studies have preferred phonemes over characters. Despite its remark- able performance, we choose byT5 as the base model. The reason for this is that while large-scale pretrained models based on text, like byT5, have been released and researched, phoneme-based en- coders have seen only small-scale models made available and studied due to limited phoneme data. It is still understood that using phonemes as input is effective, and both text and phonemes can be selectively chosen based on their scalability and robustness. We leave this exploration for future work.
D.3 EFFECTS OF CODEWORD EMITTING RATE
We conducted additional ablation studies to analyze the impact of varying codeword rates in the proposed framework. The results demonstrate trade-off: reducing the code emitting frequency
22

---

Published as a conference paper at ICLR 2024
Table 11: Results of continuation task of T5 variants, All models are base model.
Model
WER,
CER
SIM-ot
ByT5
2.79
1.00
0.3879
Im-lm-
2.88
1.17
0.3821
Flan-T5
2.92
1.21
0.3802
mT5
4.62
2.56
0.3762
T5
9.48
7.04
0.3634
T5- 15-phoneme
2.62
0.96
0.3943
degrades audio quality while increasing it diminishes language modeling performance. This finding shows that we chose the codeword rate that operates sweet spot in this trade-off.
The quality of reconstructed audios from Mel-VAE: We compared the performances of Mel- VAES trained with different codeword rates. The codeword rate in each Mel-VAE was adjusted ac- cording to the downsampling factor when generating latent representations from mel-spectrograms. We employed downsampling factors: 16, (ours), and 4; and measured PESQ and ViSQOL of the reconstructed audio from each model. Tab. 12 shows that the audio reconstruction quality of the Mel- VAEs deteriorates as the downsampling factor increases.
Table 12: Results of audio reconstruction of different Mel- MeI-VAEs.
Model
PESQ1
ViSQOL
Mel- VAE--&F (default)
2.95
4.66
Mel-VAE-dEa
3.10
4.74
Mel- MAE-VAE-
2.42
4.35
Text-to-code language model performances: We trained identical latent language models gen- erate codes for each of the VAEs and measured WER, CER, and speaker similarity. Similar to the trend in reconstructed audio quality, Tab. shows that higher code emitting frequency tends to increase speaker similarity. However, intelligibility, measured by WER and CER, performed better with a 16-fold compression compared to a 4-fold compression. This suggests that the longer the code length predicted, the more challenging it is for latent language models. Moreover, the default setting of an 8-fold compression shows the best intelligibility, indicating that our setting finds sweet spot in balancing audio quality and latent language model prediction in the trade-off. We also reported the end-to-end inference time of 10s speech, varying with frequency.
Table 13: Results of continuation task and the inference time of different Mel- VAEs.
Model
WER
CER
SIM-ot
Inference Time.
ByT5 ByT5-base-de -base-d88 (default)
2.79
1.00
0.3879
3.46s
ByT5-
4.56
2.32
0.4117
6.87s
ByT5-base-df16
3.20
1.19
0.3629
1.88s
D.4 EFFECTS OF DATA SCALE
The results in Tab. 14 show that the performance of our model improves with an increase in the volume of training data. Specifically, we trained ByT5-base models on different sizes of training datasets: 1. (small dataset) 50% of the MLS English subset; 2. (normal dataset) the full MLS English subset; and 3. (large dataset) comprehensive English dataset that includes MLS English subset, Gigaspeech. LibriTTS-R, VCTK, and LISpeech. We observed a noticeable degradation in perfor- mances (WER, CER, and speaker similarity) when the small dataset was employed. Conversely, training on the large dataset resulted in a performance enhancement in speaker similarity while maintaining WER and CER compared to the model trained on the normal dataset. This evidence strongly indicates that scaling up training data can significantly boost model's performance.
23

---

Published as a conference paper at ICLR 2024
Table 14: Results of continuation task of different dataset sizes.
Dataset
WER
CER
SIM-o
Small (22K hr)
3.06
1.15
0.3851
Normal (44K hr)
2.79
1.00
0.3879
Large (55K hr)
2.79
0.98
0.4001
undeskinngt
Figure 2: The codebook usages of the probabilistic RVQ and prior RVQ method during training. The code book usages are plotted for the Oth, 8th, 16th, 24th, and 31st depths, from left to right.
24


Reviews:
[{"review_id": "G25YadHSD9", "reviewer": "Reviewer_w2yt", "summary": "This paper introduces CLaM-TTS, a novel text-to-speech synthesis model. CLaM-TTS combines several ideas from the literature into a single model trained on several large-scale datasets and demonstrates high-quality speech generation. The input to this model is text and the output is mel spectrograms, which must then be converted to audio waveforms using a vocoder.\n\nThe model consists of the following components:\n\n1. Audio quantizer: They introduce Mel-VAE, a model for encoding mel spectrograms into a sequence of multi-dimensional discrete codes. The Mel-VAE decoder is used to convert the discrete codes back into mel spectrograms during inference.\n2. Autoregressive synthesis network: A conditional language model is used to convert text into discrete codes. The language model is conditioned upon text (encoded by a pretrained network) and autoregressively predicts discrete audio codes. \n\nThe audio quantizer is based on residual vector quantization (RVQ). Unlike traditional language models which use a softmax output layer to predict discrete codes, the CLaM-TTS synthesis network instead uses a Gaussian mixture model (GMM) in order to predict a continuous output, which is then converted into a discretized output using the Mel-VAE RVQ quantizer.\n\nThis system is then trained on 100k hours in multiple languages and evaluated on a number of metrics, including speaker similarity, MOS, and ASR-based intelligibility and WER, demonstrating competitive performance.", "questions": "Does scaling the model to 100k hours improve model performance? It is unclear from the paper if this is actually helpful. Similarly, does scaling up the model improve performance? Although scaling characteristics are not the main purpose of the paper, it would be clearer if there were some comparisons with smaller and larger models, to demonstrate that the model is able to take advantage of the scale of the training data.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper has clear and detailed presentation. The models are explained clearly, referencing prior literature appropriately while providing sufficient detail for readers unfamiliar with parts of it to understand the paper. Hyperparameters are stated in the appendix and reasoning for the architectural choices is included.\n\nThe key contribution of this paper, in my opinion, consists of using a Gaussian mixture model as the output distribution of the language model, using the RVQ quantizer to convert this output to a set of discrete codes at inference time. This approach elegantly avoids having to build a conditional multi-level sampling scheme, which is expensive at inference time. \n\nThe baselines used for this paper are recent and strong, and the metrics used to evaluate the result are also thorough and varied. Additionally, the authors highlight both successes and failures of their method, including baselines which outperform their model on certain metrics.", "weaknesses": "One of the core claims of the paper has to do with two-stage pipelines for coarse-to-fine-grained audio token generation: \"A shared characteristic among these neural codec language models is their two-stage pipeline; they autoregressively generate coarse-grained audio tokens and decode them into fine- grained representations .... we design a language model that generates from coarse to fine-grained tokens without needing a two-stage pipeline.\"\n\nThe paper makes the claim that the continuous modeling approach they apply avoids this and yields improved quality. However, there is no side-by-side comparison which shows that. To make this paper stronger, there needs to be a comparison of CLaM-TTS with and without the two stage system. This comparison would demonstrate their point effectively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6QvlIbtfLd", "reviewer": "Reviewer_yc4a", "summary": "This paper proposes Clam-TTS, a new method that learns discrete latent codecs at a much lower frequency of 10hz compared to common codecs at 50-100hz. To do so, authors propose training a Mel-VAE model that predicts several \"RQ\" codes at once. Once this part of the network is trained, the authors propose to train latent language model with a gaussian mixture based decoder that attempt to sample codes that would match the ground truth. The codes are then passed through a decoder which outputs mel spectrograms. The paper shows that this approach can yield good performance comparable to SOTA methods on continuations and cross-sentence speech generation.", "questions": "- One of the central claims of this paper is that it manages to describe speech with discrete tokens at much lower frequency than many other approaches. While there is discussion about how lower frequency may lead to worse performance, it would be nice to see empirical ablations study of how frequency affects accuracy and performance of the system\n- why are mel spectrograms necessary to obtain the desired compression? can you not progressively downsample raw wave forms to acheive the same effect?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "- Interesting approach to generating low frequency residual vector codes suitable to modeling in language models\n- good performance on a variety of tasks\n- appears to be grounded in theory", "weaknesses": "- voicebox significantly outperforms this work. while this not necessarily detracts from the contributions of this work, the fact that it includes duration predictions and works on phones is not a great excuse as many other tts systems use this recipe\n- the paper is quite dense and difficult to follow. claims such as parallel generation of multiple tokens are seen in abstract and conclusion but finding how this is actually done is not straight forward and the description of \"parallel predictors\" is obtuse.\n- Missing references: one of the first speech discretization and speech vector quantization techniques was proposed in Vq-wav2vec (https://arxiv.org/abs/1910.05453)\n- section 5.2: ablation studty -> ablation study\n\noverall my suggestion is to clean up the presentation to make this paper more accessible and then I believe it would be a useful contribution to the community", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "emUNSvczbT", "reviewer": "Reviewer_ZbF5", "summary": "In this paper, the authors propose a TTS model(CLaM-TTS) with a probabilistic residual vector quantization. They achieve superior compression in token length and allow model to generate multiple tokens at once. They evaluate the proposed CLaM-TTS in continuation and cross sentence tasks on English and Multilingual respectively. The experimental results show the model can bring clear improvements compared to the base RVQ and show its ability on both tasks.", "questions": "/", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.The paper is clear and well written. The article also offers a comprehensive mathematical analysis of using detailed mathematical derivation to illustrate the probabilistic residual vector quantization of the article.\n\n2.The paper completed a good ablation experiment, not only verified the speech quality produced with the RVQ used in the model, but also make comparison of different T5 model to illustrate the importance of pretrained language model.", "weaknesses": "1. Novelty. The VAE or latent diffusion models are not something new, and thus the overall novelty could be questioned.\n\n2. Experimental comparison. The authors mostly compare their work with the outdated model YourTTS in English Librispeech, or without comparison in multilingual TTS. Thus, some competing models are better to include. The demos are too limited, it is recommended to include demo audios from other work for comparison.\n\n3. Results. In most tasks, the model cannot achieve state-of-the-art results but claims that in the paper, and thus the performance can be questioned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xIVflABTxP", "reviewer": "Reviewer_CLFG", "summary": "This work presents a language model based approach to generate multiple neural codec tokens for zero-shot text-to-speech. This work proposed a probabilistic residual vector quantizer, which allows the language model to generate multiple neural codes at the same time, which considerably reduces the sequence length in modeling and speed up the inference time. The results demonstrate the effectiveness of the proposed method, in terms of naturalness, intelligibility and speaker similarity.", "questions": "What's y and \\hat{y} in Eq(7)? It looks like it is never defined?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper proposed a probabilistic RVQ quantizer, which enables the language model to emit multiple neural codes at each inference step. This address the long inference sequence issue in the previous methods like Vall-E and audioLm. The authors have provided a detailed derivation of how the RVQ and latent language modeling, which is very helpful to understand this method.", "weaknesses": "It is not very clear to me why the probabilistic RVQ can lead to much compressed token sequence. For example, audiolm uses the 50Hz semantic token, while this work use a 10Hz codeword rate. It would be great if the author can explain or provide experimental results showing the effect of codeword emitting frequency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mUrEB7hAMa", "reviewer": "Reviewer_D6DJ", "summary": "This paper focuses on the LLM-based zero-shot TTS task. The authors introduce CLaM-TTS, which uses probabilistic residual vector quantization for two main benefits: it significantly compresses token length and enables the generation of multiple tokens simultaneously, eliminating the need for cascaded modeling. Experimental results show that CLaM-TTS performs on par with or better than existing zero-shot TTS baselines in terms of naturalness, intelligibility, speaker similarity, and inference speed. The study also explores how the pretraining extent of language models and their text tokenization strategies affect performance.", "questions": "- It's mentioned by the authors that different codeword rates can be compared. Including a detailed analysis of the tradeoff between RTF and performance in relation to varying codeword rates would greatly enhance the paper. Such an analysis would provide a clearer understanding of how changes in codeword rate impact both the efficiency and effectiveness of the model, offering valuable insights for optimizing model performance in practical applications.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper is commendably clear and straightforward, making it accessible to its readers. All the training details are meticulously presented, contributing to the paper's clarity and ease of comprehension.\n\n- While the paper adheres to the conventional structure of token + language modeling (LM), the introduction of the CLaM approach marks a significant innovation. CLaM addresses two critical issues in the field, demonstrating sufficient novelty and contributing meaningfully to the existing body of research.\n\n- The experiments conducted are comprehensive, effectively demonstrating the efficacy of the proposed methods. The completeness of these experiments lends credibility to the findings and supports the conclusions drawn in the paper.\n\n- In terms of RTF, the CLaM-TTS model exhibits superior speed compared to the Voicebox and Vall-E TTS models. This enhanced performance highlights the advantages of the proposed approach.", "weaknesses": "It's commendable that the paper is well-organized and well-written, making it accessible and comprehensible to its audience. I don't have a comment on the weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)