VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers

## Abstract

Abstract This paper introduces VALL-E 2, the latest advancement in neural codec language models that marks a milestone in zero-shot text-to-speech synthesis (TTS), achieving human parity for the first time .
Based on its predecessor, VALL-E, the new iteration introduces two significant enhancements: Repetition Aware Sampling refines the original nucleus sampling process by accounting for token repetition in the decoding history. It not only stabilizes the decoding but also circumvents the infinite loop issue. Grouped Code Modeling organizes codec codes into groups to effectively shorten the sequence length, which not only boosts inference speed but also addresses the challenges of long sequence modeling.
Our experiments on the LibriSpeech and VCTK datasets show that VALL-E 2 surpasses previous systems in speech robustness, naturalness, and speaker similarity. It is the first of its kind to reach human parity on these benchmarks. Moreover, VALL-E 2 consistently synthesizes high-quality speech, even for sentences that are traditionally challenging due to their complexity or repetitive phrases. The advantages of this work could contribute to valuable endeavors, such as generating speech for individuals with aphasia or people with amyotrophic lateral sclerosis.
See https://aka.ms/valle2 for demos of VALL-E 2.

## 1 Introduction

Text-to-speech synthesis (TTS) aims to generate high-quality speech from text input with a high degree of clarity and intelligibility.
Along with the progress of deep learning, significant improvements have been made in TTS research in recent years (Shen et al., ; Li et al., ; Ren et al., ). Some systems, trained with clean single-speaker speech data recorded in sound-recording studios, have even achieved human-level quality for single-speaker speech generation (Tan et al., ).
However, zero-shot TTS, which requires the model to synthesize speech for unseen speakers using a short enrolled speech sample during inference, remains a challenging problem.

Our previous work, VALL-E (Wang et al., ), marked a significant breakthrough in this area. It is capable of synthesizing personalized speech using only a 3-second recording, while preserving the speaker’s voice, emotion, and acoustic environment. VALL-E is a neural codec language model that represents speech signals as discrete codec codes with a neural audio codec model. Specifically, it trains an autoregressive language model to generate the coarse codec codes and another non-autoregressive model to generate the remaining fine codec codes.
Instead of using greedy search, which continually generates silence codec codes, VALL-E uses random sampling for model inference.
However, VALL-E has two key limitations:
1) Stability: The random sampling used during inference can lead to instability in output, while nucleus sampling with a small top-p value may cause an infinite loop issue. This can be mitigated by multiple-time sampling and subsequent sorting, but this approach increases the computational cost.
2) Efficiency: The autoregressive architecture of VALL-E is bound to the same high frame rate as the off-the-shelf audio codec model, which cannot be adjusted, resulting in a slower inference speed.

Several follow-up works have been proposed to address these problems (Song et al., ; Xin et al., ; Borsos et al., ; Le et al., ; Ju et al., ). To improve stability, some works leverage text-speech alignment information in model training and inference (Song et al., ; Xin et al., ). These methods, relying on a forced-alignment model, inevitably introduces errors in the alignment result, which could affect the final performance. It also complicates the overall architecture and increases the burden for data scaling up. To improve modeling efficiency, some works explore fully non-autoregressive methods for zero-shot TTS (Borsos et al., ; Le et al., ; Ju et al., ). However, these methods require frame-aligned text-speech data for model training, facing the same problem as discussed before. Additionally, the non-autoregressive model generates the tokens with a pre-determined duration result, which constrains the search space of the generated speech and sacrifices the prosody and naturalness.

In this work, we propose VALL-E 2, the first human parity zero-shot text-to-speech synthesis system. Building upon its predecessor VALL-E, VALL-E 2 employs a neural codec language modeling method for speech synthesis and incorporates two key modifications: repetition aware sampling and grouped code modeling.
Repetition aware sampling, an improvement over the random sampling used in VALL-E, adaptively employs either random or nucleus sampling for each time step token prediction. This selection is based on the token repetition in the decoding history, enhancing the stability of the decoding process and circumventing the infinite loop issue encountered in VALL-E.
Grouped code modeling, on the other hand, partitions the codec codes into groups, each of which is modeled in a single frame in the AR modeling process. This approach not only accelerates inference by reducing the sequence length but also improves performance by mitigating the long context modeling problem.
Notably, VALL-E 2 requires only simple utterance-wise speech-transcription pair data for training, greatly simplifying the process of collecting and processing training data and facilitating potential scalability.

VALL-E 2 is trained on the large-scale Libriheavy dataset (Kang et al., ). Subsequent evaluations demonstrate that it achieves performance on par with human capabilities on both the in-domain LibriSpeech dataset (Panayotov et al., ) and the out-of-domain VCTK datasets (Veaux et al., ). As illustrated in Figure 1, VALL-E 2 significantly outperforms VALL-E and other prior works on the LibriSpeech dataset in terms of robustness, naturalness, and similarity score, even achieving human parity performance. The numbers in Figure 1 are relative numbers ($\vartriangle\text{Score}(\text{Model})=\text{Score}(\text{Model})-\text{Score}
(\text{GroundTruth})$) based on the results reported in the paper. In this context, human parity indicates that the robustness, naturalness, and similarity metrics of VALL-E 2 surpass those of the ground truth samples (meaning that $\vartriangle\text{WERR}(\text{VALL-E 2})>0$, $\vartriangle\text{CMOS}(\text{VALL-E 2})>0$, and $\vartriangle\text{SMOS}(\text{VALL-E 2})>0$), meaning that VALL-E 2 can generate accurate, natural speech in the exact voice of the original speaker, comparable to human performance. It is important to note that this conclusion is drawn solely from experimental results on the LibriSpeech and VCTK datasets. Moreover, VALL-E 2 can accelerate the decoding process by multiple times with almost no performance degradation. To specifically evaluate the stability of VALL-E 2, we synthesize speech for complex sentences that are hard to read or contain many repeated phrases, and found that VALL-E 2 can always stably generate high-quality speech. The benefits of this work could support meaningful initiatives, such as generating speech for individuals with aphasia or people with amyotrophic lateral sclerosis.
We encourage the reader to listen to our samples on the demo page [https://aka.ms/valle2](https://aka.ms/valle2).

VALL-E 2 is purely a research project. Currently, we have no plans to incorporate VALL-E 2 into a product or expand access to the public. VALL-E 2 could synthesize speech that maintains speaker identity and could be used for educational learning, entertainment, journalistic, self-authored content, accessibility features, interactive voice response systems, translation, chatbot, and so on. While VALL-E 2 can speak in a voice like the voice talent, the similarity, and naturalness depend on the length and quality of the speech prompt, the background noise, as well as other factors. It may carry potential risks in the misuse of the model, such as spoofing voice identification or impersonating a specific speaker. We conducted the experiments under the assumption that the user agrees to be the target speaker in speech synthesis. If the model is generalized to unseen speakers in the real world, it should include a protocol to ensure that the speaker approves the use of their voice and a synthesized speech detection model. If you suspect that VALL-E 2 is being used in a manner that is abusive or illegal or infringes on your rights or the rights of other people, you can report it at the Report Abuse Portal.

## 2 Related Work

### 2.1 Zero-Shot TTS

Early work in zero-shot TTS typically employed speaker adaptation and speaker encoding methods, which often required additional fine-tuning, complex pre-designed features, or heavy structure engineering (Chen et al., ; Wang et al., ; Arik et al., ; Casanova et al., ).
Inspired by the success of Large Language Models (LLMs) in natural language processing, VALL-E (Wang et al., ; Zhang et al., ) represented speech as discrete codec codes with an off-the-shelf neural codec model, and approached TTS as a conditional codec language modeling task. This approach allowed VALL-E to train a codec language model on large-scale training data and perform zero-shot TTS via prompting, achieving significant zero-shot TTS capability.

This breakthrough inspired subsequent research works to address zero-shot TTS through a language modeling approach. For instance, VALL-E X (Zhang et al., ) extended VALL-E to cross-lingual TTS tasks with an additional language ID token. SPEAR-TTS (Kharitonov et al., ) and Make-a-voice (Huang et al., ) leveraged semantic units from a speech self-supervised model as an intermediate interface between text and acoustic codec codes, enabling better training data efficiency. Mega-TTS (Jiang et al., ) and Mega-TTS 2 (Jiang et al., ) proposed to first disentangle the multiple attributes in speech, then only model partial attributes with a language modeling approach. ELLA-V (Song et al., ) and RALL-E (Xin et al., ) improved VALL-E’s robustness and stability by including speech-text alignment prediction into the decoding process. UniAudio (Yang et al., ) and BASE TTS (Łajszczak et al., ) further explored scaling the codec language model to 1b parameters and 100k hours of training data.

Meanwhile, other works explored fully non-autoregressive modeling methods to accelerate the inference speed. For example, Soundstorm (Borsos et al., ) leveraged the confidence-based parallel decoding scheme (Chang et al., ) to generate the acoustic codec codes with a non-autoregressive model. StyleTTS 2 (Li et al., ), UniCATS (Du et al., ), NaturalSpeech 2 (Shen et al., ) and NaturalSpeech 3 (Ju et al., ) used diffusion model (Ho et al., ) for the prompt-conditioned text to speech synthesis. Voicebox (Le et al., ) and Audiobox (Vyas et al., ) used flow-matching method (Lipman et al., ) and achieved better speech modeling capability.
In this work, VALL-E 2 follows the codec language modeling method of VALL-E, and enables a stable decoding process without the need for complex speech data processing and preparation, such as duration or pitch information used in previous methods. Notably, VALL-E 2 is the first to successfully achieve human parity in zero-shot TTS on both LibriSpeech and VCTK datasets.

### 2.2 Codec-based Speech Models

Inspired by the promising performance of neural codec codes in zero-shot TTS, many subsequent research works have started to explore its effectiveness on more speech tasks. For instance, PolyVoice (Dong et al., ) adopted VALL-E and built a codec-based language model for speech-to-speech translation. SpeechX (Wang et al., ) extended VALL-E with multi-task learning, demonstrating efficacy in zero-shot TTS, noise suppression, target speaker extraction, speech removal, and speech editing tasks.
In addition to speech generation, VioLA (Wang et al., ) further explored codec-based speech models for speech understanding tasks, unifying codec language models for speech recognition, synthesis, and translation tasks. AudioPaLM (Rubenstein et al., ) fused the codec tokens into the LLM PaLM 2 (Anil et al., ), and demonstrated promising results on speech recognition and translation tasks.

These works typically employ SoundStream (Zeghidour et al., ) and Encodec (Défossez et al., ), initially designed for speech compression, as the neural codec model. Inspired by these successes, several works have proposed more novel neural codecs specifically for speech processing tasks. These include Vocos (Siuzdak, ), SpeechTokenizer (Zhang et al., ), AudioDec (Wu et al., ), AcademiCodec (Yang et al., ), Descript-audio-codec (DAC) (Kumar et al., ), FunCodec (Du et al., ), and RepCodec (Huang et al., ). The Codec-SUPERB challenge (Wu et al., ) was announced to benchmark various codec codes across a wide range of speech tasks. In this work, we utilize the Encodec model to tokenize speech signals and the Vocos decoder to generate target high-quality speech signals.

## 3 VALL-E 2

### 3.1 Problem Formulation: Grouped Codec Language Modeling

Following VALL-E, we use an off-the-shelf neural audio codec model to represent speech signals as discrete codec code sequence, and regard TTS as a conditional codec language modeling task.
To improve the efficiency, VALL-E 2 introduce a grouped codec language modeling method, where we partition the codec code sequence into groups of a certain size, and model each group of codec codes as one frame.
In this way, we can get rid of the frame rate constraint of the off-the-shelf neural audio codec model, and reduce the frame rate by integer multiples. It is not only beneficial for the inference efficiency but also the overall speech quality by mitigating the long context modeling problem.

With TTS training objective, VALL-E 2 is optimized to maximize the likelihood of the grouped code sequence given the text condition.
Specifically, given an audio sample $\mathbf{y}$ and its corresponding tokenized text transcription $\mathbf{x}=[x_{0},x_{1},\ldots,x_{(L-1)}]$, where $L$ is the text sequence length, we first use a pre-trained neural audio codec model to convert the audio sample $\mathbf{y}$ into a codec code sequence $\mathbf{C}^{T\times J}=[\mathbf{c}_{0},\mathbf{c}_{1},\ldots,\mathbf{c}_{(T-1)}]$, where $T$ is the code sequence length, $J$ (here $J=8$) is the number of the quantizers in the codec model, and each $\mathbf{c}_{t}$ represents the 8 codes for each time step.
Then we partition it into the grouped code sequence $\mathbf{C}^{G}=[\mathbf{C}_{0:G},\mathbf{C}_{G:2G},\ldots,\mathbf{C}_{(T-G):T}]$ with the group size $G$, and $\mathbf{C}_{0:G}$ stands for the group $[\mathbf{c}_{0},\mathbf{c}_{1},\ldots,\mathbf{c}_{(G-1)}]$.
Due to the typical short silence at the start of an utterance, we can clip a few codes from the start of the code sequence to let the code sequence length $T$ be the integer multiple of the group size without removing any speech information.
Finally, we train the VALL-E 2 model $\theta$ to minimize the negative log-likelihood of the grouped code sequence $\mathbf{C}^{G}$ conditioned on the text sequence $\mathbf{x}$:

$$ $\displaystyle\mathcal{L}$ $\displaystyle=-\log p(\mathbf{C}^{G}|\mathbf{x};\theta)$ (1) $\displaystyle=-\sum_{t=0}^{T/G-1}\log p(\mathbf{C}_{t\cdot G:(t+1)\cdot G}| \mathbf{C}_{<t\cdot G},\mathbf{x};\theta),$ (2) $$

where $\mathbf{C}_{t\cdot G:(t+1)\cdot G}$ is the $t$-th group of codec codes $[\mathbf{c}_{t\cdot G},\ldots,\mathbf{c}_{\left(\left(t+1\right)\cdot G-1
\right)}]$, and $\mathbf{C}_{<t\cdot G}$ is all the codec codes in the previous $(t-1)$ groups.

During inference, VALL-E 2 performs zero-shot TTS task via prompting. Given a text input (containing both the transcription of speech prompt and the text to synthesis) and grouped codec codes from an unseen speaker, serving as the condition and prompt, the model can generate the target grouped codec codes with the corresponding content and speaker’s voice.
Specifically, given the text sequence $\mathbf{x}$ and the enrolled speech sample of the unseen speaker $\mathbf{y}^{\prime}$, we can obtain the corresponding grouped code sequence $\mathbf{C}^{P}=\mathbf{C}^{G}_{<T^{\prime}}=[\mathbf{C}_{0:G},\mathbf{C}_{G:2G
},\ldots,\mathbf{C}_{(T^{\prime}-G):T^{\prime}}]$.
Then, We generate the target grouped code sequence $\mathbf{C}^{T}=\mathbf{C}^{G}_{\geq T^{\prime}}=[\mathbf{C}_{T^{\prime}:(T^{
\prime}+G)},\ldots,\mathbf{C}_{(T-G):T}]$ conditioned on the text sequence $\mathbf{x}$ and code prompt $\mathbf{C}^{P}$:

$$ $\displaystyle\mathbf{C}^{T}$ $\displaystyle=\operatorname*{arg\,max}_{\mathbf{C}}p(\mathbf{C}|\mathbf{C}^{P} ,\mathbf{x};\theta)$ (3) $\displaystyle=\operatorname*{arg\,max}_{\mathbf{C}}\sum_{t=T^{\prime}/G}^{T/G- 1}\log p(\mathbf{C}_{t\cdot G:(t+1)\cdot G}|\mathbf{C}_{<t\cdot G},\mathbf{x}; \theta).$ (4) $$

Finally, we can convert the target code sequence $\mathbf{C}^{T}$ to the target speech waveform using an off-the-shelf neural codec decoder.

### 3.2 VALL-E 2 Architecture

Building upon VALL-E,VALL-E 2 also use a hierarchical structure: an Autoregressive (AR) codec language model and a Non-Autoregressive (NAR) codec language model.
The AR model generates sequence of the first codec code for each frame in an autoregressive manner, while the NAR model generates each remaining code sequence based on the preceding code sequences in a non-autoregressive manner.
Both models utilize the same Transformer architecture with a text embedding layer, a code embedding layer, and a code prediction layer. We use distinct embeddings for the codes from different codec quantizers and share the parameters of the code prediction layer with the parameters of the code embedding layer.
In addition, the AR model has a group embedding layer to project the code embedding to the group embedding, and a group prediction layer for the prediction of codes in one group .
The NAR model has a code ID embedding layer to specify the ID of the code sequence to predict.
The AR model and NAR model have different attention mask strategies: the AR model uses the causal attention strategy and the NAR model uses the full attention strategy, as shown in the right part of Figure 2.

Figure: Figure 2: Training overview of VALL-E 2, consisting of an autoregressive and a non-autoregressive Transformer. Note that the autoregressive Transformer is designed to generate grouped codec codes.
Refer to caption: x2.png

### 3.3 VALL-E 2 Training

Figure 2 shows the overview of VALL-E 2 model training. It is noteworthy that the training of VALL-E 2 requires only simple utterance-wise speech-transcription pair data, without any complex data such as force-alignment result or additional audio clips of the same speaker for reference. This greatly simplifies the process of collecting and processing training data.

Specifically, for each audio and corresponding transcription in the training dataset, we initially utilize the audio codec encoder and text tokenizer to obtain the codec codes $\mathbf{C}=[\mathbf{c}_{0},\mathbf{c}_{1},\ldots,\mathbf{c}_{(T-1)}]$ and the text sequence $\mathbf{x}=[x_{0},x_{1},\ldots,x_{(L-1)}]$, respectively. These are then used for the AR model and the NAR model training.

#### 3.3.1 Autoregressive Model Training

The AR model is trained to predict the first codec code sequence $\mathbf{c}_{:,0}=[c_{0,0},c_{1,0},\ldots,c_{(T-1),0}]$ conditioned on the text sequence $\mathbf{x}$ in an autoregressive manner.

As shown in the lower middle part of Figure 2, we first obtain the text embedding sequence $\mathbf{E}^{x}=[\mathbf{e}^{x}_{0},\mathbf{e}^{x}_{1},\ldots,\mathbf{e}^{x}_{(
L-1)}]$ and the code embedding sequence $\mathbf{E}^{c}=[\mathbf{e}^{c}_{0},\mathbf{e}^{c}_{1},\ldots,\mathbf{e}^{c}_{(
T-1)}]$ using the text embedding matrix $\mathbf{W}^{x}$ and the code embedding matrix $\mathbf{W}^{c}$.

$$ $\displaystyle\mathbf{e}^{x}_{l}$ $\displaystyle=\mathbf{W}^{x}\odot x_{l},$ (5) $\displaystyle\mathbf{e}^{c}_{t}$ $\displaystyle=\mathbf{W}^{c}\odot c_{t,0},$ (6) $$

where $l$ and $t$ denotes the indices of each item in the text sequence and code sequence, respectively, and $\odot$ denotes index selection.
Then, we partition the code embedding sequence into groups of size $G$, concatenate each group of the the code embeddings in the hidden dimension, and obtain the group embedding sequence $\mathbf{E}^{g}=[\mathbf{e}^{g}_{0},\mathbf{e}^{g}_{1},\ldots,\mathbf{e}^{g}_{(
T/G-1)}]$ using the group embedding matrix $\mathbf{W}^{g}$.

$$ $\displaystyle\mathbf{e}^{g}_{t}$ $\displaystyle=\mathbf{e}^{c}_{t\cdot G:(t+1)\cdot G}\cdot\mathbf{W}^{g}.$ (7) $$

We concatenate the text embedding sequence $\mathbf{E}^{x}$ and the group embedding sequence $\mathbf{E}^{g}$, inserting the embedding of special tokens $<\text{eos}>$ and $<\text{bos}>$ in between:

$$ $\mathbf{E}^{0}=\mathbf{E}^{x}\mathbin{\|}[\mathbf{e}_{<\text{eos}>},\mathbf{e} _{<\text{bos}>}]\mathbin{\|}\mathbf{E}^{g},$ (8) $$

where $||$ indicates concatenation in the temporal dimension.
We then separately add the learnable position embedding to the text embedding sequence and the group embedding sequence.
The AR model is fed with $\mathbf{E}^{0}$ and trained to predict corresponding code sequence with a special token $<\text{eos}>$ appended at the end using a linear mapping group prediction layer and softmax code prediction layer. Due to the causal attention mask strategy, the prediction of each code group $\mathbf{c}_{t\cdot G:(t+1)\cdot G,0}$ can only attend to the text sequence $\mathbf{x}$ and the preceding codes $\mathbf{c}_{<t\cdot G,0}$, as demonstrated in the lower right part of Figure 2.

Overall, the parameters $\theta_{\text{AR}}$ of the AR model is optimized by minimizing the negative log likelihood of the first code sequence $\mathbf{c}_{:,0}$ conditioned on the text sequence $\mathbf{x}$:

$$ $\displaystyle\mathcal{L}_{\text{AR}}$ $\displaystyle=-\log p(\mathbf{c}_{:,0}|\mathbf{x};\theta_{\text{AR}})$ (9) $\displaystyle=-\sum_{t=0}^{T/G-1}\log p(\mathbf{c}_{t\cdot G:(t+1)\cdot G,0}| \mathbf{c}_{<t\cdot G,0},\mathbf{x};\theta_{\text{AR}})$ (10) $\displaystyle=-\sum_{t=0}^{T/G-1}\sum_{t^{\prime}=t\cdot G}^{(t+1)\cdot G-1} \log p(c_{t^{\prime},0}|\mathbf{c}_{<t\cdot G,0},\mathbf{x};\theta_{\text{AR}}).$ (11) $$

In the AR model of VALL-E 2, the group sequence $\mathbf{c}_{:,0}=[\mathbf{c}_{0:G},\mathbf{c}_{G:2G,0},\ldots,\mathbf{c}_{(T-G
):T,0}]$ is modeled in an autoregressive approach, while the codec codes within each group $\mathbf{c}_{t\cdot G:(t+1)\cdot G,0}=[c_{t\cdot G,0},c_{(t\cdot G+1),0}\ldots,
c_{((t+1)\cdot G-1),0}]$ are modeled in a non-autoregressive way.

#### 3.3.2 Non-Autoregressive Model Training

Given the first code sequence generated by the AR model, the NAR model is trained to generate remaining code sequence $\mathbf{c}_{:,j}$ for each codec code ID $j$ conditioned on the text sequence $\mathbf{x}$ and the preceding code sequences $\mathbf{c}_{:,<j}$ in a non-autoregressive manner, where $j\in[1,\ldots,7]$.

As we have access to all 8 code sequences of the prompt during inference, to better model the speaker information of the prompt, during training, we explicitly split all the code sequences $\mathbf{C}$ into an acoustic condition $\mathbf{C}_{<T^{\prime}}$ and target code sequences $\mathbf{C}_{\geq T^{\prime}}$ with a randomly sampled length $T^{\prime}$. The model is then optimized to predict each target code sequence $\mathbf{c}_{\geq T^{\prime},j}$ conditioned on the text sequence $\mathbf{x}$, all $J=8$ code sequences in the acoustic condition $\mathbf{C}_{<T^{\prime}}$ and the preceding target code sequences $\mathbf{C}_{\geq T^{\prime},<j}$ in a non-autoregressive manner.

As shown in the upper middle part of Figure 2, we first obtain the text embedding sequence $\mathbf{E}^{x}=[\mathbf{e}^{x}_{0},\mathbf{e}^{x}_{1},\ldots,\mathbf{e}^{x}_{(
L-1)}]$ using the text embedding matrix $\mathbf{W}^{x}$, as denoted in Equation 5.
Then, we obtain the code embedding sequence $\mathbf{E}^{c}=[\mathbf{e}^{c}_{0},\mathbf{e}^{c}_{1},\ldots,\mathbf{e}^{c}_{(
T-1)}]$ by obtaining all the code embeddings in the acoustic condition $\mathbf{C}_{<T^{\prime}}$ and target code sequences $\mathbf{C}_{\geq T^{\prime},<j}$ with the code embedding matrix $\mathbf{W}^{c}$, and summing them along with the code ID dimension:

$$ $\mathbf{e}^{c}_{t}=\begin{cases}\sum_{k=0}^{7}\mathbf{W}^{c}\odot{c_{t,k}},&t< T^{\prime}\\ {\sum_{k=0}^{j-1}\mathbf{W}^{c}\odot{c_{t,k}},}&{t\geq T^{\prime}}\end{cases},$ (12) $$

where $t$ is the time step and $j$ is the codec code ID.
Next, we obtain the codec code ID embedding $\mathbf{e}^{j}$ with the code ID embedding matrix $\mathbf{W}^{id}$.

$$ $\mathbf{e}^{j}=\mathbf{W}^{id}\odot j.$ (13) $$

We concatenate the text embedding sequence $\mathbf{E}^{x}$, the code embedding sequence $\mathbf{E}^{c}$, and the codec code ID embedding $\mathbf{e}^{j}$, inserting the embedding of the special token $<\text{eos}>$ in the middle:

$$ $\mathbf{E}^{j}=\mathbf{E}^{x}\mathbin{\|}[\mathbf{e}_{<\text{eos}>}]\mathbin{ \|}\mathbf{E}^{c}\mathbin{\|}[\mathbf{e}_{<\text{eos}>}]\mathbin{\|}[\mathbf{e }^{j}].$ (14) $$

We then separately add the learnable position embedding to the text embedding sequence and the code embedding sequence, similar to the AR model.
The NAR model is fed with $\mathbf{E}^{j}$ and trained to predict the corresponding code sequence $\mathbf{c}_{:,j}$ for each codec code id $j$ using a code prediction layer. With the full attention mask strategy, the prediction of each token $c_{t,j}$ can attend to the entire input sequence, as depicted in the upper right part of Figure 2.

Overall, the NAR model is optimized by minimizing the negative log likelihood of each $j$-th target code sequence $\mathbf{c}_{\geq T^{\prime},j}$ conditioned on the text sequence $\mathbf{x}$, all the code sequences of the acoustic condition $\mathbf{C}_{<T^{\prime}}$ and the preceding $j$ target code sequences $\mathbf{c}_{\geq T^{\prime},<j}$.

$$ $\displaystyle\mathcal{L}_{\text{NAR}}$ $\displaystyle=-\log p(\mathbf{C}_{\geq T^{\prime},\geq 1}|\mathbf{x},\mathbf{C }_{<T^{\prime}},\mathbf{c}_{\geq T^{\prime},0};\theta_{\text{NAR}})$ (15) $\displaystyle=-\sum_{j=1}^{7}\log p(\mathbf{c}_{\geq T^{\prime},j}|\mathbf{x}, \mathbf{C}_{<T^{\prime}},\mathbf{C}_{\geq T^{\prime},<j};\theta_{\text{NAR}}).$ (16) $$

In practice, to optimize computational efficiency during training, we do not calculate the training loss by iterating over all values of $j$ and aggregating the corresponding losses, but randomly select a $j\in[1,\ldots,7]$ and optimize the model using the training loss:

$$ $\mathcal{L}_{\text{NAR_j}}=-\log p(\mathbf{c}_{\geq T^{\prime},j}|\mathbf{x}, \mathbf{C}_{<T^{\prime}},\mathbf{C}_{\geq T^{\prime},<j};\theta_{\text{NAR}}).$ (17) $$

### 3.4 VALL-E 2 Inference

Figure: Figure 3: Inference overview of VALL-E 2, which leverages the proposed repetition aware sampling method to predict grouped code sequence during autoregressive model inference.
Refer to caption: x3.png

Following VALL-E, we perform the zero-shot TTS task via prompting during inference.
As depicted in Figure 3, given the text sentence and the enrolled speech sample of the unseen speaker along with its corresponding transcription, we first concatenate the speech transcription and the text sentence, encoded into the text sequence $\mathbf{x}$ using the text tokenizer to serve as the text condition.
The speech sample is converted into the codes $\mathbf{C}^{P}=\mathbf{C}_{<T^{\prime}}=[\mathbf{c}_{0},\mathbf{c}_{1},\ldots,
\mathbf{c}_{(T^{\prime}-1)}]$ using the audio codec encoder to serve as the prompt.
By prompting the conditional codec language model, we infer the AR model and NAR model to generate the target codes $\mathbf{C}_{\geq T^{\prime}}=[\mathbf{c}_{T^{\prime}},\ldots,\mathbf{c}_{(T-1)}]$. Finally, the target codes is used by the audio codec decoder to synthesize the target personalized speech signals.

#### 3.4.1 Autoregressive Model Inference

We first infer the AR model to generate the first code sequence of the target codes $\mathbf{c}_{\geq T^{\prime},0}$ conditioned on the text sequence $\mathbf{x}$ and the code prompt $\mathbf{c}_{<T^{\prime},0}$. With the grouped codec language modeling method, we feed the grouped code sequence to the AR model and generate each group of target codes in an autoregressive way:

$$ $\displaystyle\mathbf{c}_{\geq T^{\prime},0}$ $\displaystyle=\operatorname*{arg\,max}_{\mathbf{c}_{\geq T^{\prime},0}}p( \mathbf{c}_{\geq T^{\prime},0}|\mathbf{x},\mathbf{c}_{<T^{\prime},0};\theta_{ \text{AR}})$ (18) $\displaystyle=\operatorname*{arg\,max}_{\mathbf{c}_{\geq T^{\prime},0}}\sum_{t =T^{\prime}/G}^{T/G-1}\log p(\mathbf{c}_{t\cdot G:(t+1)\cdot G,0}|\mathbf{x}, \mathbf{c}_{<t\cdot G,0};\theta_{\text{AR}})$ (19) $\displaystyle=\operatorname*{arg\,max}_{\mathbf{c}_{\geq T^{\prime},0}}\sum_{t =T^{\prime}/G}^{T/G-1}\sum_{t^{\prime}=t\cdot G}^{(t+1)\cdot G-1}\log p(c_{t^{ \prime},0}|\mathbf{x},\mathbf{c}_{<t\cdot G,0};\theta_{\text{AR}}).$ (20) $$

Figure: Algorithm 1 Repetition Aware Sampling in VALL-E 2 AR Model Decoding

Different from the random sampling method used in VALL-E, in this work, we propose a repetition aware sampling method to enhance nucleus sampling for the better decoding stability. As detailed in Algorithm 1, given the probability distribution $p(c_{t^{\prime}}|\mathbf{x},\mathbf{c}_{<t\cdot G,0};\theta_{\text{AR}})$ predicted by the AR model, we first generate the target code $c_{t^{\prime}}$ by nucleus sampling with a pre-defined top-p value $v$. Then, we calculate the repetition ratio $r$ of token $c_{t^{\prime}}$ in the preceding code sequence with a window size $K$. If the ratio $r$ exceeds a pre-defined repetition threshold ratio $t_{n}$, we replace the target code $c_{t^{\prime}}$ by random sampling from $p(c_{t^{\prime}}|\mathbf{x},\mathbf{c}_{<t\cdot G,0};\theta_{\text{AR}})$. Although the codec codes in one group are modeled in a non-autoregressive way, they are predicted autoregressively so as to calculate the repetition ratio $r$ and switch between these two sampling methods. With this repetition aware sampling method, the decoding process can not only benefit from the stability of nucleus sampling, but also avoid the infinite loop issue with the help of random sampling. It should be noted that this repetition aware sampling won’t increase the decoding latency since the runtime cost of the additional sampling operation is almost negligible compared to the model inference process.

#### 3.4.2 Non-Autoregressive Model Inference

Given the first code sequence of the target codes $\mathbf{c}_{\geq T^{\prime},0}$, we can infer the NAR model with the text condition $\mathbf{x}$ and the acoustic condition $\mathbf{C}_{<T^{\prime}}$ to generate the remaining code sequences of the target codes $\mathbf{C}_{\geq T^{\prime},\geq 1}$:

$$ $\displaystyle\mathbf{C}_{\geq T^{\prime},\geq 1}$ $\displaystyle=\operatorname*{arg\,max}_{\mathbf{C}_{\geq T^{\prime},\geq 1}}p( \mathbf{C}_{\geq T^{\prime},\geq 1}|\mathbf{x},\mathbf{C}_{<T^{\prime}}, \mathbf{c}_{\geq T^{\prime},0};\theta_{\text{NAR}})$ (21) $\displaystyle=\operatorname*{arg\,max}_{\mathbf{C}_{\geq T^{\prime},\geq 1}} \sum_{j=1}^{7}\log p(\mathbf{c}_{\geq T^{\prime},j}|\mathbf{x},\mathbf{C}_{<T^ {\prime}},\mathbf{C}_{\geq T^{\prime},<j};\theta_{\text{NAR}}).$ (22) $$

To generate the 2-8 code
sequence, we perform inference on the NAR model seven times, generating them one by one using a greedy decoding method. Together with the first codec codes generated by the AR model, the whole code matrix $\mathbf{C}_{\geq T^{\prime}}$ is used for generating the target personalized speech waveform with the corresponding audio codec decoder.

VALL-E 2 can not only use a reference utterance of an unseen speaker as prompt to generate the speech cloning his/her voice, but also be able to perform zero-shot speech continuation, in which, we use the complete transcription of the utterance as the text condition and the first 3-second prefix as the prompt for the target personalized speech generation.

## 4 Experiment

### 4.1 Setups

#### 4.1.1 Model Training

We use Libriheavy corpus (Kang et al., ) as the training data. This corpus is a labeled version of the Librilight corpus (Kahn et al., ) that contains 50k hours of speech with around 7000 distinct speakers derived from open-source English audiobooks that are part of the LibriVox project(^1^11[https://librivox.org](https://librivox.org)).
We use Byte-Pair Encoding (BPE) for text tokenization, and the pre-trained open-sourced EnCodec model (Défossez et al., ) at 6K bitrates for 24kHz audio reconstruction for speech tokenization. Additionally, we use the open-sourced pre-trained Vocos model (Siuzdak, ) as the audio codec decoder for speech generation.

Following VALL-E, both the AR model and the NAR models employ the same Transformer architecture in VALL-E 2.
In our experiments, we mainly evaluate 4 VALL-E 2 models, which share the same NAR model but different AR models.
The 4 AR models corresponds to the group size of 1, 2, 4 and 8.
Among these models, the AR model with group size of 1 is implemented without the group embedding layer and group prediction layer, and the baseline model VALL-E employs the same NAR model and AR model with group size of 1(^2^22We re-train the baseline VALL-E model with the Libriheavy dataset for fair comparison.).

Both the AR and NAR models are trained using 16 NVIDIA TESLA V100 32GB GPUs. The models are optimized with the AdamW optimizer, with the learning rate warmed up for the first 32k updates to a peak of learning rate, then linearly decayed.
For NAR model training, the length of the acoustic condition is randomly sampled to be the maximum of half of the current utterance with a random value from 3s to 30s.

#### 4.1.2 Evaluation Metrics

We employ subjective evaluation metrics, including SMOS and CMOS, to assess the speaker similarity and comparative naturalness of synthesized speech, respectively. We invite 20 external native speakers of American English to participate as contributors in a crowdsourcing effort to evaluate each speech from various perspectives.

SMOS (Similarity Mean Opinion Score) is used to evaluate the speaker similarity of the speech to the original prompt. The SMOS scale ranges from 1 to 5, with increments of 0.5 points.

CMOS (Comparative Mean Opinion Score) is used to evaluate the comparative naturalness of the synthesized speech against a given reference speech. The CMOS scale ranges from -3 (indicating the synthesized speech of the new system is much worse than the reference) to 3 (indicating the new system is much better than the reference), with intervals of 1. In our study, we use the ground truth speech as the comparison reference.

We also employ objective evaluation metrics including SIM, WER, and DNSMOS to assess the speaker similarity, robustness, and overall perceived quality of each synthesized speech.
For a better comparison in speech continuation, we evaluate the entire utterance instead of focusing solely on the continuation segment.

SIM is used to evaluate the speaker similarity between the original prompt and synthesized speech, leveraging the SOTA speaker verification model, WavLM-TDNN (^3^33We use the best speaker verification model released at [https://github.com/microsoft/UniSpeech/tree/main/downstreams/speaker_verification#pre-trained-models](https://github.com/microsoft/UniSpeech/tree/main/downstreams/speaker_verification#pre-trained-models)) (Chen et al., ). The similarity score predicted by WavLM-TDNN is in the range of $[-1,1]$, with a larger value indicating higher speaker similarity.

WER (Word Error Rate) is used to evaluate the robustness of synthesized speech. Neural TTS systems sometimes experience deletion, insertion, and replacement errors due to incorrect attention alignments, which can affect their robustness. We perform ASR on the generated audio and calculate the WER with respect to the original transcriptions. In this experiment, we employ the open-sourced Conformer-Transducer model(^4^44[https://huggingface.co/nvidia/stt_en_conformer_transducer_xlarge](https://huggingface.co/nvidia/stt_en_conformer_transducer_xlarge)) (Gulati et al., ) as the ASR model.

DNSMOS (Deep Noise Suppression Mean Opinion Score) is used to assess the overall perceived quality of the generated speech (Reddy et al., ). Specifically, we use a model trained with ground truth human ratings obtained using ITU-T P.808 (ITU, )(^5^55[https://github.com/microsoft/DNS-Challenge/tree/master/DNSMOS](https://github.com/microsoft/DNS-Challenge/tree/master/DNSMOS)) to predict the DNSMOS score, which is in the range of $[1,5]$, with a larger value indicating better quality.

#### 4.1.3 Evaluation Settings

We use LibriSpeech test-clean (Panayotov et al., ) and VCTK (Veaux et al., ) for zero-shot TTS evaluation, ensuring none of the speakers from these corpora are included in the training data.

LibriSpeech test-clean is an official test split from the LibriSpeech corpus, containing English speech sampled at 16kHz. It originates from the same domain of the LibriVox project as the training data but features different speaker IDs. Following Borsos et al. () and Wang et al. (), we use samples from LibriSpeech test-clean with lengths between 4 and 10 seconds, resulting in a 2.2 hours subset and 40 unique speakers. We evaluate each sample synthesis under two settings: 3s Prefix as Prompt and Ref Utterance as Prompt. For the first setting, we perform speech continuation and utilize the 3-second prefix of the speech as the prompt. In the second setting, we use a reference utterance from the same speaker as the prompt. Specifically, we begin by filtering the official speech list of LibriSpeech test-clean based on length. For the ordered speech list of each speaker, in the first setting, we synthesize the $i$-th speech sample using the first 3 seconds of the ground-truth $i$-th speech sample as the prompt. In the second setting, we synthesize the $i$-th speech sample using the $(i-1)$-th sample as the prompt and synthesize the first speech sample using the last sample as the prompt.

VCTK is a reading corpus with speech sampled at 48kHz by 108 English speakers.
Compared to LibriSpeech, VCTK presents a greater challenge as it encompasses speakers with a wide range of accents.
We evaluate each sample synthesis under three settings: using prompts of 3s, 5s, and 10s in length.
Specifically, for each speaker, we select an utterance whose length is closest to but less than 3s/5s/10s to serve as the prompts. We then randomly sample another utterance and use the corresponding transcription as the text input for speech synthesis.

For each sample synthesis, we first perform inference with the AR model to generate the first code sequence using the repetition aware sampling method (Section 3.4.1), where we set the hyperparameter $K=10$, $t_{r}=0.1$, and select the top-p value $v$ from $0.0$ to $0.8$ with the intervals of $0.1$.
Next, we perform inference on the NAR model seven times to generate the remaining seven code sequences using a greedy decoding method.
The sampling-based decoding method of the AR model allows us to generate diverse samples from the same input. In our experiment, we report the results of sampling once and five times for each speech synthesis. For the five-time sampling, we report the results of sorting on SIM and WER, and metric-wise maximization.

Sorting on SIM and WER: We sort the samples based on the speaker similarity and robustness scores, represented by the SIM and WER scores. Specifically, given the five samples $\{\hat{\mathbf{y}}_{i}\}_{i=1}^{5}$ with the corresponding SIM, WER, and DNSMOS scores denoted as $\hat{\mathbf{y}}_{i}^{\text{SIM}}$, $\hat{\mathbf{y}}_{i}^{\text{WER}}$, and $\hat{\mathbf{y}}_{i}^{\text{DNSMOS}}$, we sort them according to the WER score if the SIM score is greater than 0.3 and sort according to the SIM score otherwise. This sorting method can be expressed as:

$$ $\hat{\mathbf{y}}_{\text{best}}=\operatorname*{arg\,max}_{\hat{\mathbf{y}}_{i}} ([\min(\hat{\mathbf{y}}_{i}^{\text{SIM}},0.3),1-\hat{\mathbf{y}}_{i}^{\text{ WER}}]),$ (23) $$

where $\max(\cdot)$ denotes finding the lexicographically largest array (^6^66Lexicographic order: given two partially ordered sets $A$ and $B$, the lexicographical order on the Cartesian product $A\times B$ is defined as $[a,b]\leq[a^{\prime},b^{\prime}]$ if and only if $a<a^{\prime}$ or ($a=a^{\prime}$ and $b\leq b^{\prime}$).).
The resulting SIM, WER, and DNSMOS scores are $\hat{\mathbf{y}}_{\text{best}}^{\text{SIM}}$, $\hat{\mathbf{y}}_{\text{best}}^{\text{WER}}$ and $\hat{\mathbf{y}}_{\text{best}}^{\text{DNSMOS}}$.

Metric-Wise Maximization: We report the best score each system can achieve if we optimize only the value of the corresponding metric. In this case, the resulting SIM, WER, and DNSMOS scores are $\max(\hat{\mathbf{y}}_{i}^{\text{SIM}})$, $\max(\hat{\mathbf{y}}_{i}^{\text{WER}})$, and $\max(\hat{\mathbf{y}}_{i}^{\text{DNSMOS}})$.

### 4.2 LibriSpeech Evaluation

#### 4.2.1 Objective Evaluation

**Table 1: Objective evaluation results on LibriSpeech test-clean.**
| System | GroupSize |  | 3s Prefix as Prompt |  | Ref Utterance as Prompt |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SIM$\uparrow$ | WER$\downarrow$ | DNSMOS$\uparrow$ |  | SIM$\uparrow$ | WER$\downarrow$ | DNSMOS$\uparrow$ |  |  |
| GroundTruth | - |  | 0.905 | 1.6 | 3.891 |  | 0.779 | 1.6 | 3.891 |
| $\;\;\;\hookrightarrow$ Codec | - |  | 0.823 | 1.7 | 3.886 |  | 0.715 | 1.7 | 3.886 |
| Single Sampling |  |  |  |  |  |  |  |  |  |
| VALL-E | 13ms |  | 0.773 | 2.3 | 3.942 |  | 0.633 | 3.1 | 3.985 |
| VALL-E 2 | $\times 1$ |  | 0.782 | 1.6 | 3.947 |  | 0.643 | 1.5 | 3.987 |
| $\times 2$ |  | 0.777 | 1.5 | 3.966 |  | 0.635 | 1.5 | 4.000 |  |
| $\times 4$ |  | 0.773 | 1.8 | 3.950 |  | 0.615 | 2.2 | 3.967 |  |
| $\times 8$ |  | 0.766 | 2.5 | 3.937 |  | 0.566 | 4.2 | 3.875 |  |
| Five-Time Sampling (Sort on SIM and WER) |  |  |  |  |  |  |  |  |  |
| VALL-E | 13ms |  | 0.802 | 1.0 | 3.944 |  | 0.676 | 0.8 | 3.987 |
| VALL-E 2 | $\times 1$ |  | 0.807 | 1.0 | 3.943 |  | 0.687 | 0.7 | 3.994 |
| $\times 2$ |  | 0.803 | 1.0 | 3.967 |  | 0.679 | 0.6 | 3.997 |  |
| $\times 4$ |  | 0.799 | 1.1 | 3.954 |  | 0.662 | 0.7 | 3.973 |  |
| $\times 8$ |  | 0.790 | 1.0 | 3.938 |  | 0.616 | 1.0 | 3.898 |  |
| Five-Time Sampling (Metric-Wise Maximization) |  |  |  |  |  |  |  |  |  |
| VALL-E | 13ms |  | 0.806 | 1.0 | 4.055 |  | 0.686 | 0.7 | 4.124 |
| VALL-E 2 | $\times 1$ |  | 0.809 | 1.0 | 4.042 |  | 0.691 | 0.6 | 4.116 |
| $\times 2$ |  | 0.805 | 1.0 | 4.059 |  | 0.683 | 0.6 | 4.130 |  |
| $\times 4$ |  | 0.802 | 1.1 | 4.046 |  | 0.669 | 0.7 | 4.105 |  |
| $\times 8$ |  | 0.795 | 1.0 | 4.035 |  | 0.630 | 1.0 | 4.041 |  |

Table 1 presents the objective evaluation results on the LibriSpeech test-clean dataset, where VALL-E 2 significantly outperforms VALL-E in all settings, even achieving better WER and DNSMOS scores than the ground truth speech with single sampling.

The SIM, WER, and DNSMOS scores of the ground truth speech are calculated as the upper bound.
We observe a performance degradation in SIM and similar performance in WER and DNSMOS when using the off-the-shelf neural audio codec model for speech reconstruction.
The baseline VALL-E can achieve impressive overall results with five-time sampling, but lack of robustness with single sampling, which could be attributed to the instability decoding process of random sampling.

In comparison, VALL-E 2 demonstrates significant improvement in robustness, especially in the single sampling scenario.
With the repetition aware sampling, VALL-E 2 can successfully achieve better decoding stability, leads to the performance improvement in all the three metrics, and even obtain lower WER score than the ground truth speech. It indicates that our synthesized speech is highly faithful to the provided text and enrolled speech.

With the grouped code modeling, VALL-E 2 can achieve even better WER and DNSMOS scores with group size of 2 in the AR model. It demonstrates that this method can not only improve the inference efficiency by reducing the code sequence length, but also improve the model performance by mitigating the long context modeling problem. Even with group size of 4, we can still obtain similar or better results as the baseline model while greatly improve the inference efficiency by reducing the code sequence length by 4 times.
Figure 4 further demonstrates the superior decoding stability of VALL-E 2. The repetition aware sampling method significantly enhances the decoding stability, regardless of the different group size setting. It enables VALL-E 2 to perform inference with a very small top-p (even 0), which tends to introduce much less errors and generate more robust speech codec codes than decoding with a large top-p. This is the key to obtaining a good WER score, even lower than that of ground truth speech, using a small top-p.

Figure: Figure 4: Decoding stability on LibriSpeech test-clean. GS means group size and RAS stands for repetition aware sampling.
Refer to caption: x4.png

#### 4.2.2 Subjective Evaluation

**Table 2: Subjective evaluation results for 40 speakers on LibriSpeech test-clean, using a reference utterance as a prompt for each speaker.**
| System | GroupSize |  | SMOS$\uparrow$ | CMOS$\uparrow$ |  |
| --- | --- | --- | --- | --- | --- |
| GroundTruth | - |  | 4.13±0.32 | 0.00 |  |
| VALL-E | 13ms |  | 4.45±0.28 | -0.268 |  |
| VALL-E 2 | $\times 1$ |  | 4.61±0.19 | 0.033 |  |
| $\times 2$ |  | 4.51±0.26 | -0.167 |  |  |

Table 2 presents the subjective evaluation results on the LibriSpeech test-clean. For the subjective evaluation, the previous utterance from the official speech list is used as the prompt to generate the current utterance for each speaker in the LibriSpeech test-clean dataset, resulting in 40 test cases.

As indicated in the table, VALL-E 2 can successfully surpasses VALL-E in terms of both speaker similarity SMOS and speech quality CMOS, even better performance than the ground truth speech. This suggests that our proposed method can achieve human parity zero-shot TTS performance in LibriSpeech benchmark.
With group code modeling method, VALL-E 2 can also achieve better performance than VALL-E with group size of 2 for the inference of AR model.

#### 4.2.3 Ablation Study

**Table 3: Ablation study of model input on LibriSpeech test-clean. The symbol ✦ denotes that the acoustic condition is not explicitly split during the NAR model training, and the prompt is treated as the prefix of the target code matrix during the NAR model inference.**
| AR Model |  | NAR Model |  | 3s Prefix as Prompt |  | Ref Utterance as Prompt |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Prompt Input |  | Text Input | Prompt Input |  | SIM$\uparrow$ | WER$\downarrow$ | DNSMOS$\uparrow$ |  | SIM$\uparrow$ | WER$\downarrow$ | DNSMOS$\uparrow$ |
| Single Sampling |  |  |  |  |  |  |  |  |  |  |  |
| ✓ |  | ✓ | ✓ |  | 0.779 | 1.6 | 3.956 |  | 0.639 | 1.9 | 4.013 |
| ✗ |  | ✓ | ✓ |  | n/a | n/a | n/a |  | 0.169 | 2.8 | 4.001 |
| ✓ |  | ✓ | ✦ |  | 0.731 | 1.6 | 3.957 |  | 0.530 | 1.9 | 4.018 |
| ✓ |  | ✓ | ✗ |  | n/a | n/a | n/a |  | 0.385 | 1.8 | 4.015 |
| ✓ |  | ✗ | ✓ |  | 0.774 | 5.6 | 3.958 |  | 0.619 | 10.0 | 4.016 |
| Five-Time Sampling |  |  |  |  |  |  |  |  |  |  |  |
| ✓ |  | ✓ | ✓ |  | 0.804 | 1.0 | 3.952 |  | 0.684 | 0.7 | 4.016 |
| ✗ |  | ✓ | ✓ |  | n/a | n/a | n/a |  | 0.305 | 2.0 | 4.018 |
| ✓ |  | ✓ | ✦ |  | 0.765 | 1.0 | 3.956 |  | 0.583 | 0.7 | 4.020 |
| ✓ |  | ✓ | ✗ |  | n/a | n/a | n/a |  | 0.457 | 1.0 | 4.019 |
| ✓ |  | ✗ | ✓ |  | 0.793 | 1.8 | 3.960 |  | 0.647 | 3.0 | 4.018 |

Figure: Figure 5: Ablation study of the size of training data on LibriSpeech test-clean.
Refer to caption: x5.png

We conduct several ablation studies of VALL-E 2 on LibriSpeech test-clean. We use the VALL-E 2 model with group size 1, and present the results for both single-sampling and five-time sampling for each speech synthesis. For five-time sampling, we select the best candidate by sorting 5 samples based on SIM and WER scores as in Equation 23.

Ablation on Model Input:
In Table 3, we study the impact of the text and prompt input in the AR and NAR models. Removing the prompt in either AR or NAR model results in significantly lower speaker similarity scores, emphasizing the crucial role of the prompt in preserving speaker identity. Despite the NAR model having access to the prompt, the AR model’s prompt still contributes significantly to speaker similarity. In the case of the NAR model, we also discover that explicitly splitting the acoustic condition during training is essential to enhance the final speaker similarity score, as the NAR model can extract more speaker information from the entire 8 code sequences of the prompt. Interestingly, we find that the prompt in the AR model also improves the robustness of the generated speech, as evidenced by a lower WER score. This can be attributed to the prompt’s ability to constrain the search space of the one-to-many speech synthesis task, thereby enabling more stable and robust speech generation. Additionally, the text input is also crucial in the NAR model for achieving a lower WER score, despite its use in the AR model.

Ablation on Training Data:
In Figure 5, we explore the impact of the size of training data on the zero-shot TTS performance. We find that our model, with 10k training data, can already achieve performance similar to that with 50k training data on LibriSpeech test-clean. The additional 40k data only results in slight performance improvement in terms of speaker similarity and robustness. However, if we reduce the training data to less than 10k, we observe a performance degradation, especially for the setting of reference utterance as a prompt. It should be noted that this conclusion is based on the current experiment setting in the audiobook domain.

### 4.3 VCTK Evaluation

#### 4.3.1 Objective Evaluation

**Table 4: Objective evaluation results on VCTK.**
| System | GroupSize |  | 3s Prompt |  | 5s Prompt |  | 10s Prompt |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SIM$\uparrow$ | WER$\downarrow$ | DNSMOS$\uparrow$ |  | SIM$\uparrow$ | WER$\downarrow$ | DNSMOS$\uparrow$ |  | SIM$\uparrow$ | WER$\downarrow$ | DNSMOS$\uparrow$ |  |  |
| GroundTruth | - |  | 0.623 | 0.3 | 3.635 |  | 0.679 | 0.3 | 3.635 |  | 0.709 | 0.3 | 3.635 |
| $\;\;\;\hookrightarrow$ Codec | - |  | 0.563 | 0.3 | 3.609 |  | 0.616 | 0.3 | 3.609 |  | 0.644 | 0.3 | 3.609 |
| Single Sampling |  |  |  |  |  |  |  |  |  |  |  |  |  |
| VALL-E | 13ms |  | 0.430 | 2.4 | 3.667 |  | 0.455 | 3.1 | 3.664 |  | 0.533 | 5.8 | 3.575 |
| VALL-E 2 | $\times 1$ |  | 0.447 | 0.9 | 3.666 |  | 0.487 | 1.9 | 3.674 |  | 0.558 | 3.3 | 3.667 |
| $\times 2$ |  | 0.426 | 1.5 | 3.599 |  | 0.481 | 0.9 | 3.598 |  | 0.557 | 2.3 | 3.617 |  |
| $\times 4$ |  | 0.417 | 1.8 | 3.470 |  | 0.457 | 2.1 | 3.537 |  | 0.521 | 2.9 | 3.547 |  |
| $\times 8$ |  | 0.375 | 5.0 | 3.438 |  | 0.415 | 4.8 | 3.387 |  | 0.499 | 8.0 | 3.420 |  |
| Five-Time Sampling (Sort on SIM and WER) |  |  |  |  |  |  |  |  |  |  |  |  |  |
| VALL-E | 13ms |  | 0.497 | 0.3 | 3.599 |  | 0.534 | 0.3 | 3.666 |  | 0.607 | 1.5 | 3.591 |
| VALL-E 2 | $\times 1$ |  | 0.508 | 0.0 | 3.684 |  | 0.552 | 0.3 | 3.699 |  | 0.620 | 1.5 | 3.694 |
| $\times 2$ |  | 0.494 | 1.0 | 3.616 |  | 0.547 | 0.1 | 3.617 |  | 0.606 | 0.4 | 3.621 |  |
| $\times 4$ |  | 0.487 | 0.9 | 3.547 |  | 0.531 | 0.4 | 3.588 |  | 0.592 | 1.6 | 3.559 |  |
| $\times 8$ |  | 0.444 | 2.4 | 3.454 |  | 0.499 | 0.5 | 3.429 |  | 0.563 | 1.3 | 3.430 |  |
| Five-Time Sampling (Metric-Wise Maximization) |  |  |  |  |  |  |  |  |  |  |  |  |  |
| VALL-E | 13ms |  | 0.504 | 0.1 | 3.867 |  | 0.541 | 0.3 | 3.864 |  | 0.615 | 1.5 | 3.850 |
| VALL-E 2 | $\times 1$ |  | 0.513 | 0.0 | 3.860 |  | 0.555 | 0.3 | 3.868 |  | 0.621 | 1.5 | 3.855 |
| $\times 2$ |  | 0.499 | 0.1 | 3.842 |  | 0.550 | 0.1 | 3.833 |  | 0.606 | 0.4 | 3.821 |  |
| $\times 4$ |  | 0.490 | 0.5 | 3.760 |  | 0.537 | 0.3 | 3.783 |  | 0.595 | 1.4 | 3.772 |  |
| $\times 8$ |  | 0.454 | 1.0 | 3.673 |  | 0.505 | 0.4 | 3.658 |  | 0.571 | 1.3 | 3.683 |  |

Table 4 presents the objective evaluation results on the VCTK dataset, where VALL-E 2 demonstrates superior zero-shot TTS performance than VALL-E, especially in terms of speech robustness score WER.
It demonstrates the repetition aware sampling method can also effectively stable the decoding process on challenging VCTK data with speakers in diverse accents. It can roughly half the WER score in the single sampling scenario.
With five-time sampling, we can effectively filter out low-quality samples and select the best sample as the output, enabling VALL-E to generate speech of much better robustness, and mitigate the gap of the WER score between VALL-E and VALL-E 2.

When comparing different prompt lengths, we find that the grouped code modeling method can even further improve the WER score for longer prompts. The reason could be that the excessively long prompts present challenges in the long sequence modeling of the Transformer architecture and tend to yield some generation errors due to incorrect attention alignments, and the grouped code modeling method can alleviate this problem by reducing the sequence length while enhancing the AR modeling.

Figure: Figure 6: Sampling stability on VCTK dataset. GS means group size and RAS stands for repetition aware sampling.
Refer to caption: x6.png

We further presents the superior decoding stability of VALL-E 2 in Figure 6.
As found in LibriSpeech dataset, the repetition aware sampling method significantly enhances the decoding stability, and enables generating more robustness speech signals with a relatively small top-p value.

#### 4.3.2 Subjective Evaluation

**Table 5: Subjective evaluation results for 60 speakers on VCTK.**
| System | GroupSize |  | 3s Prompt |  | 5s Prompt |  | 10s Prompt |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | SMOS$\uparrow$ | CMOS$\uparrow$ |  | SMOS$\uparrow$ | CMOS$\uparrow$ |  | SMOS$\uparrow$ | CMOS$\uparrow$ |  |  |
| GroundTruth | - |  | 4.47±0.13 | 0.00 |  | 4.53±0.14 | 0.00 |  | 4.74±0.17 | 0.00 |
| VALL-E | 13ms |  | 4.32±0.16 | 0.028 |  | 4.05±0.20 | 0.144 |  | 3.50±0.49 | 0.094 |
| VALL-E 2 | $\times 1$ |  | 4.42±0.15 | 0.207 |  | 4.28±0.16 | 0.079 |  | 3.95±0.10 | 0.117 |
| $\times 2$ |  | 4.47±0.13 | 0.163 |  | 4.14±0.17 | 0.217 |  | 4.26±0.42 | 0.109 |  |

Table 5 presents the subjective evaluation results on the VCTK dataset. We conduct the subjective evaluation with 60 test cases from 60 distinct speakers.

Given the diverse speaker accents in the VCTK dataset, zero-shot TTS is much more challenging than that on LibriSpeech dataset.
The comparison result in Table 5 reveals that VALL-E 2 can successfully surpasses VALL-E in terms of both speaker similarity and speech quality, even same or better performance than the ground truth speech when using only 3s prompt.
This underscores the human parity performance of VALL-E in zero-shot TTS for a very diverse accents scenario.

Thanks to the long context modeling capability of group code modeling method, we also achieve significant performance improvement with long prompt of 10s, especially for speaker similarity.

#### 4.3.3 Ablation Study

**Table 6: Ablation study of model input on VCTK.**
| AR Model |  | NAR Model |  | 3s Prompt |  | 5s Prompt |  | 10s Prompt |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Prompt Input |  | Text Input | Prompt Input |  | SIM$\uparrow$ | WER$\downarrow$ | DNSMOS$\uparrow$ |  | SIM$\uparrow$ | WER$\downarrow$ | DNSMOS$\uparrow$ |  | SIM$\uparrow$ | WER$\downarrow$ | DNSMOS$\uparrow$ |
| Single Sampling |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| ✓ |  | ✓ | ✓ |  | 0.450 | 2.6 | 3.698 |  | 0.486 | 2.0 | 3.692 |  | 0.567 | 4.1 | 3.684 |
| ✗ |  | ✓ | ✓ |  | 0.139 | 3.0 | 3.685 |  | 0.144 | 2.9 | 3.686 |  | 0.159 | 3.5 | 3.672 |
| ✓ |  | ✓ | ✦ |  | 0.347 | 2.3 | 3.684 |  | 0.396 | 2.4 | 3.672 |  | 0.489 | 4.4 | 3.688 |
| ✓ |  | ✓ | ✗ |  | 0.224 | 2.3 | 3.686 |  | 0.245 | 2.4 | 3.679 |  | 0.284 | 3.8 | 3.690 |
| ✓ |  | ✗ | ✓ |  | 0.426 | 14.1 | 3.698 |  | 0.478 | 11.9 | 3.705 |  | 0.556 | 11.5 | 3.677 |
| Five-Time Sampling |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| ✓ |  | ✓ | ✓ |  | 0.513 | 0.0 | 3.678 |  | 0.550 | 0.0 | 3.694 |  | 0.618 | 1.6 | 3.703 |
| ✗ |  | ✓ | ✓ |  | 0.271 | 1.6 | 3.787 |  | 0.282 | 2.3 | 3.741 |  | 0.303 | 3.1 | 3.725 |
| ✓ |  | ✓ | ✦ |  | 0.418 | 0.4 | 3.665 |  | 0.472 | 1.0 | 3.700 |  | 0.550 | 1.5 | 3.675 |
| ✓ |  | ✓ | ✗ |  | 0.306 | 1.4 | 3.658 |  | 0.327 | 2.1 | 3.678 |  | 0.361 | 3.8 | 3.677 |
| ✓ |  | ✗ | ✓ |  | 0.476 | 3.0 | 3.705 |  | 0.527 | 1.5 | 3.719 |  | 0.605 | 2.4 | 3.725 |

Figure: Figure 7: Ablation study of the size of training data on VCTK.
Refer to caption: x7.png

We further conduct ablation studies of VALL-E 2 on VCTK dataset. We use the VALL-E 2 model with group size 1, and present the results for both single-sampling and five-time sampling for each speech synthesis. For five-time sampling, we sort multiple samples with Equation 23.

Ablation on Model Input:
As shown in Table 6, consistent with the observations in the LibriSpeech evaluation, the prompt is crucial in both AR and NAR models for speaker information modeling. The speaker similarity score would significantly declines when we remove the prompt input. Although the text input is consumed in the AR model, the NAR model also requires it to synthesize robust speech.

Ablation on Training Data:
As shown in Figure 7, the optimal size of training data varies for different inference prompts and metrics. The SIM score consistently benefits from larger training data, which offers more diverse speaker voice patterns. The best WER score with a 3s prompt requires more training data than the 5s prompt and 10s prompt, due to the increased challenge of zero-shot TTS with only a 3s enrolled speech. Interestingly, the best DNSMOS score is not achieved with the largest training data. A possible explanation is that, with limited model capacity, our model achieves better speaker similarity and robustness at the expense of slight losses in perceived quality.

## 5 Conclusion

We introduce VALL-E 2, a language modeling approach that achieves human parity zero-shot text to speech synthesis (TTS) for the first time.
Based on the success of VALL-E, VALL-E 2 introduce two simple but effective methods: repetition aware sampling for better decoding stability and grouped code modeling for better modeling efficiency. Furthermore, our observations reveal that VALL-E 2 is capable of reliably synthesizing speech for complex sentences, including those that are challenging to read or contain numerous repeated phrase.

Broader impacts: Since VALL-E 2 could synthesize speech that maintains speaker identity, it may carry potential risks in misuse of the model, such as spoofing voice identification or impersonating a specific speaker. We conduct the experiments under the assumption that the user agree to be the target speaker in speech synthesis. If the model is generalized to unseen speakers in the real world, it should include a protocol to ensure that the speaker approves the use of their voice and a synthesized speech detection model.
Furthermore, it is possible to build a detection model to discriminate whether an audio clip was synthesized by VALL-E 2. We will also put Microsoft AI Principles(^7^77[https://www.microsoft.com/ai/responsible-ai](https://www.microsoft.com/ai/responsible-ai)) into practice when further developing the models.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: - **Why was Vocos selected?** Given that other Vocoders with potentially better performance are available, why was Vocos specifically chosen? Additionally, what was the necessity of switching the decoder from Encodec’s original model?

- **Reason for the Significant Improvement in SIM**: It is understandable that Repetition Aware Sampling could lead to an improvement in WER; however, it is less clear how this would directly impact SIM. Furthermore, why does the subjective evaluation show significant improvement despite relatively poor objective metrics?

- **Lack of Evaluation on Difficult Cases**: The introduction references challenging cases, yet no evaluation related to these cases is provided. Why is this evaluation absent?
soundness: 2
strengths: The paper is written in a clear and accessible manner, enhancing readability and comprehension. Notably, VALL-E 2 demonstrates superior performance over ground truth in subjective evaluations, achieving higher scores in both CMOS and SMOS on both the LibriSpeech test-clean and VCTK datasets.
summary: VALL-E 2 is an LM-based TTS model based on VALL-E. It proposes two new methods:

1. **Repetition Aware Sampling**: In this method, during the sampling process, the repetition ratio is calculated based on the number of times a token has been generated. If this value exceeds a threshold, tokens are generated randomly from the original distribution.\\
2. **Grouped Code Modeling**: This method reduces sequence length by grouping adjacent tokens into fixed-size groups.

Thanks to these contributions, VALL-E 2 achieves significantly higher performance than the baseline VALL-E, particularly yielding better subjective evaluation results than the ground truth on LibriSpeech test-clean and VCTK.
weaknesses: The authors have made an effort to present this work as a promising study; however, upon closer examination, there are numerous concerns that require substantial improvement.

- **On Subjective Evaluation Results**:  
  The results discussed as a strength in Figure 1 are fundamentally flawed due to the dataset disparity with other studies (e.g., NaturalSpeech3 [1] uses the Librilight dataset). This undermines fairness in comparison. To ensure meaningful comparison, the authors should replicate NaturalSpeech3, which currently appears to be a good model, and train on the same dataset.

- **On Grouped Code Modeling**:  
  While this approach has some merit as a method to reduce the sequence length given the codec model’s high 75Hz frequency, it is rather naïve and cannot be considered innovative. In fact, similar efforts have already been undertaken in existing research, such as [2], which the authors should have cited at minimum. Additionally, the method does not lead to significant improvements in either objective or subjective evaluations, suggesting that further refinements are needed.

- **On Repetition Aware Sampling**:  
  Although this method appears to address the traditional issue of repetition in models like VALL-E effectively, it is not particularly innovative. In language modeling (LLM) contexts, penalties for repetition have long been in use [3], making the lack of reference to these approaches surprising. While the authors’ method differs slightly from these established approaches, it would be necessary to compare with them to clarify the method’s effectiveness. Moreover, the existing application of repetition penalties in TTS contexts, as seen in [4], further accentuates this concern.

- **On Ablation Studies**:  
  There is a significant lack of ablation studies. The paper includes excessive unnecessary information; for example, the equations related to the model are redundant, and condensing this information would allow the inclusion of ablation studies directly in the main text. The limited experiments in the appendix also lack relevance. Ablations such as the presence or absence of prompts and dataset size variations are not particularly noteworthy, and their results are self-evident. More critical studies, such as comparisons with traditional repetition penalties or ablations involving Vocos (a major change from VALL-E), would have been more appropriate.

- **On Baseline Comparisons**:  
  Changing the decoder from VALL-E’s original to Vocos represents a major shift and warrants stronger emphasis in comparative experiments. Additionally, the fact that subjective evaluation is best when the group size is 1 makes it very challenging to establish differentiation from the baseline.

- **Contribution to the Field**:  
  The lack of code and weight release significantly diminishes the contribution of this study to the field.


[1]: Ju, Zeqian, et al. "Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models." ICML 2024.\
[2]: Tang, Changli, et al. "Salmonn: Towards generic hearing abilities for large language models." ICLR 2024.\
[3]: Keskar, Nitish Shirish, et al. "Ctrl: A conditional transformer language model for controllable generation." arXiv preprint arXiv:1909.05858 (2019).\
[4]: Casanova, Edresson, et al. "XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model." INTERSPEECH 2024.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: [Q1. Comparison with Low-bitrate Codec] Have you compared the grouped Code Modeling with low-bitrate Codec?
soundness: 3
strengths: They enhanced the baseline model, VALL-E, the first neural codec language model, by introducing repetition-aware sampling and grouped code modeling. While the baseline models are prone to issues like word repetition or omission, the proposed methods mitigate these problems and further improve model efficiency.
summary: The authors proposed a neural codec language model for zero-shot text-to-speech, enhancing robustness by refining sampled tokens through repetition-aware sampling. They further improved robustness and efficiency by applying grouped code modeling, effectively reducing sequence length.
weaknesses: 1.	[Grouped Code Modeling1] From an engineering perspective for neural codec language models, the proposed grouped code modeling could improve model performance and efficiency. However, grouped code modeling is already a well-known technique in language models, as seen in works like [MegaByte], [RQ-Transformer], and [Block Transformer].

[MegaByte] Yu, Lili, et al. "Megabyte: Predicting million-byte sequences with multiscale transformers." Advances in Neural Information Processing Systems 36 (2023): 78808-78823.

[RQ-Transformer] Lee, Doyup, et al. "Autoregressive image generation using residual quantization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[Block Transformer] Ho, Namgyu, et al. "Block Transformer: Global-to-Local Language Modeling for Fast Inference." arXiv preprint arXiv:2406.02657 (2024).

2.	[Grouped Code Modeling2] Additionally, [UniAudio] and [GPST] have already adopted a similar structure to sample RVQ tokens more efficiently. While there may be slight differences in implementation, they have the same goal

[UniAudio] Yang, Dongchao, et al. "UniAudio: Towards Universal Audio Generation with Large Language Models." Forty-first International Conference on Machine Learning.

[GPST] Zhu, Yongxin, et al. "Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer." ACL, 2024.

3.	[Grouped Code Modeling3] Notably, the classic sequence-to-sequence text-to-speech Tacotron also utilized the grouped spectrogram sampling through a reduction factor.

[Tacotron] Wang, Yuxuan, et al. "Tacotron: Towards end-to-end speech synthesis." arXiv preprint arXiv:1703.10135 (2017).

4.	[Grouped Code Modeling4] The recently proposed model [MELLE] also claims to predict multiple frames per step, accelerating inference and mitigating robustness issues associated with long-sequence modeling while maintaining strong performance. Moreover, MELLE has been shown to outperform VALL-E2. This hurts the contribution of the proposed method.

5.	[Repetition Aware Sampling] Recently, Flow-matching and MaskGIT-based text-to-speech models have adopted iterative sampling methods similar to repetition-aware sampling. It would be beneficial for the authors to discuss and compare repetition-aware sampling with these iterative methods, particularly against models like VoiceBox and E2-TTS.Specifically, I hope to see the comparison with VoiceBox and E2-TTS. 

6.	[Weak Baseline] The authors only compared the model with VALL-E. However, VALL-E underperforms compared to VoiceBox, E2-TTS, DiTTo-TTS, and CosyVoice. 

I sincerely acknowledge the novel contribution of VALL-E in opening the door for neural codec language models; however, the novelty of VALL-E2 does not meet the standards expected for ICLR.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: - Why do this paper choose Byte-Pair Encoding (BPE) for text tokenization instead of using phonemes? How many BPE tokens are used in the model? Given that large datasets like LibriHeavy typically require thousands of BPE classes, while phoneme-based tokenization usually involves only a few dozen classes, how do you anticipate this choice impacts the model’s performance?

- Including punctuation marks in modeling units could benefit text-to-speech (TTS) systems. I’m interested to know if the BPE units in this work incorporate punctuation marks. How might this decision impact the model’s performance?
soundness: 4
strengths: - VALL-E 2 achieving human parity in zero-shot TTS is a promising advancement, marking a new benchmark for text-to-speech systems. Its potential applications are particularly promising in assistive technologies for individuals with speech impairments.

- The introduction of repetition-aware sampling and grouped code modeling is a simple but effective approach that enhances the model's stability and efficiency in generating speech. These methods could be easily adapted to speech-to-speech language models.

- The paper demonstrates strong experimental validation with comprehensive evaluations on datasets including LibriSpeech and VCTK, showing clear improvements in robustness, naturalness, and speaker similarity​.

- The technical explanations and results are clearly presented, making the contributions and performance enhancements easy to understand.
summary: VALL-E represents a breakthrough in neural codec language modeling for zero-shot text-to-speech synthesis. It can synthesize personalized speech from just a 3-second recording, while preserving the speaker's voice, emotion, and acoustic environment. VALL-E uses an autoregressive transformer to model coarse codec codes (1st group of EnCodec) and a non-autoregressive transformer to generate fine codec codes (2nd-8th groups of EnCodec). However, VALL-E faces two key limitations: 1) Stability: Random sampling during inference can cause instability, while small top-p nucleus sampling risks infinite loops. 2) Efficiency: Its autoregressive architecture is constrained by a fixed high frame rate, slowing inference.

The paper introduces VALL-E 2, which addresses the aforementioned issues with two innovations: Repetition Aware Sampling, which stabilizes decoding without increasing computational costs, and Grouped Code Modeling, which reduces sequence length and speeds up inference. These improvements make VALL-E 2 more robust, natural, and efficient in zero-shot TTS, achieving human parity for the first time on benchmarks including LibriSpeech and VCTK. VALL-E 2 can stably generate high-quality speech for complex sentences that are hard to read or contain many repeated phrases.

Considering that VALL-E (pre-printed on January 5, 2023) has not been published in any conference or journal, yet has already garnered 542 citations and opened the field for neural codec models, and that VALL-E 2 builds on this foundation by achieving human parity for the first time, I believe the combining contributions makes it deserving of acceptance.
weaknesses: See above

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: NA
soundness: 3
strengths: see above
summary: This work is an extension of VALL-E, which aims to solve its two problems. 1. inference repetitions --> degrade performance 2. too long codec sequence for modeling --> degrade speed.

Specifically, they propose 1. Repetition Aware Sampling to remove repititions by accounting for token repetition in the decoding history, thus improve the synthesis quality, 2. Grouped Code Modeling to re-organize codec sequence into groups to shorten the length, thus improve the modeling efficiency.

From my side, it is a good extension of VALL-E series and solve its practical issues. But from research perpective, this work does not convey much novelty or insights, especially given the high requirement of ICLR conference.
weaknesses: see above

Meta-review: The paper proposes to incorporate a new sampling procedure and a different grouping of codes to improve VALL-E for zero-shot TTS.

I recommend a rejection because all reviewers unanimously agree that paper lacks novelty despite the strong performance. 

There are various other suggestions provided by the reviewers that the authors should consider to improve the paper. In particular, the comparison across multiple systems should be done carefully to avoid comparing apples and oranges.

Decision: Reject