Analyzing and Mitigating Inconsistency in Discrete Audio Tokens for Neural Codec Language Models

## 1 abstract

Building upon advancements in Large Language Models (LLMs), the field of audio processing has seen increased interest in training audio generation tasks with discrete audio token sequences. However, directly discretizing audio by
neural audio codecs often results in sequences that fundamentally differ from text sequences. Unlike text, where text token sequences are deterministic, discrete audio tokens can exhibit significant variability based on contextual factors, while still producing perceptually identical audio segments. We refer to this phenomenon as Discrete Representation Inconsistency (DRI). This inconsistency can lead to a single audio segment being represented by multiple divergent sequences, which creates confusion in neural codec language models and results in omissions and repetitions during speech generation. In this paper, we quantitatively analyze the DRI phenomenon within popular audio tokenizers such as EnCodec. Our approach effectively mitigates the DRI phenomenon of the neural audio codec. Furthermore, extensive experiments on the neural codec language model over LibriTTS and large-scale MLS datases (44,000 hours) demonstrate the effectiveness and generality of our method. The demo of audio samples is available online (^1^11[https://consistencyinneuralcodec.github.io](https://consistencyinneuralcodec.github.io)).

## 2 Introduction

Recently, speech Large Language Models (LLMs) (Zhan et al., ; Anastassiou et al., ; Du et al., ) have demonstrated significant strides in generating high-quality speech, largely due to the contributions of neural audio codecs in high-fidelity audio reconstruction (Zeghidour et al., ; Défossez et al., ; Yang et al., ). The neural codec language model (Wang et al., ; Yang et al., ; Zhang et al., ) employs the neural audio codec as the audio tokenizer to quantize continuous audio signals into discrete tokens, and it can generate discrete tokens autoregressively (Zhang et al., ; Yang et al., ), and then detokenize them back to audio signals by the neural audio codec. Despite the advantages of autoregressive modeling can assist those works achieve better zero-shot performance and naturalness, the synthesized speech frequently yields higher Word Error Rate (WER) due to the issue of instability in discrete token generation (Song et al., ; Xin et al., ; Du et al., ).

Figure: Figure 1: Discrete Representation Inconsistency (DRI) phenomenon. Subfigure (a) shows that text, whether it includes contextual information or not, can be encoded by the text tokenizer into the same text tokens. In contrast, Subfigure (b) illustrates that audio, with or without contextual information, is encoded by the audio tokenizer into different audio tokens. The DRI phenomenon within the audio tokenizer poses a many-to-one mapping problem, and the complexity of this many-to-one mapping raises the uncertainty for neural codec language models in predicting the next token.
Refer to caption: extracted/5903017/figure/notion/all_tokenizer.png

The discrete sequence of text is context-independent. In contrast, acoustic discrete representations are encoded by integrating the contextual information. The advantage of this approach is that discrete audio tokens consider a larger receptive field of information, thus achieving a higher compression ratio of information. However, the drawback is that the representation itself becomes more fragile, sensitive, and easily affected by minor signal changes, leading to drastic drifts in the entire sequence as demonstrated in Figure 1.

The previous work (Yang et al., ) has noticed that audio segments containing the same sound events aren’t encoded into completely consistent discrete acoustic tokens by the neural audio codec. In this paper, we call this phenomenon Discrete Representation Inconsistency (DRI), and further dig into the problem on Vector Quantization (VQ) (Défossez et al., ) based acoustic tokens due to its popularity as an audio tokenizer and its high-quality reconstruction capabilities. We compare the consistency of the discrete sequences of audio segments with and without context on a large amount of audio. Our quantitative analyses reveal that the existing audio tokenizers suffer from low consistency. In particular, we find that for Residual Vector Quantization (RVQ) (Défossez et al., ) approaches, consistency declines significantly with deeper layer of codebooks.

Although audio with or without contextual audio is encoded into different discrete audio token sequences, both sequences can be used to reconstruct the original audio information, which leads to a many-to-one mapping problem that becomes more complex as the sequence length increases. This complexity results in increased uncertainty for neural codec language models in predicting the next token. One direct approach to address this issue is to prevent the audio tokenizer from considering contextual information during sequence encoding, such as by setting the convolutional layer’s kernel size to 1 in the encoder. While this method allows for independent discretization of each audio frame, it also significantly reduces encoding efficiency and degrades the quality of the reconstructed audio. Therefore, this study aims to maintain the original receptive field while enabling the model to address the trade-offs between audio reconstruction quality and resolving the many-to-one problem. To achieve this objective, we introduce the slice-consistency method, wherein a segment of audio is randomly sliced, and the encoded representation from this segment is required to closely approximate the corresponding representation obtained from the entire audio. In addition, in order to further alleviate the issue of many-to-one mapping, we propose perturbation-consistency method, whereby the representation of an audio and its representation after applying slight spectral perturbation should closely align. Compared to EnCodec (Défossez et al., ), our method has shown an average consistency improvement of 21.47%, 29.17%, and 36.29% in the first layer, the first 3 layers, and the first 8 layers, respectively. Extensive experiments on the neural codec language model (e.g., VALL-E (Wang et al., )) on LibriTTS (Zen et al., ) dataset (960 hours) and large-scale MLS (Pratap et al., ) dataset (44,000 hours) confirm that improving consistency results in better performance. Our contributions are summarized as below:

- •
We shed light on the Discrete Representation Inconsistency (DRI) phenomenon and conduct quantitative analyses for various neural audio codecs. We find that the existing audio tokenizers suffer from low consistency.
- •
Inspired by our analyses, we propose two methods, slice-consistency method and perturbation-consistency method, to enhance the consistency of the neural audio codec from two particular perspectives and mitigate the many-to-one problem.
- •
Experiments show that our method achieves an average consistency improvement of 21.47%, 29.17%, and 36.29% in the first layer, the first 3 layers, and the first 8 layers, respectively. Additionally, we conduct extensive experiments on the neural codec language model, VALL-E, on the LibriTTS dataset (960 hours) and further expand the training dataset to the large-scale MLS dataset (44,000 hours), resulting in 3.72% WER reduction and 5.68% speaker similarity improvement. These findings confirm that enhancing consistency leads to improved performance.

## 3 Analysis on consistency of neural audio codecs

Figure: Figure 2: Results of consistency accuracy for popular neural audio codecs under different layers and slice lengths. Subfigure (a)(b)(c) show slice lengths across 0.2s, 0.3s and 0.4s, respectively, and all of them exhibit similar conclusions that consistency accuracy declines significantly in the deeper layers of the codebook, indicating that the DRI phenomenon becomes more pronounced with layers in neural audio codecs increasing.
Refer to caption: extracted/5903017/figure/notion/8_layer_fig.png

In this section, we extract discrete audio tokens from audio segments with and without context using popular neural audio codecs (Défossez et al., ; Yang et al., ; Zhang et al., ; Du et al., ; Kumar et al., ; Ju et al., ) to analyze the DRI phenomenon. First, we introduce the overall experiment design. Then we propose using consistency accuracy as an evaluation metric to conduct quantitative analyses. Finally, we analyze the results and discuss the potential implications of the DRI phenomenon.

### 3.1 Experimental Design On DRI Phenomenon

Recent advancements on neural audio codecs have adopted an encoder-decoder architecture combined with the RVQ module to effectively compress continuous audio signals into discrete audio tokens (Défossez et al., ; Yang et al., ; Zhang et al., ; Du et al., ; Kumar et al., ; Ju et al., ), which is typically composed of 3 components: (1) An encoder, composed of convolutional layers to capture contextual information, maps the audio signal into a latent representation $Z$. (2) A RVQ module contains $N$ quantization layers to quantize the latent representation $Z$ into the discrete audio tokens at each time step. (3) A decoder reconstructs the quantized latent representation back to the audio signal.

To analyze the DRI phenomenon, we use popular neural audio codecs (Défossez et al., ; Yang et al., ; Zhang et al., ; Du et al., ; Kumar et al., ; Ju et al., ) as audio tokenizers to quantize both the entire audio and an audio segment within that audio, and then compare the results of their corresponding discrete audio tokens. Obviously, these two audio segments are exactly identical with the only difference being whether there is context, and we expect that both discrete audio tokens should be identical after quantization. But the encoders in current neural audio codecs introduce the contextual information that gives rise to the DRI phenomenon, leading to both discrete audio token sequences showing significant differences.

### 3.2 Consistency Accuracy

To quantitatively analyze the degree of the DRI phenomenon in neural audio codecs, we propose using consistency accuracy as an evaluation metric:

$$ $Acc_{\text{consistency}}=\frac{1}{T}\frac{1}{N}\sum_{t=1}^{T}\sum_{i=1}^{N} \mathbb{I}(\text{RVQ}(Z^{\text{slice}})[t,i]=\text{RVQ}(Z)[t,i]),$ $$

where $Z$ is the latent representation of the original audio after encoding by the encoder, and $N$ represents the number of codebooks in the RVQ module. We randomly extract an audio segment of length $T$ from the original audio, and encode it by the encoder to obtain $Z^{\text{slice}}$.

### 3.3 Results And Analysis

Audio tokenizer vs. text tokenizer. As shown in Figure 1 (a), regardless of whether the context is included, the same text is tokenized into the same text tokens, indicating that the text tokenizer is context-independent. In contrast, Figure 1 (b) demonstrates that using a neural audio codec as the audio tokenizer produces different discrete audio token sequences for identical audio segments. Although it is difficult for human auditory perception to distinguish the reconstructed audio from both sequences, the many-to-one mapping caused by the DRI phenomenon still increases the difficulty for model training, leading to a decline in speech reconstruction and generation performance.

The results of consistency accuracy. To quantitatively analyze the DRI phenomenon, we calculate the consistency accuracy for popular neural audio codecs under different layers and slice lengths. The results are shown in Figure 2 and the low consistency accuracy reveals that the DRI phenomenon is present in the current neural audio codecs (Défossez et al., ; Yang et al., ; Zhang et al., ; Du et al., ; Kumar et al., ; Ju et al., ). Furthermore, we find that with deeper layers of codebooks, neural audio codecs demonstrate lower consistency. This may be attributed to the fact that audio tokens in shallow layers exhibit a high alignment with context-independent semantic information, resulting in better consistency. In contrast, deeper layers focus on more fragile and sensitive acoustic information that can easily change due to minor disturbances, leading to a decrease in consistency accuracy (Zhang et al., ).

The potential implications of the DRI phenomenon. There are many minor disturbances that can cause the DRI phenomenon, such as contextual information and phase perturbation (Lee et al., ) that do not alter the auditory perception of the reconstructed audio but can lead to changes in the discrete audio token sequences, which can greatly confuse models. Especially when neural codec language models need to predict different audio tokens due to the DRI phenomenon, this confusion can cause the predicted probability distributions of the next token to converge towards uniformity, resulting in inaccurate predictions and negatively impacting overall performance. Therefore, it is necessary to ease the many-to-one mapping problem to improve the consistency of neural audio codecs, which in turn enhances the performance of downstream speech generation.

## 4 Method

Figure: Figure 3: The overview of the proposed consistency constraint method. For the slice-consistency method, a segment of audio is randomly sliced, and its encoded representation must closely match the representation derived from the entire audio. For the perturbation-consistency method, the representation of an audio and its representation after slight spectral perturbation should be closely aligned.
Refer to caption: extracted/5903017/figure/notion/model.png

According to the analysis in Section 3, we can draw a conclusion that an ideal neural audio codec should balance the trade-offs between high audio reconstruction quality and addressing the many-to-one problem. To achieve this objective, we introduce two consistency constraint methods: slice-consistency method and perturbation-consistency method, which enhance the consistency of the neural audio codec from two particular perspectives. Since these methods can be integrated into any neural audio codec, we demonstrate their application using a neural audio codec based on RVQ which utilizes an encoder to transform the audio signal into the latent representation $Z$ and reconstructs the waveform from the quantized latent representation.

### 4.1 Consistency Constraint Methods

Slice-consistency requests that audio segments with and without context should be encoded into consistent latent representations. To achieve this object, as shown in Figure 3, we slice a segment of audio from the original audio, and then encode it using the encoder in the neural audio codec to obtain the latent representation $Z^{\text{slice}}$. Compared with the latent representation $Z$ from the entire audio, $Z^{\text{slice}}$ is not influenced by contextual information. To reduce the influence of context on the latent representation $Z$, we use Mean Squared Error (MSE) as a constraint to enhance the consistency between $Z^{\text{slice}}$ and the corresponding latent representation in $Z$:

$$ $\mathcal{L}_{\text{slice}}=\frac{1}{T}\sum_{t=1}^{T}\text{MSE}(Z^{\text{slice} }[t],Z[t]).$ $$

As analyzed in Appendix 9.1 about the receptive field, the convolutional layers in the encoder of neural audio codecs introduce contextual information, leading to identical audio segments being tokenized into different discrete audio token sequences. It is clear that reducing the kernel size of the convolutional layers in the encoder can enhance consistency, but this can also result in a decline in both reconstruction efficiency and quality. Therefore, applying the slice-consistency method is necessary to maintain the original receptive field while enabling models to balance the trade-offs between audio reconstruction quality and alleviating the DRI phenomenon.

Perturbation-Consistency refers to the latent representations of audio, which should remain consistent before and after being applied imperceptible perturbations to human ears. Specifically, as shown in Figure 3, we slightly adjust the phase of the the original audio without significantly altering the waveform structure, and encode it using the encoder in the neural audio codec to obtain the latent representation $Z^{\text{perception}}$. Since human ears have a limited ability to directly perceive phase changes, we hope that the robustness of the model can also eliminate inconsistency caused by such slight perturbations. Therefore, we utilize MSE to maintain consistency of both latent representations with and without phase perturbation (Lee et al., ):

$$ $\mathcal{L}_{\text{perception}}=\text{MSE}(Z^{\text{perception}},Z).$ $$

It is evident that the perturbation-consistency method differs from audio-based data augmentation methods. Data augmentation methods such as SpecAugment (Park et al., ) and environment noise (Snyder et al., ) significantly alter the original audio to create new audio. The newly generated audio has a considerable difference in perception compared to the original audio, which aims to expand the training data and increase the robustness of models. In contrast, the perception-consistency method requires that changes to audio should be imperceptible to human ears to avoid severe perturbations that disrupt the audio reconstruction quality. Since the phase is difficult to be perceived by human, we apply phase perturbation (Lee et al., ) as a slight perturbation method, which can enhance the perturbation-consistency without expanding the training data.

### 4.2 Implementation Details

In order to satisfy both methods and enhance training efficiency, we align the latent representation $Z^{perception}$ obtained by the slice-consistency method and the latent representation $Z^{perception}$ obtained by the perturbation-consistency method:

$$ $\mathcal{L}_{\text{consistency}}=\frac{1}{T}\sum_{t=1}^{T}\text{MSE}(Z^{slice} [t],Z^{perception}[t]).$ $$

By introducing consistency constraint $\mathcal{L}_{\mathrm{consistency}}$, our method can be applied to any neural audio codec and we build our method on RVQ-GAN framework (Kumar et al., ) that also includes reconstruction loss $\mathcal{L}_{\mathrm{rec}}$, adversarial loss $\mathcal{L}_{\mathrm{adv}}$, feature matching loss $\mathcal{L}_{\mathrm{fm}}$, and commit loss $\mathcal{L}_{\mathrm{rvq}}$:

$$ $\mathcal{L}=\mathcal{L}_{\mathrm{rec}}+\lambda_{\mathrm{adv}}\mathcal{L}_{ \mathrm{adv}}+\lambda_{\mathrm{fm}}\mathcal{L}_{\mathrm{fm}}+\lambda_{\mathrm{ rvq}}\mathcal{L}_{\mathrm{rvq}}+\lambda_{\mathrm{con}}\mathcal{L}_{\mathrm{ consistency}}.$ $$

## 5 Experiment Setting

### 5.1 Experimental Configuration

Datasets. During the training process of the neural audio codec and the neural codec language model, we utilize LibriTTS (Zen et al., ) dataset, which comprises 960 hours of transcribed speech data. The test set of LibriTTS (Zen et al., ) is used as test data, which is randomly selected 2,300 audio samples to verify the consistency of neural audio codecs and 350 audio samples to validate speech generation performance. To verify the effectiveness of data scaling on our proposed method, we expand the training dataset to 44,000 hours of large-scale MLS (Pratap et al., ) dataset for both speech reconstruction and speech generation tasks.

Training settings. To validate effectiveness of consistency constraint in speech reconstruction, we apply it on the RVQ based neural audio codec (denoted as Ours) that uses the Adam optimizer (Diederik, ), with an initial learning rate of 3e-4 and beta parameters set to (0.5, 0.9), to train for 350,000 iterations. All audio samples are truncated to a fixed length of 1.28 seconds and resampled to 16 kHz with the batch size of 384. To demonstrate the effectiveness of our method for the downstream speech generation task, we take the neural audio codec based on our method as the audio tokenizer for the neural codec language model, VALL-E (Wang et al., ) that generates audio by predicting the first layer of audio tokens in an autoregressive manner, and then predicting the remaining audio tokens in a non-autoregressive manner. The reproducted VALL-E (Wang et al., ) is trained for 1,300,000 steps with the batch size of 56 and optimized by the Adam optimizer (Diederik, ), with parameters $\beta$ = (0.9, 0.95). And we set the weight $\lambda_{con}$ for the consistency constraint to 10.0.

Baseline models. For speech reconstruction, we use the official open-source checkpoints from EnCodec (Défossez et al., ), HiFiCodec (Yang et al., ), SpeechTokenizer (Zhang et al., ), DAC (Kumar et al., ), and FunCodec (Du et al., ) as baseline models. To ensure fair comparison, we set the bandwidth of different neural audio codecs closely to 4.0 kbps or 8.0 kbps. For speech generation, we employ the SOTA neural codec language models as baselines, including SpeechGPT (Zhang et al., ), SpeechTokenizer-based USLM (Zhang et al., ), AnyGPT (Zhan et al., ), VoiceCraft (Peng et al., ), XTTS v2 (Casanova et al., ) and VALL-E (Wang et al., ) reproduced by Amphion (Zhang et al., ). More details about baseline neural codec language models in Appendix 9.2.

### 5.2 Evaluation Metrics

#### 5.2.1 Evaluation of Speech Reconstruction

We adopt consistency accuracy across all layers of neural audio codecs to measure the DRI phenomenon. Given that the codewords in the first few layers of the neural audio codec store the most information, their consistency significantly affects the performance of downstream neural codec language models. Therefore, we especially present the first 3 layers’ consistency accuracy. Since the conclusions obtained from different lengths are generally consistent, we set $T$ in the consistency accuracy to 0.2. In addition, we also use ViSQOL (Chinen et al., ) and PESQ (Rix et al., ) to measure the quality of the reconstructed speech (Défossez et al., ; Zeghidour et al., ), with higher scores indicating the better speech quality.

#### 5.2.2 Evaluation of Speech Generation

Objective evaluation. We use Whisper (Radford et al., ) model to transcribe the generated speech and calculate the WER. To evaluate speaker similarity, we firstly use 3D-speaker (Chen et al., ) toolkit to extract speaker embeddings from the generated speech and reference speech, and then compute the cosine similarity between the normalized embeddings. We also employ UTMOS (Saeki et al., ) as an automatic Mean Opinion Score (MOS) prediction system to assess the naturalness of the speech.

Subjective evaluation. We randomly select 50 audio samples from the LibriTTS (Zen et al., ) test set to conduct MOS (Chu & Peng, ) and Similarity Mean Opinion Score (SMOS) (Chu & Peng, ) test. MOS assesses the naturalness of the generated speech, while SMOS measures the similarity between the generated speech and the original speaker’s voice. Both MOS and SMOS range from 1 to 5, with higher values indicating better speech quality and greater speaker similarity.

## 6 Result

In this section, we evaluate the effectiveness of slice-consistency method and perturbation-consistency method. Firstly, We compare the speech reconstruction results of the neural audio codec wiht consistency constraint and baseline models. And then we demonstrate that introducing our method to speech generation model like VALL-E (Wang et al., ) can effectively enhance speech generation performance on both small-scale and large-scale data. Finally, we conduct ablation studies to illustrate the effects of the slice-consistency method and perturbation-consistency method, respectively.

### 6.1 Speech Reconstruction Results

**Table 1: The speech reconstruction results on LibriTTS test set. Bold means the best result, and underline means the second-best result. Ours denotes the neural audio codec with consistency constraint.**
| Neural<br>Audio Codec | Bandwidth | Sampling<br>Rate | Number of<br>Codebooks | Consistency↑ | First 3 Layers’<br>Consistency↑ | ViSQOL↑ | PESQ↑ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| EnCodec | 4.5 kbps | 24kHz | 6 | 47.43% | 61.49% | 4.25 | 2.41 |
| 6.0 kbps | 8 | 40.46% | 61.49% | 4.35 | 2.73 |  |  |
| 8.25 kbps | 11 | 32.77% | 61.49% | 4.44 | 3.02 |  |  |
| HiFiCodec | 3.0 kbps | 24kHz | 4 | 40.77% | 46.92% | 4.32 | 2.76 |
| SpeechTokenizer | 4.0 kbps | 16kHz | 8 | 14.70% | 26.91% | 4.36 | 2.62 |
| DAC | 4.0 kbps | 16kHz | 8 | 39.14% | 48.43% | 4.44 | 2.68 |
| FunCodec | 4.0 kbps | 16kHz | 8 | 6.86% | 16.39% | 4.47 | 3.26 |
| 8.0 kbps | 16 | 3.58% | 15.49% | 4.57 | 3.62 |  |  |
| Ours | 4.0 kbps | 16kHz | 8 | 71.03% | 88.82% | 4.45 | 3.25 |
| 8.0 kbps | 16 | 56.32% | 90.66% | 4.64 | 3.59 |  |  |

We evaluate the effectiveness of our method from the perspectives of consistency and reconstructed speech quality. First, we compare the consistency accuracy between the neural audio codec with consistency constraint and baseline models. The results presented in Table 1 demonstrate that the neural audio codec based on our method can reconstruct speech with superior consistency accuracy compared to baseline models, achieving 71.03% across all layers at the bandwidth setting of 4.0 kbps and 90.66% across the first 3 layers at the bandwidth setting of 8.0 kbps. In contrast, the baseline models suffer from low consistency accuracy, indicating that the same audio segments are encoded into different discrete audio token sequences.

Figure: Figure 4: Consistency accuracy of each layer in neural audio codecs. Ours denotes the neural audio codec with consistency constraint.
Refer to caption: extracted/5903017/figure/notion/ours_consistency.png

As shown in Figure 4, we can observe that consistency accuracy declines significantly in the deeper layers of the codebooks, particularly in baseline models. This observation may stem from the fact that the semantic information in the shallow layers is relevant to text that is more context-independent, resulting in higher consistency accuracy. In contrast, the acoustic information in the deeper layers is more fragile and sensitive, making it more susceptible to contextual influences, which may pose challenges for downstream neural codec language models when predicting audio tokens from these deeper layers (Zhang et al., ). More details about the accuracy of each layer can be found in Appendix 9.3. Then we evaluate ViSQOL (Chinen et al., ) and PESQ (Rix et al., ) to evaluate the reconstructed speech quality. The results in Table 1 show that the ViSQOL (Chinen et al., ) of the neural audio codec based on our method surpasses all baseline models, achieving the score of 4.64. Additionally, its PESQ (Rix et al., ) is also comparable to that of the baseline models, with only 0.03 lower than the best result. This suggests that our method can be confidently applied to neural audio codecs without negatively impacting reconstruction performance.

### 6.2 Speech Generation Results

**Table 2: The speech generation results on LibriTTS test set. Bold means the best result, and underline means the second-best result. Ours and Ours w/o consistency constraint denote the same neural audio codecs with and without consistency constraint. The subscripts of the neural codec language models (e.g., $330M,44Kh$) denote the model size and data scale.**
| Neural<br>Audio Codec | Bandwidth | Neural Codec<br>Language Model | Objective | Subjective |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| WER↓ | SIM↑ | UTMOS↑ | MOS↑ | SMOS↑ |  |  |  |
| Ground Truth | / | / | 1.37 | / | 4.15 | 4.43 | 4.23 |
| mHuBERT | 0.5 kbps | SpeechGPT72Kh | 13.39 | 11.87% | 4.10 | 3.08 | 1.63 |
| EnCodec | 2.2 kbps | VoiceCraft330M,9Kh | 2.57 | 71.05% | 3.55 | 3.58 | 3.47 |
| VoiceCraft830M,9Kh | 2.80 | 78.26% | 3.76 | 3.72 | 3.43 |  |  |
| Mel VQ-VAE | / | XTTS_v227Kh | 1.64 | 83.96% | 3.92 | 3.58 | 3.85 |
| SpeechTokenizer | 4.0 kbps | USLM960h | 6.86 | 43.36% | 3.05 | 3.07 | 2.90 |
| AnyGPT57Kh | 18.93 | 34.60% | 3.15 | 2.77 | 2.63 |  |  |
| VALL-E960h | 9.55 | 66.77% | 3.82 | 3.56 | 3.27 |  |  |
| Ours w/o<br>consistency constraint | 4.0 kbps | VALL-E960h | 4.73 | 76.95% | 4.10 | 3.73 | 3.50 |
| VALL-E44Kh | 5.09 | 78.46% | 4.14 | 3.92 | 3.40 |  |  |
| Ours | 4.0 kbps | VALL-E960h | 1.84 | 83.71% | 4.31 | 3.97 | 3.73 |
| VALL-E44Kh | 1.37 | 84.14% | 4.30 | 4.02 | 3.95 |  |  |

In this section, we utilize the neural audio codec both with and without the consistency constraint to replicate VALL-E (Wang et al., ). From both subjective and objective perspectives, we evaluate the generated speech to demonstrate the effectiveness of our method for the downstream speech generation task. Furthermore, we increase the training data from 960 hours of LibriTTS  (Zen et al., ) dataset to 44,000 hours of large-scale MLS (Pratap et al., ) dataset to verify that our method is equally effective on larger scale datasets.

Objective Evaluation. According to Table 2, we have the following observations: (1) VALL-E (Wang et al., ), which is based on our method and trained by large-scale MLS (Pratap et al., ) dataset, outperforms all other baseline models on WER, SIM and UTMOS, indicating that our method can help speech generation models synthesize speech with better intelligibility, similarity and naturalness. (2) Compared to the VALL-E model without the consistency constraint, our method can help VALL-E achieve significant improvement in intelligibility and similarity, with 3.72% WER reduction and 5.68% SIM improvement. This indicates that improving the consistency of the neural audio codec can reduce the complexity of predicting discrete audio tokens and result in better performance. (3) The results show that VALL-E (Wang et al., ), which is based on our method and trained by 44,000 hours, shows superior speech generation results than that trained on 960 hours, achieving the average reduction of 0.47 in WER and improvement of 0.43% in SIM, illustrating the scalability of our method across different dataset scales.

Subjective Evaluation. In subjective evaluation, we conduct MOS and SMOS tests to assess speech quality and speaker similarity for all of the neural
codec language models, as shown in Table 2. The results of MOS and SMOS show similar outcomes to objective evaluations, showing that VALL-E (Wang et al., ) based on our method achieves higher speech quality and speaker similarity.

### 6.3 Ablation Study

**Table 3: Ablation study on the slice-consistency method and perturbation-consistency method. In the Slice column, the percentage (e.g., 20%) represents the proportion of the sliced audio segments to the entire audio. In the Perturbation column, phase perturb means whether to use perturbation-consistency method.**
| Neural Audio Codec | Neural Codec Language Model |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Slice | Perturbation | Consistency↑ | First 3 Layers’<br>Consistency↑ | Objective |  |  |
| WER↓ | SIM↑ | UTMOS↑ |  |  |  |  |
| 20% | phase perturb | 76.75% | 90.66% | 1.84 | 83.71% | 4.31 |
| / | phase perturb | 7.03% | 16.20% | 2.24 | 77.09% | 4.15 |
| 20% | / | 75.91% | 90.85% | 2.36 | 81.84% | 4.14 |
| / | / | 6.94% | 15.49% | 4.73 | 76.95% | 4.10 |
| 40% | phase perturb | 64.74% | 85.44% | 1.90 | 82.81% | 4.27 |
| 60% | phase perturb | 31.79% | 60.95% | 3.02 | 82.41% | 4.25 |

In this section, we conduct ablation experiments to respectively verify the effects of slice-consistency method and perturbation-consistency method. As shown in Table 3, we use the case of slicing the audio at 20% and applying perturbation-consistency method as a reference, which achieves the best results in both speech reconstruction and speech generation. Then we remove the design of slice-consistency method or perturbation-consistency method. The drop in all evaluation metrics demonstrates that both slice-consistency method and perturbation-consistency method are beneficial for speech reconstruction and generation. Finally, we conduct ablation studies on the proportion of slicing audio segments, and the results show that the slice percentage of 20% outperforms the model with the slice percentages of 40% and 60%. This suggests that shorter audio segments containing less contextual information can effectively alleviate the contextual dependence of original audio representation during the alignment process, thereby enhancing its consistency and ultimately leading to better performance in the downstream speech generation model.

Considering the better consistency in shallow layers, we further analyze VALL-E (Wang et al., ) based on our method with fewer codebooks in Appendix 9.4. The experimental results demonstrate that our method is effective across different numbers of codebooks, indicating the generalizability of the proposed method.

## 7 Related work

Discrete speech representations. Discrete speech representations can be categorized into semantic and acoustic tokens. Discrete semantic tokens are extracted from self-supervised speech models like HuBERT (Hsu et al., ) and WavLM (Chen et al., ), or Automatic Speech Recognition (ASR) models like SenseVoice (SpeechTeam, ). K-means or VQ models serve as information bottlenecks, filtering out paralinguistic information while retaining semantic information. In contrast, discrete acoustic tokens are encoded by neural audio codecs, preserving complete acoustic information and aiming to reconstruct high-fidelity audio. SoundStream (Zeghidour et al., ) and EnCodec (Défossez et al., ) adopt RVQ framework to encode speech into multi-level discrete acoustic tokens. SingleCodec (Li et al., ) and Disen-TF-Codec (Jiang et al., ) utilize a reference encoder to capture global time-invariant information, thereby reducing the number of codebooks. SpeechTokenizer (Zhang et al., ) and FACodec (Ju et al., ) decouple speech into different attributes, making discrete tokens more suitable for downstream speech modeling tasks. However, the synthesized speech from the neural codec language model, which relies on discrete audio tokens, often leads to a higher WER (Song et al., ; Xin et al., ; Du et al., ). This is because these discrete audio tokens are fragile and sensitive, easily affected by minor changes in the audio signal. Inspired by the context-independent text tokens, we propose enhancing the consistency of audio token sequences to address the many-to-one mapping problem and improve the stability of discrete audio tokens.

Audio tokenizers and neural codec language models. After tokenizing continuous audio signals into discrete tokens by a neural audio codec, a neural codec language model can be trained on these discrete audio tokens. VALL-E (Wang et al., ) and SpearTTS (Kharitonov et al., ) employ EnCodec (Défossez et al., ) and SoundStream (Zeghidour et al., ) as audio tokenizers to extract discrete acoustic tokens, aiming to retain all acoustic information. SpearTTS (Kharitonov et al., ) and SoundStorm (Borsos et al., ) adopt a coarse-to-fine approach to generate discrete acoustic tokens. VoiceCraft (Peng et al., ) rearrange audio tokens through an autoregressive way to perform speech generation and editing tasks. LLM-Codec (Yang et al., ) represents audio tokens with words or subwords from the vocabulary of LLMs, aligning audio modality with text modality. Although LLM-Codec (Yang et al., ) has noticed that even when audio segments contain the same sound events, the discrete tokens generated by the audio tokenizer may still exhibit inconsistency. Therefore, to address this DRI phenomenon, we propose slice-consistency method and perturbation-consistency method to enhance the consistency within neural audio codecs, thereby improving the performance of downstream speech generation.

## 8 Conclusion

We conduct a detailed analysis on the consistency of the discrete audio token sequences, and shed light on the Discrete Representation Inconsistency (DRI) phenomenon within the existing neural audio codecs. To mitigate the DRI phenomenon, we propose two consistency enhancement methods: (1) The slice-consistency method requires that the representation from a randomly sliced audio segment should match the corresponding representation from the entire audio. (2) The perturbation-consistency method aims to align the representation obtained from the audio after applying slight spectral perturbations with that from the original audio. Experimental results indicate that our proposed methods can successfully increase the consistency of discrete audio token sequences, thereby enabling the neural codec language model based on these audio tokens to outperform the SOTA speech generation model and show better performance by scaling up data. For future work, we plan to try more consistency enhancement methods and apply our method to various modalities to assess its generalizability.

## 9 Appendix

### 9.1 Analysis of Inconsistency Caused by Receptive Field Sizes

**Table 4: The parameters of the convolutional layers and the receptive field size in neural audio codec’s encoder.**
| Layer ID | Kernel Size | Stride | Dilation | Strides of<br>Previous Layers | Receptive Field Size |
| --- | --- | --- | --- | --- | --- |
| 1 | 7 | 1 | 1 | 0 | $7$ |
| 2 | 3 | 1 | 1 | 1 | $7+(3-1)\times 1=9$ |
| 3 | 1 | 1 | 1 | 1 | $9+(1-1)\times 1=9$ |
| 4 | 1 | 1 | 1 | 1 | $9+(1-1)\times 1=9$ |
| 5 | 4 | 2 | 1 | 1 | $9+(4-1)\times 1=12$ |
| 6 | 3 | 1 | 1 | 2 | $12+(3-1)\times 2=16$ |
| 7 | 1 | 1 | 1 | 2 | $16+(1-1)\times 2=16$ |
| 8 | 1 | 1 | 1 | 2 | $16+(1-1)\times 2=16$ |
| 9 | 8 | 4 | 1 | 2 | $16+(8-1)\times 2=30$ |
| 10 | 3 | 1 | 1 | 8 | $30+(3-1)\times 8=46$ |
| 11 | 1 | 1 | 1 | 8 | $46+(1-1)\times 8=46$ |
| 12 | 1 | 1 | 1 | 8 | $46+(1-1)\times 8=46$ |
| 13 | 10 | 5 | 1 | 8 | $46+(10-1)\times 8=118$ |
| 14 | 3 | 1 | 1 | 40 | $118+(3-1)\times 40=198$ |
| 15 | 1 | 1 | 1 | 40 | $198+(1-1)\times 40=198$ |
| 16 | 1 | 1 | 1 | 40 | $198+(1-1)\times 40=198$ |
| 17 | 16 | 8 | 1 | 40 | $198+(16-1)\times 40=798$ |
| 18 | 7 | 1 | 1 | 320 | $798+(7-1)\times 320=2718$ |

The size of the receptive field is related to the number of convolutional layers and pooling layers:

$$ $\text{RF}_{i}=\text{RF}_{i-1}+(k-1)\times S_{i},$ $$

where $\text{RF}_{i}$ represents the receptive field size of the current layer, and $\text{RF}_{i-1}$ denotes the receptive field size of the previous layer. $S_{i}$ represents the product of the strides of all previous layers (excluding the current layer), and is given by:

$$ $S_{i}=\prod_{i=1}^{L_{i}}stride_{i}.$ $$

As shown in Table 4, a larger receptive field in the encoder of neural audio codec brings more contextual information. Although this can enhance audio quality and improve encoding efficiency, it also leads to a significant decline in consistency and gives rise to the DRI phenomenon. Therefore, it is crucial to preserve the original receptive field while allowing the model to balance the trade-offs between audio reconstruction quality and addressing the many-to-one problem.

### 9.2 Evaluation baselines

SpeechGPT (^2^22[https://huggingface.co/fnlp/SpeechGPT-7B-com](https://huggingface.co/fnlp/SpeechGPT-7B-com)) (Zhang et al., ) is a neural codec language model based on HuBERT (Hsu et al., ) with conversational abilities, capable of providing various styles of speech responses based on context and human instructions.

USLM (^3^33[https://huggingface.co/fnlp/USLM](https://huggingface.co/fnlp/USLM)) (Zhang et al., ) is built upon SpeechTokenizer (Zhang et al., ) and consists of both autoregressive and non-autoregressive models to hierarchically model information in speech. The autoregressive model captures the content information, while the non-autoregressive model complements it by adding paralinguistic information.

AnyGPT (^4^44[https://huggingface.co/fnlp/AnyGPT-chat](https://huggingface.co/fnlp/AnyGPT-chat)) (Zhan et al., ) is an any-to-any multimodal neural codec language model that utilizes discrete representations for various modalities, including speech, text, images, and music. It also uses SpeechTokenizer (Zhang et al., ) to quantize speech.

VoiceCraft (^5^55[https://huggingface.co/pyp1/VoiceCraft_giga330M](https://huggingface.co/pyp1/VoiceCraft_giga330M)) (^6^66[https://huggingface.co/pyp1/VoiceCraft_830M_TTSEnhanced](https://huggingface.co/pyp1/VoiceCraft_830M_TTSEnhanced)) (Peng et al., ) is a token-infilling neural codec language model. It introduces a token rearrangement procedure that combines causal masking and delayed stacking to enhance voice cloning ability.

XTTS v2 (^7^77[https://huggingface.co/coqui/XTTS-v2](https://huggingface.co/coqui/XTTS-v2)) (Casanova et al., ) is a multilingual speech generation model and employs a VQ-VAE (Van Den Oord et al., ) module to discretize the mel-spectrogram.

VALL-E (^8^88[https://github.com/open-mmlab/Amphion/tree/main/egs/tts/VALLE_V2](https://github.com/open-mmlab/Amphion/tree/main/egs/tts/VALLE_V2)) (Wang et al., ) is reproduced by Amphion (Zhang et al., ) that adopts SpeechTokenizer (Zhang et al., ) as the neural audio codec.

### 9.3 Consistency accuracy of each layer

**Table 5: Detailed results of consistency accuracy of each layer in neural audio codecs. Ours denotes the neural audio codec with consistency constraint.**
| Neural Audio Codec | Every Layer’s Consistency |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
| EnCodec | 74.66% | 61.20% | 48.62% | 41.30% | 32.47% | 26.30% | 21.25% | 17.89% |
| HiFiCodec | 61.87% | 55.73% | 23.15% | 22.34% | / | / | / | / |
| SpeechTokenizer | 41.52% | 23.13% | 16.09% | 11.64% | 8.59% | 6.21% | 5.08% | 5.31% |
| DAC | 63.44% | 46.17% | 36.88% | 32.77% | 33.75% | 34.92% | 34.26% | 30.90% |
| FunCodec | 29.34% | 10.12% | 7.03% | 4.10% | 2.54% | 1.02% | 0.78% | 0.59% |
| Ours | 96.13% | 91.09% | 84.77% | 79.57% | 73.44% | 68.71% | 63.13% | 57.19% |

As shown in Table 5, we provide a detailed display of the consistency accuracy at each layer for all neural audio codecs, and the accuracy of the neural audio codec with consistency constraint surpasses that of the baseline models at every layer. Specifically, compared to EnCodec (Défossez et al., ), our method has shown an average consistency improvement of 21.47%, 29.17%, and 36.29% in the first layer, the first 3 layers, and the first 8 layers, respectively. We can observe that consistency accuracy significantly decreases as the number of layers increases, particularly in baseline models. This may suggest that the semantic information in the shallow layers is more relevant to context-independent text, which results in higher consistency accuracy. In contrast, the acoustic information in the deeper layers is more fragile and sensitive, making it more influenced by context (Zhang et al., ). This could create challenges for downstream neural codec language models when predicting audio tokens from these deeper layers.

### 9.4 Explore generality of our method on the number of codebooks.

**Table 6: The speech generation results on LibriTTS test set for VALL-E based on our method with different number of codebooks.**
| Neural Audio Codec | Neural Codec Language Model |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Number of<br>Codebooks | Bandwidth | Slice | Perturbation | Objective | Subjective |  |  |  |
| WER↓ | SIM↑ | UTMOS↑ | MOS↑ | SMOS↑ |  |  |  |  |
| 4 | 2.0 kbps | / | / | 1.93 | 69.86% | 4.29 | 3.88 | 3.35 |
| 20% | phase perturb | 1.15 | 72.50% | 4.24 | 4.06 | 3.40 |  |  |
| 40% | phase perturb | 1.39 | 72.54% | 4.21 | 3.93 | 3.37 |  |  |
| 8 | 4.0 kbps | / | / | 4.73 | 76.95% | 4.10 | 3.73 | 3.50 |
| 20% | phase perturb | 1.84 | 83.71% | 4.31 | 3.97 | 3.73 |  |  |
| 40% | phase perturb | 1.90 | 82.81% | 4.27 | 3.84 | 3.70 |  |  |

We find that consistency declines significantly with deeper layers. To validate that our method is still effective on shallow layers, we evaluate the speech generation performance of VALL-E (Wang et al., ) based on our method with different numbers of codebooks. As shown in Table 6, with different numbers of codebooks, the improvement across all evaluation metrics demonstrates the generalizability of our proposed method.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Consistency Accuracy and WER Correlation

Is consistency accuracy truly correlated with WER? In Figure 4, SpeechTokenizer performs worse than EnCodec and DAC in terms of consistency accuracy, which seems counterintuitive. From my experience, when training VALL-E, SpeechTokenizer typically yields better WER results compared to EnCodec and DAC.
soundness: 3
strengths: 1. The proposed methods are easy to understand and straightforward to implement, which enhances their practical applicability.

2. The identification and in-depth analysis of Discrete Representation Inconsistency in neural audio codecs are novel contributions that highlight an important challenge in the field.

3. The methods substantially enhance the consistency of audio tokens, leading to better performance in speech synthesis tasks, as evidenced by significant improvements in WER and speaker similarity metrics.
summary: The paper investigates the phenomenon of Discrete Representation Inconsistency (DRI) in audio token sequences generated by neural audio codecs. Unlike text tokens, which are deterministic, discrete audio tokens can vary significantly even if the perceptual audio remains identical. This inconsistency complicates the prediction of subsequent tokens in neural codec language models, leading to potential errors in audio generation. The authors propose two novel methods to address DRI: the slice-consistency and perturbation-consistency methods. They demonstrate the effectiveness of these methods through extensive experiments, showing significant improvements in token consistency, reduced Word Error Rate (WER), and enhanced speaker similarity in speech synthesis tasks.
weaknesses: 1, Missing Baseline Comparison

In your discussion of WER improvements, your paper lacks a comprehensive comparison with other works that have also improved VALL-E to reduce WER. Since many follow-up studies on VALL-E aim to lower WER, it’s important to systematically compare your method with these works to highlight your approach's advantages. 

For example, MELLE [1] uses mel-spectrogram to replace codec enhancing model robustness. VALL-E 2 [2] proposes the repetition aware sampling and grouped code modeling to enhance the stability. 


2, Evaluation Metric

Your paper uses different WER and SIM metrics compared to those used in the original VALL-E (WavLM-TDNN,  HuBERT-Large-finetune) and its subsequent improvement papers. This inconsistency makes it difficult for readers to understand the extent of your improvements, leading to confusion. It’s recommended to align your metrics with those in the original VALL-E paper for clearer comparative analysis.

3, Fairness of Baseline Experiments


There are concerns about the fairness of your baseline experiments. It is unclear whether the training configurations for your VALL-E model using your codec were the same as those for the baseline VALL-E models, such as the one using SpeechTokenizer. Ensuring consistent training settings is crucial for a fair comparison.

Additionally, it is puzzling that the WER of your baseline codec outperforms that of SpeechTokenizer. From my understanding, VALL-E with SpeechTokenizer typically achieves a lower WER compared to standard codecs.


[1] Meng L, Zhou L, Liu S, et al. Autoregressive speech synthesis without vector quantization[J]. arXiv preprint arXiv:2407.08551, 2024.


[2] Chen S, Liu S, Zhou L, et al. VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers[J]. arXiv preprint arXiv:2406.05370, 2024.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. The paper assumes that inconsistencies between sliced and full audio representations in codecs lead directly to issues like omissions and repetitions in LLM outputs. However, there is a lack of detailed evidence or analysis showing the extent of this impact. While codec inconsistencies are highlighted, it’s unclear how strongly this correlates with LLM performance issues. Text tokenizers, which don’t face consistency problems, still experience omissions and repetitions, suggesting that other factors could be influencing these issues. More evidence linking codec inconsistency to LLM performance would strengthen this argument.

2. The results in Table 3 show that the scenario with no slicing but with phase perturbation has the lowest consistency (around 7%), yet achieves the second-best WER. This outcome contradicts the paper’s central argument that higher consistency improves model performance. This suggests that low consistency doesn’t necessarily result in poor performance, which weakens the foundation of the paper’s claims.

3. The performance difference between SpeechTokenizer and EnCodec in Table 2 is unusually large, diverging significantly from the results in the original SpeechTokenizer paper[1]. This raises concerns about the fairness and consistency of the comparisons. Without clear justification for these differences, the validity of the conclusions drawn from these comparisons is questionable. The setup for these evaluations needs more transparency to ensure fair and reliable comparisons.
[1] Zhang X, Zhang D, Li S, et al. "Speechtokenizer: Unified speech tokenizer for speech large language models," *arXiv preprint*, arXiv:2308.16692, 2023.
soundness: 2
strengths: 1. The paper introduces a new problem, Discrete Representation Inconsistency (DRI), in discrete audio tokens for neural codec language models, proposing two effective methods, Slice-Consistency and Perturbation-Consistency, to improve token consistency while maintaining audio quality.

2. Thorough experiments demonstrate that the proposed methods effectively reduce DRI and improve key metrics like Word Error Rate (WER) and speaker similarity, validated on both small (LibriTTS) and large (MLS) datasets.

3. The paper is well-organized and easy to follow, clearly explaining the DRI problem, proposed solutions, and their effectiveness, making the study accessible and understandable.
summary: This paper  introduces and analyzes the Discrete Representation Inconsistency (DRI) phenomenon, which occurs in discrete audio tokens. DRI leads to confusion in large language models (LLMs) during prediction, causing omissions and repetitions in speech generation. The inconsistency arises because the same audio segment can be encoded into different discrete sequences depending on the context, complicating the model’s prediction process.

To address this issue, the authors propose two solutions:

Slice-Consistency Method: This method ensures that a randomly sliced segment of audio has an encoded representation consistent with that of the entire audio. It helps maintain consistency without reducing the model's ability to understand the audio.

Perturbation-Consistency Method: This method aligns the encoded representations of audio before and after slight spectral perturbations, enhancing the model's robustness. The perturbations are subtle enough to be imperceptible to human hearing but help the model remain stable against small variations.

The authors conduct extensive experiments to validate the effectiveness of these methods. The results demonstrate that these approaches significantly alleviate the DRI phenomenon and improve speech generation performance. Metrics such as reduced Word Error Rate (WER), increased speaker similarity, and enhanced naturalness of generated speech confirm the success of the proposed solutions.
weaknesses: 1. Lack of Direct Evidence Linking Codec Inconsistency to Performance：The paper says that codec inconsistency causes omissions and repetitions in outputs, but there’s no direct evidence showing how much this affects performance. While the DRI issue is clear, its exact impact on results isn’t. Adding experiments that compare models trained with consistent and inconsistent audio tokens would make this point stronger.

2. Alternative Perturbations and the Importance of Perturbation-Consistency：The paper doesn’t talk about whether other types of perturbations, besides spectral perturbations, could help improve consistency. It also doesn’t clearly explain why Perturbation-Consistency is necessary or how it works. Trying out other types of perturbations and explaining why spectral perturbations were chosen would make the method more convincing.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: See weaknesses above.
soundness: 3
strengths: 1. The paper is well-written and easy to follow.

2. The performance is comparative.

3. The Discrete Representation Inconsistency seems an important question.
summary: This paper is about analysis and solve the problem of Discrete Representation Inconsistency.
weaknesses: 1. Fair Comparison: The authors expanded the training dataset to 44,000 hours. However, in Table 1, I do not think other models, including EnCodec and DAC, were trained on the same data. Is this fair for the results in Table 1?
2. Unprofessional Expressions: The format of the abstracts violates the conference rules. In Figure 3, the word "perception" is presented in both italics and normal text. The equations are not tagged with numbers like "(1), (2)." There is no underline in Table 1, Column "ViSQOL." The bold "Ours and Ours w/o consistency constraint" in the caption of Table 2 should be replaced with quotes. This raises serious questions about the paper, as there is still a long way to go.
3. Efficiency: How large is the model? Is there any insight regarding the parameters and FLOPS? Could the performance boost come from a larger model?
4. Implementation Details: Some important details are not mentioned. The variables related to $\lambda$ in the equation in line 285 are not further detailed in this paper. Are they all set to 1?
5. Novelty: The novelty of this paper is rather weak. There is no structural improvement, only further utilization of the method in Phaseaug. However, the analysis of Discrete Representation Inconsistency is interesting. I consider this as a minor weakness.

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 2
questions: 1. Please clarify the use of re-ranking methods for TTS evaluation. If re-ranking was used, the authors should report performance across different re-ranking levels (e.g., 5, 10, 20 samples, or no re-ranking) to provide a more comprehensive view of its impact on results.

2. To comprehensively assess the TTS system's robustness, I recommend conducting additional evaluations using challenging sentences, such as "22222222 hello 22222222" (some examples can be found in https://ralle-demo.github.io/RALL-E/ or any reasonable test set is acceptable).

3. The baseline codec models in this work are trained on short utterances (around 1 second). It remains unclear whether training a codec model on longer utterances could improve consistency accuracy. Intuitively, codec models trained on short clips may be more susceptible to acoustic variance (the semantic context to leverage is limited). An additional useful experiment would be to include the Mimi codec (https://github.com/kyutai-labs/moshi) as a baseline, given its training on 12-second speech clips and a 7 million-hour dataset. This comparison could offer valuable insights into the effect of training duration and codec capability on consistency accuracy.

4. I recommend adding more perturbations, such as non-audible noise, reverb, or slight volume augmentation when measuring consistency accuracy. This would help in evaluating the robustness of the model's consistency accuracy under varied conditions.

5.  Please include a breakdown of the WER metrics, specifying insertion, deletion, and replacement errors. As the aim of the proposed method is to address omissions and repetitions during speech generation, a detailed report on these error types is crucial to understanding the system's performance.

6. Further evaluation is recommended on music and sound effects benchmarks to validate the effectiveness of the proposed method, as suggested in the weakness part.
soundness: 1
strengths: The problem is interesting.
summary: This paper investigated the problem of Discrete Representation Inconsistency (DRI), where a single audio segment is represented by multiple divergent token sequences. This causes confusion in neural codec language models and leads to omissions and repetitions in speech generation. The authors proposed several consistency regularization techniques that enhance codec reconstruction quality to address this problem. The authors also show that text-to-speech (TTS) systems trained on Encodec using these proposed methods achieve better performance.
weaknesses: 1. There is no theoretical evidence presented to support that improved consistency accuracy in codec models leads to better performance in codec and downstream audio LLM tasks. It would be valuable to see further analysis on this aspect. While the authors highlight the consistency improvement achieved by the proposed method, both the evaluation and training methods (i.e., slice method) are consistent. Therefore, the observed improvement is expected.
2. Neural codec language models for music and sound effects generation (e.g., AudioGen, MusicGen) are also highly relevant. Showing the proposed method used for codec and language models on music and sound effect benchmarks and comparing them with state-of-the-art models is essential to establish the effectiveness of the approach.
3. The reported evaluation of TTS models appears inconsistent with existing literature, particularly regarding WER results. It seems likely that re-ranking may have been used before the WER evaluation, where multiple samples are generated, and the one with the best WER is selected. For instance, VALL-E, known for its limitations in generating speech precisely aligned with text transcriptions, typically achieves around 10% WER on LibriSpeech when trained on LibriLight, which can be improved to approximately 5% through re-ranking. In my opinion, the 1.37% WER reported in the paper is a groundbreaking achievement for TTS. While I recognize that the proposed method could enhance TTS robustness, such a substantial improvement is surprising without further modifications to the training pipeline or model architecture. Additionally, VoiceCraft generally achieves around 20% WER on LibriTTS without re-ranking, whereas the results in the paper show a considerably lower WER of about 2-3%. The paper does not specify how TTS samples were generated; if re-ranking was applied, please clarify this and provide results without re-ranking for an accurate comparison.
4. Although the studied question is interesting, its scientific contribution is limited, or at least needs to be supported by more experiments and results.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Can the authors include the two baselines suggested in the weaknesses section? The first is a kernel size of 1, and the second is the baseline performance of a model trained on the same data without consistency loss. If this isn't possible or necessary, could the authors provide a justification?
soundness: 3
strengths: - The authors identify an interesting problem with audio tokenizers where audio, with or without contextual information,
is encoded by the audio tokenizer into different audio tokens unlike text tokenizer which decode to identical text tokens
- The proposed method aims to retain the original receptive field while enabling the model to address the trade-offs between audio reconstruction quality and resolving the many-to-one problem. To achieve this the method uses a segment of audio randomly sliced, and the encoded representation from this sliced segment is required to closely approximate the corresponding representation obtained from the entire audio. To address many-to-one problem, the representation of an audio and its representation after applying slight spectral
perturbation are forced to be aligned. 
- The experiments cover various neural audio codecs and datasets across different bandwidth settings and model sizes, supporting the generality of the proposed solution. The results demonstrate large improvements in consistency metrics (up to 36.29% in deeper layers), WER reduction (3.72%), and speaker similarity increase (5.68%)
summary: The paper addresses the Discrete Representation Inconsistency (DRI) phenomenon in neural audio codecs, where identical audio segments are represented by different discrete token sequences, reducing downstream task performance on speech generation. The authors propose two methods, slice-consistency and perturbation-consistency, to mitigate DRI by aligning token representations from audio slices and perturbed audio. They show significant consistency improvements in popular neural audio codecs and reduced word error rates (WER) in speech generation tasks using large datasets.
weaknesses: - The paper mainly focuses on consistency-based methods and doesn't extensively compare these approaches to other potential techniques for handling the many-to-one mapping problem, such as the simpler methods introduced in the introduction section (e.g., setting the kernel size to 1). Including such baselines would be helpful for comparison in Table 2.
- The paper would benefit from a baseline without the consistency constraint. The current baselines are different methods/codecs like Encodec, DAC, etc., which might be trained on different datasets. This raises the possibility that the better reconstruction in the author's proposed method might be due to larger training data, rather than the addition of consistency-based loss. To verify this, it would be helpful to include a baseline RVQ-GAN or a model without the additional losses.

Meta-review: > The paper investigates the phenomenon of Discrete Representation Inconsistency (DRI) in audio token sequences generated by neural audio codecs. Unlike text tokens, which are deterministic, discrete audio tokens can vary significantly even if the perceptual audio remains identical. This inconsistency complicates the prediction of subsequent tokens in neural codec language models, leading to potential errors in audio generation. The authors propose two novel methods to address DRI: the slice-consistency and perturbation-consistency methods. They demonstrate the effectiveness of these methods through extensive experiments, showing significant improvements in token consistency, reduced Word Error Rate (WER), and enhanced speaker similarity in speech synthesis tasks.

Reviewer aNo3 is unduly harsh as the authors addressed most of his concerns. However, even by totally discounting reviewer aNo3 score, the paper has scores 6-5-5 and should be rejected. Several of the weaknesses raised by reviewers are real. Even though the paper improved significantly (with more experiments) during the rebuttal period, it is more likely the paper could have been accepted if in its current state from submission time. The initial version was lacking several experiments and explanations.

Decision: Reject