Analyzing and Mitigating Inconsistency in Discrete Audio Tokens for Neural Codec Language Models

Authors: Wenrui Liu, Zhifang Guo, Jin Xu, Yuanjun Lv, Yunfei Chu, Zhou Zhao, Junyang Lin
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: TTS, speech generation, neural audio codec, neural codec lanugage model

Abstract:
Building upon advancements in Large Language Models (LLMs), the field of audio processing has seen increased interest in training audio generation tasks with discrete audio token sequences. However, directly discretizing audio by
neural audio codecs often results in sequences that fundamentally differ from text sequences. Unlike text, where text token sequences are deterministic, discrete audio tokens can exhibit significant variability based on contextual factors, while still producing perceptually identical audio segments. We refer to this phenomenon as \textbf{Discrete Representation Inconsistency (DRI)}. This inconsistency can lead to a single audio segment being represented by multiple divergent sequences, which creates confusion in neural codec language models and results in omissions and repetitions during speech generation. In this paper, we quantitatively analyze the DRI phenomenon within popular audio tokenizers such as EnCodec. Our approach effectively mitigates the DRI phenomenon of the neural audio codec. Furthermore, extensive experiments on the neural codec language model over LibriTTS and large-scale MLS datasets (44,000 hours) demonstrate the effectiveness and generality of our method. The demo of audio samples is available online~\footnote{\url{https://consistencyinneuralcodec.github.io}}.

Decision: reject
Reviewer scores: 5, 6, 5, 1, 6

Reviews and discussion:
<thread>
creation date:
1729471969469

thread_type:
official_review

<thread_object>
creator: reviewer_xpes

<content>
summary:
The paper addresses the Discrete Representation Inconsistency (DRI) phenomenon in neural audio codecs, where identical audio segments are represented by different discrete token sequences, reducing downstream task performance on speech generation. The authors propose two methods, slice-consistency and perturbation-consistency, to mitigate DRI by aligning token representations from audio slices and perturbed audio. They show significant consistency improvements in popular neural audio codecs and reduced word error rates (WER) in speech generation tasks using large datasets.

strengths:
- The authors identify an interesting problem with audio tokenizers where audio, with or without contextual information,
is encoded by the audio tokenizer into different audio tokens unlike text tokenizer which decode to identical text tokens
- The proposed method aims to retain the original receptive field while enabling the model to address the trade-offs between audio reconstruction quality and resolving the many-to-one problem. To achieve this the method uses a segment of audio randomly sliced, and the encoded representation from this sliced segment is required to closely approximate the corresponding representation obtained from the entire audio. To address many-to-one problem, the representation of an audio and its representation after applying slight spectral
perturbation are forced to be aligned. 
- The experiments cover various neural audio codecs and datasets across different bandwidth settings and model sizes, supporting the generality of the proposed solution. The results demonstrate large improvements in consistency metrics (up to 36.29% in deeper layers), WER reduction (3.72%), and speaker similarity increase (5.68%)

weaknesses:
- The paper mainly focuses on consistency-based methods and doesn't extensively compare these approaches to other potential techniques for handling the many-to-one mapping problem, such as the simpler methods introduced in the introduction section (e.g., setting the kernel size to 1). Including such baselines would be helpful for comparison in Table 2.
- The paper would benefit from a baseline without the consistency constraint. The current baselines are different methods/codecs like Encodec, DAC, etc., which might be trained on different datasets. This raises the possibility that the better reconstruction in the author's proposed method might be due to larger training data, rather than the addition of consistency-based loss. To verify this, it would be helpful to include a baseline RVQ-GAN or a model without the additional losses.

questions:
- Can the authors include the two baselines suggested in the weaknesses section? The first is a kernel size of 1, and the second is the baseline performance of a model trained on the same data without consistency loss. If this isn't possible or necessary, could the authors provide a justification?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645908077

thread_type:
official_review

<thread_object>
creator: reviewer_ano3

<content>
summary:
This paper investigated the problem of Discrete Representation Inconsistency (DRI), where a single audio segment is represented by multiple divergent token sequences. This causes confusion in neural codec language models and leads to omissions and repetitions in speech generation. The authors proposed several consistency regularization techniques that enhance codec reconstruction quality to address this problem. The authors also show that text-to-speech (TTS) systems trained on Encodec using these proposed methods achieve better performance.

strengths:
The problem is interesting.

weaknesses:
1. There is no theoretical evidence presented to support that improved consistency accuracy in codec models leads to better performance in codec and downstream audio LLM tasks. It would be valuable to see further analysis on this aspect. While the authors highlight the consistency improvement achieved by the proposed method, both the evaluation and training methods (i.e., slice method) are consistent. Therefore, the observed improvement is expected.
2. Neural codec language models for music and sound effects generation (e.g., AudioGen, MusicGen) are also highly relevant. Showing the proposed method used for codec and language models on music and sound effect benchmarks and comparing them with state-of-the-art models is essential to establish the effectiveness of the approach.
3. The reported evaluation of TTS models appears inconsistent with existing literature, particularly regarding WER results. It seems likely that re-ranking may have been used before the WER evaluation, where multiple samples are generated, and the one with the best WER is selected. For instance, VALL-E, known for its limitations in generating speech precisely aligned with text transcriptions, typically achieves around 10% WER on LibriSpeech when trained on LibriLight, which can be improved to approximately 5% through re-ranking. In my opinion, the 1.37% WER reported in the paper is a groundbreaking achievement for TTS. While I recognize that the proposed method could enhance TTS robustness, such a substantial improvement is surprising without further modifications to the training pipeline or model architecture. Additionally, VoiceCraft generally achieves around 20% WER on LibriTTS without re-ranking, whereas the results in the paper show a considerably lower WER of about 2-3%. The paper does not specify how TTS samples were generated; if re-ranking was applied, please clarify this and provide results without re-ranking for an accurate comparison.
4. Although the studied question is interesting, its scientific contribution is limited, or at least needs to be supported by more experiments and results.

questions:
1. Please clarify the use of re-ranking methods for TTS evaluation. If re-ranking was used, the authors should report performance across different re-ranking levels (e.g., 5, 10, 20 samples, or no re-ranking) to provide a more comprehensive view of its impact on results.

2. To comprehensively assess the TTS system's robustness, I recommend conducting additional evaluations using challenging sentences, such as "22222222 hello 22222222" (some examples can be found in https://ralle-demo.github.io/RALL-E/ or any reasonable test set is acceptable).

3. The baseline codec models in this work are trained on short utterances (around 1 second). It remains unclear whether training a codec model on longer utterances could improve consistency accuracy. Intuitively, codec models trained on short clips may be more susceptible to acoustic variance (the semantic context to leverage is limited). An additional useful experiment would be to include the Mimi codec (https://github.com/kyutai-labs/moshi) as a baseline, given its training on 12-second speech clips and a 7 million-hour dataset. This comparison could offer valuable insights into the effect of training duration and codec capability on consistency accuracy.

4. I recommend adding more perturbations, such as non-audible noise, reverb, or slight volume augmentation when measuring consistency accuracy. This would help in evaluating the robustness of the model's consistency accuracy under varied conditions.

5.  Please include a breakdown of the WER metrics, specifying insertion, deletion, and replacement errors. As the aim of the proposed method is to address omissions and repetitions during speech generation, a detailed report on these error types is crucial to understanding the system's performance.

6. Further evaluation is recommended on music and sound effects benchmarks to validate the effectiveness of the proposed method, as suggested in the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730654636676

thread_type:
official_review

<thread_object>
creator: reviewer_ee2q

<content>
summary:
This paper is about analysis and solve the problem of Discrete Representation Inconsistency.

strengths:
1. The paper is well-written and easy to follow.

2. The performance is comparative.

3. The Discrete Representation Inconsistency seems an important question.

weaknesses:
1. Fair Comparison: The authors expanded the training dataset to 44,000 hours. However, in Table 1, I do not think other models, including EnCodec and DAC, were trained on the same data. Is this fair for the results in Table 1?
2. Unprofessional Expressions: The format of the abstracts violates the conference rules. In Figure 3, the word "perception" is presented in both italics and normal text. The equations are not tagged with numbers like "(1), (2)." There is no underline in Table 1, Column "ViSQOL." The bold "Ours and Ours w/o consistency constraint" in the caption of Table 2 should be replaced with quotes. This raises serious questions about the paper, as there is still a long way to go.
3. Efficiency: How large is the model? Is there any insight regarding the parameters and FLOPS? Could the performance boost come from a larger model?
4. Implementation Details: Some important details are not mentioned. The variables related to $\lambda$ in the equation in line 285 are not further detailed in this paper. Are they all set to 1?
5. Novelty: The novelty of this paper is rather weak. There is no structural improvement, only further utilization of the method in Phaseaug. However, the analysis of Discrete Representation Inconsistency is interesting. I consider this as a minor weakness.

questions:
See weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721158111

thread_type:
official_review

<thread_object>
creator: reviewer_2nt9

<content>
summary:
This paper  introduces and analyzes the Discrete Representation Inconsistency (DRI) phenomenon, which occurs in discrete audio tokens. DRI leads to confusion in large language models (LLMs) during prediction, causing omissions and repetitions in speech generation. The inconsistency arises because the same audio segment can be encoded into different discrete sequences depending on the context, complicating the model’s prediction process.

To address this issue, the authors propose two solutions:

Slice-Consistency Method: This method ensures that a randomly sliced segment of audio has an encoded representation consistent with that of the entire audio. It helps maintain consistency without reducing the model's ability to understand the audio.

Perturbation-Consistency Method: This method aligns the encoded representations of audio before and after slight spectral perturbations, enhancing the model's robustness. The perturbations are subtle enough to be imperceptible to human hearing but help the model remain stable against small variations.

The authors conduct extensive experiments to validate the effectiveness of these methods. The results demonstrate that these approaches significantly alleviate the DRI phenomenon and improve speech generation performance. Metrics such as reduced Word Error Rate (WER), increased speaker similarity, and enhanced naturalness of generated speech confirm the success of the proposed solutions.

strengths:
1. The paper introduces a new problem, Discrete Representation Inconsistency (DRI), in discrete audio tokens for neural codec language models, proposing two effective methods, Slice-Consistency and Perturbation-Consistency, to improve token consistency while maintaining audio quality.

2. Thorough experiments demonstrate that the proposed methods effectively reduce DRI and improve key metrics like Word Error Rate (WER) and speaker similarity, validated on both small (LibriTTS) and large (MLS) datasets.

3. The paper is well-organized and easy to follow, clearly explaining the DRI problem, proposed solutions, and their effectiveness, making the study accessible and understandable.

weaknesses:
1. Lack of Direct Evidence Linking Codec Inconsistency to Performance：The paper says that codec inconsistency causes omissions and repetitions in outputs, but there’s no direct evidence showing how much this affects performance. While the DRI issue is clear, its exact impact on results isn’t. Adding experiments that compare models trained with consistent and inconsistent audio tokens would make this point stronger.

2. Alternative Perturbations and the Importance of Perturbation-Consistency：The paper doesn’t talk about whether other types of perturbations, besides spectral perturbations, could help improve consistency. It also doesn’t clearly explain why Perturbation-Consistency is necessary or how it works. Trying out other types of perturbations and explaining why spectral perturbations were chosen would make the method more convincing.

questions:
1. The paper assumes that inconsistencies between sliced and full audio representations in codecs lead directly to issues like omissions and repetitions in LLM outputs. However, there is a lack of detailed evidence or analysis showing the extent of this impact. While codec inconsistencies are highlighted, it’s unclear how strongly this correlates with LLM performance issues. Text tokenizers, which don’t face consistency problems, still experience omissions and repetitions, suggesting that other factors could be influencing these issues. More evidence linking codec inconsistency to LLM performance would strengthen this argument.

2. The results in Table 3 show that the scenario with no slicing but with phase perturbation has the lowest consistency (around 7%), yet achieves the second-best WER. This outcome contradicts the paper’s central argument that higher consistency improves model performance. This suggests that low consistency doesn’t necessarily result in poor performance, which weakens the foundation of the paper’s claims.

3. The performance difference between SpeechTokenizer and EnCodec in Table 2 is unusually large, diverging significantly from the results in the original SpeechTokenizer paper[1]. This raises concerns about the fairness and consistency of the comparisons. Without clear justification for these differences, the validity of the conclusions drawn from these comparisons is questionable. The setup for these evaluations needs more transparency to ensure fair and reliable comparisons.
[1] Zhang X, Zhang D, Li S, et al. "Speechtokenizer: Unified speech tokenizer for speech large language models," *arXiv preprint*, arXiv:2308.16692, 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730813197230

thread_type:
official_review

<thread_object>
creator: reviewer_iyga

<content>
summary:
The paper investigates the phenomenon of Discrete Representation Inconsistency (DRI) in audio token sequences generated by neural audio codecs. Unlike text tokens, which are deterministic, discrete audio tokens can vary significantly even if the perceptual audio remains identical. This inconsistency complicates the prediction of subsequent tokens in neural codec language models, leading to potential errors in audio generation. The authors propose two novel methods to address DRI: the slice-consistency and perturbation-consistency methods. They demonstrate the effectiveness of these methods through extensive experiments, showing significant improvements in token consistency, reduced Word Error Rate (WER), and enhanced speaker similarity in speech synthesis tasks.

strengths:
1. The proposed methods are easy to understand and straightforward to implement, which enhances their practical applicability.

2. The identification and in-depth analysis of Discrete Representation Inconsistency in neural audio codecs are novel contributions that highlight an important challenge in the field.

3. The methods substantially enhance the consistency of audio tokens, leading to better performance in speech synthesis tasks, as evidenced by significant improvements in WER and speaker similarity metrics.

weaknesses:
1, Missing Baseline Comparison

In your discussion of WER improvements, your paper lacks a comprehensive comparison with other works that have also improved VALL-E to reduce WER. Since many follow-up studies on VALL-E aim to lower WER, it’s important to systematically compare your method with these works to highlight your approach's advantages. 

For example, MELLE [1] uses mel-spectrogram to replace codec enhancing model robustness. VALL-E 2 [2] proposes the repetition aware sampling and grouped code modeling to enhance the stability. 


2, Evaluation Metric

Your paper uses different WER and SIM metrics compared to those used in the original VALL-E (WavLM-TDNN,  HuBERT-Large-finetune) and its subsequent improvement papers. This inconsistency makes it difficult for readers to understand the extent of your improvements, leading to confusion. It’s recommended to align your metrics with those in the original VALL-E paper for clearer comparative analysis.

3, Fairness of Baseline Experiments


There are concerns about the fairness of your baseline experiments. It is unclear whether the training configurations for your VALL-E model using your codec were the same as those for the baseline VALL-E models, such as the one using SpeechTokenizer. Ensuring consistent training settings is crucial for a fair comparison.

Additionally, it is puzzling that the WER of your baseline codec outperforms that of SpeechTokenizer. From my understanding, VALL-E with SpeechTokenizer typically achieves a lower WER compared to standard codecs.


[1] Meng L, Zhou L, Liu S, et al. Autoregressive speech synthesis without vector quantization[J]. arXiv preprint arXiv:2407.08551, 2024.


[2] Chen S, Liu S, Zhou L, et al. VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers[J]. arXiv preprint arXiv:2406.05370, 2024.

questions:
Consistency Accuracy and WER Correlation

Is consistency accuracy truly correlated with WER? In Figure 4, SpeechTokenizer performs worse than EnCodec and DAC in terms of consistency accuracy, which seems counterintuitive. From my experience, when training VALL-E, SpeechTokenizer typically yields better WER results compared to EnCodec and DAC.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732626436093

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To all reviewers - part1

comment:
Thanks for your valuable feedback, and we hope our response fully resolves your concerns.

## Question 1: 
For **Reviewer iYgA**'s Weakness 2: Evaluation Metric
Your paper uses different WER and SIM metrics compared to those used in the original VALL-E (WavLM-TDNN, HuBERT-Large-finetune) and its subsequent improvement papers. This inconsistency makes it difficult for readers to understand the extent of your improvements, leading to confusion. It’s recommended to align your metrics with those in the original VALL-E paper for clearer comparative analysis.

For **Reviewer 2NT9**'s Question 3:
The performance difference between SpeechTokenizer and EnCodec in Table 2 is unusually large, diverging significantly from the results in the original SpeechTokenizer paper[1]. This raises concerns about the fairness and consistency of the comparisons. Without clear justification for these differences, the validity of the conclusions drawn from these comparisons is questionable. The setup for these evaluations needs more transparency to ensure fair and reliable comparisons. 

__Reponse 1:__ In our study, we compared various TTS systems. And the original papers presenting these TTS systems employed different testing configurations [1, 2, 3]. To provide a fair and unbiased comparison, we conducted tests on a subset extracted from LibriTTS, utilizing the Whisper v3 as the ASR model to evaluate WER like XTTS[3], and employing the 3D-speaker toolkit to assess SIM. Additionally, to validate our model's effectiveness, we have followed the testing configuration outlined in VoiceCraft[2] for further experimentation. The results of these experiments are detailed in Table 1 in this section.

Furthermore, to ensure the reproducibility of our results, we have made our code and training scripts publicly available here <https://github.com/ConsistencyInNeuralCodec/ConsistencyInNeuralCodec>.

Table 1: The speech generation results on LibriTTS test set. Bold means the best result, and
underline means the second-best result. **Ours** and **Ours w/o consistency constraint** denote the
same neural audio codecs with and without consistency constraint. The subscripts of the neural
codec language models (e.g., 330M, 44Kh) denote the model size and data scale.

| Neural Audio Codec              | Neural Codec Language Model       | WER↓  | SIM↑   | UTMOS↑ |
|---------------------------------|-----------------------------------|-------|--------|--------|
| /                               | ground_truth                      | 0.0   | 71.38% | 4.12   |
| EnCodec                         | VoiceCraft_330m                   | 8.26  | 51.10% | 3.54   |
|                                 | VoiceCraft_880m                   | 4.72  | 55.78% | 3.73   |
| Mel VQ-VAE                      | XTTS_v2                           | $\underline{3.50}$  | 60.06% | 3.95   |
| SpeechTokenizer                 | USLM                              | 8.01  | 56.82% | 3.09   |
|                                 | AnyGPT                            | 25.75 | 25.66% | 3.19   |
|                                 | VALL-E (reproduced by Amphion[4]) | 14.32 | 48.90% | 3.87   |
| ours w/o consistency constraint | VALL-E_960h                       | 8.51  | 55.90% | 4.08   |
|                                 | VALL-E_44kh                       | 5.11  | 56.20% | 4.12   |
| ours w consistency constraint   | VALL-E_960h                       | 3.51  | $\underline{60.97%}$ | $\underline{4.32}$   |
|                                 | VALL-E_44kh                       | **3.13**  | **61.72%** | **4.34**   |

[1] Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

[2] VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild

[3] XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model

[4] Amphion: An Open-Source Audio, Music, and Speech Generation Toolkit
</content>
</thread_object>

</thread>
<thread>
creation date:
1732626625334

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To all reviewers - part2

comment:
Thanks for your valuable feedback, and we hope our response fully resolves your concerns.

## Question 2: 

For **Reviewer iYgA**'s question 1: Consistency Accuracy and WER Correlation

For **Reviewer 2NT9**'s Weakness 1: Lack of Direct Evidence Linking Codec Inconsistency to Performance
The paper says that codec inconsistency causes omissions and repetitions in outputs, but there’s no direct evidence showing how much this affects performance. While the DRI issue is clear, its exact impact on results isn’t. Adding experiments that compare models trained with consistent and inconsistent audio tokens would make this point stronger.

For **Reviewer aNo3**'s weakness 1: There is no theoretical evidence presented to support that improved consistency accuracy in codec models leads to better performance in codec and downstream audio LLM tasks. It would be valuable to see further analysis on this aspect. While the authors highlight the consistency improvement achieved by the proposed method, both the evaluation and training methods (i.e., slice method) are consistent. Therefore, the observed improvement is expected.

__Reponse 2:__ In our research, we obtain a series of codecs with various consistency accuracy, which we then utilized as audio tokenizers to train VALL-E and observe how consistency impacts WER. As shown in Table 2 in this section, there is a positive correlation between consistency accuracy and WER improvement. Specifically, as the consistency accuracy increases, the WER correspondingly decreases. To visually illustrate the relationship between WER and consistency, we also plotted the figure <https://github.com/ConsistencyInNeuralCodec/consistencyinneuralcodec.github.io/blob/main/fig/consistency_wer/consistency_wer.jpg>.

Table 2: how different consistency accuracy within neural audio codec affect the WER of speech generated by VALL-E. 
| Consistency Accuracy | 49.15 | 64.74 | 76.75% |
| Consistency Accuracy | WER  |
|----------------------|------|
| 49.15%               | 4.27 |
| 64.74%               | 3.66 |
| 76.75%               | 3.51 |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732833292777

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To all reviewers - part3

comment:
In the part 1 of Section to All Reviewers, there are two VALL-E reproductions from different works: SpeechTokenizer+VALL-E reproduced by Amphion[1], and Ours+VALL-E, which is our reproduction. Our reproduced VALL-E demonstrated better speech generation metrics compared to the original VALL-E, not inferior results. Therefore, when assessing the consistency constraints, our experiments conduct on our own VALL-E reproduction provide stronger evidence. And we still report the result of SpeechTokenizer+VALL-E in hope to provide reviewers and readers with a reference.

Although we have provided a detailed explanation of the origins of these two VALL-E models in the last version of the paper's PDF, Reviewer iYgA is still confused by this comparison between two reproduced VALL-E models. Considering that Reviewer iYgA's suggestion makes sense, as comparing the open-source VALL-E with other models could cause unnecessary misunderstandings, and since the open-source VALL-E does not affect our experimental conclusions, we have removed the open-source VALL-E from the TTS systems to be compared in the new version of the PDF.

[1] Amphion: An Open-Source Audio, Music, and Speech Generation Toolkit
</content>
</thread_object>

</thread>


Metareview:
metareview:
> The paper investigates the phenomenon of Discrete Representation Inconsistency (DRI) in audio token sequences generated by neural audio codecs. Unlike text tokens, which are deterministic, discrete audio tokens can vary significantly even if the perceptual audio remains identical. This inconsistency complicates the prediction of subsequent tokens in neural codec language models, leading to potential errors in audio generation. The authors propose two novel methods to address DRI: the slice-consistency and perturbation-consistency methods. They demonstrate the effectiveness of these methods through extensive experiments, showing significant improvements in token consistency, reduced Word Error Rate (WER), and enhanced speaker similarity in speech synthesis tasks.

Reviewer aNo3 is unduly harsh as the authors addressed most of his concerns. However, even by totally discounting reviewer aNo3 score, the paper has scores 6-5-5 and should be rejected. Several of the weaknesses raised by reviewers are real. Even though the paper improved significantly (with more experiments) during the rebuttal period, it is more likely the paper could have been accepted if in its current state from submission time. The initial version was lacking several experiments and explanations.