HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis

Authors: Yuto Nishimura, Takumi Hirose, Masanari Ohi, Hideki Nakayama, Nakamasa Inoue
Primary area: generative models
Keywords: Text-to-speech synthesis, LLM-based TTS, neural audio codec, long-form generation

Abstract:
Recently, Text-to-speech (TTS) models based on large language models (LLMs)
that translate natural language text into sequences of discrete audio tokens have
gained great research attention, with advances in neural audio codec (NAC) mod-
els using residual vector quantization (RVQ). However, long-form speech synthe-
sis remains a significant challenge due to the high frame rate, which increases the
length of audio tokens and makes it difficult for autoregressive language models
to generate audio tokens for even a minute of speech. To address this challenge,
this paper introduces two novel post-training approaches: 1) Multi-Resolution Re-
quantization (MReQ) and 2) HALL-E. MReQ is a framework to reduce the frame
rate of pre-trained NAC models. Specifically, it incorporates multi-resolution
residual vector quantization (MRVQ) module that hierarchically reorganizes dis-
crete audio tokens through teacher-student distillation. HALL-E is an LLM-based
TTS model designed to predict hierarchical tokens of MReQ. Specifically, it incor-
porates the technique of using MRVQ sub-modules and continues training from a
pre-trained LLM-based TTS model. Furthermore, to promote TTS research, we
create MinutesSpeech, a new benchmark dataset consisting of 40k hours of filtered
speech data for training and evaluating speech synthesis ranging from 3s up to
180s. In experiments, we demonstrated the effectiveness of our approaches by ap-
plying our post-training framework to VALL-E. We achieved the frame rate down
to as low as 8 Hz, enabling the stable minitue-long speech synthesis in a single
inference step. Audio samples, dataset, codes and pre-trained models are available
at https://yutonishimura-v2.github.io/HALL-E_DEMO.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730113207220

thread_type:
official_review

<thread_object>
creator: reviewer_jbx3

<content>
summary:
The paper introduces a novel text-to-speech (TTS) model designed to generate extended speech sequences. To address the inherent limitations of autoregressive language models in handling long outputs, the authors propose the use of *multi-resolution audio codes* (MReQ). Furthermore, the paper presents HALL-E, an autoregressive language model specifically developed to generate these MReQ audio codes efficiently. In addition to the model, the authors construct a new benchmark dataset comprising 40,000 hours of curated speech data. The experimental results demonstrate that the proposed approach can achieve a low frame rate of 8 Hz, highlighting its potential for generating high-quality long-form speech with reduced computational overhead.

strengths:
1. This paper introduces the Multi-Resolution Requantization (MReQ) model, designed to enhance signal processing across multiple granularities.  
2. It presents the development of HALL-E, an autoregressive language model capable of generating extended speech waveforms with high fidelity.  
3. Furthermore, the study proposes a novel benchmark dataset, MinutesSpeech, specifically curated to evaluate the performance of models on long-duration speech sequences.

weaknesses:
1. The concept of time-varying audio codes is not a new development. For reference, please see the work by [Variable-rate Discrete Representation Learning (2021)] [https://arxiv.org/pdf/2103.06089].

2. A notable issue lies in maintaining temporal coherence and acoustic fidelity, as evidenced by the declining performance observed in the SIM score.

3. The model continues to face difficulties in handling spontaneous speech, highlighting an ongoing challenge in this area.

questions:
--
</content>
</thread_object>

</thread>
<thread>
creation date:
1730224629525

thread_type:
official_review

<thread_object>
creator: reviewer_azfo

<content>
summary:
The paper presents three contributions: (1) Multi-Resolution Requantization (MReQ), a framework that post-trains a conventional RVQ codec into one with a different token rate in each layer; (2) HALL-E, an extension of the VALL-E framework to utilize an MReQ-based codec; and (3) the introduction of MinutesSpeech, a new benchmark for minute-long speech synthesis. Leveraging the MReQ concept, the authors successfully train a codec whose first layer operates at only 8Hz. With this low-token rate modeling, the HALL-E model generates minutes-long speech with significantly better WER and RTF compared to the original VALL-E model.

strengths:
The three contributions listed above are valid and well-received. The topic of minute-long speech synthesis is also one that will attract a wide audience.

weaknesses:
While the paper is generally well-written, there are several points that seem unclear to me. It is difficult to fully accept their proposal unless these points are clarified. I am open to reconsidering the score later.

- Unreasonable usage of PESQ raises doubts about their experiments.
  - Why and how can PESQ be used to filter out noisy data in the data curation pipeline? PESQ requires reference audio to compute the score, and I cannot understand how it could be applied for data curation purposes.
  - Why is the PESQ score for the ground truth so low in Table 3? The PESQ score for the ground truth should be 4.64.
- Questionable inference and evaluation settings
  - Zero-shot TTS experiment was conducted by “continual” setting or “cross-utterance” setting? From the description, it’s unclear.
  - Assuming that a "continual" setting is used, did the authors exclude the audio prompt portion of the synthesized audio when computing the speaker SIM? This is the approach taken in the VALL-E paper and other related works. The increase in speaker SIM in Table 9 with respect to the prompt length is unusually large, raising concerns that the authors may not have excluded the prompt from the speaker similarity computation.
  - Why is DNSMOS used? DNSMOS is specifically designed and trained to assess the cleanness of speech rather than its naturalness. I strongly suggest replacing it with a naturalness MOS metric, such as UTMOS.
- Other minor request to assess the quality of the model
  - Please present the speaker similarity and MOS scores (UTMOS?) for Table 3 and Table 10.
  - Please provide a breakdown of the AR and NAR model RTFs for Table 7.

Additionally, I found several minor issues in the description. They are not critical, but they need to be revised:
  - Equation 2 is likely incorrect. I believe it should be x_l = x_{l-1} - \tilde{z_l}
  - Similarly, Equation 9 is likely incorrect.
  - In Table 8, the upsampling ratio should be explicitly described instead of simply listing "HALL-E" in the last row.

questions:
Please address the questions raised in the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730359399432

thread_type:
official_review

<thread_object>
creator: reviewer_uzgt

<content>
summary:
To deal with the long-form speech synthesis problem, this paper propose a multi-resolution re-quantization method which hierarchically reorganizes discrete tokens with teacher-student distillations. Based on the hierarchical codec codes, an LLM-based TTS model HALL-E is proposed for speech synthesis. Besides these, a new benchmark dataset MinutesSpeech is introduced for minute-long speech synthesis.

This paper is well written and easy to follow.

strengths:
1. This paper has three contributions 1). A multi-resolution requantization method MReQ is proposed to generate hierarchical codec codes. 2). A hierarchical LLM-base TTS model is proposed to predict MReQ codec codes. 3). A minute-long speech synthesis dataset is introduced.
2. This paper is well written and easy to follow.

weaknesses:
1. Some important codec models are not compared with, such as DAC and Vocos.
2. MReQ is only tested on the speech reconstruction, without evaluated with audio dataset.
3. Some latest zero-shot TTS models are not compared with, such as Voicebox, E2 and VALL-E 2.

questions:
1. Have you tried to train MReQ with a much smaller sampling rate, such as 2 or 4 Hz for the first layer? 
2. What is the accuracy of the sub-encoder E to predict b_{k+1} based on a_{k+1}? Will this accuracy affect the final performance a lot?
3. For the L_{total}， are the weights of the three sub losses equal and all set to 1.0 ? Have you tried other settings?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677871020

thread_type:
official_review

<thread_object>
creator: reviewer_1npx

<content>
summary:
This paper presents two post-training approaches to resolve the long context length issues for modern transformer-based autoregressive TTS models based on NAC models. MReQ as a framework reduces the framerate by introducing a novel Multi-resolution vector quantization module that allows the user to decompose an existing NAC model into several different code books, each of which operate at a different framerate. This allows them to reduce the bottleneck of the AR component of LLM-based TTS by reducing the frequency of the first component down to 8 Hz. To train this MRVQ module, student-teacher distillation is required. Using this module, this work presents HALL-E, a hierarchical TTS model that generates tokens based on the MReQ tokens. This paper also introduces MinutesSpeech, a benchmark dataset for TTS synthesis that is curated for long-form speech.

strengths:
Idea is pretty interesting and novel and well motivated. The MReQ module is fairly complex, but diagrams and descriptions of the technique are well written. Manuscript is very detailed and training and post-training are well documented. Along with the code release, results look to be easily replicable. 

Authors go through great lengths to train and evaluate everything from scratch and provide extensive results for reconstruction and zero-shot TTS which included both automatic and subjective results. Experiments were run on multiple NAC models, showing that this technique can generalize to pre-existing NAC models.

Benchmark dataset is a welcome addition and the generation and evaluation of the dataset are well documented. Results show RTF improvements, indicating that this technique indeed reduces the bottleneck behind transformer-based TTS.

weaknesses:
* One weakness in the study is that authors do not consider distribution shifts between the different dataset used to train the NAC model and the dataset used for post training. In their experiments, they pretrain their own NAC models (Encodec and Speechtokenizer) on the MinutesSpeech training set and did post training with MReQ with the same training data. However, in practice, researchers will not have access to the datasets used to pretrain these NAC models, so there may be some distribution shift. A similar issue occurs with quantization, where you need some training data to calibrate the model. Additionally, while SpeechTokenizer is meant to tokenize speech, Encodec was originally designed to encode speech along with other kinds of audio. It is unclear whether this capability will be preserved after post-training. Ablation studies do not cover what happens when you try to use this technique with an off-the shelf NAC model like SpeechTokenizer or Encodec off the shelf. 

* The statement on line 211/212 that Training NAC models with the MRVQ module does not seem to be justified or elaborated on. Is this just saying that training this kind of codec model from scratch rather than from a pretrained model is difficult? It seems like from Table 10, that for the MReQ model, starting this kind of training w/o pretraining seems to not affect the WER or RESQ significantly.

* The qualitative results in the ablation study near line 495 are not well explained. The statement that this waveform is unnatural with almost no silence is not clear without a transcription. From a glance, this seems to be a problem with the prosody of the generation, but this is unclear without hearing the samples. Appendix D1 also does not seem to elaborate on this much. I tried to listen to the samples from MinutesSpeech test-90s, and it seems like this is primarily a prosody issue as the Valle model seems to speak at a very consistent rate with much louder inhalation sounds.

* Furthermore, the ablation study for “Can HALL-E handle long audio prompts?” seems trivial, it should be expected that longer voice prompts gives the model more context on the user’s speech. It would be more interesting to see if this effect exists or is less pronounced in the VALLE model.

* There are also some wording errors in the manuscript. For instance, in the implementation details on page 4, lines 199 to 201, the manuscript says that “Figure 2a shows the MRVQ module applied to the Encodec model”, however, Figure 2a actually just shows the normal Encodec model as a teacher model. This should actually be Figure 2b that shows the MRVQ module applied to the Encodec model. Furthermore, Line 201 states that “Figure 2b shows the LRVQ block”, however, it is actually Figure 2c that shows the LRVQ block.  Also, should the statement in line 485/486: “The results indicate that both losses and post-training play important roles in the overall performance.” say that both losses and “pre-training” play important roles, since the second row of Table 10 is about the pre-training of the Encodec model?

questions:
* This technique seems well suited for LLM-based TTS solutions, similar to Valle with AR and NAR components. However, can you also say anything about how this kind of encoding might be used as a general purpose codec for speech in general? It would be interesting if this model could have lower bitrates or higher robustness. Furthermore, does this means that the PreQ and PostQ components of your LRVQ blocks would be unnecessary since they are primarily used to train the NAR components of the TTS model?

* With a VALLE model based on Encodec, once can remove RVQ code sequences from the NAR generation to get faster generation, in exchange for coarser generation. For this MReQ based paradigm, how does the audio quality here change?

* For the second ablation in section 7.4, how does this effect appear with the VALLE model that you trained?

* For the third ablation in section 7.4, it is very surprising that even w/o pretraining, it does not seem like this technique suffers significantly (compared to the performance drop in Table 11 for HALL-E). Why is that?

* The splits of the dataset in Table 2 and the description of why these splits were created in Section 7.1 seems very arbitrary and over designed for these experiments. Are these training splits deduplicated from each other?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705685211

thread_type:
official_review

<thread_object>
creator: reviewer_fwrz

<content>
summary:
The paper presents HALL-E, a neural codec language model aimed at addressing challenges in minute-long zero-shot text-to-speech (TTS) synthesis. It introduces Multi-Resolution Requantization (MReQ) to reduce frame rates in neural audio codec (NAC) models, and proposes HALL-E, which leverages MReQ for efficient and effective long-form speech synthesis. The work also introduces MinutesSpeech, a new speech dataset for long-context conversational TTS. The experimental results are extensive, showing that the proposed method can outperform baseline models both quantitatively and qualitatively.

strengths:
* The paper offers thorough quantitative and qualitative results, demonstrating that the proposed generative modeling outperforms baseline methods in long-form speech synthesis.
* The introduction of MinutesSpeech is a noteworthy addition to TTS research, providing high-quality, long-context speech data. The illustrated curation and preprocessing techniques further enhance its utility.
* The ablation studies on MRVQ provide insights into the necessity of each component, justifying their inclusion in the architecture.

weaknesses:
* Although the MReQ module achieves low-resolution quantization while maintaining sample quality, its design is excessively complicated, posing challenges for future research to build upon. The LRVQ design incorporates multiple RVQ modules—pre-quantizer, main quantizer, and post-quantizer—alongside the additional HSR loss, making it significantly more complex than prior RVQ methods. Furthermore, the use of multiple RVQ quantizations within the NAR transformer of HALL-E adds another layer of complexity, reducing the accessibility of the proposed approach.
* The paper only compares the proposed method with VALL-E and does not include recent zero-shot TTS baselines. Considering that non-autoregressive modeling can be advantageous for long speech generation, incorporating comparisons with state-of-the-art non-autoregressive zero-shot TTS models would better demonstrate the effectiveness and relevance of the proposed method.

questions:
* The description of the LRVQ modules appears to omit the use of commitment loss for residual vector quantization. If the LRVQ module does not require commitment loss, the authors would explain the reasoning behind this and how the training is stabilized without it.
* Providing a detailed description of the pre-trained LLM-based TTS model that HALL-E is post-trained on would greatly aid in understanding the training process and setup of the proposed model.
</content>
</thread_object>

</thread>
<thread>
creation date:
1740106826085

thread_type:
public_comment

<thread_object>
creator: ~ewald_enzinger1

<content>
title:
Issues running tts_demo.py script

comment:
Dear authors,

I very much enjoyed reading your paper! Thank you for releasing the source code as supplementary material.
I tried to run the tts_demo.py script, but ran into some issues:

It appears the SpeechGenSolver class implementation is missing from audiocraft/solvers/valle_ar.py:
```
(halle) ~/HALL-E/scripts $ python tts_demo.py
Traceback (most recent call last):
  File "/home/user/HALL-E/scripts/tts_demo.py", line 6, in <module>
    from audiocraft.models import (
  File "/home/user/HALL-E/audiocraft/__init__.py", line 24, in <module>
    from . import data, models, modules
  File "/home/user/HALL-E/audiocraft/models/__init__.py", line 15, in <module>
    from .halle import Halle
  File "/home/user/HALL-E/audiocraft/models/halle.py", line 20, in <module>
    from ..solvers.valle_ar import SpeechGenSolver
  File "/home/user/HALL-E/audiocraft/solvers/__init__.py", line 21, in <module>
    from .halle_ar import HalleARSolver
  File "/home/user/HALL-E/audiocraft/solvers/halle_ar.py", line 35, in <module>
    from .valle_ar import SpeechGenSolver
ImportError: cannot import name 'SpeechGenSolver' from 'audiocraft.solvers.valle_ar' (/home/user/HALL-E/audiocraft/solvers/valle_ar.py)
```
With the assumption that ValleARSolver is equivalent to SpeechGenSolver, I tried to fix the imports, but ran into another issue: The function "load_hier_lm_model" is missing from audiocraft/models/loaders.py:
```
$ python tts_demo.py
Traceback (most recent call last):
  File "/home/user/HALL-E/scripts/tts_demo.py", line 6, in <module>
    from audiocraft.models import (
  File "/home/user/HALL-E/audiocraft/__init__.py", line 24, in <module>
    from . import data, models, modules
  File "/home/user/HALL-E/audiocraft/models/__init__.py", line 16, in <module>
    from .halle import Halle
  File "/home/user/HALL-E/audiocraft/models/halle.py", line 25, in <module>
    from .loaders import load_compression_model, load_hier_lm_model
ImportError: cannot import name 'load_hier_lm_model' from 'audiocraft.models.loaders' (/home/user/HALL-E/audiocraft/models/loaders.py)
```
Are you planning to release an updated source code repository?

Thanks,
Ewald
</content>
</thread_object>

</thread>
<thread>
creation date:
1742302995736

thread_type:
public_comment

<thread_object>
creator: ~yuancheng_wang1

<content>
title:
Repo Github not Found

comment:
Dear authors,

Thanks for your great work! However, it seems like the repo github https://yutonishimura-v2.github.io/HALL-E_DEMO can not found. And I would like to ask if you are going to open source the MinutesSpeech dataset?

Thanks, Yuancheng
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this work, the authors targeted to resolve the challenge of long-form TTS. The high frame rate results in the long length of audio tokens, which makes it different for autoregressive language models to generate tokens. Representative works such as VALL-E encounter challenges in generating high-quality audio for extended sequences. Consequently, this research is significant for advancing speech generation beyond 10+ seconds. 

There are two main contributions of this paper: 1) Multi-Resolution Requantization (MReQ) which is used to reduce the frame rate of neural codec. 2) HALL-E which is an LLM-based TTS model to predict hierarchical tokens of MReQ. Furthermore, the authors have developed a new dataset comprising 40,000 hours of training data, which will support the community in researching long-form speech generation. 

The work is sufficiently novel to address a very meaningful problem of speech generation. The experiments are well-executed with convincing results.  During the rebuttal, the authors actively answered reviewers’ questions with clarified details. Therefore, the work is worth publication in ICLR.

In summary, the strength of this paper is effectively addressing the challenging problem of synthesizing speech longer than 10 seconds by utilizing the proposed MReQ to reduce the frame rate and HALL-E to model the hierarchical tokens of MReQ. However, a noted weakness is the lack of comparison with recent advanced zero-shot TTS models such as VALL-E 2, which has shown significant improvement over VALL-E used in the paper, despite the inclusion of a comparison with MaskGCT during the rebuttal.