HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis

## Abstract

Abstract Recently, Text-to-speech (TTS) models based on large language models (LLMs) that translate natural language text into sequences of discrete audio tokens have gained great research attention, with advances in neural audio codec (NAC) models using residual vector quantization (RVQ).
However, long-form speech synthesis remains a significant challenge due to the high frame rate, which increases the length of audio tokens and makes it difficult for autoregressive language models to generate audio tokens for even a minute of speech.
To address this challenge, this paper introduces two novel post-training approaches: 1) Multi-Resolution Requantization (MReQ) and 2) HALL-E.
MReQ is a framework to reduce the frame rate of pre-trained NAC models.
Specifically, it incorporates multi-resolution residual vector quantization (MRVQ) module that hierarchically reorganizes discrete audio tokens through teacher-student distillation.
HALL-E is an LLM-based TTS model designed to predict hierarchical tokens of MReQ.
Specifically, it incorporates the technique of using MRVQ sub-modules and continues training from a pre-trained LLM-based TTS model.
Furthermore, to promote TTS research, we create MinutesSpeech, a new benchmark dataset consisting of 40k hours of filtered speech data for training and evaluating speech synthesis ranging from 3s up to 180s.
In experiments, we demonstrated the effectiveness of our approaches by applying our post-training framework to VALL-E.
We achieved the frame rate down to as low as 8 Hz, enabling the stable minitue-long speech synthesis in a single inference step.
Audio samples, dataset, codes and pre-trained models are available at https://yutonishimura-v2.github.io/HALL-E_DEMO .

## 1 Introduction

Recent advances in large language models (LLMs) have enabled us to model complex linguistic structures and patterns with unprecedented precision in natural language processing tasks (Brown et al., ; Chowdhery et al., ; Zeng et al., ).
Motivated by these developments, LLM-based text-to-speech (TTS) models have gained significant research interest for their ability to model complex speech structures and patterns, enabling the synthesis of more natural-sounding speech in a zero-shot manner (Wang et al., ; Zhang et al., ; Song et al., ; Han et al., ; Chen et al., ; Xin et al., ; Meng et al., ).
The core concept behind LLM-based TTS models is to translate natural language text into a sequence of audio tokens, typically using frozen natural audio codec (NAC) models that quantize audio signals into discrete tokens via vector quantization techniques (Zeghidour et al., ; Défossez et al., ; Kumar et al., ; Zhang et al., ; Wu et al., ; Huang et al., ; Du et al., ; Yang et al., ).

Since LLMs are becoming capable of capturing long context in text (Guo et al., ; Chen et al., ; Han et al., ), LLM-based TTS models are also expected to handle long context to synthesize speech over extended periods.
However, this presents significant challenges, mainly because the length of audio tokens per second is typically large.
More specifically, when using an autoregressive language model to predict audio tokens, as in the VALL-E architecture (Wang et al., ), the high frame rate(^1^11We refer to the number of audio tokens per second as the frame rate (Hz).) at the first layer of residual vector quantization (RVQ) (Zeghidour et al., ) in NAC models often becomes a major factor that hinders the synthesis of long speech.
As recently discussed and investigated by Han et al. (); Chen et al. (), reducing the frame rate is essential, but not straightforward. Simply reducing the frame rate in NAC models results in degraded audio quality and incorrect phoneme pronunciation.
Addressing this issue requires novel solutions that bridge NAC models and LLM-based TTS models.

In this paper, we propose a novel approach to tackle the challenge of minitue-long speech synthesis in LLM-based TTS models by introducing a hierarchical post-training framework that effectively manages the trade-off between reducing frame rate and producing high-quality speech.
Our contributions are summarized as follows:

- 1)
We propose Multi-Resolution Requantization (MReQ), a post-training framework for hierarchically reorganizing pre-trained RVQ module to reduce the frame rate at lower quantization layers. MReQ incorporates a multi-resolution residual vector quantization (MRVQ) module into a pre-trained NAC model and continues training in a teacher-student distillation manner.
This results in reducing the frame rate at the first quantization layer to 8 Hz.
- 2)
We propose HALL-E, a hierarchical LLM-based TTS model designed to predict hierarchical tokens of MReQ.
The AR model is trained using 8Hz tokens, while the NAR model is trained by using sub-modules in MRVQ, and continues training from a pre-trained LLM-based TTS model.
- 3)
We introduce MinutesSpeech, a new benchmark dataset to promote TTS research, particularly for minute-long speech synthesis.
The training set consists of 40k hours of automatically filtered and balanced speech data.
The test set consists of 8 hours of speech data with transcriptions created by professional transcribers.
- 4)
We thoroughly conducted experiments to provide best practices for managing the trade-off between reducing frame rate and producing high-quality speech, while demonstrating the effectiveness and efficiency of our approach.
We open-source dataset, codes and pre-trained models along with audio samples at [https://yutonishimura-v2.github.io/HALL-E_DEMO](https://yutonishimura-v2.github.io/HALL-E_DEMO).

## 2 Related Work

Neural audio codec models.
NAC models produce discrete audio tokens by quantizing audio signals.
SoundStream (Zeghidour et al., ) and Encodec (Défossez et al., ) are pioneering NAC models, which significantly improved compression efficiency over traditional audio codecs.
Recent studies have proposed NAC models with a focus on maintaining performance in speech processing tasks. Examples include
SpeechTokenizer (Zhang et al., ),
Descript Audio Code (Kumar et al., ),
AcademiCodec (Yang et al., ),
AudioDec (Wu et al., ),
RepCodec (Huang et al., ), and FunCodec (Du et al., ).
Many studies have focused on reducing bps, while few have explored lowering the frame rate. Défossez et al. () achieved the lowest frame rate for a NAC model, reaching 12.5Hz by incorporating Transformer models and SpeechTokenizer.
We report achieving an even lower frame rate of 8Hz, which is about 1.5 times shorter than it.

Zero-shot TTS.
Zero-Shot TTS aims to synthesize speech from text in the voice of a target speaker using only a short reference audio segment from that speaker.
Early models relied on speaker embeddings or speaker adaptation (Casanova et al., ; Arik et al., ; Chen et al., ),
while recent studies have focused on LLM-based models that use NAC models in conjunction with LLMs.
VALL-E (Wang et al., ) was the first LLM-based model, demonstrating impressive capabilities in zero-shot TTS tasks.
Follow-up studies have explored various extensions such as VALL-E X for cross-lingual TTS (Zhang et al., ), ELLA-V using Montreal forced aligner (Song et al., ),
RALL-E using prosody features (Xin et al., ), VALL-E R using monotonic alignment (Han et al., ), VALL-E 2 using grouped code modeling (Chen et al., ), and MELLE using mel-spectrogram features (Meng et al., ).
Prosody information can also be modeled by LLMs
latent language mode
Mega-TTS (Jiang et al., ) and Mega-TTS 2 (Jiang et al., ) introduced prosody LLMs to generate more natural prosody.
Meanwhile, diffusion-based models (e.g., NaturalSpeech2/3 (Shen et al., ; Ju et al., ) and Voicebox (Le et al., )) and prompt-based models (e.g., Prompt-TTS2 (Guo et al., ; Leng et al., )) are also known to be effective to generate high-quality controllable speech.
Beyond speech synthesis, several studies proposed audio generation models such as
UniAudio (Yang et al., ),
Audiobox (Vyas et al., ).
In contrast, this work explores post-training methods to reduce the frame rate of LLM-based models, aiming at minute-long speech synthesis given a pre-trained NAC model such as Encodec.

## 3 Preliminaries

Neural audio codec.
A NAC model typically consists of three components: an encoder $\mathrm{Enc}(\cdot)$, a vector quantizer $\mathrm{VQ}(\cdot)$, and a decoder $\mathrm{Dec}(\cdot)$.
The quantizer is the core component that produces discrete audio tokens.
This work assumes that an RVQ module (Zeghidour et al., ) is used as the quantizer, which is defined as follows:
$\displaystyle\bm{z}_{l}=\mathrm{VQ}_{l}(\bm{x}_{l-1}),$
(1)
$\displaystyle\bm{x}_{l+1}=\bm{x}_{l}-\tilde{\bm{z}}_{l},$
(2)
$\displaystyle\bm{h}=\sum_{l=1}^{L}\tilde{\bm{z}}_{l},$
(3)
where $\bm{x}_{0}=\mathrm{Enc}(\bm{x}_{\mathrm{in}})\in\mathbb{R}^{d\times n}$ is the encoder output for the input audio $\bm{x}_{\mathrm{in}}$,
$d$ is the latent dimension,
$n$ is the sequence length,
$\mathrm{VQ}_{l}$ is a vector quantizer,
$\bm{z}_{l}\in\mathbb{N}^{n}$ is a discrete token sequence,
$\tilde{\bm{z}}_{l}=\mathrm{Emb}_{l}(\bm{z}_{l})\in\mathbb{R}^{d\times n}$ is a sequence of embeddings corresponding to $\bm{z}_{l}$ obtained through a learnable embedding layer $\mathrm{Emb}_{l}(\cdot)$(^2^22In this paper, the tilde symbol ( $\tilde{}$ ) denotes the embeddings corresponding to a token sequence.),
$l\in\{1,2,\cdots,L\}$ is the layer index, and $L$ is the number of layers. The output $\bm{h}\in\mathbb{R}^{d\times n}$ is then fed into the decoder to reconstruct the input audio as $\bm{y}=\mathrm{Dec}(\bm{h})$.

LLM-based TTS.
An LLM-based TTS typically consists of two decoder-only language models: an autoregressive (AR) model $T_{\mathrm{ar}}$ and a non-autoregressive (NAR) model $T_{\mathrm{nar}}$ (Wang et al., ).
The speech synthesis procedure is given by the following equations:
$\displaystyle\hat{\bm{z}}_{1}=T_{\mathrm{ar}}(\bm{t},\bm{z}_{1}^{\mathrm{pr}}),$
(4)
$\displaystyle\hat{\bm{z}}_{l+1}=T_{\mathrm{nar}}(\bm{t},\bm{h}^{\mathrm{pr}}_{
L},\hat{\bm{h}}_{l},l),$
(5)
$\displaystyle\hat{\bm{h}}_{l}=\sum_{l^{\prime}=1}^{l}\hat{\tilde{\bm{z}}}_{l^{
\prime}},$
(6)
$\displaystyle\hat{\bm{y}}=\mathrm{Dec}([\bm{h}^{\mathrm{pr}}_{L},\hat{\bm{h}}_
{L}]),$
(7)
where $[\bm{h}^{\mathrm{pr}}_{L},\hat{\bm{h}}_{L}]$ denotes the concatenation of these two matrices along the time axis.
In Eq. (4), $T_{\mathrm{ar}}$ generates an audio token sequence $\hat{\bm{z}}_{1}\in\mathbb{N}^{n^{\prime}}$ corresponding to the first layer of RVQ given two inputs: a text prompt $\bm{t}$ and an audio prompt $\bm{z}^{\mathrm{pr}}_{1}=\mathrm{VQ}_{1}(\mathrm{Enc}(\bm{x}_{\mathrm{pr}}))$ extracted from an audio input $\bm{x}_{\mathrm{pr}}$.
In Eq. (5), $T_{\mathrm{nar}}$ iteratively generates token sequences $\hat{\bm{z}}_{l+1}\in\mathbb{N}^{n^{\prime}}$ from the accumulated hidden features $\hat{\bm{h}}_{l}$ in Eq. (6) and the audio prompt’s hidden features $\bm{h}^{\mathrm{pr}}_{L}$.
Finally, in Eq. (7), speech $\hat{\bm{y}}$ is generated.
Note that $\mathrm{Enc},\mathrm{VQ}_{1}$, and $\mathrm{Dec}$ are from a frozen NAC model.

Figure: Figure 1: Preliminary experiments on varying the frame rate of Encodec.
Refer to caption: x1.png

Preliminary experiments.
LLM-based TTS models have a predefined context window size and are typically trained with speech data ranging from several seconds to several tens of seconds. To generate long speech segments, a straightforward approach is to reduce the frame rate in the NAC model.
However, reducing the frame rate below 48 Hz significantly
decreases speech reconstruction performance as shown in
Figure 1, where we evaluated the performance of Encodec (Défossez et al., ) in terms of the Perceptual Evaluation of Speech Quality (PESQ) scores and word error rates (WERs) as functions of frame rates.
Specifically, it is confirmed that training becomes entirely difficult at 8Hz.
Therefore, in this study, we propose a NAC model that works even at an 8Hz, demonstrating a significant improvement over existing limitations.

## 4 MReQ: Multi-Resolution Requantization

This section introduces MReQ, a post-training framework for hierarchically reorganizing a pre-trained RVQ module to reduce the frame rate.
Specifically, MReQ incorporates a multi-resolution residual vector quantization (MRVQ) module to a pre-trained NAC model as shown in Figure 2, and continues training the NAC model in a teacher-student distillation manner.
For a pre-trained 48Hz Encodec model, MReQ reduces the frame rate at the first quantization layer to 8 Hz. This enables LLM-based TTS models to handle longer contexts.

### 4.1 Architecture

The MRVQ module is a nested structure of RVQ.
Specifically, it consists of a residual structure composed of multiple low frame-rate residual vector quantization (LRVQ) blocks, each of which is itself a residual structure operating at a different frame rate.
The definition is given as follows.

Definition 1 (MRVQ module).
Let $\bm{x}_{0}\in\mathbb{R}^{d\times n_{0}}$ be an encoder output, where $d$ is the latent dimension and $n_{0}=Ts_{0}$ is the sequence length depending on the time length $T$ (sec) and the frame rate $s_{0}$ (Hz).
The MRVQ module is defined as follows:
$\displaystyle\bm{c}_{k}=\mathrm{LRVQ}^{(k)}_{\alpha\mathchar 45\relax\beta
\mathchar 45\relax\gamma}(\bm{x}_{k-1}),$
(8)
$\displaystyle\bm{x}_{k+1}=\bm{x}_{k}-\tilde{\bm{c}}_{k},$
(9)
$\displaystyle\bm{h}=\sum_{k=1}^{K}\tilde{\bm{c}}_{k},$
(10)
where $\mathrm{LRVQ}^{(k)}_{\alpha\mathchar 45\relax\beta\mathchar 45\relax\gamma}$ is an LRVQ block, $K$ is the number of blocks, $\alpha\mathchar 45\relax\beta\mathchar 45\relax\gamma$ is a triplet of hyperparameters to determine the block structure.

Definition 2 (LRVQ block).
Each LRVQ block $\mathrm{LRVQ}^{(k)}_{\alpha\mathchar 45\relax\beta\mathchar 45\relax\gamma}$ consists of five components:
a pre-quantizer $\mathrm{PreQ}^{(k)}_{\alpha}$,
a sub-encoder $E_{k}$ for down sampling,
a main quantizer $\mathrm{Quant}^{(k)}_{\beta}$,
a sub-decoder $D_{k}$ for upsampling, and a post-quantizer $\mathrm{PostQ}^{(k)}_{\gamma}$. The quantization procedure is given by
$\displaystyle\bm{a}_{k}=\mathrm{PreQ}^{(k)}_{\alpha}(\bm{x}_{k-1}),$
(11)
$\displaystyle\bm{b}_{k}=\mathrm{Quant}^{(k)}_{\beta}(E_{k}(\tilde{\bm{a}}_{k})),$
(12)
$\displaystyle\bm{c}_{k}=\mathrm{PostQ}^{(k)}_{\gamma}(D_{k}(\tilde{\bm{b}}_{k}
)),$
(13)
where $\bm{a}_{k},\bm{b}_{k},\bm{c}_{k}$ are token sequences.
The three quantizers $\mathrm{PreQ}^{(k)}_{\alpha},\mathrm{Quant}^{(k)}_{\beta}$ and $\mathrm{PostQ}^{(k)}_{\gamma}$ are implemented using RVQ with $\alpha$, $\beta$, and $\gamma$ layers, respectively.
Note that $\bm{b}_{k}\in\mathbb{N}^{\beta\times n_{k}}$ is the token sequence representing audio in a low frame rate. Its length is given by $n_{k}=Ts_{k}$, where $s_{k}$ is the frame rate satisfying $s_{1}<s_{2}<\cdots<s_{K}$ and $s_{K}=s_{0}$.
The other two sequences $\bm{a}_{k}$ and $\bm{c}_{k}$ are used only for facilitating training of NAR models used in LLM-based TTS models.

Figure: Figure 2: MReQ post-training based on teacher-student distillation. (a) Pre-trained RVQ module is used to extract teacher embeddings ${\bm{h}_{t}}$. (b) MRVQ module consists of multiple LRVQ blocks and learns to reduce the frame rates. Student embeddings ${\bm{h}_{s}}$ are extracted. (c) Each LRVQ block consists of a pre-quantizer $\mathrm{PreQ}$, a sub-encoder $E$, a main quantizer $\mathrm{Quant}$, a sub-decoder $D$, and a post-quantizer $\mathrm{PostQ}$ to reduce frame rate from $s_{0}$ to $s_{k}$.
Refer to caption: x2.png

**Table 1: Implementation details**
| $k$ | $s_{k}$ | $\alpha\mathchar 45\relax\beta\mathchar 45\relax\gamma$ | Stride |
| --- | --- | --- | --- |
| 1 | 8 | 1-6-1 | 6 |
| 2 | 16 | 2-6-2 | 3 |
| 3 | 24 | 2-4-2 | 2 |
| 4 | 48 | 3-0-0 | - |

Implementation details.
Figure 2a shows the MRVQ module applied to the Encodec model, where the frame rate is reduced from $s_{0}=48$ Hz to $s_{1}=8$ Hz using $4$ LRVQ blocks.
Figure 2b shows the LRVQ block.
Each sub-encoder $E_{k}$ consists of a convolution layer followed by two bi-LSTM layers, which reduces the frame rate from $s_{0}$ to $s_{k}$.
Each sub-decoder $D_{k}$ consists of two bi-LSTM layers followed by a transposed convolution layer, which is symmetric to $E_{k}$.
Table 1 lists frame rates $s_{k}$, hyperparameter triplets $\alpha\mathchar 45\relax\beta\mathchar 45\relax\gamma$, and strides for the convolution and transposed convolution layers.
For $k=4$, only the pre-quantize is used, and the other components are replaced with identical functions, reducing Eqs. (12, 13) to $\bm{b}_{4}=\bm{a}_{4}$ and $\bm{c}_{4}=\bm{b}_{4}$, respectively.

### 4.2 Post-Training with Teacher-Student Distillation

Training NAC models with the MRVQ module is challenging because quantization layers with lower frame rates are prone to be ignored.
To address this, we introduce a post-training technique based on teacher-student distillation, where a NAC model pre-trained with a high frame rate serves as the teacher model.
As shown in Figure 2a, teacher embeddings (in green) are extracted from the frozen RVQ module, while student embeddings (in purple) are extracted from the MRVQ module.
We then minimize the feature-level distillation (FLD) loss and the hidden-state reconstruction (HSR) loss.

FLD loss.
The FLD loss is a mean absolute error (MAE) loss between teacher and student embeddings, defined as follows:
$\displaystyle L_{\mathrm{FLD}}=\sum_{({s},{t})\in\mathcal{P}}\lambda^{\mathrm{
FLD}}_{({s},{t})}\|{\hat{\bm{h}}_{s}}-{\bm{h}_{t}}\|_{1},$
(14)
$\displaystyle{\hat{\bm{h}}_{s}}=\sum_{k=1}^{{s}}\tilde{\bm{c}}_{k},$
(15)
$\displaystyle{\bm{h}_{t}}=\sum_{l=1}^{{t}}\tilde{\bm{z}}_{l},$
(16)
where
${\bm{h}_{s}}$ is a student embedding,
${\bm{h}_{t}}$ is a teacher embedding,
$\mathcal{P}$ is a set of student-teacher index pairs,
and $\lambda^{\mathrm{FLD}}_{({s},{t})}$ is a weight coefficient.
We use $\mathcal{P}=\{({1},{1}),({2},{3}),({3},{5}),({4},{8})\}$ for RVQ with eight layers and MRVQ with four LRVQ blocks.
Note that $\tilde{\bm{c}}_{k}$ and $\tilde{\bm{z}}_{l}$ are obtained from Eqs. (8) and Eqs. (1), respectively.
The student-teacher pairs in $\mathcal{P}$ are determined so that the cumulative number of student’s post-quantization layers matches that of the teacher’s quantization layers.

HSR loss.
The HSR loss is introduced to further facilitate training of each LRVQ block:

$$ $\displaystyle L_{\mathrm{HSR}}=\sum_{k=1}^{K}\lambda^{\mathrm{HSR}}_{k}\| \tilde{\bm{a}}_{k}-D_{k}(\tilde{\bm{b}}_{k})\|_{1}$ (17) $$

where $\tilde{\bm{a}}_{k},\tilde{\bm{b}}_{k}$ are from Eqs. (11, 12), and $\lambda^{\mathrm{HSR}}_{k}$ is a weight coefficient.

Total loss.
The total loss is given by
$L_{\mathrm{total}}=L_{\mathrm{NAC}}+L_{\mathrm{FLD}}+L_{\mathrm{HSR}}$, where $L_{\mathrm{NAC}}$ is the loss used to train the NAC model.
We continue to train the encoder and decoder with the MRVQ module using a copied weight from the NAC model, which is used as a teacher model.
Compared to training from scratch, this allows for more efficient and stable convergence.

Discussion.
Our post-training approach is designed to be independent of the encoder-decoder architecture of the original NAC model, as we only assumed the use of RVQ.
Consequently, by utilizing state-of-the-art NAC models such as SpeechTokenizer (Zhang et al., ) instead of Encodec (Défossez et al., ), it is possible to achieve higher performance at a lower frame rate.

## 5 Hierarchical Neural Codec Language Model

This section introduces HALL-E, a hierarchical LLM-based TTS model that learns to generate hierarchical tokens. Inspired by VALL-E (Wang et al., ),
HALL-E consists of a pair of language models: an AR model and an NAR model.
There are two key differences compared to VALL-E.
First, our AR model handles low frame-rate sequences, as low as 8 Hz in our experiments, enabling stable generation of audio tokens for longer speech segments.
Second, we incorporate frozen sub-modules obtained by decomposing the MRVQ module into the token prediction process using the NAR model. This integration facilitates training and results in high-quality speech synthesis.
The whole inference process is illustrated in Figure 3.

Figure: Figure 3: HALL-E architecture. (a) AR model generates a low frame-rate token sequence $\hat{\bm{b}}_{1}$. (b) NAR model predicts $\hat{\bm{b}}_{k+1}$ from $\hat{\bm{b}}_{k}$ iteratively by utilizing frozen sub-modules of MRVQ.
Refer to caption: x3.png

AR model.
Given a text prompt $\bm{t}$ and an audio $\bm{x}_{\mathrm{pr}}$ as a prompt, the AR model predicts a low frame-rate token sequence $\hat{\bm{b}}_{1}$ corresponding to $\bm{t}$.
This step is formulated as $\hat{\bm{b}}_{1}=T_{\mathrm{ar}}(\bm{t},\bm{b}^{\mathrm{pr}}_{1})$, similar to Eq. (4) for VALL-E, where $T_{\mathrm{ar}}$ is an AR transformer decoder and $\bm{b}^{\mathrm{pr}}_{1}$ is the audio prompt obtained by the first LRVQ block.

NAR model.
Given a token sequence $\hat{\bm{b}}_{k}$, the NAR model predicts the next token sequence $\hat{\bm{b}}_{k+1}$ iteratively.
Specifically, $\hat{\bm{b}}_{k+1}$ is predicted in three steps by utilizing frozen sub-modules obtained from the MRVQ module.
First, the sub-decoder and the post-quantizer are applied to $\hat{\bm{b}}_{k}$ to obtain $\hat{\bm{c}}_{k}=\mathrm{PostQ}^{(k)}_{\gamma}(D_{k}(\hat{\tilde{\bm{b}}}_{k}))$. Second, an NAR transformer decoder $T_{\mathrm{nar}}$ is employed to predict $\hat{\bm{a}}_{k+1}$ from $\hat{\bm{c}}_{k}$.
Because MRVQ has a nested structure, this step requires applying the decoder $\alpha_{k+1}$ times, where $\alpha_{k+1}$ is the number of layers in the pre-quantizer at the block $k+1$.
Specifically, for the layer $l+1$ in block $k+1$, denoted as $\hat{\bm{a}}_{k+1,l+1}$, is computed $\alpha_{k+1}$ times using the following equation:
$\hat{\bm{a}}_{k+1,l+1}=T_{\mathrm{nar}}(\bm{t},\bar{\bm{h}}^{\mathrm{pr}}_{K,L
},\bar{\bm{h}}_{k+1,l},\ell_{k+1,l})$ similar to Eq. (5), where $\bar{\bm{h}}_{k+1,l}=\sum_{k^{\prime}=1}^{k}\hat{\tilde{\bm{c}}}_{k^{\prime}}+
\sum_{l^{\prime}=1}^{l}\hat{\tilde{\bm{a}}}_{k+1,l^{\prime}}$ is the accumulated feature and $\bar{\bm{h}}^{\mathrm{pr}}_{K,L}$ is the one for prompt.
The layer id $\ell_{k+1,l}=\sum_{k^{\prime}=1}^{k}\alpha_{k}+l$ is the cumulative sum of the number of pre-quantization layers.
Finally, $\hat{\bm{b}}_{k+1}$ is obtained by applying the sub-encoder and the main quantizer to $\hat{\bm{a}}_{k+1}$ as $\hat{\bm{b}}_{k+1}=\mathrm{Quant}^{(k+1)}(E_{{k+1}}(\hat{\tilde{\bm{a}}}_{k+1}))$.
This process enables the NAR model to effectively generate high-fidelity token sequences.

Training.
The AR model is trained using a cross-entropy loss $L_{\mathrm{CE}}(\hat{\bm{b}}_{1},\bm{b}_{1})$ measured between a prediction $\hat{\bm{b}}_{1}$ and the corresponding ground truth $\bm{b}_{1}$ obtained from training speech data.
The delay pattern used in MusicGen (Copet et al., ) is also applied.
The NAR model is also trained using a cross-entropy loss $L_{\mathrm{CE}}(\hat{\bm{a}}_{k+1,l+1},\bm{a}_{k+1,l+1})$, where $\hat{\bm{a}}_{k+1,l+1}$ is a prediction and $\bm{a}_{k+1,l+1}$ is the ground truth.
HALL-E is also post-trained given a pre-trained LLM-based TTS model.

Discussion.
This work focuses on post-training approaches for reducing frame rate, but further exploration in token merging and grouping structures (Han et al., ; Chen et al., ) would also be interesting in future research.
Another architecture design involves incorporating learnable upsampling layers into the NAR model to directly predict $\hat{\bm{b}}_{k+1}$ from $\hat{\bm{b}}_{k}$.
However, handling different frame rates with a single NAR transformer is challenging.
Additionally, a sub-encoder and sub-decoder are placed between $\hat{\bm{b}}_{k+1}$ and $\hat{\bm{b}}_{k}$, making the relationships between them more complex compared to the RVQ-based approach. To mitigate this, we employ the MRVQ sub-modules like above, which simplify the relationships to resemble the previous method, thereby improving the efficiency of the training.

## 6 MinutesSpeech benchmark dataset

This section introduces MinutesSpeech,
a benchmark dataset for minutes-long TTS synthesis.
Unlike previous datasets such as LibriSpeech, which are primarily designed for automatic speech recognition, our dataset is curated to advance TTS research from the following three perspectives.

1) Benchmarking Minutes-long TTS Synthesis. We provide two subsets for benchmarking: MinutesSpeech-90s and MinutesSpeech-180s, consisting of speech segments ranging from 3 seconds to 90 seconds and 3 seconds to 180 seconds, respectively.
All test audio files are under Creative Commons licenses.
For each speech segment, we provide transcriptions created by two professional native transcribers.
Since LLM-based TTS models have been typically evaluated using audio segments ranging from 4 to 10 seconds from LibriSpeech in previous studies, our dataset promotes research on longer speech synthesis by providing substantially longer segments.

2) Balanced Audio Length for Training. We also provide a training set consisting of 40,000 hours of audio data. To successfully train TTS models capable of stably generating audio ranging from a few seconds to over a minute in a single inference, we carefully designed the distribution of audio lengths in the training dataset, covering durations from 3 seconds up to 180 seconds. Automatically generated transcriptions are provided for this subset to facilitate large-scale training.

3) Variety of Speaking Styles. To encompass a variety of conversational speech styles, we curated data from podcasts. Compared to the audiobook speech in LibriSpeech, synthesizing conversational speech is more challenging due to its spontaneous nature. We believe this is an important research direction that can help bridge the gap between LLMs and LLM-based TTS models.

Data Curation Pipeline.
To create the training set, we first curated 296,464 podcast files, ranging from 10 to 90 minutes, totaling about 186k hours.
We evaluated these audio files using the PESQ score. The audio was segmented every 30 seconds, and the score was calculated for each segment. We then computed the mean and standard deviation for each audio file, retaining only those with a mean score higher than 2.5 and a standard deviation lower than 0.6. As a result, approximately 25% of the data remained.
We then applied automatic speech recognition and speaker diarization to extract speech segments, each associated with a single speaker. Specifically, Whisper distil Large v3(^3^33[https://huggingface.co/distil-whisper/distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3)) was used for automatic transcriptions, and Pyannote(^4^44[https://huggingface.co/pyannote/speaker-diarization-3.1](https://huggingface.co/pyannote/speaker-diarization-3.1)) was employed for speaker diarization.
We then segmented each audio file into segments ranging from 3 to 90 or 180 seconds.
Lastly, we removed a certain proportion of utterances with text lengths that were either too long or too short,
based on their respective distributions,
resulting in a final dataset of approximately 40k hours.

Figure: Figure 4: Duration and word count distributions.
Refer to caption: x4.png

For the test set, we manually collected audio files under
Creative Commons licenses and asked professional transcribers to select high-quality audio and create accurate transcriptions.
Note that filtering based on the maximum and minimum text lengths used in the training set was also applied to each test set.

Dataset Statistics. Figure 4 shows the distributions of speech durations and the number of words per segment in the test sets. As shown, our dataset provides a diverse range of speech lengths, facilitating the development and evaluation of TTS models capable of handling varying durations and linguistic content.

**Table 2: Dataset statistics calculated on audio segments longer than 3s. $\dagger$ marks the results at 8Hz.**
| Dataset | Split | Clips | Audio duration | Words | Max Token length |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Average (sec) | Total (hours) | Audio | Text | Sum |  |  |  |
| LibriSpeech | train-clean-100 | 27949 | 12.90 $\pm$ 3.29 | 100.18 | 986k | 1176 | 240 | 1416 |
| LibriSpeech | test-35s | 1937 | 8.29 $\pm$ 5.10 | 4.46 | 43k | 1680 | 340 | 2020 |
| MinutesSpeech | train-28s | 7385k | 19.11 $\pm$ 7.31 | 39.21k | 407501k | 1344 | 314 | 1658 |
| MinutesSpeech | train-90s | 4181k | 34.37 $\pm$ 28.32 | 39.92k | 407689k | 720^† | 933 | 1655 |
| MinutesSpeech | train-54s | 4984k | 28.82 $\pm$ 17.45 | 39.91k | 409261k | 2592 | 579 | 3171 |
| MinutesSpeech | train-180s | 3762k | 37.69 $\pm$ 41.94 | 39.39k | 401816k | 1440^† | 1733 | 3173 |
| MinutesSpeech | test-90s | 685 | 45.28 $\pm$ 31.35 | 8.61 | 83k | 720^† | 925 | 1645 |
| MinutesSpeech | test-180s | 536 | 56.76 $\pm$ 55.24 | 8.45 | 81k | 1440^† | 1715 | 3155 |

## 7 Experiments

### 7.1 Experimental setup

Baseline models.
As a competitive comparison, we chose VALL-E (Wang et al., ) using Encodec (Défossez et al., ) at 48Hz as the baseline model, which is already lower than 75Hz used in previous studies, based on our preliminary experiments (see Figure 1).
We also demonstrate the effectiveness of our approach on SpeechTokenizer (Zhang et al., ).

Datasets.
For training, we used the MinutesSpeech training set, specifically train-90s and -180s for HALL-E, and train-28s, -54s, -90s, and -180s for VALL-E.
The train-28s and -54s are designed for 48Hz to match the token length in train-90s and -180s at 8Hz (see Table 2).
For evaluation, MinutesSpeech test-90s, -180s, and LibriSpeech test_clean set are used.
The minimum audio length was set to 4s, while the maximum audio lengths were set to 90s, 180s, and 35s, respectively. The audio length for prompt was consistently set to 3s.

Evaluation metrics.
Two evaluation metrics are used for speech reconstruction experiments: WER and PESQ.
WER is calculated using the conformer-transducer(^5^55[https://huggingface.co/nvidia/stt_en_conformer_transducer_xlarge](https://huggingface.co/nvidia/stt_en_conformer_transducer_xlarge)).
Six evaluation metrics are used for zero-shot TTS experiments: WER, speaker similarity (SIM), deep noise suppression mean opinion score (DNSMOS), Wasserstein distance (WD) with respect to the duration distribution, subjective evaluation of naturalness (QMOS), and subjective evaluation of speaker similarity (SMOS).
SIM is calculated using WavLM-TDNN
(^6^66[https://github.com/microsoft/UniSpeech/tree/main/downstreams/speaker_verification](https://github.com/microsoft/UniSpeech/tree/main/downstreams/speaker_verification)).
DNSMOS is calculated using the model trained with ground truth human ratings obtained using ITU-T P.808 (ITU, )(^7^77[https://github.com/microsoft/DNS-Challenge/tree/master/DNSMOS](https://github.com/microsoft/DNS-Challenge/tree/master/DNSMOS)).
WD is calculated between the duration distributions of the generated speech and the ground truth speech.
For QMOS and SMOS, 40 utterances were randomly selected from each test set, and three native English speakers rated their naturalness on a scale from 1 to 5. We then calculated the mean and confidence intervals.

Training details.
Encodec and SpeechTokenizer were pre-trained using the Adam optimizer for 100k iters with a batch size of 704 for Encodec and 674 for SpeechTokenizer on four H100 GPUs, and a learning rate of $9\times 10^{-4}$.
MReQ post-training was performed on a single H100 GPU for 160k iters with a batch size of 160 and a learning rate of $3\times 10^{-4}$.
VALL-E was pre-trained using the AdamW optimizer for 100k iters on four H100 GPUs.
To fully utilize GPU memory, the batch size was adjusted based on the audio length of the training samples. A cosine annealing learning rate schedule was employed with an initial learning rate of $1\times 10^{-4}$. HALL-E was trained using the same settings. More details are provided in Appendix A.

### 7.2 Speech reconstruction

**Table 3: Speech reconstruction on LibriSpeech**
| NAC model | Frame rates (Hz) | WER$\downarrow$ | PESQ$\uparrow$ |
| --- | --- | --- | --- |
| GT | - | 1.96 | 3.56 |
| Encodec | 8 | 100.0 | 1.25 |
| Encodec | 24 | 2.01 | 3.27 |
| Encodec | 48 | 2.00 | 3.45 |
| + MReQ (ours) | (8,16,24,48) | 2.02 | 3.47 |
| SpeechTokenizer | 48 | 2.00 | 3.52 |
| + MReQ (ours) | (8,16,24,48) | 1.99 | 3.53 |

Table 3
shows the speech reconstruction performance of Encodec and SpeechTokenizer
before and after applying MReQ.
As shown, MReQ achieved WER and PESQ scores comparable to the original values while significantly reducing the frame rate at the first quantization layer to as low as 8 Hz.

### 7.3 Zero-Shot Speech Synthesis

MinutesSpeech evaluation.
Table 5 compares performance of zero-shot TTS synthesis on the MinutesSpeech test sets.
Overall, HALL-E significantly outperformed the VALL-E, achieving comparable or even better WER and DNSMOS scores than the ground truth audio. This demonstrates the effectiveness of our approach.
VALL-E resulted in a high WER and a low QMOS score when trained on train-28s because it cannot generate audio longer than 28 seconds.
When trained on train-90s, the WER decreased but still remained significantly higher than that of ground truth. This is because training an AR model with a high frame rate is unstable, indicating that simply changing the training data is not sufficient for successful training.
Our approach addressed this issue by lowering the frame rate via MReQ.
In terms of SIM, VALL-E outperformed HALL-E because lowering the frame rate sacrifices acoustic information.
However, HALL-E surpassed VALL-E in SMOS, as it generates the speaker’s important style element, duration, much more naturally than VALL-E (see Figure 5).

LibriSpeech evaluation.
Table 5 shows results on LibriSpeech for short speech synthesis up to 35 seconds.
As shown, VALL-E’s WER increased as the training audio length increases. In contrast, HALL-E achieved a WER lower than 5% even with train-180s.
The gap in WER between the model and the ground truth is due to the difference between LibriSpeech, which primarily contains read speech, and MinutesSpeech, which includes a significant amount of spontaneous speech.
Balancing speech synthesis of both read and spontaneous speech remains a challenge for future work.

**Table 4: Zero-shot TTS performance on MinutesSpeech test sets. Best results are marked in bold.**
|  | TTS model | NAC model | Training | WER$\downarrow$ | SIM$\uparrow$ | WD$\downarrow$ | DNSMOS$\uparrow$ | QMOS$\uparrow$ | SMOS$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| test-90s | GT | – | – | 10.30 | – | – | 3.79 $\pm$ 0.24 | 3.83 $\pm$ 0.30 | - |
| VALL-E | Encodec | train-28s | 39.77 | 0.726 | 23.62 | 3.84 $\pm$ 0.19 | 2.29 $\pm$ 0.32 | 2.04 $\pm$ 0.25 |  |
| VALL-E | Encodec | train-90s | 16.14 | 0.712 | 2.68 | 3.87$\pm$ 0.17 | 2.48 $\pm$ 0.28 | 2.36 $\pm$ 0.26 |  |
| HALL-E | MReQ-Encodec | train-90s | 9.79 | 0.685 | 4.00 | 3.91 $\pm$ 0.21 | 3.35 $\pm$ 0.26 | 3.15 $\pm$ 0.26 |  |
| test-180s | GT | – | – | 10.20 | – | – | 3.78 $\pm$ 0.25 | 4.26 $\pm$ 0.22 | - |
| VALL-E | Encodec | train-54s | 36.52 | 0.706 | 25.66 | 3.55 $\pm$ 0.53 | 1.68 $\pm$ 0.25 | 1.70$\pm$ 0.26 |  |
| VALL-E | Encodec | train-180s | 21.71 | 0.702 | 12.52 | 3.76 $\pm$ 0.30 | 2.11 $\pm$ 0.26 | 2.15$\pm$ 0.26 |  |
| HALL-E | MReQ-Encodec | train-180s | 10.53 | 0.660 | 5.79 | 3.91 $\pm$ 0.19 | 3.38 $\pm$ 0.25 | 3.31$\pm$ 0.23 |  |

NAC models.
Table 7 compares the results obtained by Encodec and SpeechTokenizer.
As shown, all metrics except SIM improve with the use of SpeechTokenizer.
Since SpeechTokenizer aims to preserve more linguistic information in the first VQ layer,
it is likely that this aligns well with maintaining linguistic information better than acoustic information when the frame rate is reduced.
These results suggest the potential for further performance enhancement by employing more advanced NAC models in the future.

Computational efficiency. Table 7 compares the real-time factor (RTF) of VALL-E and HALL-E, measured on an RTX 4090 GPU using the 4s to 10s segments from LibriSpeech. The results show that HALL-E generates audio approximately 3.4 times faster than VALL-E.
In LLM-based TTS, the computational bottleneck typically lies in the AR model.
Significant speed improvements were achieved by HALL-E because it reduces the number of tokens required for generation by a factor of six. This enhancement shows promising potential not only for LLM-based TTS but also for various applications, such as spoken language modeling, as part of future work.

**Table 6: Results for Encodec and SpeechTokenizer. MinutesSpeech train-90s and test-90s are used for training and testing, respectively.**
| TTS model | NAC model | WER$\downarrow$ | SIM$\uparrow$ | WD$\downarrow$ | DNSMOS$\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| GT | - | 10.30 | - | - | 3.79 $\pm$ 0.23 |
| VALL-E | Encodec | 39.77 | 0.726 | 23.62 | 3.84 $\pm$ 0.19 |
| HALL-E | MReQ-Encodec | 9.79 | 0.685 | 4.00 | 3.91 $\pm$ 0.21 |
| VALL-E | SpeechTokenizer | 35.62 | 0.724 | 24.20 | 3.86 $\pm$ 0.18 |
| HALL-E | MReQ-SpeechTokenizer | 9.12 | 0.678 | 3.09 | 3.95 $\pm$ 0.19 |

### 7.4 Analysis and ablation studies

Is the hierarchical structure essential?
The results in Table 11 demonstrate the impact of different frame rates adopted in MReQ on MinutesSpeech test-90s. For further details, please refer to Appendix C.1.
From this table, it is evident that the proposed method achieves the best WER, indicating the importance of gradually increasing the frame rate in a hierarchical manner.
We also observed a trade-off between WER and SIM, as the SIM deteriorates with a reduction in the number of 48Hz layers in the proposed method.

Can HALL-E handle long audio prompts? Table 11 shows the SIM as a function of the audio length for prompt. The SIM values were calculated using audio segments longer than 25 seconds from the MinutesSpeech test-90s.
As expected, longer audio length resulted in better performance. In zero-shot TTS, due to the limited context, the voice-cloning results are typically inferior to standard fine-tuning (Anastassiou et al., ). Our proposed method has the potential to push the limits of such zero-shot TTS capabilities.

**Table 8: Impact of varying hierarchical structure.**
| Upsampling | WER$\downarrow$ | SIM$\uparrow$ |
| --- | --- | --- |
| (8,48) | 10.27 | 0.693 |
| (8,16,48) | 10.71 | 0.693 |
| HALL-E | 9.79 | 0.685 |

Ablation study for MReQ:
Table 11 shows the ablation study on the codec in the proposed method, using the LibriSpeech test set. The results indicate that both losses and post-training play important
roles in the overall performance.
It is important to note that although the impact of the FLD loss may appear minor, this does not mean that its influence on TTS is negligible. The codec model has a
hierarchical structure with redundancy, meaning that information lost at the first layer can be compensated for in subsequent layers. However, when training the AR model, it is crucial for sufficient information to be present in the first layer, and the FLD loss is essential for achieving this. For further details, please refer to Appendix C.1.

Ablation study for HALL-E:
Table 11 shows the ablation study on HALL-E on MinutesSpeech test-90s. As shown, all the proposed methods were found to be effective.
In particular, the results of training only with PreQ or PostQ highlight the importance of using the MRVQ sub-modules with NAR model as pointed out in Section 5.
This underscores the necessity of designing the codec model with downstream tasks in mind, ensuring that it is optimized for the specific requirements of those tasks.

Figure: Figure 5: Samples of generated waveforms
Refer to caption: extracted/5904728/fig/fig_waveforms_2.png

Qualitative results.
Figure 5 shows the ground truth audio from MinutesSpeech test-90s and audio generated by each models. Both HALL-E and VALL-E models were trained on MinutesSpeech train-90s. As can be observed, HALL-E is able to generate durations very similar to the ground truth. On the other hand, VALL-E produces an unnatural waveform with almost no silence.
Training the model at a low frame rate not only enables the generation of long speech segments but also allows for capturing more natural long-term temporal dynamics.
For more examples, please refer to Appendix D.1.

## 8 Conclusion

We introduced two novel approaches for minute-long zero-shot text-to-speech synthesis: MReQ and HALL-E.
MReQ reduced the frame rate of the Encodec model to 8 Hz by reorganizing audio tokens via MRVQ.
HALL-E efficiently synthesized minute-long speech by using 8Hz tokens in the AR model and MRVQ sub-modules in the NAR model.
We demonstrated the effectiveness of these approaches on MinutesSpeech, the newly introduced dataset consisting of 40,000 hours of speech data.
Our work contributed to promote zero-shot TTS research.

Limitations and future work.
By reducing the frame rate to 8 Hz, our AR model can utilize longer contexts, enhancing the naturalness of the synthesized speech. We believe that to handle extended context is particularly advantageous for larger AR models such as AudioLM (Borsos et al., ), SpeechGPT (Zhang et al., ), and PSLM (Mitsui et al., ). Demonstrating the effectiveness of our approach not only in TTS but also with these models remains a future work.
Furthermore, as shown in Table 2, we have achieved shorter audio token length than the corresponding text token length. However, in our current AR model, we concatenate these tokens, which results in the text tokens becoming a bottleneck in terms of sequence length. Small-E (Lemerle et al., ) propose methods to mitigate this issue by processing each token individually and fusing them using cross-attention. Exploring such architectural enhancements is an important direction for future work.
Lastly, as shown in Table 7, our method brought significant improvements with SpeechTokenizer, even more so than when applied to Encodec. It means that our approach can further enhance the objective of preserving linguistic information. This indicates that our method could serve as a replacement for traditional SSL models (Hsu et al., ; Chen et al., ) or ASR encoders (Lakhotia et al., ; Chu et al., ), marking an important direction for future research.

## Acknowledgement

This work was supported by JSPS KAKENHI Grant Number 22K12089.
Computational resource of AI Bridging Cloud Infrastructure (ABCI) provided by National Institute of Advanced Industrial Science and Technology (AIST) was used.

## Appendix A Model details

### A.1 MReQ

**Table 12: Hyper-parameters of Encodec**
| Module | Hyper-Parameter | Values |
| --- | --- | --- |
| Encoder & Decoder | ConvBlock Number | 4 |
|  | ConvBlock Filter Sizes | [128, 256, 512, 1024] |
|  | ConvBlock Strides | [10, 5, 5, 2] |
|  | ConvBlock Kernel size | 7 |
|  | ConvBlock Norm | Weight norm |
|  | Activate Function | ELU (Clevert, ) |
|  | LSTM number | 2 |
| RVQ | Codebook Number | 8 |
|  | Codebook Dim | 128 |
|  | Codebook Vocabulary | 1024 |
| Discriminator | MS-STFT (Défossez et al., ) |  |
|  | Window lengths | [2048, 1024, 512, 256, 128] |
| Loss Function | Adversarial Loss | 4 |
| Weight Coefficients | Feature Matching Loss | 4 |
|  | L1 Loss | 0.1 |
|  | MS-SPEC Loss (Yamamoto et al., ) | 2 |

Encodec:
Table 13 lists the various hyperparameters of the Encodec used in this study. Essentially, we utilized the configuration from the original Audiocraft (Copet et al., ) implementation. The main difference is the modification of the strides within the encoder and decoder from the original [8, 5, 4, 2] to [10, 5, 5, 2]. This change increases the overall stride from 320 to 500, resulting in an improvement in the frame rate of the generated token sequences from 75Hz to 48Hz.

MReQ-Encodec:
Table 13 presents the list of hyperparameters for the proposed MReQ-Encodec method. Modules added by MReQ, such as the sub-encoder, sub-decoder, and LRVQ, are the only ones listed, as all other modules are identical to those in Encodec. Essentially, the parameters for the sub-encoder, sub-decoder, and LRVQ are based on the hyperparameters of the original encoder, decoder, and RVQ, respectively. We anticipate that applying MReQ to architectures other than Encodec will similarly yield stable hyperparameters by basing them on those of the original architecture, ensuring ease of adaptation.

SpeechTokenizer:
In this paper, the implementation of SpeechTokenizer is almost identical to the Encodec architecture presented in Table 13. The only difference is that a bidirectional LSTM was used instead of an LSTM.
For the teacher used in the teacher forcing process of Layer 1’s RVQ, we followed the original paper and employed Hubert base ls960 (^8^88[https://huggingface.co/facebook/hubert-base-ls960](https://huggingface.co/facebook/hubert-base-ls960)).

In the original paper, 16,000Hz audio was used as the input data, with a stride of 320, resulting in a frame rate of 50Hz. Additionally, the SSL model typically employs a window size of 20ms with a 16,000Hz input, also yielding a frame rate of 50Hz. This setup enabled the application of the teacher forcing framework without any special adjustments in the temporal direction.

However, in this study, to ensure a fair comparison with Encodec, we adopted a 24,000Hz input and a stride of 500. Consequently, the frame rate became 48Hz, creating a mismatch with the 50Hz frame rate of the SSL model. To address this issue, we adjusted the audio input to the SSL model by speeding it up by a factor of $50/48$, ensuring that the duration of the SSL output aligns with that of the first layer of RVQ.

Moreover, while the original work utilized not only the MS-STFT discriminator employed by Encodec but also added a multi-period discriminator (MPD) and a multi-scale discriminator (MSD), as in HiFi-Codec (Yang et al., ), it was observed that the discriminators were too strong under the aforementioned settings. As a result, using only the MS-STFT discriminator, similar to Encodec, yielded the best performance.

MReQ-SpeechTokenizer:
As described above, the SpeechTokenizer ultimately uses an SSL model and, apart from the Bi-LSTM, matches the Encodec architecture in all respects. Therefore, when extending to MReQ, the architecture is identical to that of MReQ-Encodec, that is, the only difference between MReQ-SpeechTokenizer and MReQ-Encodec lies in the pretrained weights used.

In this study, we did not use SSL models to train the MReQ-SpeechTokenizer. This is because an SSL model was already utilized during the training of SpeechTokenizer, and since the features from that model were used for distillation, we deemed it unnecessary to reapply the SSL model.

A similar concept can be applied to models like Language-Codec (Ji et al., ). The goal of this model is to restrict the quantizers of the first three channels to learn only the compressed audio frame information in the specified space by applying masking during RVQ training. However, since the features obtained in this manner are used in MReQ training, there is no need to perform masking again. In other words, MReQ is flexible enough to be easily adapted and extended to new NAC models that may be developed in the future to learn better features.

### A.2 HALL-E

**Table 14: Hyper-parameters of AR LM in VALL-E**
| Module | Hyper-Parameter | Values |
| --- | --- | --- |
| Transformer | Dim | 1280 |
|  | Number of Heads | 20 |
|  | Number of Layers | 36 |
|  | Dim of FFN | 5120 |
|  | Norm | Layer norm |
|  | Activate Function | GELU (Hendrycks & Gimpel, ) |
|  | Positional Encoding | sin (Vaswani et al., ) |
|  | Text Tokenizer | Byte-Pair Encoding (BPE) (Sennrich, ) |
|  | Text Vocab | 258 |

VALL-E:
Table 15 presents the hyperparameters of the AR LM in the VALL-E model. For this implementation, we based our work on the LM of Audiocraft’s MusicGen (Copet et al., ) and referred to an unofficial VALL-E implementation (^9^99[https://github.com/lifeiteng/vall-e](https://github.com/lifeiteng/vall-e)).

Table 15 presents the hyperparameters of the NAR LM in the VALL-E model. The fundamental hyperparameters are similar to those of the AR LM, with differences in the values for Dim, Heads, and Layers, resulting in a smaller number of parameters compared to the AR LM. It is generally known that the task of generating acoustic tokens from acoustic input tokens handled by the NAR LM is simpler than the task of generating tokens, including linguistic elements, from text handled by the AR LM. Indeed, in our preliminary experiments, increasing the model size of the NAR LM to match that of the AR LM resulted in minimal differences in performance.

HALL-E:
The architecture of the AR LM in HALL-E is almost identical to that of VALL-E, as shown in Table 15. The only difference lies in the input tokens being fed in a delayed format. This method was introduced in MusicGen and adopted in TTS systems such as Lyth & King (). By employing this approach, it becomes possible to efficiently train multi-layer token sequences within the framework of next-token prediction.

In HALL-E’s AR LM, although it operates at a very low frame rate of 8Hz, it needs to handle six layers of tokens. If this were trained in a flattened format, the sequence length would be equivalent to 48Hz, rendering the frame rate reduction meaningless. However, by handling it in a delayed format, the sequence length can be maintained at nearly the 8Hz frame rate, allowing for the optimal utilization of tokens obtained through MReQ.

The architecture of HALL-E’s NAR model similarly shares many points with VALL-E’s NAR LM, as shown in Table 15. There are two major differences: 1) a cross-attention layer is additionally inserted into each transformer layer, using text tokens as input for the cross-attention, and 2) as described in Section 5, the output tokens of PostQ and PreQ are combined for input and output.

1) In this study, HALL-E handles very long audio segments, such as 90s or 180s. However, as the NAR LM processes tokens at 48Hz, it is difficult to train with all the tokens as input. Therefore, we consider training with only a portion of these long tokens. Specifically, by clipping the audio length corresponding to 90s or 180s so that it becomes 28s or 54s, it is possible to train sequences of the same length as VALL-E’s NAR LM. However, VALL-E’s NAR LM requires the text corresponding to the audio tokens to be concatenated with the audio tokens as input. In this study, where there is no alignment between the text and audio, it is difficult to extract the corresponding text portion for the clipped audio tokens and concatenate them.

To address this, we stopped using the traditional concatenation method for inputting text information and instead used cross-attention to input the text information. This allows the attention mechanism to automatically learn the alignment with the input audio tokens. To our knowledge, no model in the VALL-E framework has incorporated this kind of cross-attention mechanism to add text information. Small-E (Lemerle et al., ) introduces this idea in a more refined form with Linear Causal Language Models.

During inference, only audio up to 28s or 54s can be processed at most, so a method is employed where new audio is generated sequentially by sliding 5s at a time. For more detailed inference algorithms and other information, please refer to our publicly available GitHub repository(^10^1010[Comingsoon](Comingsoon)).

2) The technique of appropriately combining PostQ and PreQ for input, introduced in Section 5, was implemented with the aim of promoting learning by bringing the structure closer to the original form of the VALL-E NAR LM, as explained in that section. While this is indeed another major difference from VALL-E, it should be noted that this operation involves using the modules acquired through MReQ to properly transform tokens outside of the NAR LM, and no changes have been made to the architecture of the NAR LM itself.

## Appendix B Dataset details

### B.1 MinutesSpeech test set details

As mentioned in Section 6, the test set was gathered from broadcasts available under a Creative Commons license. The goal was to collect redistributable data to facilitate future research on long-form speech generation. The collection method involved first filtering English podcast descriptions for those containing Creative Commons-related terms. Subsequently, podcasts explicitly labeled with Creative Commons were manually selected. In most cases, the Creative Commons license applied only to the music used in the broadcasts, with few instances of the license covering the broadcasts themselves. Next, ASR was used to generate provisional transcripts, which were then refined by native English speakers. Simultaneously, the quality of the audio was evaluated, and broadcasts with high audio quality, suitable for speech synthesis, were selected. The list of podcast broadcasts included in the final dataset is presented in Table 16.

**Table 16: The list of podcast broadcasts under the Creative Commons license included in the MinutesSpeech test set**
| URLs | Total Duration (h) | License |
| --- | --- | --- |
| [https://anchor.fm/s/6f51ed88/podcast/rss/](https://anchor.fm/s/6f51ed88/podcast/rss/) | 6.0 | CC BY 4.0 |
| [https://feeds.captivate.fm/research-culture//](https://feeds.captivate.fm/research-culture//) | 1.8 | CC BY-SA 4.0 & CC BY-ND 4.0 |
| [https://anchor.fm/s/640d7168/podcast/rss/](https://anchor.fm/s/640d7168/podcast/rss/) | 0.9 | CC BY-NC-ND 4.0 |
| [https://feeds.hubhopper.com/b38dfaea594789a83b87cb96d3f79004.rss](https://feeds.hubhopper.com/b38dfaea594789a83b87cb96d3f79004.rss) | 0.4 | CC BY-NC |

### B.2 Examples of the transcripts

Table 17 shows a portion of an actual transcript. This example from the MinutesSpeech test-90s was generated by combining segments using ASR timestamps and speaker identification annotations by native English speakers, ensuring that the maximum duration does not exceed 90 seconds.
As illustrated in this example, podcasts often feature a dialogue format between two individuals, which leads to variability in the duration of a single speaker’s continuous utterance. In fact, even within this table, the durations range from a minimum of 3.68 seconds to a maximum of 79.82 seconds. For the complete dataset and further details, please refer to [Comingsoon](Comingsoon).

**Table 17: Example transcript from [https://anchor.fm/s/6f51ed88/podcast/rss/](https://anchor.fm/s/6f51ed88/podcast/rss/)**
| Start (s) | End (s) | Duration (s) | Transcript |
| --- | --- | --- | --- |
| 8.11 | 27.47 | 19.36 | Welcome to Deconstructing Management, a podcast made by college students for college students. We’ve interviewed the chapter authors of the OpenStack’s Principles of Management textbook… |
| 34.57 | 92.55 | 57.98 | I’m here with Chapter 7 author, Siri Terjesen. Dr. Terjesen holds a postdoctoral degree from the Queensland University of Technology… |
| 96.19 | 99.99 | 3.80 | Very good, Eric. Thanks so much for inviting me to join you in your class today. |
| 100.53 | 104.65 | 4.12 | Do you believe that successful entrepreneurs are born, or are they made? |
| 104.65 | 110.61 | 5.96 | That’s a great question. And I think there are quite a lot of entrepreneurs that are born, right? |
| 123.79 | 142.33 | 18.54 | And we didn’t instigate that. But I think that they can also be made… |
| 156.81 | 168.57 | 11.76 | I would have to agree with you because they could be born or, like my cousin… |
| 172.33 | 180.27 | 7.94 | He’s going to start, like a party business, kind of like where he has bouncy houses… |
| 180.85 | 252.06 | 71.21 | That’s neat. So really your cousin is a great example of a serial or even a portfolio entrepreneur… |
| 252.06 | 259.86 | 7.80 | So going straight to another question, what are the most important traits a person must have to become a successful entrepreneur? |
| 260.12 | 313.72 | 53.60 | That’s a great question, Eric. And you may have seen this in your cousin, right?… |
| 318.13 | 357.97 | 39.84 | Well, you’re never too old! So there’s a huge opportunity there… |
| 365.94 | 445.76 | 79.82 | That’s a great question. So, I think folks who have low-risk preferences may not make good entrepreneurs… |
| 459.27 | 526.33 | 67.06 | That’s the first step, right? It’s thinking about what the business would be… |
| 527.01 | 541.68 | 14.67 | It’s mainly, it’s also like, to start with like, shirts, like something so simple, like shirts… |
| 544.02 | 547.70 | 3.68 | So, so like, the controllers for like, Xbox, and like, PS4. |

## Appendix C Ablation studies

### C.1 MReQ

Further information of the ablation study for a hierarchical structure in MReQ.
In Section 7.4, we demonstrated that the proposed method achieves better results compared to other upsampling approaches. The parameters used are as follows:
For (8, 48), $K=2$, $(\alpha_{1},\beta_{1},\gamma_{1})=(1,6,1)$, $(\alpha_{2},\beta_{2},\gamma_{2})=(7,0,0)$.
For (8, 16, 48), $K=3$, $(\alpha_{1},\beta_{1},\gamma_{1})=(1,6,1)$, $(\alpha_{2},\beta_{2},\gamma_{2})=(2,6,2)$, $(\alpha_{3},\beta_{3},\gamma_{3})=(5,0,0)$.

Additionally, there is an important point that needs to be addressed.
when the upsampling rate is (8, 48), the loss diverged after about 60% of training.
This is likely due to the instability caused by the difference in the 1st layer between the Encodec used for the pre-training and MReQ-Encodec used for the post-training, which led to unstable learning.
This highlights the importance of hierarchical modeling to prevent sudden distribution shifts.

**Table 18: Ablation study for FLD loss in MReQ**
| Model | WER$\downarrow$ | SIM$\uparrow$ | WD$\downarrow$ | DNSMOS$\uparrow$ |
| --- | --- | --- | --- | --- |
| HALL-E | 9.79 | 0.685 | 4.00 | 3.91 $\pm$ 0.21 |
| w/o FLD loss | 9.07 | 0.704 | 4.51 | 3.88 $\pm$ 0.20 |

Is FLD loss needed?
Table 18 compares the case where MReQ-Encodec was trained without using the FLD loss and HALL-E was trained using it. Here, MinutesSpeech-90s was used for evaluation. Interestingly, when the FLD loss is excluded, the results do not worsen across all metrics—WER and SIM actually improve. However, WD and DNSMOS deteriorate to a non-negligible extent. This indicates that using the FLD loss allows for generating more natural, human-like speech. Below, we will discuss the reasons for this.

According to preliminary experiments, when MReQ is trained entirely without a pre-trained model, LRVQ at frame rates like 8Hz is ignored, and the learning process progresses to supplement all the information at higher frame rates. Since post-training with pre-trained model weights is applied here, this issue is relatively mitigated, and relatively good results are achieved even without the FLD loss.

On the other hand, when the frame rate is compressed, there is a tendency for finer temporal information to be lost. This means, in principle, that acoustic information is more likely to be lost than linguistic information. The first layer of Encodec’s RVQ, which serves as the pre-trained model, contains not only linguistic information but also a rich amount of acoustic information. Therefore, applying the FLD loss facilitates learning that retains both linguistic and acoustic information. This is believed to contribute to more natural speech generation. In contrast, when the FLD loss is not used, more acoustic information is lost. As a result, while the retention of linguistic information may improve WER, the overall outcome is less natural.

For this reason, we have adopted the FLD loss, which enables the generation of more natural speech. Further improvements in WER and SIM while using this method remain a task for future work.

Is it possible to use an even lower frame rate?
Table 19 shows the results of HALL-E trained using a lower frame rate adopted in MReQ. The evaluation was conducted similarly using MinutesSpeech-90s. Here, a very low frame rate of 4Hz was applied to the first LRVQ quantization. As the results indicate, this caused a significant deterioration in WER. In general, the average duration of each phoneme is known to be around 100ms, which corresponds to approximately 10Hz. Therefore, using a frame rate as low as 4Hz is likely to be fundamentally challenging.

**Table 19: Ablation study for lower frame rate**
| Upsampling | WER$\downarrow$ | SIM$\uparrow$ | WD$\downarrow$ | DNSMOS$\uparrow$ |
| --- | --- | --- | --- | --- |
| $(8,16,24,48)$ | 9.79 | 0.685 | 4.00 | 3.91 $\pm$ 0.21 |
| $(4,8,16,24,48)$ | 20.07 | 0.690 | 1.95 | 3.90 $\pm$ 0.21 |

On the other hand, we can see a substantial improvement in WD. This suggests that lowering the frame rate to 4Hz makes it easier to learn longer-term temporal dynamics, enabling the generation of more natural durations. From this, it is evident that while further lowering the frame rate involves significant difficulties, the potential benefits are also considerable.

**Table 20: Ablation study for lower bps in MReQ**
| $\beta$s | WER$\downarrow$ | PESQ$\uparrow$ |
| --- | --- | --- |
| (6, 6, 4, 0) | 2.02 | 3.47 |
| (3, 3, 2, 0) | 2.13 | 3.42 |

Is it possible to reduce a bps?
In MReQ, the number of Quant layers can be freely adjusted. Therefore, we considered reducing the number of Quant layers by half. In the original MReQ, the bitrate is set to match that of Encodec, which is 3.84 kbps, but by reducing the number of Quant layers in this way, it can be decreased to 2.64 kbps. Table 20 shows the performance changes of MReQ on the LibriSpeech test set when the bitrate is reduced in this manner. As the table indicates, significant performance degradation occurs even for a relatively simple task like Speech Reconstruction. This suggests that it is preferable to reduce the bitrate through improvements on the NAC model side, rather than by reducing the frame rate within the MReQ framework.

**Table 21: Ablation study for lower bps in HALL-E**
| $\beta$s | WER$\downarrow$ | SIM$\uparrow$ | WD$\downarrow$ | DNSMOS$\uparrow$ |
| --- | --- | --- | --- | --- |
| (6, 6, 4, 0) | 9.79 | 0.685 | 4.00 | 3.91 $\pm$ 0.21 |
| (3, 3, 2, 0) | 13.31 | 0.713 | 2.76 | 3.92 $\pm$ 0.21 |

Table 21 shows the performance of HALL-E using this MReQ evaluated on the MinutesSpeech-90s dataset. As indicated by the WER degradation already observed in MReQ, WER worsens significantly when training HALL-E. On the other hand, improvements are seen in all other metrics besides WER. This can be attributed to the fact that the reduction in the number of layers handled by the AR LM—from 6 layers to 3—makes the training process much easier, allowing for better learning of aspects like duration. Additionally, reducing the number of Quant layers, and thereby the amount of information contained in each block, results in smaller relative differences between blocks, making it easier for the NAR LM to learn, which likely contributes to improvements in metrics like SIM. However, the deterioration in WER is too significant to ignore, indicating that it is generally better to maintain the same bitrate.

### C.2 HALL-E

**Table 22: Zero-shot TTS performances of HALL-E with different $\alpha$-$\beta$-$\gamma$ pairs**
| Dataset | TTS model | $(\alpha_{1},\alpha_{2},\alpha_{3},\alpha_{4})$ | WER$\downarrow$ | SIM$\uparrow$ | WD$\downarrow$ | DNSMOS$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
|  | GT | - | 10.30 | - | - | 3.79 $\pm$ 0.24 |
|  | HALL-E | (1, 1, 2, 4) | 9.77 | 0.688 | 4.25 | 3.92 $\pm$ 0.23 |
| MinutesSpeech-90s | HALL-E | (1, 1, 3, 3) | 9.60 | 0.702 | 4.59 | 3.93 $\pm$ 0.22 |
|  | HALL-E | (1, 2, 2, 3) | 9.79 | 0.685 | 4.00 | 3.91 $\pm$ 0.21 |
|  | HALL-E | (1, 2, 3, 2) | 10.58 | 0.696 | 4.09 | 3.90 $\pm$ 0.19 |
|  | GT | - | 1.74 | - | - | 3.84 $\pm$ 0.19 |
|  | HALL-E | (1, 1, 2, 4) | 4.37 | 0.706 | 0.201 | 3.89 $\pm$ 0.19 |
| LibriSpeech | HALL-E | (1, 1, 3, 3) | 4.67 | 0.703 | 0.165 | 3.88 $\pm$ 0.20 |
|  | HALL-E | (1, 2, 2, 3) | 4.63 | 0.701 | 0.196 | 3.88 $\pm$ 0.19 |
|  | HALL-E | (1, 2, 3, 2) | 5.68 | 0.688 | 0.178 | 3.89 $\pm$ 0.19 |

Is different $\alpha$-$\beta$-$\gamma$ pair more effective?
Table 22 shows the performance changes when the $\alpha$-$\beta$-$\gamma$ combinations used in each LRVQ of MReQ are modified. It should be noted that in all cases, the bps is adjusted to be equal to that of Encodec, and $\beta_{4}=\gamma_{4}=0$. As the results indicate, there is no consistently strong combination across all metrics. On the other hand, when $\alpha_{4}=2$, the results are consistently the worst. This implies that reducing the number of layers corresponding to 48Hz too much leads to performance degradation. As shown in Figure 1, significant performance deterioration starts from 48Hz in Encodec, suggesting that this layer should be retained as much as possible.

In this study, we propose $(1,2,2,3)$ as the recommended method, as it consistently achieved balanced and good results across both datasets. However, other configurations also produced satisfactory results, demonstrating the robustness of our method to variations in hyperparameters.

## Appendix D More qualitative results

### D.1 Duration Analysis

Figure: Figure 6: A sample from MinutesSpeech test-180s. Both the HALL-E and VALL-E models were trained using MinutesSpeech train-180s.
Refer to caption: extracted/5904728/fig/fig_waveforms_180s.png

Figure: Figure 7: Comparison of the distribution of generated speech durations across different methods in the MinutesSpeech test sets. The left plot shows the distribution for the test-90s set, while the right plot shows the distribution for the test-180s set. Both the HALL-E and VALL-E models were trained using the corresponding training sets (train-90s and train-180s).
Refer to caption: extracted/5904728/fig/fig_duration_boxplot_90s.png

Figure 6 shows the waveform of a different sample compared to Figure 5. Similar to the observations in Figure 5, VALL-E generates an unnatural waveform with almost no silence, while HALL-E produces a more natural waveform, closely resembling the ground truth with appropriate amounts of silence.

Figure 7 presents the duration distribution for the entire dataset, calculated from MinutesSpeech test-90s and test-180s. As consistently demonstrated by the WD in Table 5, HALL-E’s distribution is consistently closer to the GT compared to VALL-E. In addition to the issue of failing to insert silent pauses, VALL-E tends to overly repeat breathing sounds and fillers, resulting in longer durations. HALL-E, by utilizing a very small frame rate of 8Hz, overcomes these issues specific to AR models and achieves a more natural duration.

### D.2 Filler Analysis

**Table 23: The comparison of filler generation abilities between HALL-E and VALL-E. The left table presents the analysis results for the MinutesSpeech test-90s, while the right table shows the results for the test-180s. In each case, HALL-E and VALL-E are trained on the corresponding train-90s and train-180s datasets.**
| TTS model | Avg. Fillers per text | Total Fillers |
| --- | --- | --- |
| GT | 3.74 $\pm$ 3.66 | 2567 |
| HALL-E | 3.45 $\pm$ 3.35 | 2364 |
| VALL-E | 3.22 $\pm$ 3.30 | 2207 |

Fillers are an essential component for generating more natural human-like speech (Mitsui et al., ). In this study, we propose and use the MinutesSpeech dataset collected from podcasts. This dataset includes numerous instances of human dialogue, naturally containing many fillers. The ability to generate fillers at a frequency similar to human speech could be considered a future benchmark for TTS systems.

Here, using the MinutesSpeech test set, we demonstrate the spontaneous speech generation capability of HALL-E by analyzing and comparing the frequency of filler words in both ground truth and generated speech. The list of filler words present in the MinutesSpeech test set is shown in Table 24.

**Table 24: Filler Words List used in MinutesSpeech test set**
| The list of filler words |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| so | you know | like | actually | right | well | i mean | um | you see | basically | literally | uh |

Table 23 shows the statistics of fillers in each MinutesSpeech test set. In all cases, HALL-E generates fillers at a frequency closer to the ground truth compared to VALL-E. This improvement is likely due to the use of an 8Hz frame rate, which facilitates better language understanding in the AR LM.

Notably, in the 180s results for VALL-E, the standard deviation is significantly large. This is because VALL-E, as an AR model, sometimes exhibits repeated generation, producing fillers consecutively. Such cases were not observed in HALL-E, suggesting that a lower frame rate is crucial for suppressing repeats.

However, there is still a gap between the generated results and the GT. This is likely because more advanced role-playing capabilities are needed. It is expected that utilizing models such as SLMs, which extend LLMs to speech, could enable more human-like filler generation with their advanced language processing abilities.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 1
questions: * The description of the LRVQ modules appears to omit the use of commitment loss for residual vector quantization. If the LRVQ module does not require commitment loss, the authors would explain the reasoning behind this and how the training is stabilized without it.
* Providing a detailed description of the pre-trained LLM-based TTS model that HALL-E is post-trained on would greatly aid in understanding the training process and setup of the proposed model.
soundness: 2
strengths: * The paper offers thorough quantitative and qualitative results, demonstrating that the proposed generative modeling outperforms baseline methods in long-form speech synthesis.
* The introduction of MinutesSpeech is a noteworthy addition to TTS research, providing high-quality, long-context speech data. The illustrated curation and preprocessing techniques further enhance its utility.
* The ablation studies on MRVQ provide insights into the necessity of each component, justifying their inclusion in the architecture.
summary: The paper presents HALL-E, a neural codec language model aimed at addressing challenges in minute-long zero-shot text-to-speech (TTS) synthesis. It introduces Multi-Resolution Requantization (MReQ) to reduce frame rates in neural audio codec (NAC) models, and proposes HALL-E, which leverages MReQ for efficient and effective long-form speech synthesis. The work also introduces MinutesSpeech, a new speech dataset for long-context conversational TTS. The experimental results are extensive, showing that the proposed method can outperform baseline models both quantitatively and qualitatively.
weaknesses: * Although the MReQ module achieves low-resolution quantization while maintaining sample quality, its design is excessively complicated, posing challenges for future research to build upon. The LRVQ design incorporates multiple RVQ modules—pre-quantizer, main quantizer, and post-quantizer—alongside the additional HSR loss, making it significantly more complex than prior RVQ methods. Furthermore, the use of multiple RVQ quantizations within the NAR transformer of HALL-E adds another layer of complexity, reducing the accessibility of the proposed approach.
* The paper only compares the proposed method with VALL-E and does not include recent zero-shot TTS baselines. Considering that non-autoregressive modeling can be advantageous for long speech generation, incorporating comparisons with state-of-the-art non-autoregressive zero-shot TTS models would better demonstrate the effectiveness and relevance of the proposed method.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: * This technique seems well suited for LLM-based TTS solutions, similar to Valle with AR and NAR components. However, can you also say anything about how this kind of encoding might be used as a general purpose codec for speech in general? It would be interesting if this model could have lower bitrates or higher robustness. Furthermore, does this means that the PreQ and PostQ components of your LRVQ blocks would be unnecessary since they are primarily used to train the NAR components of the TTS model?

* With a VALLE model based on Encodec, once can remove RVQ code sequences from the NAR generation to get faster generation, in exchange for coarser generation. For this MReQ based paradigm, how does the audio quality here change?

* For the second ablation in section 7.4, how does this effect appear with the VALLE model that you trained?

* For the third ablation in section 7.4, it is very surprising that even w/o pretraining, it does not seem like this technique suffers significantly (compared to the performance drop in Table 11 for HALL-E). Why is that?

* The splits of the dataset in Table 2 and the description of why these splits were created in Section 7.1 seems very arbitrary and over designed for these experiments. Are these training splits deduplicated from each other?
soundness: 3
strengths: Idea is pretty interesting and novel and well motivated. The MReQ module is fairly complex, but diagrams and descriptions of the technique are well written. Manuscript is very detailed and training and post-training are well documented. Along with the code release, results look to be easily replicable. 

Authors go through great lengths to train and evaluate everything from scratch and provide extensive results for reconstruction and zero-shot TTS which included both automatic and subjective results. Experiments were run on multiple NAC models, showing that this technique can generalize to pre-existing NAC models.

Benchmark dataset is a welcome addition and the generation and evaluation of the dataset are well documented. Results show RTF improvements, indicating that this technique indeed reduces the bottleneck behind transformer-based TTS.
summary: This paper presents two post-training approaches to resolve the long context length issues for modern transformer-based autoregressive TTS models based on NAC models. MReQ as a framework reduces the framerate by introducing a novel Multi-resolution vector quantization module that allows the user to decompose an existing NAC model into several different code books, each of which operate at a different framerate. This allows them to reduce the bottleneck of the AR component of LLM-based TTS by reducing the frequency of the first component down to 8 Hz. To train this MRVQ module, student-teacher distillation is required. Using this module, this work presents HALL-E, a hierarchical TTS model that generates tokens based on the MReQ tokens. This paper also introduces MinutesSpeech, a benchmark dataset for TTS synthesis that is curated for long-form speech.
weaknesses: * One weakness in the study is that authors do not consider distribution shifts between the different dataset used to train the NAC model and the dataset used for post training. In their experiments, they pretrain their own NAC models (Encodec and Speechtokenizer) on the MinutesSpeech training set and did post training with MReQ with the same training data. However, in practice, researchers will not have access to the datasets used to pretrain these NAC models, so there may be some distribution shift. A similar issue occurs with quantization, where you need some training data to calibrate the model. Additionally, while SpeechTokenizer is meant to tokenize speech, Encodec was originally designed to encode speech along with other kinds of audio. It is unclear whether this capability will be preserved after post-training. Ablation studies do not cover what happens when you try to use this technique with an off-the shelf NAC model like SpeechTokenizer or Encodec off the shelf. 

* The statement on line 211/212 that Training NAC models with the MRVQ module does not seem to be justified or elaborated on. Is this just saying that training this kind of codec model from scratch rather than from a pretrained model is difficult? It seems like from Table 10, that for the MReQ model, starting this kind of training w/o pretraining seems to not affect the WER or RESQ significantly.

* The qualitative results in the ablation study near line 495 are not well explained. The statement that this waveform is unnatural with almost no silence is not clear without a transcription. From a glance, this seems to be a problem with the prosody of the generation, but this is unclear without hearing the samples. Appendix D1 also does not seem to elaborate on this much. I tried to listen to the samples from MinutesSpeech test-90s, and it seems like this is primarily a prosody issue as the Valle model seems to speak at a very consistent rate with much louder inhalation sounds.

* Furthermore, the ablation study for “Can HALL-E handle long audio prompts?” seems trivial, it should be expected that longer voice prompts gives the model more context on the user’s speech. It would be more interesting to see if this effect exists or is less pronounced in the VALLE model.

* There are also some wording errors in the manuscript. For instance, in the implementation details on page 4, lines 199 to 201, the manuscript says that “Figure 2a shows the MRVQ module applied to the Encodec model”, however, Figure 2a actually just shows the normal Encodec model as a teacher model. This should actually be Figure 2b that shows the MRVQ module applied to the Encodec model. Furthermore, Line 201 states that “Figure 2b shows the LRVQ block”, however, it is actually Figure 2c that shows the LRVQ block.  Also, should the statement in line 485/486: “The results indicate that both losses and post-training play important roles in the overall performance.” say that both losses and “pre-training” play important roles, since the second row of Table 10 is about the pre-training of the Encodec model?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Have you tried to train MReQ with a much smaller sampling rate, such as 2 or 4 Hz for the first layer? 
2. What is the accuracy of the sub-encoder E to predict b_{k+1} based on a_{k+1}? Will this accuracy affect the final performance a lot?
3. For the L_{total}， are the weights of the three sub losses equal and all set to 1.0 ? Have you tried other settings?
soundness: 3
strengths: 1. This paper has three contributions 1). A multi-resolution requantization method MReQ is proposed to generate hierarchical codec codes. 2). A hierarchical LLM-base TTS model is proposed to predict MReQ codec codes. 3). A minute-long speech synthesis dataset is introduced.
2. This paper is well written and easy to follow.
summary: To deal with the long-form speech synthesis problem, this paper propose a multi-resolution re-quantization method which hierarchically reorganizes discrete tokens with teacher-student distillations. Based on the hierarchical codec codes, an LLM-based TTS model HALL-E is proposed for speech synthesis. Besides these, a new benchmark dataset MinutesSpeech is introduced for minute-long speech synthesis.

This paper is well written and easy to follow.
weaknesses: 1. Some important codec models are not compared with, such as DAC and Vocos.
2. MReQ is only tested on the speech reconstruction, without evaluated with audio dataset.
3. Some latest zero-shot TTS models are not compared with, such as Voicebox, E2 and VALL-E 2.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: Please address the questions raised in the weaknesses section.
soundness: 3
strengths: The three contributions listed above are valid and well-received. The topic of minute-long speech synthesis is also one that will attract a wide audience.
summary: The paper presents three contributions: (1) Multi-Resolution Requantization (MReQ), a framework that post-trains a conventional RVQ codec into one with a different token rate in each layer; (2) HALL-E, an extension of the VALL-E framework to utilize an MReQ-based codec; and (3) the introduction of MinutesSpeech, a new benchmark for minute-long speech synthesis. Leveraging the MReQ concept, the authors successfully train a codec whose first layer operates at only 8Hz. With this low-token rate modeling, the HALL-E model generates minutes-long speech with significantly better WER and RTF compared to the original VALL-E model.
weaknesses: While the paper is generally well-written, there are several points that seem unclear to me. It is difficult to fully accept their proposal unless these points are clarified. I am open to reconsidering the score later.

- Unreasonable usage of PESQ raises doubts about their experiments.
  - Why and how can PESQ be used to filter out noisy data in the data curation pipeline? PESQ requires reference audio to compute the score, and I cannot understand how it could be applied for data curation purposes.
  - Why is the PESQ score for the ground truth so low in Table 3? The PESQ score for the ground truth should be 4.64.
- Questionable inference and evaluation settings
  - Zero-shot TTS experiment was conducted by “continual” setting or “cross-utterance” setting? From the description, it’s unclear.
  - Assuming that a "continual" setting is used, did the authors exclude the audio prompt portion of the synthesized audio when computing the speaker SIM? This is the approach taken in the VALL-E paper and other related works. The increase in speaker SIM in Table 9 with respect to the prompt length is unusually large, raising concerns that the authors may not have excluded the prompt from the speaker similarity computation.
  - Why is DNSMOS used? DNSMOS is specifically designed and trained to assess the cleanness of speech rather than its naturalness. I strongly suggest replacing it with a naturalness MOS metric, such as UTMOS.
- Other minor request to assess the quality of the model
  - Please present the speaker similarity and MOS scores (UTMOS?) for Table 3 and Table 10.
  - Please provide a breakdown of the AR and NAR model RTFs for Table 7.

Additionally, I found several minor issues in the description. They are not critical, but they need to be revised:
  - Equation 2 is likely incorrect. I believe it should be x_l = x_{l-1} - \tilde{z_l}
  - Similarly, Equation 9 is likely incorrect.
  - In Table 8, the upsampling ratio should be explicitly described instead of simply listing "HALL-E" in the last row.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: --
soundness: 3
strengths: 1. This paper introduces the Multi-Resolution Requantization (MReQ) model, designed to enhance signal processing across multiple granularities.  
2. It presents the development of HALL-E, an autoregressive language model capable of generating extended speech waveforms with high fidelity.  
3. Furthermore, the study proposes a novel benchmark dataset, MinutesSpeech, specifically curated to evaluate the performance of models on long-duration speech sequences.
summary: The paper introduces a novel text-to-speech (TTS) model designed to generate extended speech sequences. To address the inherent limitations of autoregressive language models in handling long outputs, the authors propose the use of *multi-resolution audio codes* (MReQ). Furthermore, the paper presents HALL-E, an autoregressive language model specifically developed to generate these MReQ audio codes efficiently. In addition to the model, the authors construct a new benchmark dataset comprising 40,000 hours of curated speech data. The experimental results demonstrate that the proposed approach can achieve a low frame rate of 8 Hz, highlighting its potential for generating high-quality long-form speech with reduced computational overhead.
weaknesses: 1. The concept of time-varying audio codes is not a new development. For reference, please see the work by [Variable-rate Discrete Representation Learning (2021)] [https://arxiv.org/pdf/2103.06089].

2. A notable issue lies in maintaining temporal coherence and acoustic fidelity, as evidenced by the declining performance observed in the SIM score.

3. The model continues to face difficulties in handling spontaneous speech, highlighting an ongoing challenge in this area.

Meta-review: In this work, the authors targeted to resolve the challenge of long-form TTS. The high frame rate results in the long length of audio tokens, which makes it different for autoregressive language models to generate tokens. Representative works such as VALL-E encounter challenges in generating high-quality audio for extended sequences. Consequently, this research is significant for advancing speech generation beyond 10+ seconds. 

There are two main contributions of this paper: 1) Multi-Resolution Requantization (MReQ) which is used to reduce the frame rate of neural codec. 2) HALL-E which is an LLM-based TTS model to predict hierarchical tokens of MReQ. Furthermore, the authors have developed a new dataset comprising 40,000 hours of training data, which will support the community in researching long-form speech generation. 

The work is sufficiently novel to address a very meaningful problem of speech generation. The experiments are well-executed with convincing results.  During the rebuttal, the authors actively answered reviewers’ questions with clarified details. Therefore, the work is worth publication in ICLR.

In summary, the strength of this paper is effectively addressing the challenging problem of synthesizing speech longer than 10 seconds by utilizing the proposed MReQ to reduce the frame rate and HALL-E to model the hierarchical tokens of MReQ. However, a noted weakness is the lack of comparison with recent advanced zero-shot TTS models such as VALL-E 2, which has shown significant improvement over VALL-E used in the paper, despite the inclusion of a comparison with MaskGCT during the rebuttal.

Decision: Accept (Poster)