HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis
Authors: Yuto Nishimura
Takumi Hirose
Masanari Ohi
Hideki Nakayama
Nakamasa Inoue
Venue: ICLR 2025 Poster (2025)
Keywords: Text-to-speech synthesis
LLM-based TTS
neural audio codec
long-form generation
Primary area: generative models
TL;DR: We introduce MReQ and HALL-E, methods that reduce frame rates in TTS models to enable minute-long speech synthesis, and present the MinutesSpeech dataset.

Abstract: Recently, Text-to-speech (TTS) models based on large language models (LLMs)
that translate natural language text into sequences of discrete audio tokens have
gained great research attention, with advances in neural audio codec (NAC) mod-
els using residual vector quantization (RVQ). However, long-form speech synthe-
sis remains a significant challenge due to the high frame rate, which increases the
length of audio tokens and makes it difficult for autoregressive language models
to generate audio tokens for even a minute of speech. To address this challenge,
this paper introduces two novel post-training approaches: 1) Multi-Resolution Re-
quantization (MReQ) and 2) HALL-E. MReQ is a framework to reduce the frame
rate of pre-trained NAC models. Specifically, it incorporates multi-resolution
residual vector quantization (MRVQ) module that hierarchically reorganizes dis-
crete audio tokens through teacher-student distillation. HALL-E is an LLM-based
TTS model designed to predict hierarchical tokens of MReQ. Specifically, it incor-
porates the technique of using MRVQ sub-modules and continues training from a
pre-trained LLM-based TTS model. Furthermore, to promote TTS research, we
create MinutesSpeech, a new benchmark dataset consisting of 40k hours of filtered
speech data for training and evaluating speech synthesis ranging from 3s up to
180s. In experiments, we demonstrated the effectiveness of our approaches by ap-
plying our post-training framework to VALL-E. We achieved the frame rate down
to as low as 8 Hz, enabling the stable minitue-long speech synthesis in a single
inference step. Audio samples, dataset, codes and pre-trained models are available
at https://yutonishimura-v2.github.io/HALL-E_DEMO.

Published as a conference paper at ICLR 2025
HALL-E: HIERARCHICAL NEURAL CODEC LAN- GUAGE MODEL FOR MINUTE-LONG ZERO-SHOT TEXT-TO-SPEECH S SYNTHESIS
Yuto Nishimura Takumi Hirose", Masanari Ohi', Hideki Nakayama', Nakamasa Inoue The University of Tokyo, Institute of Science Tokyo
yutonishimurav2eg. u-tokyo. ic.
ABSTRACT
Recently, fext-to-speech (TTS) models based on large language models (LLMs) that translate natural language text into sequences of discrete audio tokens have gained great research attention, with advances in neural audio codec (NAC) mod- els using residual vector quantization (RVQ). However, long-form speech synthe- sis remains a significant challenge due to the high frame rate, which increases the length of audio tokens and makes it difficult for autoregressive language models to generate audio tokens for even a minute of speech. To address this challenge, this paper introduces two novel post-training approaches: 1) Resllii-eso Re- quantization (MReQ) and 2) HALL-E. MReQ is framework to reduce the frame rate of pre-trained NAC models. Specifically, incorporates multi-resolution residual vector quantization (MRVQ) module that hierarchically reorganizes dis- crete audio tokens through teacher-student distillation. HALL-E is an M-baeeed TTS model designed predict hierarchical tokens of MReQ. Specifically, it incor- porates the technique using MRVQ sub-modules and continues training from a pre-trained LLM-based TTS model. Furthermore, to promote TTS research, we create MinutesSpeech, a new benchmark dataset consisting of 40k hours of filtered speech data for training and evaluating speech synthesis ranging from 3s up 180s. In experiments, we demonstrated the effectiveness of our approaches by ap- plying our post-training framework VALL-E. We achieved the frame rate down to as low as 8 Hz, enabling the stable minitue-long speech synthesis in single inference step. Audio samples, dataset, codes and pre-trained models are available at htttt:::.. //yttn/ shi mu r v2. gi..gi...... gig.ttthhe hub. i HALL- DEMO
INTRODUCTION
Recent advances large language models (LLMs) have enabled us to model complex linguistic structures and patterns with unprecedented precision natural language processing tasks (Brown) et al. 2020 Chowdhery et al. 2023 Zeng et al. 2023). J. Motivated by these developments, LLM- based text-to-specch (TTS) models have gained research interest for their ability model complex speech structures and patterns, enabling the synthesis of more -atural-sounding speech a zero-shot
manner (Wang al., 2023 Zhang al. 2023b (onn al. 2024 Han al. 2024a Chen al., 2024a Xin et al. 2024; Meng et al., 2024 Wang et al. 2024a). The core concept behind L.MM- based TTS models is translate natural language text into sequence of audio tokens, typically using frozen natural audio codec (NAC) models that quantize audio signals into discrete tokens via
vector quantization techniques (Zeghidour al. 2021
Défossez al. 2022 Kumar al. 2023b
Zhang et al. 2024 Wu et al., (023] Huang et al.
2023 Du et al. 2024 Yang et al. 2023a).
Since LLMs are becoming capable of capturing long context in text (Guo et al. 2022 Chen al., 2024b Han et al. (2024b). L.L.-based TTS models are also expected to handle long context to synthesize speech over extended periods. However, this presents significant challenges, mainly because the length of audio tokens per second is typically large. More specifically, when using an autoregressive language model to predict audio tokens, as in the VALL-E architecture (Wang et al. 2023), the high frame r at the first layer of residual vector quantization (RVQ) (Zeghidour al.
We refer the number of audio tokens per second the frame rate (Hz).

---

Published as a conference paper at ICLR 2025
2021 in NAC models often becomes major factor that hinders the synthesis of long speech. As recently discussed and investigated by Hann et al.] (2024a): Chen et al. (2024a), reducing the frame rate is essential, but not straightforward. Simply reducing the frame rate in NAC models results in degraded audio quality and incorrect phoneme pronunciation. Addressing this issue requires novel solutions that bridge NAC models and LLLM-based TTS models.
In this paper, we propose a novel approach to tackle the challenge of minitue-long speech synthesis in LLM-based TTS models by introducing a hierarchical post-training framework that effectively manages the trade-off between reducing frame rate and producing high-quality speech. Our contri- butions are summarized as follows:
1) We propose Relti-Resotttion Requantization (MReQ), a post-training framework for hicrar- chically reorganizing pre-trained RVQ module reduce the frame rate at lower quantization layers. MReQ incorporates a multi-resolution residual vector quantization (MRVQ) module into pre-trained NAC model and continues training in teacher-student distillation manner. This results in reducing the frame rate at the first quantization layer to 8 Hz.
2) We propose HALL-E, hierarchical LLM-based TTS model designed to predict hierarchical tokens of MReQ. The AR model trained using 8Hz tokens, while the NAR model trained by using sub-modules in MRVQ, and continues training from pre-trained LLM-based TTS model.
3) We introduce MinutesSpeech, new benchmark dataset promote TTS research, particularly for minute-long speech synthesis. The training set consists of 40k hours of automatically filtered and balanced speech data. The test set consists of hours of speech data with transcriptions created by professional transcribers.
4) We thoroughly conducted experiments to provide best practices for managing the trade-off be- tween reducing frame rate and producing high-quality speech, while demonstrating the effective- ness and efficiency of our approach. We open- dataset, codes and pre-trained models along with audio samples at
shimura - w22.gg.tub..... 10/ HALL- EDMMO
RELATED WORK
Neural audio codec models. NAC models produce discrete audio tokens by quantizing audio sig- nals. SoundStream (Zeghdoorr et al. 2021) and Encodec (Défossez et al. (022)) are pioneering NAC models, which significantly improved compression efficiency over traditional audio codecs. Recent studies have proposed NAC models with a focus on maintaining performance in speech processing tasks. Examples include SpeechTokenize
(Zhang et al. 2024), Descript Audio Code (Kumar al. 2023b), AcademiCodec (Yang et al. (2023a), AudioDec (Wu et al. 2023), RepCodec (Huang et al. 2023]. and FunCodec (Du et al.) 2024). Many studies have focused on reducing bps, while few have explored lowering and varying the frame rate Dieleman et al. (202T) introduced the concept of time-varying audio codes. Défossez et al. (2024) achieved the lowest frame rate for NAC model, reaching 12.5Hz by incorporating Transformer models and SpeechTokenizer. We report achieving an even lower frame rate of 8Hz, which is about 1.5 times shorter than it.
Zero-shot TTS. Shoo TTS aims to synthesize speech from text in the voice of a target speaker using only a short reference audio segment from that speaker. Early models relied on speaker em- beddings speaker adaptation (Casanova et al. 2022; Arik et al. 2018 Chen et al., 2019). while recent studies have focused on LLM-based models that use NAC models in conjunction with LLMs. VALL-E WWang et al.) (023) was the first LLM based model, demonstrating impressive capabilities in zero-shot TTS tasks. Follow-up studies have explored various extensions such as VALL-E X for cross-lingual TTS (Zhang al. 2023b), ELLA-V using Montreal forced aligner (Song al. (024)) RALL-E using prosody Teatures (Xin et al. 2024). VALL-E R using monotonic alignment (Han al. 2024a), VALL-E 2 using grouped code modeling (Chen et al. 2024a). and MELLE using mel-spectrogram features (Meng et al., 2024). Prosody information can also be modeled by LLMS latent language mode Mega-TTSS (Jiang et al. (023) and Mega-TTS (Jiang et al. 2024) introduced prosody LLMs to generate more natural prosody. Meanwhile, diffusion- based models (e.g., Natu- ralSpeech2/3 (Shen et al. 2024 Ju et al. 2024) and Voicebox (Le et al., 2024)) and prompt-based models (e.g., Prompt-TTS2 (Guo et al. 2023 Leng al. (0244) are also known to be effecttve to generate high-quality controllable speech. Beyond speech synthesis, several studies proposed audio generation models such as UniAudio (Yang et al. 2023b), Audiobox (Vyas et al. 2023). In contrast, this work explores post- cost-training methods to reduce the frame rate of LLM- models, aiming at minute-long speech synthesis given pre-trained NAC model such Encodec.
2

---

Published as a conference paper at ICLR 2025
3 PRELIMINARIES
Neural audio codec. A NAC model typically consists of three components: an encoder Gnc(-), vector quantizer VQ(-), and decoder Doc(-). The quantizer the core component that produces discrete audio tokens. This work assumes that an RVQ module (Zeghidour et al., (021) is used as the quantizer, which defined as follows:
(21-1)))),
(1)
21-1 21,
(2)
21
(3)
where 20 Enc(wn) E Iddx" is the encoder output for the input audio Win+ is the latent di- mension, is the sequence length, VQ, is vector quantizer,
N is discrete token sequence, Σ = -Emb(22 21) E TR is a sequence of embeddings corresponding to 21 obtained through learn- able embedding layer Embr(
{1, 2.... L} is the layer index, and is the number of layers. The output h E Rdxa is then fed into the decoder to reconstruct the input audio as Dec(h).
LLM-based TTS. An LLM-based TTS typically consists of two decoder-only language models: autoregressive (AR) model Tar and a non-autoregressive (NAR) model Thar [Wang et al. 2023)
The speech synthesis procedure is given by the following equations:
21 = Tar (t, ),",, (4) 21+1 = Thar (t, h h4 1), (5) hi Σ Se, (6)
ÿ = D h2l), (7)
where [/2,,, hr]] denotes the concatenation of these two matrices along the time axis. In Eq. Tar generates audio token sequence &1 NT' corresponding to the first layer of RVQ given two inputs: a text prompt t and an audio prompt =!" = VQ, (Enc pp))) extracted from an audio input Epr- In Eq. 5). Taae iteratively generates token sequences ê(+1 E N" from the accumulated hidden features h; in Eq. (6) and the audio prompt's hidden features h Finally, in Eq. 7 speech ý is generated. Note that Enc, VQ₁, and Dec are from a frozen NAC model.
Preliminary experiments. LLM-based TTS models have predefined context window size and typically trained with speech data ranging from several seconds to several tens of seconds. To gener- ate long speech segments, straightforward approach is to reduce the frame rate in the NAC model. However, reducing the frame rate below 48 Hz significantly decreases speech reconstruction perfor- mance as shown in Figure where we evaluated the performance of Encodec in terms of the estimated Perceptual Evaluation of Speech Quality (PESQ) scores (Kumar et al. (023a) and word eтгor rates (WERs) as functions of frame rates. Specifically, it is confirmed that training becomes entirely difficult at 8Hz. Therefore, in this study, we propose NAC model that works significant improvement over existing limitations.
4.0
100
WEB (%)
@
2.0
15 24
48
15
75
Hz
Hz
Reconstruction
Ground truth
Figure 1: Preliminary experiments on vary- ing the frame rate of Encodec.
8Hz, demonstrating
4 MREQ: MULTI-RESOLUTIOO REQUANTIZATION
This section introduces MReQ, post-training framework for hierarchically reorganizing pre- trained RVQ module to reduce the frame rate. Specifically, MReQ incorporates a multi-resolution residual vector quantization (MRVQ) module to pre- pre-trained NAC model as shown in Figure 2 and continues training the NAC model in a teacher-studee Encodec model, MReQ reduces the frame rate at the first quantization layer to 8 Hz. This enables LLM--aase TTS models to handle longer contexts.
distillation manner. For a pre-trained 48Hz
4.1 ARCHITECTURE
The MRVQ module is a nested structure of RVQ. Specifically, it consists of a residual structure composed of multiple low frame-rate residual vector quantization (LRVQ) blocks, each of which is itself residual structure operating at a different frame rate. The definition is given as follows.
In this paper, the tilde symbol denotes the emheddings corresponding token sequence.
3

---

Published as a conference paper at ICLR 2025
Teacher RVQ
(b) Studeat MRVQ
VQ+
* 4
LBV0
Frame
Student
Decoder
401fc
VQu
Distillation loss
1EVV
Proqu
Ap
Queari
Postigt vi
RVQ
MRVQ
VV
Conee
48tt
VQu
LevQ
Encoder
Encoder
layees
Ispere
Pa.
layees
401tc
VQu
Tavy
Reconstruction loss
Downsampling
Dssampling
Teacher
Audio (24kt)))
Student
(c) Architecture of LRVQ262 for 16Hss
Pre trained RVQ
MRVQ madule For MROQ
Figure 2: MReQ post-training based teacher-studen distillation. (a) Pre-trained RVQ module is used to extract teacher embeddings ht. (b) MRVQ module consists of multiple LRVQ blocks and learns to reduce the frame rates. Student embeddings h, extracted. (c) Each LRVQ block consists of a pre-quantizer PreQ, a sub-encoder F, a main quantizer Quant, a sub-decoder D, and post-quantizer Post to reduce frame rate from Sq to 8g-
Definition 1 (MRVQ module). Let xp E Rdx= be an encoder output, where is the latent dimension and no Tso is the sequence length depending on the time length T (sec) and the frame rate 8o (Hz). The MRVQ module is defined follows:
K
T.VO (K) O---y (xx-1), (8)
Ck₁
h
CK
(10)
where LRVO is an LRVQ block, K is the number of blocks, a- is triplet of hyperparam- eters to determine the block structure.
Definition 2 (LRVQ block). Each LRVQ block IRVO a-9-y consists of five components: pre- (k) sub-decoder
quantizer (k) (k) sub-encoder Fg. for down sampling, main quantizer Quant
D, for upsampling, and a post-quantizer Post() The quantization procedure is given by
0g ( (11) by = Quant (M) (Ex()))), (12)
Ca = Post (Dxx(b)) (13)
where ag. by, C2 are token sequences. The three quantizers PreQ (R) Quant (k) and PostQ are implemented using RVQ with a, 3, and ? layers, respectively. Note that by E Nax=* is the token sequence representing audio in a low frame rate. Its length is given by пı = Tsk,, where Sy is the frame rate satisfying 81 < 82 <
< 8K and 8 80- The other two sequences as and eg are used only for facilitating training of NAR models used in L.-M-based TTS models.
Implementation details. Figure 2b shows the MRVQ module applied to the Encodec model, where the frame rate is reduced from so 48 Hz to 81 8 Hz using LRVQ blocks. Figure 2c shows the LRVQ block. Each sub-encoder Eg consists of convolution layer followed by two bi-LSTM layers, which reduces the frame rate from Sy to Sk- Each sub- decoder D₁ consists of two bi-LSTM layers followed by transposed convolution layer, which is symmetric to Fg. Table [ lists frame rates 8k, hyperparameter triplets a-3--, and strides for the convolution and transposed convolution layers. For k = 4, only the pre-quantize is used, and the other components
Table 1: Implementation details
k S
a-3-y
Stride
1
8
1-6-1
6
2
16
2-6-2
3
3 24
2-4-2
2
4 48
3-0-0
are replaced with identical functions, reducing Eqs. 112 13 b = a4 and C4 b4, respectively.
4.2
WITH
DISTILLATION
Training NAC models with the MRVQ module challenging because quantization layers with lower frame rates are prone to be ignored. To address this, we introduce post-training technique based on teacher-studed teacher model. As shown in Figure 2a teacher embeddings (in green) are extracted from the frozen RVQ module, while student embeddings (in purple) are extracted from the MRVQ module. We then minimize the feature-level distillation (FLD) loss and the hidden-state reconstruction (HSR) loss.
distillation, where a NAC model pre-trained with a high frame rate serves as the
4

---

Published as a conference paper at ICLR 2025
(a) Low frame-rate rate AR, model
(b) NAR model
Next
b
8Hz
0c-1,2+1
Quant
AR Transformer Tar
NAR Transformer Take
Sub-encader B
brof
Hello....
||||||
b
Hello,
||||||
TTex prompt
Text
Speech
Text
Speech
PostQ
by
Miin
Sub=docoder D
Auxiliary vokeme and featuures
Next
moddueeedeseonn LRVQ
Figure 3: HALL-E architecture. (a) AR model generates low frame-rate token sequence bj. (b)
NAR model predicts b from by iteratively by utilizing frozen sub-modules of MRVQ.
FLD loss. The FLD loss is MAE loss between teacher and student embeddings, defined
LFLD
Σ (s.1)EP
FLD (s.)) [h - helli, (14) h.
C
(15)
S1 (16)
where h, is a student embedding, h, is teacher embedding, P is set of student-teacher index pairs, and XFLD is weight coefficient. We use
(4,8)) for RVQ with eight layers and MRVQ with four LRVQ blocks. Note that ég and ž are obtained from Eqs. (8) and Eqs. (T). respectively. The student- teccher pairs in P determined so that the cumulative number of student's 's post-quantization layers matches that of the teacher's quantization layers.
{(1, (2,3),
HSR loss. The HSR loss is introduced to further facilitate training of each LRVQ block:
LHSR
HSR A
a (x(b))ll
(17)
k=1
where åg by are from Eqs. [1
12 and A is weight coefficient.
Total loss. The total loss is given by Liotal = LAAC + LFLD + LHSR. where LNAC is the loss used to train the NAC model. We continue to train the encoder and decoder with the MRVQ module using a copied weight from the NAC model, which is used as teacher model. Compared to training from scratch, this allows for more efficient and stable convergence.
Discussion. Our post-training approach is designed to independent of the encoder-decoder ar- only assumed the of RVQ. Consequently, by uti-
chitecture of the original NAC model, lizing state-of-the-arr NAC models such as Speech Tokenizer (Zhang et al.) (024) instead of En- codec (Défossez al. 2022), is possible achieve higher performance lower frame rate.
HIERARCHICAL NEURAL CODEC LANGUAGE MODEL.
This section introduces HALL-E, hierarchical LLM-based TTS model that learns to generate hi- erarchical tokens. Inspired by VALL-E (Wang et al.) 2023 HALL-E consists of pair of language models: an AR model and an NAR model. There are two key differences compared to VALL-E. First, our AR model handles low frame-rate sequences, low Hz in our experiments, enabling stable generation of audio tokens for longer speech segments. Second, we incorporate frozen sub- modules obtained by decomposing the MRVQ module into the token prediction process using the NAR model. This integration facilitates training and results in high-quality speech synthesis. The whole inference process illustrated in Figure
AR model. Given a text prompt and an audio Upr as prompt, the AR model predicts a low frame- rate token sequence b, corresponding to t. This step is formulated as b, = Tar (t, bft).. similar to Eq. 4 for VALL-E, where Tar an AR transformer decoder and 6%" the audio prompt obtained by the first LRVQ block.
NAR model. Given a token sequence b,, the NAR model predicts the next token sequence b iteratively. Specifically, bx+1 is predicted in three steps by utilizing frozen sub-modules obtained from the MRVQ module. First, the sub-decoder and the post-quantizer applied de ob-

---

Published as a conference paper at ICLR 2025
tain és = Q
Second, an NAR transformer decoder Tuar is employed to pre- dict dx-1 from Cx. Because MRVQ has nested structure, this step requires applying the de- coder 0x.+1 times, where 04+11 is the number of layers in the pre-quantizer at the block A + 1. Specifically, for the layer ( (+] 1 in block k 1, denoted as d.+1/411, is computed ax+1 times using the following equation: (141,411
(Dx((b))
Taar (t, K,,, h 1x+1,1) similar to Eq. ( where h Ek δx + Σx= â is the accumulated feature and K, is the one for prompt. The layer id la+1,1 = Ek- On + is the cumulative sum of the number of pre-quantization layers. Finally, b is obtained by applying the sub-encoder and the main quantizer to åx+1 as by+1 = Quant (k+1) (0++((xx+1)) This process enables the NAR model to effectively generate high-fidelity token sequences.
Training. The AR model is trained using a cross-entropy loss LCB b1) measured between prediction b and the corresponding ground truth b1 obtained from training speech data. The delay pattern used in MusicGen (Copet et al. 2023a) is also applied. The NAR model is also trained using a cross-entropy loss (C ground truth. HALL-E is also post-trained given pre-trained LLM-basee TTS model.
1,1+1 where 01++1,141 is prediction and 1,/1, the
Discussion, This work focuses on post-training for reducing frame rate, but further exploration in token merging and grouping structures (Han et al.) [024a] Chen et al. (024a) would also be inter- esting in future research. Another architecture design involves incorporating learnable upsampling layers into the NAR model to predict bx+1 from 6y. However, handling different frame rates with a single NAR transformer is challenging. Additionally, a sub-encoder and sub-decoder are placed between b and ba. making the relationships between them more complex compared to the RVQ- based approach. To mitigate this, we employ the MRVQ sub-modules like above, which simplify the relationships to resemble the previous method, thereby improving the efficiency of the training.
6 MINUTESSPEECH BENCHMARK DATASET
This section introduces MinutesSpeech, a benchmark dataset for minutes-long TTS synthesis. Un- like previous datasets such as LibriSpeech, which are primarily designed for automatic speech recog- nition, our dataset curated to advance TTS research from the following three perspectives.
1) Benchmarking Minutes-long TTS Synthesis. We provide two subsets for benchmarking: MinutesSpeech--99 and MinutesSpeech-115 180s, consisting of speech segments ranging from 3 sec- onds to 90 seconds and 3 seconds to 180 seconds, respectively. All test audio files are under Creative Commons licenses. For each speech segment, we provide transcriptions created by two professional native transcribers. Since LLM- TTS models have been typically evaluated using audio seg- ments ranging from to 10 seconds from LibriSpeech in previous studies, our dataset promotes research longer speech synthesis by providing substantially longer segments.
2) Balanced Audio Length for Training, We also provide a training set consisting of 40,000 hours of audio data. To successfully train TTS models capable of stably generating audio ranging from few seconds to over a minute in single inference, we carefully designed the distribution of audio lengths in the training dataset, covering durations from seconds up 180 seconds. Automatically generated transcriptions are provided for this subset to facilitate large-scale training.
3) Variety of Speaking Styles. To encompass a variety of conversational speech styles, we curated data from podcasts. Compared to the audiobook speech in LibriSpeech, synthesizing conversational speech is more challenging due its spontaneous nature. We believe this is an important research direction that can help bridge the gap between L.MSs and LLMM-bseed TTS models.
Data Curation Pipeline. To create the training set, we first curated 296,464 podeast files, rang- ing from 10 to 90 minutes, totaling about 186k hours. We evaluated these audio files using the PESQ score estimated using TorchAudio-Squim (Kumar et al., (023a). The audio was seg- mented every 30 seconds, and the score was calculated for each segment. We then computed the mean and standard deviation for each audio file, retaining only those with a mean score higher than 2.5 and standard deviation lower than 0.6. As result, approximately 25% of the data remained. We then applied automatic speech recognition and speaker diarization to ex- tract speech segments, each associated with single speaker. Specifically, Whisper distil Large

---

Published as a conference paper at ICLR 2025
Table 2: Dataset statistics calculated on audio segments longer than 3s.
marks the results at 8Hz.
Audio duration
Max Token length Audio
Dataset
Split
Clips
Words
Average (sec)
Total (hours)
Text
Sum
LibriSpeech
train-clean-100
27949
12.90 3.29
100.18 4.46
986k
1176
240
1416
LibriSpeech
test-35s
1937
8.29 5.10
43k
1680
340
2020
MinutesSpeech
train-28s
7385k
19.11 + 7:11
39.21k
407501k
1344
314
1658
SinutesSpeech
train-90s
4181k
34.37
39.92k
407689k
720T
933
1655
SinucesSpeech
train-54s
4984k
28.82
+ 17:45
39.91k
40926 lk
2592
579
3171
SinutesSpeech
train-1801
3762k
37.69
4:49
39.39k
401816k
14401
1733
3173
MinutesSpeech
test-90s
685
45.28
31.35
8.61
833
720"
925
1645
SpeutesSpeech
10st-180s
536
56.76
55.24
8.45
81k
1440
1715
3155
was used for automatic transcriptions, and Pyannote was employed for speaker diarization. We then segmented each audio file into segments ranging from 3 to 90 or 180 seconds. Lastly,
we removed certain proportion of utterances with text lengths that were either too long or too short, based on their respective distributions, resulting in a final dataset of approximately 40k hours. For the test set, we manually collected audio files under Creative Commons licenses and asked professional transcribers to select high-quality audio and create accurate transcriptions.
Note that filtering based on the maximum and minimum text lengths used in the training set was also applied to each test set.
hours
hours
8.45
Dataset Statistics. Figure 4 shows the distri- butions of speech durations and the number of words per segment in the test sets. As shown, our dataset provides a diverse range of speech lengths, facilitating the development and evalu- ation of TTS models capable of handling vary- ing durations and linguistic content.
43k words
words
222
140
Wert Sourt
Figure 4: Duration and word count distributions.
7
EXPERIMENTS
7.1 EXPERIMENTAL SETUP
Baseline models. As a competitive comparison, we chose VALL-E (Wang et al., (023) using En- codec (Défossez al., 2022) at 48Hz as the baseline model, which is already lower than 75Hz used previous studies, based on our preliminary experiments (see Figure M We also demonstrate the
effectiveness of approach SpeechTokenizer (Zhang et al. 2024).
Datasets. For training, we used the MinutesSpeech training set, specifically train-90s and -180s for HALL-E, and train-28s, -54s, -90s, and -180s for VALL-E. The train-28s and -54s are designed for 48Hz to match the token length in train-90s and -180s at 8Hz (see Table 2). For evaluation, MinutesSpeech test-90s, 180s, and LibriSpeech test_clean set are used. The minimum audio length was set to 4s, while the maximum audio lengths were set to 90s, 180s, and 35s, respectively. The audio length for prompt was consistently
38.
Evaluation metrics. Two evaluation metrics are used for speech reconstruction experiments: WER and PESQ. WER is calculated using the conformer-transdduce
Zero-shot TTS experiments
are conducted in the continual setting (Wang et al. 2023). Seven evaluation metrics are used: WER,
speaker similarity (SIM). DNSMOS using ITU-T P.S0S ITU 2018 6 UTMOS (Sacki al. 2022).
Wasserstein distance (WD) with respect to the duration distribution, subjective evaluation of natu- ralness (QMOS), and subjective evaluation of speaker similarity (SMOS). SIM calculated using WavL M WD is calculated between the duration distributions of the generated speech and the ground truth speech. For QMOS and SMOS, 40 utterances were randomly selected from each
heepst/7g1
Spseeehn 1887 Ma1n7 aowns ET camar www.kar.www..faaal......

---

Published as a conference paper at ICLR 2025
test set, and three native English speakers rated their naturalness a scale from 1 to 5. We then calculated the mean and confidence intervals.
Training details. Encodec and SpeechTokenizer were pre-trained using the Adam optimizer for 100k iters with batch size of 704 for Encodec and 674 for Speech Tokenizer on four H100 GPUs, and a learning rate of 9 x 10-4. MReQ post-training was performed on single H100 GPU for 160k iters with a batch size of 160 and a learning rate of 3 x 10 VALL-E was trained using the AdamW optimizer for 100k iters on four H100 GPUs. To fully utilize GPU memory, the batch size was adjusted based on the audio length of the training samples. A cosine annealing learning rate schedule was employed with an initial learning rate of xx10= HALL-E was trained using the same settings, where VALL-E is used as pre-trained model. More details are provided in Appendix A
7.2 SPEECH RECONSTRUCTION
Table 3: Speech reconstruction on LibriSpeech.
Table 3] shows the speech re-
NAC rate, Hz) WERI
PESQ
STOMt
SIMI UTMOS1
construction performance of En- codec and SpeechTokenizer be- fore and after applying MReQ. MReQ maintains clarity and naturalness in terms of WER, STOI, SIM and UTMOS, while significantly reducing the frame rate at the first quantization layer to as low as 8 Hz.
GT
1.96
4.64 0:00
1.00 0.00 1.00
4.05
Encodec (8)
100.0
1.22.4088
0.32
0.52
1.33
Encodec (24)
2.01
3.68
0.94
0.90
3.37
Encodec (48) (8.Q(8.16,24444
2.00
4.02
0.96
0,94
3.86+0.36
2.02
3,89
0.95+a.02
0.92
+.39
SpeochTokenizer (48)
2.00
4.10
0.96
0.95
3.97 +0.53
1.99
3,96+H.14
0.95
0.93 4.01 +0.32
7.3 ZERO-SHOV SPEECH SYNTHESIS
MinutesSpeech evaluation. Table 4 compares performance of zero-shot TTS synthesis on the Min- utesSpeech test sets. Overall, HALL-E significantly outperformed the VALL-E, achieving compa- rable or even better WER and DNSMOS scores than the ground truth audio. This demonstrates the effectiveness of our approach. VALL-E resulted in a high WER and low QMOS score when trained on train-28s because it cannot generate audio longer than 28 seconds. When trained on train-90s, the WER decreased but still remained significantly higher than that of ground truth. This is because training an AR model with a high frame rate is unstable, indicating that simply changing the training data not sufficient for successful training. Our approach addressed this issue by lowering the frame rate via MReQ. In terms of SIM, VALL-E outperformed HALL-E because lowering the frame rate sacrifices acoustic information. However, HALL-E surpassed VALL-E in SMOS, as it generates the speaker's important style element, duration, much more naturally than VALL-E (see Figure 6).
LibriSpeech evaluation. Table 5 shows results on LibriSpeech for short speech synthesis up to 35 seconds. As shown, VALL-E's WER increased the training audio length increases. In contrast, HALL-E achieved a WER lower than 5% even with train-180s. The gap in WER between the model and the ground truth due the difference between LibriSpeech, which primarily contains read speech, and MinutesSpeech, which includes significant amount of spontaneous speech. Balancing speech synthesis of both read and spontancous speech remains challenge for future work.
NAC models. Table 6 compares the results obtained by Encodec and Speech As shown, all metrics except SIM improve with the use of SpeechTokenizer. Since SpeechTokenizer aims to preserve more linguistic information in the first VQ layer, it is likely that this aligns well with maintaining linguistic information better than acoustic information when the frame rate reduced. These results suggest the potential for further performance enhancement by employing more ad- vanced NAC models in the future.
Computational efficiency. Table comppree the real-time factor (RTF) of VALL-E and HALL-E, measured on an RTX 4090 GPU using the 4s to 10s segments from LibriSpeech. The results show that HALL-E generates audio approximately 3.4 times faster than VALL-E. In LLM-based TTS, the computational bottleneck typically lies in the AR model. Significant speed improvements were achieved by HALL-E because it reduces the number of tokens required for generation by a factor of six. This enhancement shows promising potential not only for LLM- based TTS but also for various applications, such spoken language modeling, as part of future work.
8

---

Published as a conference paper at ICLR 2025
Table 4: Zero-shot TTS performance on MinutesSpeech test sets, Best results are marked in bold.
TTS model
NAC model
Training
WWR,, SIMI WDL DNSMOST
UTMOS?
QMOST
SMOST
test- -0s
GT
10.30
3.79 024
3.28
0380
VALL-E
Encodec
mrain-28s
39.77
0.726
23.62
3.84 0:19
3.61 :.48
2.29
2.04 2:05
VALL-E
Encodec
train-90s
16.14
0.712
2.68
an
3.58 0m0
2.48
0.2
2.36 0.26
HALL-E
MR-Q-Encoder
train-90s
9.79
0.685
4.00
3.74
3.35 e.26
3.15
0.26
test- 180s
GT
10.20
3.78 0:5
3.25 0.50
4,26
0:22
VALL-E
Encodec
train-54s
36.52
0.706
25.66
3.55 0.3
3.15 6.9
1.68 o
1.70= 0.26
VALL-E
Encodec
train-180s
21.71
0.702
12.52
3.76
3.55
2.11
2.15#
HALL-E
MReQ-Encodee train-180s
10.53
0.660
5.79
3.91 + 0.19
3.74 0.37 3.38 0.25 3.31 0.23
Table 5: Zero-shot TTS performance on LibriSpeech test.clean set. Best results in the same group (1 or 2) are marked in bold. QMOS and SMOS were conducted in the same group.
TTS model
NAC model
Training
WERL SIM WD4 DNSMOS UTMOS1
QMOS SMOS
GT
1.74
4.07 0.19
+02
GT
1.74
3.84 0.19
=.37
4.16
VALL-E
Encodee
train-28s
4.05
0.740
0.442
3 0.19
3.99 2.39
2.71 0:000
2.662031
VALL-E2
Encodee
train-54s
6.03
0.741
0.414
3.95 0.20
2.61 0:25
2.60
VALL-E'
Encodec
train-90s
7.17
0.678
1.12
3.78
+ 022
3.74 + 0.48
2.08 0.1 2. 1111+10 +:22
VALL-E²
Encodec
train- 180s
95.46
0.711
12.77
3.83
3,81
1.74
1.71 +0221
HALL-E'
MReQ-Encodec
train-90s
4.63
0.701
0.196
3.88
3.84
2.74
2.79
HALL-E
MReQ-Encodec train-180s
4.49
0.676
0.166
3.88
0:21
3.81
0.35
2.58 0.27 2.58 +:22
Table 6: Results for Encodee and SpeechTokeniizer MinutesSpeech train-90s and test-90s are used for training and testing, respectively.
Real: time factor
0.05
0.10
0.15
0,20
VALLA BALL-S
TTS model
NAC model
WER| SIMI WDJ DNSMOS UTMOS1
0.213
GT
10.30
3.79 023 3.28 0.51
VALL-E HALL-E
Encodee
39.77 0.726 23.62
3.84 0.19
3.61
= 0.48
3.002200.008
MReQ-EEcodec
9.79 0.685
4.00
3,91 = 021 3.74
= 0.37
Takenisation
AR trannformct
NAR transfommer Decoder
VALL-E
SpeechTokenizer
35.62
0.724
24.20
3.86
- 0.18
3.65
: 0.38
Figure 5: Real-time factor (RTF).
HALL-E MRcM- Seeeo--Spechhooeniee
9.12 0.678 3.09
3,95
0.19
3.77
0.33
7.4 ANALYSIS AND ABLATION STUDIES
Is the hierarchical structure essential? The results in Table 7 demonstrate the impact of different frame rates adopted MReQ on Mieehh test-90s. For further details, please refer to Ap- pendix[C.) From this table, is evident that the proposed method achieves the best WER, indicating the importance of gradually increasing the frame rate in a hicrarchical manner. We also ob- served a trade-off between WER and SIM, the SIM deteriorates with a reduction in the number of 48Hz layers in the proposed method.
Table 7: Impact of varying hier- archical structure.
Upsampling
WER
SIM
(8,48)
10.27
0.693
(8,16,48)
10.71
0.693
18.16,24,48)
9.79
0.685
Table 8: SIM as
a function of audio prompt length.
Long audio prompts. Table g shows the SIM as a function of the audio length for prompt. The SIM values were calculated using audio segments longer than 65 seconds from the MinutesSpeecc test-90s. As expected, longer audio length resulted in better performance. In zero-shot TTS, due to the limited context, the voice-cloning results are typically inferior to stan- dard fine-tuning (Anastassiou al., 2024). Our proposed method has the potential to push the limits of such zero-shot TTS capabilities.
Len, VALL-E HALL-E
0.727
0.685
10s
0.801
0.750
20s
0.830
0.809
40s
0.851
0.846
60s
0.856
0.859
Ablation study for MReQ: Table 9 shows the ablation study on the codec in the proposed method, using the LibriSpeech test set. The results indicate that the HSR loss most contributes to recon- struction performance. It is important to note that although the impact of the FLD loss may appear

---

Published as a conference paper at ICLR 2025
minor, this does not mean that its influence on TTS is negligible. The codec model has a hierarchical structure with redundancy, meaning that information lost at the first layer can be compensated for in subsequent layers. How- ever, when training the AR model, it is crucial for suffi- cient information to be present in the first layer, and the FLD loss is essential for achieving this. For further de- tails, please refer to Appendix
Table Ablation study for MReQ
Model
WER] PESQT UTMOS1
Proposed
2.02 3,89 2.39
w/o FLD loss
2.03 3.89 3,87
w/ HSR loss
2.15
3.79 3,86
w/o pre-training
2.08
3.83 2.85
Ablation study for HALL-E: Table 10 shows the ab- lation study on HALL-E on MinutesSpeech test-90s. As shown, all the proposed methods were found to be effective. In particular, the results of training only with PreQ or PostQ highlight the importance of using the MRVQ sub-modules with NAR model as pointed out in Section 5 This underscores the necessity of designing the codec model with downstream tasks in mind, ensuring that it is optimized for the specific requirements of those tasks.
Table 10: Ablation study for HALL-E
Model
WERJ SIMI UTMOS1
Proposed
9.79 0.685 3.74
wlo pre-training
49.80 0.647 2.55
w/ PreQ only training
10.36 0,682 3.52+
w/ PostQ only training
10.11 0.683 3.73
Qualitative results. Figure 6 shows the ground truth audio from Min- utesSpeech test-90s and audio gener- ated by each models. Both HALL-E and VALL-E models were trained on MinutesSpeech train-90s. As shown, the speech synthesized by HALL-E has duration similar to the ground truth, while VALL-E struggles with duration prediction. Training the model at a low frame rate not only enables the generation of long speech segments but also allows for capturing ples, please refer to Appendix D.I
VALLE HALLE
Figure 6: Samples of generated waveforms. Each sentence is displayed in a different color.
more natural long-term temporal dynamics. For more exam-
CONCLUSION
We introduced two novel approaches for minute-long zero-shot text-to-speech synthesis: MReQ and HALL-E. MReQ reduced the frame rate of the Encodec model to 8 Hz by reorganizing audio tokens via MRVQ. HALL-E efficiently synthesized minute-long speech by using 8Hz tokens the AR model and MRVQ nub-modules in the NAR model. We demonstrated the effectiveness of these approaches on MinutesSpecch, the newly introduced dataset consisting of 40,000 hours of speech data. Our work contributed promote zero-shot TTS research.
Limitations and future work. By reducing the frame rate to 8 Hz, our AR model can utilize longer contexts, enhancing the naturalness of the synthesized speech. We believe that to handle extended context is particularly advantageous for larger AR models such as AudioLM (Borsos et al. 2023). SpeechGPT (Zhang et al. 2023a), and PSLM (Mitsui et 2024). Demonstrating the effectiveness of our approach not only in TTS but also with these models remains a future work. Furthermore, as shown Table 2 we have achieved shorter audio token length than the corresponding text token length. However, in our current AR model, we concatenate these tokens, which results in the text tokens becoming a bottleneck in terms of sequence length. Small-E (Lemerle et al., 2024) propose methods to mitigate this issue by processing each token individually and fusing them using cross- attention. Exploring such architectural enhancements is an important direction for future work. Lastly, shown in Table 6 our method brought significant improvements with SpkeczTo, even more so than when applied to Encodec. means that our approach can further enhance the ob- jective of preserving linguistic information. This indicates that our method could serve replace- ment for traditional SSL models (Hsu et al., [22T;: Chen et al., 2022) or ASR encoders (Lakhotial al. 2021 Chu et al. 2023), marking important direction for future research.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was supported by JSPS KAKENHI Grant Numbers 22K12089 and 23H00490.
REFERENCES
Philip Anastassiou, Jiawei Chen, Jitong Chen, Yuanzhe Chen, Zhuo Chen, Ziyi Chen, Jian Cong, Lelai Deng, Chuang Ding, Lu Gao, al. Seed-Its: A family of high-quality versatile speech generation models. arXiv preprint arXiv:2406.024433 2024.
Sercan Ömer Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou. Neural voice cloning with a few samples. Proc. Annual Conference on Neural Information Processing Systems (NeurIPS), pp. 10040-10050, 2018
Zalán Borsos, Raphaël Marinier, Damien Vincent, Rugene Kharitonov, Olivier Pietquin, Matt Shar- ifi, Dominik Roblek, Olivier Teboul, David Grangier. Marco Tagliasacchi, et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Proc. Annual Conference Neural Information Processing Systems (NeurIPS), pp. 1877-1901, 2020.
Edresson Casanova, Julian Weber, Christopher Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir A Ponti. YourTTS: Towards zero-shot multi-speaker TTS and zero-shot voice conversion for everyone. In Proc. International Conference on Machine Learning (ICML), pp. 2709-2720, 2022.
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujic Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. Waylm: Large-scale sepe-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, (6(6):1505 1518, 2022.
Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, and Furu Wei. VALL-E 2: Neural codec language models are human parity zero-shot text to speech synthesizers. arXiv preprint arXiv:2406.05.. 406.05370, 2024a.
Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. Longlora: Efficient fine- tuning of long-context large language models. In Proc. International Conference Learning Representations (ICLR), 2024b.
Yutian Chen, Yannis M. Assael, Brendan Shillingford, David Budden, Scott F. Reed, Heiga Zen, Quan Wang, Luis C. Cobo, Andrew Trask, Ben Laurie, Çaglar Gülçehre, Aäron van den Oord, Oriol Vinyals, and Nando de Freitas. Sample efficient adaptive text-to-speech. In Proc. Interna- tional Conference on Learning Representations (ICLR), 2019.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, and Sebastian Gehrmann et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research (JMLR), 24: 1-113, 2023.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023.
Djork-Amé Clevert. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv: 1002299 2015.
Jade Copet, Felix Kreuk, Itai Gat. Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défossez. Simple and controllable music generation. In Proc. Annual Conference on Neural Information Processing Systems (NeurIPS). 2023a.

---

Published as a conference paper at ICLR 2025
Jade Copet, Felix Kreuk, Itai Gat. Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défossez. Simple and controllable music generation. In Thirty- Conference on Neural Information Processing Systems, 2023b.
Alexandre Défossez, Jade Copet. Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. Transactions on Machine Learning Research (TMLR), 2022.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dia- logue. 2024.
Sander Dieleman, Charlie Nash, Jesse Engel, and Karen Simonyan. Variable-rate discrete represen- tation learning. arXiv preprint arXiv:2 2021.
Zhihao Du, Shiliang Zhang, Kai Hu, and Siqi Zheng. Funcodec: A fundamental, reproducible and integrable open-source toolkit for neural speech codec. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 591-595, 2024.
Qipeng Guo, Yunfan Shao, Pengfei Liu, Zheng Zhang, and Xiangyang Xue. Longt5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics (NAACL Findings), pp. 724-736, 2022.
Zhifang Guo. Yichong Leng, Yihan Wu, Sheng Zhao, and Xu Tan. Prompitts: Controllable text-to- speech with text descriptions. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 1-5, 2023.
Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, and Furu Wei. VALL-E R: Robust and efficient zero-shot text-to-speech synthesis via monotonic alignment. arXiv preprint arXiv:2406.07855, 2024a.
Insu Han, Rajesh Jayaram, Amin Karbasi, Vahab Mirrokni, David P. Woodruff, and Amir Zandieh. Hyperattention: context attention in near-linear time. In Proc. International Conference on Learning Representations (ICLR), 2024b.
Dan Hendrycks and Kevin Gimpel.
Gaussian error linear units (gelus). arXiv preprint
1XXi::6066.08155. 2016.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Zhichao Huang, Chutong Meng, and Tom Ko. Repcodec: A speech representation codec for speech tokenization. arXiv preprint arXiv:2309.00166 2023.
Rec ITU. P. 808: Subjevtive evaluation of speech quality with crowdsoucing approach. Interna- lional Telecommunication Standardization Sector (ITU-T), 2018
Shengpeng Ji, Minghui Fang. Ziyue Jiang, Rongjie Huang, Jialung Zuo, Shulei Wang, and Zhou Zhao. Language-codec, Reducing the gaps between discrete codec representation and speech language models. arXiv preprint 2402 /2208, 2024.
Ziyue Jiang. Yi Ren, Zhenhui Ye, Jinglin Liu, Chen Zhang, Qian Yang, Shengpeng Ji, Rongjic Huang, Chunfeng Wang, Xiang Yin, Zejun Ma, and Zhou Zhao. Mega-tts: Zero-shot text-to- speech at scale with intrinsic inductive bias. arXiv preprint 2306v:2306.03509, 2023.
Ziyue Jiang, Jinglin Liu, Yi Ren, Jinzheng He, Zhenhui Ye, Shengpeng Ji, Qian Yang, Chen Zhang, Pengfei Wei, Chunfeng Wang, Xiang Yin, Zejun Ma, and Zhou Zhao. Mega-tts 2: Boosting prompting mechanisms for zero-shot speech synthesis. In Proc. International Conference on Learning Representations (ICLR), 2024.
12

---

Published as a conference paper at ICLR 2025
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang. Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, Zhizheng Wu, Tao Qin, Xiang- Yang Li, Wei Ye, Shikun Zhang, Jiang Bian, Lei He, Jinyu Li, and Sheng Zhao. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models. In Proc. International Conference on Machine Learning (ICML), 2024.
Anurag Kumar, Ke Tan, Zhaoheng Ni, Pranay Manocha, Xiaohui Zhang, Ethan Henderson, and Buye Xu. Torchaudio- squim: Reference-less speech quality and intelligibility measures in tor- chaudio. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 1-5, 2023a.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved RVQGAN. In Proc. Annual Conference on Neural Information Processing Systems (NeurIPS). 2023b.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 1-113---355 2021.
Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances neural information processing systems, 36, 2024.
Théodor Lemerle, Nicolas Obin, and Axel Roebel. Small-e: Small language model with linear attention for efficient speech synthesis. arXiv preprint arXiv:2406.0444 2024.
Yichong Leng, Zhifang Guo, Kai Shen, Zeqian Ju, Xu Tan, Eric Liu, Yufei Liu, Dongehao Yang. Leying Zhang, Kaitao Song, Lei He, Xiangyang Li, Sheng Zhao, Tao Qin, and Jiang Bian. PromptTTS 2: Describing and generating voices with text prompt. In Proc. International Confer- ence on Learning Representations (ICLR), 2024.
Dan Lyth and Simon King. Natural language guidance of fighfiidlity text-to- -otto---eeechh with synthetic annotations. arXiv preprint arXiv:2402.01912 2024.
Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, and Furu Wei, Autoregressive Speech Synthesis without Vector Quantization. arXiv preprint arXiv:2407.0855// 2024.
Kentaro Mitsui, Yukiya Hono, and Kei Sawada. Towards human-like spoken dialogue generation between ai agents from written dialogue. arXiv preprint arXiv:2310.01008 2023.
Kentaro Mitsui, Koh Mitsuda, Toshiaki Wakatsuki, Yukiya Hono, and Kei Sawada. Pslm: Parallel generation of text and speech with llms for low-latency spoken dialogue systems. arXiv preprint arXiv:2406. 12428, 2024.
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. Utmos: Utokyo-sarllab system for voicemos challenge 2022. pp. 4521-4525, 2022.
Rico Sennrich. Neural machine translation of rare words with subword units. arXiv preprint
arXiv: 1508: 07909, 2015.
Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. In Proc. International Conference on Learning Representations (ICLR), 2024.
Yakun Song, Zhuo Chen, Xiaofei Wang, Ziyang Ma, and Xie Chen. ELLA-V: Stable neu- ral codec language modeling with alignment-guide sequence reordering. arXiv preprint arXiv:2401.073333 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
13

---

Published as a conference paper at ICLR 2025
Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, Jeff Wang, Ivan Cruz, Bapi Akula, Akinniyi Akinyemi, Brian Ellis, Rashel Moritz, Yacl Yungster, Alice Rakotoarison, Liang Tan, Chris Sum- mers, Carleigh Wood, Joshua Lane, Mary Williamson, and Wei-Ning Hsu. Audiobox: Unified audio generation with natural language prompts. arXiv preprint arXiv: 15821/ 2023.
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301. 02.300002111 11,1111 2023.
Xiaofei Wang. Manthan Thakker, Zhuo Chen, Naoyuki Kanda, Sefik Emre Eskimez, Sanyuan Chen, Min Tang, Shujie Liu, Jinyu Li, and Takuya Yoshioka. Speechx: Neural codec language model as a versatile speech transformer. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 32:1-14, 2024a.
Yuancheng Wang. Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu. Maskget: Zero-shot text-to-speech with masked generative codec transformer. arXiv preprint arXiv:2409.000 00750, 2024b.
Yi-Chiao Wu, Israel D Gebru, Dejan Marković, and Alexander Richard. Audiodec: An open-source streaming high-fidelity neural audio codec. In Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 1-5, 2023.
Detai Xin, Xu Tan, Kai Shen, Zeqian Ju, Dongchao Yang, Yuancheng Wang, Shinnosuke Takamichi, Hiroshi Saruwatari, Shujie Liu, Jinyu Li, and Sheng Zhao. Rall-e: Robust codec lan- guage modeling with chain-of-thought prompting for text-to-speech synthesis. arXiv preprint 00X10:2404.03204. 2024.
Ryuichi Yamamoto, Eunwoo Song, and Jae-Min Kim. Parallel wavegan: fast waveform gen- eration model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6199-6203. IEEE, 2020.
Dongchao Yang. Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou. Hifi-codec: Group-residual vector quantization for high fidelity audio codec. arXiv preprint 026:05530 2023a.
Donghhao Yang. Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, Zhou Zhao, and Helen Meng. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00704, 2023b.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing, 30:495-507 507,55572222 2021.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. Glm-130b: An open bilingual pre-trained model. In Proc. International Conference on Learning Representations (ICLR), 2023.
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abil- ities. arXiv preprint arXiv:2305. 11000, 2023a.
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. Speechtokenizer: Unified speech tokenizer for speech language models. In Proc. International Conference Learning Representations (ICLR), 2024.
Ziqiang Zhang, Long Zhou, Chengyi Wang. Sanyuan Chen, Yu Wu, Shujic Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, al. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. arXiv preprint arXiv:2303.03926 2023b.
14

---

Published as a conference paper at ICLR 2025
A
MODEL DETAILS
A.1 MREQ
Table 11: Hyper- of Encodec
Module
Hyper-Parameter Parameter
Values
Encoder & Decoder
ConvBlock Number
4 [128, 256, 512, 1024] [10, 5, 5,
ConvBlock Filter Sizes
ConvBlock Strides
ConvBlock Kernel size
7
ConvBlock Norm
Weight norm
Activate Function
ELU
Clevert 2015
LSTM number
RVQ
Codebook Number
Codebook Dim
128
Codebook Vocabulary
1024
Discriminator
MS-STFT Défossez et al. 2022)
Window lengths
[2048, 1024, 512, 256, 128]
Loss Function
Adversarial Loss
Weight Coefficients
Feature Matching Loss
4
L1 Loss
0.1
MS-SPEC Loss Yamaamoto et al. 2020
2
Table 12: Hyper-parameters of MReQ-Encodec
Module
Hyper- Parameerr
Values
Sub- Sub-Encoders & Sub-Decoders
ConyBlock Number
2 [512, 1024]
ConyBlock Filter Sizes
ConyBlock Strides
k=1
[6. 1]
k=2
[3. 1]
k=3
[2, 1]
ConyBlock Kernel size
ConyBlock Norm
Weight norm
Activate Function
ELU (Clevert 2015 2
Bidiractional LSTM number
LRVQ
Codebook Number
a-3-
=
1-6-1
2 k=
2-6-2
2--2
k=
3-0-0
Codebook Dim
128
Codebook Vocabulary
1024
Loss Function
FLD Loss
(DELD FLD
(8, 6, 4, 2) HSR HSR HSR. (8, 6, 4,
FLD
FLD
(2.3)
(3.5)
(4.))
Weight Coefficients
HSR Loss
(2HIST X A
Encodee: Table [T lists the various hyperparameters of the Encodec used in this study. Essentially, we utilized the configuration from the original Audiocraft (Copet et al., (023b) implementation. The main difference is the modification of the strides within the encoder and decoder from the original [8, 5, 4, 2] to [10, 5, 5, 2]. This change increases the overall stride from 320 to 500, resulting in an improvement in the frame rate of the generated token sequences from 75Hz to 48Hz.
MReQ-Encoded Table 12 presents the list of hyperparameters for the proposed TiReQ-Encoder method. Modules added by MReQ, such the sub-encoder, sub-decoder, and LRVQ, are the only
15

---

Published as a conference paper at ICLR 2025
ones listed, as all other modules are identical to those in Encodec. Essentially, the parameters for the sub-encoder, sub-decoder, and LRVQ are based on the hyperparameterss of the original en- coder, decoder, and RVQ, respectively. We anticipate that applying MReQ to architectures other than Encodec will similarly yield stable hyperparameters by basing them on those of the original architecture, ensuring ease of adaptation.
In Section 4.2 we expressed the total loss function as Laotal = INAC + LFLD + LHSR. The detailed formulation of LNaC in MReQ-Encodec is given by the following equation:
LNAC A (xin, y) + >y- - ((in, v) + >g (v()) Afear (Xin, v) + v) Cw.),,
(18)
where each À represents the weight of its corresponding loss function. The terms l and & denote the reconstruction errors calculated using LI loss and MS-SPEC loss, respectively. The terms &g and é feet correspond the adversarial loss and the feature matching loss, respectively. The last term, fw, represents the quantization enrors across all quantizers, specifically PreQ, PostQ, and Quant.
SpeechTokenizeer: In this paper, the implementation of SpeechTokenizer
almost identical to the Encodec architecture presented in Table [1] The only difference is that a bidirectional LSTM was used instead of an LSTM. For the teacher used in the teacher foreing process of Layer I's RVQ, we followed the original paper and employed Hubert base Is960
In the original paper, 16,000Hz audio was used as the input data, with a stride of 320, resulting in a frame rate of 50Hz. Additionally, the SSL model typically employs a window size of 20ms with a 16,000Hz input, also yielding a frame rate of 50Hz. This setup enabled the application of the teacher forcing framework without any special adjustments in the temporal direction.
However, in this study, to ensure a fair comparison with Encodec, we adopted a 24,000Hz input and stride of 500. Consequently, the frame rate became 48Hz, creating mismatch with the 50Hz frame rate of the SSL model. To address this issue, we adjusted the audio input to the SSL model by speeding it up by a factor of 50/48, ensuring that the duration of the SSL output aligns with that of the first layer of RVQ.
Moreover, while the original work utilized not only the MS-STFT discriminator employed by En- codee but also added a multi-period discriminator (MPD) and a multi-scale discriminator (MSD), as in HiFi-Codec (Yang et al.) (2023a), it was observed that the discriminators were too strong under the aforementioned settings. As a result, using only the MS-STFT discriminator, similar to Encodec, yielded the best performance.
As described above, the Speech Tokenizer ultimately uses an SSL model and, apart from the Bi-LSTM, matches the Encodec architecture in all respects. Therefore, when ex- tending to MReQ, the architecture is identical to that of and MReQ-Encodec lies in the pretrained weights used.
SRke--SpeechTokennnee
that is, the only difference
between
In this study, we did not use SSL models to train the Sareo-SSeechhokeeee SSL model was already utilized during the training of SpeechTokenizer, and since the features from that model were used for distillation, we deemed it unnecessary to reapply the SSL model.
This is because an
similar concept can be applied to models like Language-Coon Codee (Ji et al., 2024). The goal of this model is to restrict the quantizers of the first three channels to learn only the compressed audio frame information in the specified space by applying masking during RVQ training. However, since the features obtained in this manner are used in MReQ training, there is no need to perform masking again. other words, MReQ is flexible enough to be easily adapted and extended to new NAC models that may be developed in the future to learn better features.
A.2 HALL-E
VALL-E: Table presents the hyperparameters of the AR LM in the VALL-E model. For this implementation, we based our work on the LM of Audiocraft's MusicGen (Copet et al. 2023b) and referred to an unofficial VALL-E implementation 9
Table 14 presents the hyperparameters of the NAR LM in the VALL-E model. The fundamental hyperparameters are similar to those of the AR LM, with differences in the values for Dim, Heads,
htops:://gchhbs wf wasebberrt-aseesppeee
16

---

Published as a conference paper at ICLR 2025
Table 13: Hyper-parameter of AR LM in VALL-E
Module
HlperrParamette
Values
Transformer
Dim
1280
Number of Heads
20
Number of Layers
36
Dim of FFN Norm
5120
Layer norm
Activate Function
GELU Henndrycks & Gimpel 2016
Positional Encoding
sin (Vaswan et al.
2017
Text Tokenizer
Byte-Pair Encoding (BPE)
Sennrich
2015
Text Vocab
258
Table 14: pappeeerrrmeeerss of NAR LM in VALL-E
Module
Hlyper-Paramees
Values
Transformer
Dim
1024
Number of Heads
16
Number of Layers
24
Dim of FFN Norm
4096
Layer norm
Activate Function
GELU Hendryycks Gimpel 2016
Positional Encoding
sin Vaswani et al. 2017
Text Tokenizer
Byte-Pair Encoding (BPE) Sennrich 2015 258
Text Vocab
and Layers, resulting smaller number of parameters compared to the AR LMM. It is generally known that the task of generating acoustic tokens from acoustic input tokens handled by the NAR LM simpler than the task of generating tokens, including linguistic elements, from text handled by the AR LM. Indeed, in our preliminary experiments, increasing the model size of the NAR LM to match that the AR LM resulted in minimal differences in performance.
The architecture of the AR LM in HALL-E is almost identical to that of VALL-E, as shown in Table I3 The only difference lies in the input tokens being fed in a delayed format. This method was introduced in MusicGen and adopted in TTS systems such Lyth & King (2024). By employing this approach, it becomes possible to efficiently train multi-layer token sequences within the framework of next-token prediction.
HALL-E:
In HALL-E's AR LM, although it operates at a very low frame rate of 8Hz, needs to handle six layers of tokens. If this were trained flattened format, the sequence length would be equivalent to 48Hz, rendering the frame rate reduction meaningless. However, by handling it in delayed format, the sequence length can be maintained nearly the 8Hz frame rate, allowing for the optimal utilization of tokens obtained through MReQ.
The architecture of HALL-E's NAR model similarly shares many points with VALL-E's NAR LM, as shown in Table 14 There are two major differences: 1) cross-attention layer is additionally inserted into each transformer layer, using text tokens as input for the cross-attention, and 2) as described in Section 5 the output tokens of PostQ and PreQ are combined for input and output.
1) In this study, HALL-E handles very long audio segments, such as 90s or 180s. However, as the NAR LM processes tokens at 48Hz, it is difficult to train with all the tokens as input. Therefore, we consider training with only a portion of these long tokens. Specifically, by clipping the audio length corresponding to 90s or 180s that it becomes 28s or 54s, it is possible to train sequences of the same length as VALL-E's NAR LM. However, VALL-E's s NAR LM requires the text corresponding to the audio tokens to be concatenated with the audio tokens input. In this study, where there is no alignment between the text and audio, it is difficult to extract the corresponding text portion for the clipped audio tokens and concatenate them.
To address this, we stopped using the traditional concatenation method for inputting text information and instead used cross-attention to input the text information. This allows the attention mechanism
17

---

Published as a conference paper at ICLR 2025
to automatically learn the alignment with the input audio tokens. To our knowledge, model in the VALL-E framework has incorporated this kind of cross-attention mechanism to add text informa- tion. Small-E (Lemerle al. 2024) introduces this idea in more refined form with Linear Causal Language Models.
During inference, only audio up 28s 54s be processed most, method is employed where new audio generated sequentially by sliding 5s at a time. For more detailed inference algorithms and other information, please refer our publicly available GitHub repository
2) The technique of appropriately combining PostQ and PreQ for input, introduced in Section 5 was implemented with the aim of promoting learning by bringing the structure closer to the original form of the VALL-E NAR LM, as explained in that section. While this is indeed another major difference from VALL-E, should be noted that this operation involves using the modules acquired through MReQ to properly transform tokens outside of the NAR LM, and no changes have been made to the architecture of the NAR LM itself.
B DATASET DETAILS
B. MINUTESSPEECH TEST SET DETAILS
As mentioned in Section 6 the test was gathered from broadcasts available under Creative Commons license. The goal was to collect redistributable data to facilitate future research on long- form speech generation. The collection method involved first filtering English podcast descriptions for those containing Creative Commons-related terms. Subsequently, podcasts explicitly labeled with Creative Commons were manually selected. In most cases, the Creative Commons license applied only to the music used in the broadcasts, with few instances of the license covering the broadcasts themselves. Next, ASR was used generate provisional transcripts, which were then refined by native English speakers. Simultaneously, the quality of the audio was evaluated, and broadcasts with high audio quality, suitable for speech synthesis, were selected. The list podcast broadcasts included in the final dataset is presented in Table 15
Table 15: The list of podcast broadcasts under the Creative Commons license included in the Min- upesSpeech test set
URLs
Total Duration (h) License
https::/anchor. h
6.0
CC BY 4.0
podcast/rss/
http:://feedss.ccttivteee
1.8
CC BY-SA 4.0 &
research-oulttree/
CC BY-ND 4.0
httpss//anchorrff////0O0000
0.9
CC BY-NC-ND 4.0
podcast/rss/
0.4
CC BY-NC
E85
B.2 EXAMPLES OF THE TRANSCRIPTS
Table 16 shows a portion of an actual transcript. This example from the MinutesSpeech test-90s was generated by combining segments using ASR timestamps and speaker identification annota- tions by native English speakers, ensuring that the maximum duration does not exceed 90 sec- onds. As illustrated in this example, podcasts often feature dialogue format between two in- dividuals, which leads to variability in the duration a single speaker's continuous utterance. In fact, even within this table, the durations range from minimum of 3.68 seconds max- imum of 79.82 seconds. For the complete dataset and further details, please refer to https:
18

---

Published as a conference paper at ICLR 2025
7000
Average: 3.10
6000
5000
4000
Count
3000
2000
1000
2.50
3.00
3.50
4,00
Estimated PESQ
Figure 7: Distribution of estimated PESQ scores.
B.3 QUALITY
shows the distribution of PESQ scores estimated by TorchAudio-Squim Kumar et al. (2023a) for the MinutesSpeech training dataset, with an average score 3.10. Compared to Lib- niSpeech, which has an average score of 3.56. MinutesSpeech has lower-quality audio data. How- ever, audio reconstructed by the Encodec and SpeechTokenizer models trained on MinutesSpeech achieved scores of 3.45 and 3.52, respectively, as shown in Table This indicates that the dataset possesses sufficient quality for TTS purposes.
Figure 7
C ABLATION STUDIES
C.I MREQ
Further information of the ablation study for hierarchical structure in MReQ. In Section 7.4 we demonstrated that the proposed method achieves better results compared to other upsampling approaches. The parameters used are as follows: For (8, 48), = 2, (a,,81, 11, m)) (1, 1),
(a2, 32, 72) (7, 0)..) For (8, 16, 48), K 3, B1, 71) (1, 6, 1), (a2, 82, 72) = (2, (a3, Sa. 73) = 0,
Additionally, there is important point that needs to be addressed. when the upsampling rate is (8, 48), the loss diverged after about 60% of training. This likely due to the instability caused by the difference in the 1st layer between the Encodec used for the pre-training and MReQ-Encode used for the post-training, which led to unstable learning. This highlights the importance of hierarchical modeling prevent sudden distribution shifts.
Is FLD loss needed? Table 7 com- pares the where Ennoeec was trained without using the FLD loss and HALL-E was trained us- ing it. Here, MinutesSpeech-995 was used for evaluation. Interestingly, when the FLD loss is excluded, the results do not worsen across all metrics-Wer and SIM ac- tually improve. However, WD and DNSMOS deteriorate to a non-negligible extent. This indicates that using the FLD loss allows for generating more natural, human-like speech. Below, we will discuss the reasons for this.
Table 17: Ablation study for FLD loss in MReQ
Model
WERL
SIMI
WDJ
DNSMOST
HALL-E
9.79
0.685
4.00
3.91 : 0.21
w/o FLD loss
9.07
0.704
4.51
3.88 0.20
According to preliminary experiments, when MReQ is trained entirely without pre-trained model, LRVQ at frame rates like 8Hz is ignored, and the learning process progresses to supplement all the information at higher frame rates. Since post-training with pre-trained model weights is applied here, this issue is relatively mitigated, and relatively good results are achieved even without the FLD loss.
On the other hand, when the frame rate is compressed, there a tendency for finer temporal infor- mation be lost. This means, in principle, that acoustic information is more likely to be lost than
19

---

Published as a conference paper at ICLR 2025
Table 16:
Example transcript from
Start (s)
End (s)
Duration (s)
Transcript
8.11
27.47
19.36
Welcome to Deconstructing Management, podcast made by college students for college students. We've interviewed the chapter authors of the OpenStack's Principles of Management textbook...
34.57
92.55
57.98
I'm here with Chapter author, Siri Terjesen. Dr. Terjesen holds a postdoctoral degree from the Queensland University of Tech- nology...
96.19
99.99
3.80
Very good, Eric. Thanks so much for inviting me to join you in your class today.
100.53
104.65
4.12
Do you believe that successful entrepreneurs are born, or are they made?
104.65
110.61
5,96
That's a great question. And I think there are quite a lot of en- trepreneurs that are born, right?
123.79
142.33
18.54
And we didn't 't instigate that. But think that they can also be made...
156.81
168.57
11.76
I would have to agree with you because they could be born like my cousin...
172.33
180.27
7.94
He's going to start, like a party business, kind of like where he has bouncy houses...
180.85
252.06
71.21
That's neat. So really your cousin is great example of scrial or even portfolio entrepreneur...
252.06
259.86
7.80
So going straight to another question, what are the most im- portant traits a person must have to become a successful en- trepreneur?
260.12
313.72
53.60
That's great question, Eric. And you may have seen this in your cousin, right?...
318.13
357.97
39.84
Well, you're never too old! So there's a huge opportunity there...
365.94
445.76
79.82
That's great question. So, I think folks who have low-risk pref- erences may not make good entrepreneurs..
459.27
526.33
67.06
That's the first step. right? It's thinking about what the business would be...
527.01
541.68
14.67
It's mainly, it's also like, to start with like, shirts, like something so simple, like shirts...
544.02
547.70
3.68
So, so like, the controllers for like, Xbox, and like, PS4.
linguistic information. The first layer of Encodec's RVQ, which serves as the pre-trained model, contains not only linguistic information but also rich amount of acoustic information. Therefore, applying the FLD loss facilitates learning that retains both linguistic and acoustic information. This is believed to contribute to more natural speech generation. In contrast, when the FLD loss is not used, more acoustic information is lost. As a result, while the retention of linguistic information may improve WER, the overall outcome is less natural.
For this reason, we have adopted the FLD loss, which enables the generation of more natural speech. Further improvements in WER and SIM while using this method remain task for future work.
Is possible to use an even lower frame rate? Table 18 shows the results of HALL-E trained using lower frame rate adopted in MReQ. The evaluation was conducted similarly using MieutesSpeech- 90s. Here, a very low frame rate of 4Hz was applied to the first LRVQ quantization. As the results indicate, this caused a significant deterioration in WER. In general, the average duration of each
20

---

Published as a conference paper at ICLR 2025
phoneme is known to be around 100ms, which corresponds approximately 10Hz. Therefore, using a frame rate as low as 4Hz is likely to be fundamentally challenging.
On the other hand, we see substantial improvement in WD. This suggests that lowering the frame rate to 4Hz makes it easier learn longer-term temporal dy- namics, enabling the generation of more natural durations. From this, is evident that while further lowering the frame rate involves significant difficulties, the potential benefits are also considerable.
Table 18: Ablation study for lower frame rate
Upsampling
WER
SIM1
WD4
DNSMOS
(8, 16, 24, 48)
9.79
0.685
4.00
3.91 2021
24, 48)
20.07
0.690
1.95
3.90 =
Is it possible to reduce a bps? In MReQ, the number of Quant layers can be freely adjusted. Therefore, we considered reducing the number of Quant layers by half. In the original MReQ. the bitrate is set to match that of Encodec, which is 3.84 kbps, but by reducing the number of Quant layers in this way, it can de- creased to 2.64 kbps. Table 19 shows the performance changes of MReQ on the LibriSpeech test set when the bitrate is reduced in this manner. As the table indicates, significant performance degradation occurs tively simple task like Speech Reconstruction, This suggests that it is preferable to reduce the bitrate
Table 19: Hyperparameter study for lower bps in MReQ
As
WERI
PESQ1
(6, 6, 4, 0)
2.02
3,47
(3. 2, 0)
2.13
3.42
for rela-
through improvements the NAC model side, rather than by reducing the frame rate within the MReQ framework.
Table 20 shows the performance of HALL-E using this MReQ evaluated the MinutesSpeech- 90s dataset. As indicated by the WER degradation already observed in MReQ, WER wors- ens significantly when training HALL-E. On the other hand, improvements are seen in all other metrics besides WER. This can be attributed to the fact that the reduction in the number of
layers handled by the AR LM-from 6 layers to 3-makes the training process much easier, allowing for better learning of aspects like duration. Additionally, re- ducing the number of Quant layers, and thereby the amount of information con-
Table 20: Ablation study for lower bps HALL-E
3s
WER|
SIM1
WD4
DNSMOST
(6, 6, 4, 0)
9.79
0.685
4.00
3.91
(3, 3, 2, 0)
13.31
0.713
2.76
3.92 0.21
tained in each block, results in smaller relative differences between blocks, making it easier for the NAR LM to learn, which likely contributes improvements in metrics like SIM. However, the deterioration in WER is too significant to ignore, indicating that it generally better to maintain the same bitrate.
C.2 HALL-E
Table 21: Zero-shot TTS performances of HALL-E with different a-8-y pairs
Dataset
TTS model (01, 02 03, 04)
WER.
SIMt
WD4
DNSMOS
GT
3.79 4: 024
HALL-E
(1, 2,
9.77
0.688
4.25
3.92 023
MinutesSpecch-906
HALL-E
1,
9.60
0.702
4.59
3.93 i: 0.22
HALL-E
2,
9.79
0.685
4.00
3.91 1:011
HALL-E
(1,2,3,2)
10.58
0.696
4.09
3.90 0.19
GT
1.74
3.84
HALL-E
(1, 2,
4.37
0.706
0.201
3,89 1: 0.19
LibriSpeech
HALL-E
(1. 1, 1,
4.67
0.703
0.165
3.88 4:030
HALL-E
21 2,
4.63
0.701
0.196
3.88 + 0.39
HALL-E
2,
5.68
0.688
0.178
3.89 0.39
Is different a-3-y pair more effective? Table 21 shows the performance changes when the a-3-> combinations used in each LRVQ of MReQ are modified. It should be noted that in all cases, the bps is adjusted be equal that of Encodec, and B4 7 0. As the results indicate, there is no consistently strong combination across all metrics. On the other hand, when a2 = 2, the results are consistently the worst. This implies that reducing the number of layers corresponding to 48Hz too
21

---

Published as a conference paper at ICLR 2025
much leads to performance degradation. As shown in Figure significant performance deterioration starts from 48Hz in Encodec, suggesting that this layer should be retained as much as possible.
In this study, propose 2, as the recommended method, it consistently achieved bal- anced and good results across both datasets. However, other configurations also produced satisfac- tory results, demonstrating the robustness of our method to variations in hyperparameters.
D MORE QUALITATIVE RESULTS
D.1 DURATION ANALYSIS
Figure shows an qualitative example with manually annotated segments for each word. the first sentence, highlighted in red, the duration of the proper noun "Cam Ross" differs significantly across the models. In the speech generated by HALL-E, this phrase is pronounced more slowly, closely resembling the ground truth. This effect can be attributed the low sampling rate tokens, which likely made it easier to learn contextual relationships between words. Additionally, VALL-E tends introduce more filler sounds and breaths in the latter part of the utterance, potentially disrupting the duration prediction.
VALLE
herpttok
Wolp
Ampliuds
H
+ +
VALLE
Figure 8: Qualitative example with transcription (zoom in to view details).
Figure shows the waveform different sample compared to Figure 6 Similar the observa- tions in Figure VALL-E generates unnatural waveform with almost no silence, while HALL-E produces more natural waveform, closely resembling the ground truth with appropriate amounts of silence.
Figure I0 presents the duration distribution for the entire dataset, calculated from MinutesSpeech test-90s and test-180s. As consistently demonstrated by the WD in Table HALL-E's distribution is consistently closer to the GT compared to VALL-E. In addition to the issue of failing to insert silent pauses, VALL-E tends to overly repeat breathing sounds and fillers, resulting in longer durations. HALL-E, by utilizing a very small frame rate of 8Hz, overcomes these issues specific to AR models and achieves natural duration.
22

---

Published as a conference paper at ICLR 2025
Amplituc plitucce
HALL-E
Amplltude
VALL-E
Amplitude
20
40
60
80
100
120
140
Time (scc)
Figure 9: A sample from MieutesSpeech test- 18ss Both the HALL-E and VALL-E models were trained using MinutesSpeech train- 180s.
WM1-F
VALLE
HALLE
HALL-2
GT
Ouration155 15)
Duration
Figure 10: Comparison of the distribution of generated speech durations across different methods in the MinutesSpeech test sets. The left plot shows the distribution for the test-90s set, while the right plot shows the distribution for the test-180s set. Both the HALL-E and VALL-E models trained using the corresponding training sets (train-90s and train-180s, 180s).
The comparison of filler generation abilities between HALL-E and VALL-E. The left table presents the analysis results for the MinutesSpeech test-90s, while the right table shows the results for the test- test-180s. 180s. In each case. HALL-E and VALL-E are trained the corresponding train-90s and train-188 180s datasets.
Table 22:
TTS model
Avg. Fillers per text
Total Fillers
TTS model
Avg. Fillers per text
Total Fillers
GT
3.74 +: 3,66
2567
GT
4.64 5.22
2493
HALL-E
3,45 + 335
2364 2207
HALL-E VALL-E
4.19 4.66
2254
VALL-E
3.22 4: 3.30
3.87 7.05
2081
D.2
FILLER ANALYSIS
Fillers are an essential component for generating more natural human-like speech (Mitsui et al.) 2023). In this study, propose and use the MinutesSpeech dataset collected from podcasts. This dataset includes numerous instances of human dialogue, naturally containing many fillers. The abil- ity generate fillers frequency similar to human speech could be considered future benchmark for TTS systems.
Here, using the MinutesSpeech test set, demonstrate the spontaneous speech generation capabil- ity of HALL-E by analyzing and comparing the frequency of filler words in both ground truth and generated speech. The list of filler words present in the MinutesSpeech test set shown in Table 23
23

---

Published as a conference paper at ICLR 2025
Table 23: Filler Words List used in MinutesSpeech test set
The list of filler words
you know
like
actually
right
well
i mean
you basically
literally
uh
Table 22 shows the statistics of fillers in each MinutesSpeech test set. In cases, HALL-E generates fillers at frequency closer to the ground truth compared to VALL-E. This improvement is likely due to the use an 8Hz frame rate, which facilitates better language understanding in the AR LM.
Notably, in the 180s results for VALL-E, the standard deviation is significantly large. This is because VALL-E, as an AR model, sometimes exhibits repeated generation, producing fillers consecutively. Such cases were not observed in HALL-E, suggesting that lower frame rate crucial for suppress- ing repeats.
However, there is still gap between the generated results and the GT. This is likely because more advanced role-playing capabilities are needed. expected that utilizing models such as SLMs, which extend LLMs speech, could enable more human-like filler generation with their advanced language processing abilities.
E COMPARISON WITH NAR TTS
Table 24 analyzes the performance of MaskGCT (Wang et al. 2024b) by speech duration using the pre-trained model provided by the authors. We observed that the performance of MaskGCT significantly decreases as the length of synthesized speech increases. This indicates that even with NAR TTS, generating long speech in a single inference procedure challenging. Tables 25 and 26 compare VALL-E, HALL-E and MaskGCT in the test-30s and test-90s settings, respectively. In the 30-second test setting, the WER MaskGCT is lower than that of VALL-E and HALL-E. However, its UTMOS is worse compared to HALL-E. In the 90-second test setting, only HALL-E achieves WER lower than 10.0%, along with the highest UTMOS. These results highlight the effectiveness of the proposed method.
Table 24: MaskGCT performance by test duration.
TTS Model
Duration (sec)
WER
SIMT
DNSMOST
UTMOS1
MaskGCT
30
7.74
0.763
3.95 = 0.23
3,42 = 0.46
MaskGCT
45
10.75
0.763
3.94 + 0.22
3.39 = 0.49
MaskGCT
60
23.08
0.757
3.92 + 0.23
3.30
0.54
MaskGCT
90
51.12
0.691
3.57 = 0.51
2.57 = 1.02
Table 25: Performance comparison on test-30s.
TTS Model
WER
SIM
DNSMOS
UTMOS1
GT
10.65
3.74 + 0.29
3.20 + 0.51
VALL-E (train-28s))
11.84
0.731
3.84 0.19
3.61 0.41
VALL-E (train-90s)
14.83
0.695
3.82 1.22 0.20
3.46
HALL-E (train-90s)
10.88
0.685
3.87 0.21
3.61 0.38
MaskGCT
7.74
0.763
3.95 + 0.23
3.42 + 0.46
Table 26: Performance comparison on test-90s.
TTS Model
WER
SIMT
DNSMOST
UTMOS1
GT
10.30
3.79 1.79:00 0.24
3.28 + 0.51
VALL-E (1r)in-28s)
39.77
0.726
3.84 1 0.19
3.61
+ 0.48
VALL-E (train-90s)
16.14
0.712
3.87 + 0.17
3.58
+ 0.60
HALL-- (train-90s)
9.79
0.685
4.91+0.21
3.74
+ 0.37
MaskGCT
51.12
0.691
3.57
2.57
- 1.02
24


Reviews:
[{"review_id": "fMYhlGs3rY", "reviewer": "Reviewer_Fwrz", "summary": "The paper presents HALL-E, a neural codec language model aimed at addressing challenges in minute-long zero-shot text-to-speech (TTS) synthesis. It introduces Multi-Resolution Requantization (MReQ) to reduce frame rates in neural audio codec (NAC) models, and proposes HALL-E, which leverages MReQ for efficient and effective long-form speech synthesis. The work also introduces MinutesSpeech, a new speech dataset for long-context conversational TTS. The experimental results are extensive, showing that the proposed method can outperform baseline models both quantitatively and qualitatively.", "questions": "* The description of the LRVQ modules appears to omit the use of commitment loss for residual vector quantization. If the LRVQ module does not require commitment loss, the authors would explain the reasoning behind this and how the training is stabilized without it.\n* Providing a detailed description of the pre-trained LLM-based TTS model that HALL-E is post-trained on would greatly aid in understanding the training process and setup of the proposed model.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "* The paper offers thorough quantitative and qualitative results, demonstrating that the proposed generative modeling outperforms baseline methods in long-form speech synthesis.\n* The introduction of MinutesSpeech is a noteworthy addition to TTS research, providing high-quality, long-context speech data. The illustrated curation and preprocessing techniques further enhance its utility.\n* The ablation studies on MRVQ provide insights into the necessity of each component, justifying their inclusion in the architecture.", "weaknesses": "* Although the MReQ module achieves low-resolution quantization while maintaining sample quality, its design is excessively complicated, posing challenges for future research to build upon. The LRVQ design incorporates multiple RVQ modules—pre-quantizer, main quantizer, and post-quantizer—alongside the additional HSR loss, making it significantly more complex than prior RVQ methods. Furthermore, the use of multiple RVQ quantizations within the NAR transformer of HALL-E adds another layer of complexity, reducing the accessibility of the proposed approach.\n* The paper only compares the proposed method with VALL-E and does not include recent zero-shot TTS baselines. Considering that non-autoregressive modeling can be advantageous for long speech generation, incorporating comparisons with state-of-the-art non-autoregressive zero-shot TTS models would better demonstrate the effectiveness and relevance of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uGS1FWyrjS", "reviewer": "Reviewer_1NPx", "summary": "This paper presents two post-training approaches to resolve the long context length issues for modern transformer-based autoregressive TTS models based on NAC models. MReQ as a framework reduces the framerate by introducing a novel Multi-resolution vector quantization module that allows the user to decompose an existing NAC model into several different code books, each of which operate at a different framerate. This allows them to reduce the bottleneck of the AR component of LLM-based TTS by reducing the frequency of the first component down to 8 Hz. To train this MRVQ module, student-teacher distillation is required. Using this module, this work presents HALL-E, a hierarchical TTS model that generates tokens based on the MReQ tokens. This paper also introduces MinutesSpeech, a benchmark dataset for TTS synthesis that is curated for long-form speech.", "questions": "* This technique seems well suited for LLM-based TTS solutions, similar to Valle with AR and NAR components. However, can you also say anything about how this kind of encoding might be used as a general purpose codec for speech in general? It would be interesting if this model could have lower bitrates or higher robustness. Furthermore, does this means that the PreQ and PostQ components of your LRVQ blocks would be unnecessary since they are primarily used to train the NAR components of the TTS model?\n\n* With a VALLE model based on Encodec, once can remove RVQ code sequences from the NAR generation to get faster generation, in exchange for coarser generation. For this MReQ based paradigm, how does the audio quality here change?\n\n* For the second ablation in section 7.4, how does this effect appear with the VALLE model that you trained?\n\n* For the third ablation in section 7.4, it is very surprising that even w/o pretraining, it does not seem like this technique suffers significantly (compared to the performance drop in Table 11 for HALL-E). Why is that?\n\n* The splits of the dataset in Table 2 and the description of why these splits were created in Section 7.1 seems very arbitrary and over designed for these experiments. Are these training splits deduplicated from each other?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Idea is pretty interesting and novel and well motivated. The MReQ module is fairly complex, but diagrams and descriptions of the technique are well written. Manuscript is very detailed and training and post-training are well documented. Along with the code release, results look to be easily replicable. \n\nAuthors go through great lengths to train and evaluate everything from scratch and provide extensive results for reconstruction and zero-shot TTS which included both automatic and subjective results. Experiments were run on multiple NAC models, showing that this technique can generalize to pre-existing NAC models.\n\nBenchmark dataset is a welcome addition and the generation and evaluation of the dataset are well documented. Results show RTF improvements, indicating that this technique indeed reduces the bottleneck behind transformer-based TTS.", "weaknesses": "* One weakness in the study is that authors do not consider distribution shifts between the different dataset used to train the NAC model and the dataset used for post training. In their experiments, they pretrain their own NAC models (Encodec and Speechtokenizer) on the MinutesSpeech training set and did post training with MReQ with the same training data. However, in practice, researchers will not have access to the datasets used to pretrain these NAC models, so there may be some distribution shift. A similar issue occurs with quantization, where you need some training data to calibrate the model. Additionally, while SpeechTokenizer is meant to tokenize speech, Encodec was originally designed to encode speech along with other kinds of audio. It is unclear whether this capability will be preserved after post-training. Ablation studies do not cover what happens when you try to use this technique with an off-the shelf NAC model like SpeechTokenizer or Encodec off the shelf. \n\n* The statement on line 211/212 that Training NAC models with the MRVQ module does not seem to be justified or elaborated on. Is this just saying that training this kind of codec model from scratch rather than from a pretrained model is difficult? It seems like from Table 10, that for the MReQ model, starting this kind of training w/o pretraining seems to not affect the WER or RESQ significantly.\n\n* The qualitative results in the ablation study near line 495 are not well explained. The statement that this waveform is unnatural with almost no silence is not clear without a transcription. From a glance, this seems to be a problem with the prosody of the generation, but this is unclear without hearing the samples. Appendix D1 also does not seem to elaborate on this much. I tried to listen to the samples from MinutesSpeech test-90s, and it seems like this is primarily a prosody issue as the Valle model seems to speak at a very consistent rate with much louder inhalation sounds.\n\n* Furthermore, the ablation study for “Can HALL-E handle long audio prompts?” seems trivial, it should be expected that longer voice prompts gives the model more context on the user’s speech. It would be more interesting to see if this effect exists or is less pronounced in the VALLE model.\n\n* There are also some wording errors in the manuscript. For instance, in the implementation details on page 4, lines 199 to 201, the manuscript says that “Figure 2a shows the MRVQ module applied to the Encodec model”, however, Figure 2a actually just shows the normal Encodec model as a teacher model. This should actually be Figure 2b that shows the MRVQ module applied to the Encodec model. Furthermore, Line 201 states that “Figure 2b shows the LRVQ block”, however, it is actually Figure 2c that shows the LRVQ block.  Also, should the statement in line 485/486: “The results indicate that both losses and post-training play important roles in the overall performance.” say that both losses and “pre-training” play important roles, since the second row of Table 10 is about the pre-training of the Encodec model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hqeBVXUu3y", "reviewer": "Reviewer_UZgt", "summary": "To deal with the long-form speech synthesis problem, this paper propose a multi-resolution re-quantization method which hierarchically reorganizes discrete tokens with teacher-student distillations. Based on the hierarchical codec codes, an LLM-based TTS model HALL-E is proposed for speech synthesis. Besides these, a new benchmark dataset MinutesSpeech is introduced for minute-long speech synthesis.\n\nThis paper is well written and easy to follow.", "questions": "1. Have you tried to train MReQ with a much smaller sampling rate, such as 2 or 4 Hz for the first layer? \n2. What is the accuracy of the sub-encoder E to predict b_{k+1} based on a_{k+1}? Will this accuracy affect the final performance a lot?\n3. For the L_{total}， are the weights of the three sub losses equal and all set to 1.0 ? Have you tried other settings?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper has three contributions 1). A multi-resolution requantization method MReQ is proposed to generate hierarchical codec codes. 2). A hierarchical LLM-base TTS model is proposed to predict MReQ codec codes. 3). A minute-long speech synthesis dataset is introduced.\n2. This paper is well written and easy to follow.", "weaknesses": "1. Some important codec models are not compared with, such as DAC and Vocos.\n2. MReQ is only tested on the speech reconstruction, without evaluated with audio dataset.\n3. Some latest zero-shot TTS models are not compared with, such as Voicebox, E2 and VALL-E 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4Ew30uGH7O", "reviewer": "Reviewer_aZfo", "summary": "The paper presents three contributions: (1) Multi-Resolution Requantization (MReQ), a framework that post-trains a conventional RVQ codec into one with a different token rate in each layer; (2) HALL-E, an extension of the VALL-E framework to utilize an MReQ-based codec; and (3) the introduction of MinutesSpeech, a new benchmark for minute-long speech synthesis. Leveraging the MReQ concept, the authors successfully train a codec whose first layer operates at only 8Hz. With this low-token rate modeling, the HALL-E model generates minutes-long speech with significantly better WER and RTF compared to the original VALL-E model.", "questions": "Please address the questions raised in the weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The three contributions listed above are valid and well-received. The topic of minute-long speech synthesis is also one that will attract a wide audience.", "weaknesses": "While the paper is generally well-written, there are several points that seem unclear to me. It is difficult to fully accept their proposal unless these points are clarified. I am open to reconsidering the score later.\n\n- Unreasonable usage of PESQ raises doubts about their experiments.\n  - Why and how can PESQ be used to filter out noisy data in the data curation pipeline? PESQ requires reference audio to compute the score, and I cannot understand how it could be applied for data curation purposes.\n  - Why is the PESQ score for the ground truth so low in Table 3? The PESQ score for the ground truth should be 4.64.\n- Questionable inference and evaluation settings\n  - Zero-shot TTS experiment was conducted by “continual” setting or “cross-utterance” setting? From the description, it’s unclear.\n  - Assuming that a \"continual\" setting is used, did the authors exclude the audio prompt portion of the synthesized audio when computing the speaker SIM? This is the approach taken in the VALL-E paper and other related works. The increase in speaker SIM in Table 9 with respect to the prompt length is unusually large, raising concerns that the authors may not have excluded the prompt from the speaker similarity computation.\n  - Why is DNSMOS used? DNSMOS is specifically designed and trained to assess the cleanness of speech rather than its naturalness. I strongly suggest replacing it with a naturalness MOS metric, such as UTMOS.\n- Other minor request to assess the quality of the model\n  - Please present the speaker similarity and MOS scores (UTMOS?) for Table 3 and Table 10.\n  - Please provide a breakdown of the AR and NAR model RTFs for Table 7.\n\nAdditionally, I found several minor issues in the description. They are not critical, but they need to be revised:\n  - Equation 2 is likely incorrect. I believe it should be x_l = x_{l-1} - \\tilde{z_l}\n  - Similarly, Equation 9 is likely incorrect.\n  - In Table 8, the upsampling ratio should be explicitly described instead of simply listing \"HALL-E\" in the last row.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lxtIlcPUFk", "reviewer": "Reviewer_JBx3", "summary": "The paper introduces a novel text-to-speech (TTS) model designed to generate extended speech sequences. To address the inherent limitations of autoregressive language models in handling long outputs, the authors propose the use of *multi-resolution audio codes* (MReQ). Furthermore, the paper presents HALL-E, an autoregressive language model specifically developed to generate these MReQ audio codes efficiently. In addition to the model, the authors construct a new benchmark dataset comprising 40,000 hours of curated speech data. The experimental results demonstrate that the proposed approach can achieve a low frame rate of 8 Hz, highlighting its potential for generating high-quality long-form speech with reduced computational overhead.", "questions": "--", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper introduces the Multi-Resolution Requantization (MReQ) model, designed to enhance signal processing across multiple granularities.  \n2. It presents the development of HALL-E, an autoregressive language model capable of generating extended speech waveforms with high fidelity.  \n3. Furthermore, the study proposes a novel benchmark dataset, MinutesSpeech, specifically curated to evaluate the performance of models on long-duration speech sequences.", "weaknesses": "1. The concept of time-varying audio codes is not a new development. For reference, please see the work by [Variable-rate Discrete Representation Learning (2021)] [https://arxiv.org/pdf/2103.06089].\n\n2. A notable issue lies in maintaining temporal coherence and acoustic fidelity, as evidenced by the declining performance observed in the SIM score.\n\n3. The model continues to face difficulties in handling spontaneous speech, highlighting an ongoing challenge in this area.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)