Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer (2024)

Abstract: While recent advancements in speech language models have achieved significant progress, they face remarkable challenges in modeling the long acoustic sequences of neural audio codecs. In this paper, we introduce Generative Pretrained Speech Transformer (GPST), a hierarchical transformer designed for efficient speech language modeling. GPST quantizes audio waveforms into two distinct types of discrete speech representations and integrates them within a hierarchical transformer architecture, allowing for a unified one-stage generation process and enhancing Hi-Res audio generation capabilities. By training on large corpora of speeches in an end-to-end unsupervised manner, GPST can generate syntactically consistent speech with diverse speaker identities. Given a brief 3-second prompt, GPST can produce natural and coherent personalized speech, demonstrating in-context learning abilities. Moreover, our approach can be easily extended to spoken cross-lingual speech generation by incorporating multi-lingual semantic tokens and universal acoustic tokens. Experimental results indicate that GPST significantly outperforms the existing speech language models in terms of word error rate, speech quality, and speaker similarity. See https://youngsheen.github. io/GPST/demo for demo samples.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1764–1775 August 11-16, 2024 ©2024 Association for Computational Linguistics
While recent advancements in speech language models have achieved significant progress, they face remarkable challenges in modeling the long acoustic sequences of neural audio codecs. In this paper, we introduce Generative Pretrained Speech Transformer (GPST), a hierarchical transformer designed for efficient speech language modeling. GPST quantizes audio waveforms into two distinct types of discrete speech representations and integrates them within a hierarchical transformer architecture, allowing for a unified one-stage generation process and enhancing Hi-Res audio generation capabilities. By training on large corpora of speeches in an end-to-end unsupervised manner, GPST can generate syntactically consistent speech with diverse speaker identities. Given a brief 3-second prompt, GPST can produce natural and coherent personalized speech, demonstrating in-context learning abilities. Moreover, our approach can be easily extended to spoken cross-lingual speech generation by incorporating multi-lingual semantic tokens and universal acoustic tokens. Experimental results indicate that GPST significantly outperforms the existing speech language models in terms of word error rate, speech quality, and speaker similarity. See https://youngsheen.github. io/GPST/demo for demo samples.

heading: 1 Introduction
text: Speech quantization has emerged as a crucial technique for speech language models to generate controllable, high-quality speech waveforms (Borsos et al., 2023b; Lakhotia et al., 2021; Wang et al., 2023a; Kreuk et al., 2023; Kharitonov et al., 2023; Borsos et al., 2023a). Specifically, a speech waveform can be quantized into two distinct types of discrete representations: semantic tokens (Lakhotia et al., 2021) and acoustic tokens (Défossez et al.,
*Work done during an internship at Tencent AI Lab. †Corresponding author.
2022; Zeghidour et al., 2022). Semantic tokens are typically obtained by applying the K-means clustering algorithm to the continuous activation space of self-supervised speech models (Hsu et al., 2021; Baevski et al., 2020). Notably, GSLM (Lakhotia et al., 2021) finds that auto-regressive models trained on semantic tokens can capture high-level linguistic content, supporting language modeling and resynthesis (Polyak et al., 2021). However, semantic tokens fail to retain acoustic details such as speaker identity, resulting in suboptimal reconstruction. In contrast, acoustic tokens generated by neural codec models (Zeghidour et al., 2022; Défossez et al., 2022) effectively compress speech at low bitrates while capturing the nuances of speech waveforms. Consequently, a speech language model can maintain long-term consistency with semantic tokens and produce high-quality synthesis with acoustic tokens.
However, neural codec models require an excessive number of codes for high-quality speech synthesis. For example, EnCodec (Défossez et al., 2022) generates codec embeddings at 75 Hz for audio waveforms at 24 kHz. Subsequently, these codec embeddings are modeled using residual vector quantization (RVQ), wherein high-quality synthesis typically requires eight or more hierarchical quantizers with 1024 entries. Therefore, a mere 10- second waveform results in at least 75× 8× 10 = 6000 codes, which constitutes an excessively long sequence for language models due to the quadratic complexity with respect to the sequence length for calculating self-attention (Vaswani et al., 2017). Consequently, addressing the trade-off between the perceptual quality and computational complexity remains a core challenge for speech language models.
Recently, some methods have been proposed to address the issue of lengthy acoustic sequences. Acoustic tokens inherently possess a hierarchical structure because of residual vector quantiza-
1764
tion: tokens from the preceding quantizers restore acoustic properties such as speaker identity, while the subsequent quantizers capture finer acoustic details. Each quantizer is trained to model the residuals from the previous quantizers. Recent approaches (Borsos et al., 2023b; Wang et al., 2023a; Kharitonov et al., 2023) treat the acoustic token generation process as a multi-stage framework to avoid learning excessively long sequences simultaneously.
In this work, we present Generative Pre-trained Speech Transformer (GPST), a model that facilitates controllable, high-quality speech generation in single stage. Our approach combines speech quantization with the architecture of a hierarchical transformer (Lee et al., 2022b; Yu et al., 2023). GPST initially models the semantic sequence with a next token prediction task, followed by modeling the acoustic sequence with the task of predicting the next D stack codes. The semantic sequence serves as a prompt for the acoustic token as a condition. We design a specialized hierarchical architecture to model the underlying hierarchical structure of the acoustic sequence, which comprises of a large global transformer and a small local transformer. The global transformer learns the highlevel relationships between the semantic tokens and the stacked acoustic tokens, while the local transformer models the hierarchical details in the stacked acoustic codes. By incorporating semantic and acoustic tokens within one hierarchical transformer, GPST can significantly reduce computational costs and effortlessly learn the long-term interactions of semantic tokens and local dependencies among residual codes. Furthermore, we propose a training technique called “local-drop” to further improve the training efficiency of Hi-Res speech generation, which is typically impractical in current speech language models because of a large number of residual quantizers. Consequently, our model can generate high-quality and semantically coherent speeches in one stage efficiently.
Our main contributions are summarized as follows.
• We propose a novel generative pre-trained speech language model GPST that enables controllable, high-quality speech generation in a single stage. By integrating semantic tokens and acoustic tokens within a hierarchical transformer, GPST significantly reduces computational costs while efficiently learning the
long-term interactions of semantic tokens and local dependencies among residual codes simultaneously.
• We demonstrate GPST’s capacity not only to generate coherent speech unconditionally but also to generate speech while preserving the speaker’s identity with only a 3-second short prompt. Experimental results reveal its superiority over existing speech language models with only 33% parameters.
• To the best of our knowledge, GPST is the first work that supports spoken multilingual speech generation and Hi-Res speech synthesis.

heading: 2 Related Work

heading: 2.1 Discrete Speech Representation
text: Speech quantization has become a fundamental technique in speech language modeling (Borsos et al., 2023b; Kreuk et al., 2023; Wang et al., 2023a; Kharitonov et al., 2023). Typically, a speech waveform can be quantized into two distinct types of discrete representations: semantic tokens and acoustic tokens. Benefiting from the development of self-supervised learning in the field of speech understanding, Textless NLP (Lakhotia et al., 2021; Polyak et al., 2021) proposes to model speech based on HuBERT codes (Hsu et al., 2021) or semantic tokens, which are obtained by applying a K-means clustering algorithm on the activation hidden space of HuBERT. Auto-regressive modeling of these tokens (Lakhotia et al., 2021) can facilitate generating syntactically and semantically plausible speech continuations. SeamlessM4T (Communication et al., 2023) learns a spoken multi-lingual SSL model XLSR (Babu et al., 2022) to build a multi-lingual semantic vocabulary for speech translation. However, semantic tokens fail to synthesize the acoustic details in speech such as the speaker’s identity. Neural audio codecs (Zeghidour et al., 2022; Défossez et al., 2022) are proposed to quantize speech into stacked codes with residual vector quantization (RVQ) at low bitrates while preserving high-quality reconstruction. These acoustic tokens can capture the details of audio waveforms as diverse as multi-speaker speech (Borsos et al., 2023b), music (Agostinelli et al., 2023) and audio effects (Kreuk et al., 2023). In comparison, the proposed GPST integrates semantic tokens and acoustic tokens within one model in a single stage, effectively unifying their strengths.

heading: 2.2 Speech Language Models
text: Recently, speech language models have achieved remarkable progress in generating controllable, high-quality speech waveforms. Among them, AudioLM (Borsos et al., 2023b) introduces acoustic tokens into semantic token modeling and proposes a multi-stage generative framework to model semantic tokens, coarse acoustic tokens, and fine acoustic tokens sequentially. SPEAR-TTS (Kharitonov et al., 2023) extends AudioLM to the TTS task by training an additional text-to-semantic model. SoundStorm (Borsos et al., 2023a) speeds up the generation process of AudioLM by introducing confidence-based parallel decoding on acoustic tokens. VALL-E (Wang et al., 2023a) proposes a multi-stage language model for TTS with phonemes as input and acoustic tokens as output. VALL-E X (Zhang et al., 2023b) extends VALL-E to cross-lingual TTS tasks based on a text-based translation system. SpeechGPT (Zhang et al., 2023a) conducts further pre-training and instruction tuning on a speech dataset of semantic tokens, empowering text-based LLMs such as LLaMA (Touvron et al., 2023) to handle crossmodal instruction recognition and speech dialogues. PolyVoice (Dong et al., 2024) proposes a speech language model trained with text instructions for speech-to-speech translation. Viola (Wang et al., 2023b) proposes a multi-task framework built upon VALL-E to support multiple speech tasks. However, they are compelled to model acoustic tokens in a multi-stage framework due to the high complexity of learning long acoustic sequences. The proposed GPST circumvents this limitation by proposing a hierarchical transformer architecture that unifies semantic tokens and stacked hierarchical acoustic tokens within one stage. Concurrently, a parallel work (Yang et al., 2023) also explores similar methods in their study.

heading: 3 Generative Pre-trained Speech Language Model (GPST)
text: In this section, we start with the formulation of speech language modeling, along with the modeling challenges in speech generation. Next, we elaborate on our proposed model GPST in detail, followed by an efficient training technique for GPST to train Hi-Res speech model. Additionally, we discuss various inference modes with in-context learning.

heading: 3.1 Generative Speech Pre-training
text: Given an audio waveform sequence x ∈ RT , we quantize it into the sequence of semantic tokens S = (s1, . . . , sT1) ∈ {1, . . . , Ns}T1 and acoustic tokens A = (a11, . . . , a D 1 , . . . , a D T2 ) ∈ {1, . . . , Na}T2×D, with T1, T2 ≪ T . The acoustic sequence is a two-dimensional matrix and has a hierarchical structure such that aqt is derived from the residual of the previous token aq−1t . The learning objective of the speech language model can be auto-regressively factorized as
p(S,A) = p(S)p(A|S)
=
T1∏
t=1
p(st|s<t) D,T2∏
q,t=1
p(aqt |a≤D<t , a<qt , S) (1)
A naive approach can unfold the acoustic sequence A into a one-dimensional sequence of length T2 × D in raster-scan order and feed it to a transformer model. However, T2 × D is typically a large number, and the transformer would suffer from the quadratic cost of its self-attention mechanism.
AudioLM (Borsos et al., 2023b) adopts a threestage approach for modeling speech, as depicted in Figure 1(a). The first stage involves auto-regressive pre-training on semantic tokens to capture the longterm temporal structure. Next, the acoustic sequence, which is of size T2 ×D, is divided into a coarse part of size T2 ×D′ and a fine part of size T3× (D−D′), where D′ is typically much smaller than D − D′. The fine part is a small subset of the coarse sequence to reduce the sequence length since T2 × (D −D′) is still too large. AudioLM designs two individual transformers to model the coarse and fine acoustic sequences separately. The learning objective is approximately factorized as
p(S,A) = p(S)p(A|S)
≈ T1∏
t=1
p(st|s<t; θS) D′,T2∏
q1,t=1
p(aq1t |a≤D ′ <t , a <q1 t , S; θC)
D,T3∏
q2=D′+1,t=1
p(aq2t |a>D ′ <t , a <q2 t , a ≤D′ ≤T3 ; θF )
(2)
where q1 ≤ D′ < q2 ≤ D and T3 < T2. The fine acoustic transformer only models a small subset of the coarse acoustic tokens to reduce the sequence length. The learnable parameters θS , θC , θF correspond to three independent transformers respectively.
As shown in Figure 1(b), VALL-E (Wang et al., 2023a) uses phoneme sequences derived from text with a G2P tool as the condition, rather than semantic tokens. We slightly abuse the notation here since phonemes serve a similar purpose with semantic tokens. VALL-E also divides the acoustic token generation process into two stages, where the acoustic tokens from the first quantizer layer are generated in an auto-regressive manner while the subsequent acoustic tokens are generated non-autoregressively. Note that VALL-E can not generate semantically coherent sequences unconditionally since it does not model p(S). The learning objective is approximately factorized as
p(A|S) = D,T2∏
q,t=1
p(aqt |a≤D<t , a<qt , S) ≈
T2∏
t=1
p(a1t |a1<t, S; θAR) D,T2∏
q=2,t=1
p(aqt |a<q≤T2 , S; θNAR)
(3)
where θAR, θNAR refer to different models respectively.
The speech language models above are necessitated to split the acoustic generation into a multistage process due to the considerable length of acoustic sequences.

heading: 3.2 Efficient Hierarchical Transformer
text: Considering the hierarchical structure underlying acoustic sequence, we propose GPST, a hierarchical transformer architecture to effectively and efficiently learn the codes extracted by EnCodec. As shown in Figure 2, GPST is composed of (1) a semantic token extractor that integrates a speech SSL encoder and a K-means clustering model (Communication et al., 2023), as well as a neural codec
model EnCodec (Défossez et al., 2022), (2) a large global transformer that contextualizes representations by applying causal attention over previous semantic tokens and stacked acoustic tokens, and (3) a smaller local transformer that takes a contextualized hidden state from the global model, and auto-regressively predicts subsequent acoustic codes. We adopt the setting of a large global module with a small local module to simulate potential applications that use LLMs as the global module, which we leave for future work. The learning objective is exactly factorized as
p(S,A) = p(S)p(A|S) = T1∏
t=1
p(st|s<t; θglobal)
D,T2∏
q,t=1
p(aqt |a≤D<t , a<qt , S; θglobal, θlocal)
(4)
The end-to-end learning process is within one model in one stage, which mitigates the error propagation issues that can arise in a multi-stage formulation.
Global Transformer. The global transformer is an Ng layer decoder-only transformer with a causal mask. It has two types of tokens concatenated into a single sequence as input. The first type comprises semantic tokens, which can capture long-term consistency. The second type is the sum of the acoustic tokens obtained by RVQ
E(st) = Es(st) + PEg(t), for 1 ≤ t ≤ T1
E(at) =
D∑
q=1
Ea(a q t ) + PEg(t+ T1), for 1 ≤ t ≤ T2
ht = GlobalTransformer(s1, . . . , sT1 , a1, . . . , aT2), 1 ≤ t ≤ T1 + T2 (5)
where Es and Ea are embedding functions for semantic and acoustic tokens respectively. PEg is a positional embedding for the global transformer. We add special tokens at the first position and the segment boundary of the sequence to inform the model to switch the generation space.
Local Transformer. The local transformer consists of Nl layers. Given the contextualized hidden states ht from the global transformer, the local transformer auto-regressively predicts D acoustic codes a1t , . . . , a D t at position t
E(aqt ) = Ea(a q t ) + PEl(q), 1 ≤ q ≤ D at = LocalTransformer(ht, a1t , . . . , a D t )
(6)
where PEl is a positional embedding for the local transformer, which is shared across position t. GPST is trained to minimize the negative loglikelihood:
L = T1∑
t=1
− log p(st|s<t; θglobal)
− T2∑
t=1
D∑
q=1
log p(aqt |a<qt , S; θglobal, θlocal) (7)
Local-drop. The number of residual quantizers increases when generating Hi-Res speech, which would cause high computational complexity. We propose a technique named local-drop to improve the training efficiency of GPST further. Since the local transformer only models individual stacks of acoustic tokens, it has an input shape of (Batch × T2, D). The dimension of acoustic sequence length
T2 is unfolded to the first batch dimension, which means the stack of codes is not attended by selfattention. We randomly drop some tokens a≤Dt to decrease the size of the first dimension.

heading: 3.3 Inference
text: Speech language models can generate semantically coherent speech for unseen speakers with incontext learning, which is an emerging capability of auto-regressive pre-trained language models like GPT (Brown et al., 2020) for zero-shot learning. Suppose we have the semantic tokens Sp and the acoustic tokens Ap from the prompt, the semantic tokens St and the acoustic tokens At from the target. Based on the usage of the prompt, we can categorize the generation mode into four cases.
Unconditional Generation. In this setting, we unconditionally generate the semantic tokens, which are subsequently used as the prefix for acoustic generation. The randomly sampled semantic sequence can generate diverse, syntactically and semantically consistent linguistic content. The acoustic tokens vary in speaker identity with the semantic content serving as a guideline. We provide some transcription cases in Appendix A.1.
Semantic to Acoustic. In this setting, we use the ground-truth semantic tokens St as a condition for acoustic generation, which is similar to the task of TTS. The generated speech preserves the content of the spoken sentence while varying in speaker identity. We also follow SPEAR-TTS (Kharitonov et al., 2023) and train a toy decoder-only transformer named GPST-TTS on the LibriSpeech 960h
dataset to generate semantic tokens with text as a condition, supporting the TTS task.
Speaker Identity Transfer. In this setting, we are interested in the task of voice conversion that transfers the speaker identity of the prompt speech into the target speech. The sequence input to the model is concatenated in the following order [Sp, St, Ap]. GPST is encouraged to generate subsequent acoustic tokens that share the speaker identity with Ap while remaining consistent with the content of St. We find that directly concatenating linguistically inconsistent Sp and St together would cause unstable generation around the interface boundary. To address this issue, we propose artificially inserting a very short silence excerpt (0.1 second) between Sp and St to explicitly break the linguistic continuation. In this way, the model would not struggle to mitigate the discontinuity of Sp and St and can generate stable speeches.
Acoustic Continuations. Different from the speaker identity transfer mode, where the prompt and target are from different utterances, the prompt of the acoustic continuations mode is the first 3 seconds of the target. The model is asked to generate the acoustic continuation after 3 seconds.

heading: 3.4 Spoken Multilingual Learning
text: We adopt the multi-lingual XLSR encoder from SeamlessM4T (Communication et al., 2023) as the semantic token extractor. The semantic vocabulary of SeamlessM4T naturally supports the multilingual speech representation. For acoustic tokens, we adopt the pre-trained neural audio codec model EnCodec (Défossez et al., 2022) as the acoustic token extractor. Although EnCodec is trained on the English data, we find that it can synthesize other languages as well. We take it as the universal acoustic extractor.

heading: 3.5 Efficiency Analysis
text: Transformer (Vaswani et al., 2017) is criticized for the quadratic complexity with respect to sequence lengths during self-attention calculations. Considering an acoustic matrix A of size T2×D, the naive approach of unfolding it into a one-dimensional sequence like AudioLM would result in a computational complexity of O(NT 22D
2), where N is the number of transformer layers. In contrast, GPST has Ng global layers and Nl local layers, with the global transformer dealing with a sequence length of T2 and the local transformer with a sequence length of D. Suppose N = Ng +Nl for simplicity.
The overall computational complexity for GPST is O(NgT 22 + NlT2D
2), which is smaller than O(NT 22D
2). Furthermore, self-attention is not the primary computational cost factor in large transformers. The embedding size and the dimension of the feedforward network dominate the model’s overall computational cost (Kaplan et al., 2020). A forward pass with a large transformer with m nonembedding parameters on a sequence of length T2 uses roughly 2mT2 FLOPS. Therefore, for GPST with a global dimension mg and a local dimension ml, the required FLOPS is 2T2(mg +mlD). Since ml is typically much smaller than mg, the FLOPS for GPST is approximately 2T2mg, which is D times faster than the standard transformer with 2T2Dmg FLOPS.

heading: 4 Experiments

heading: 4.1 Experiment Setup

heading: 4.1.1 Datasets
text: We follow Borsos et al. (2023b) and use LibriLight (Kahn et al., 2020) as the training data which contains 60K hours of unlabelled speech in English. We randomly crop 10 seconds out of each audio clip for training. We choose LibriSpeech test-clean dataset (Panayotov et al., 2015) for evaluation since there is no speaker overlap with LibriLight. Following Borsos et al. (2023b); Wang et al. (2023a), we select samples with lengths between 4 and 10 seconds as the test dataset. For the multi-lingual task, we test in a bi-lingual setting with the tone language Chinese and non-tone language English for simplicity. We choose LibriSpeech 960h as the English training data and Aishell-2 1000h (Du et al., 2018) as the Chinese training data, both of which share similar sizes. We also test a larger bitrate setting for GPST-Hi-Res with 16 quantizers, which is not applicable to other baselines. All experiments are conducted three times and the average scores are reported. We describe the implementation details in Appendix A.2.

heading: 4.1.2 Baselines
text: We choose speech language models GSLM (Lakhotia et al., 2021), AudioLM (Borsos et al., 2023b) and VALL-E (Wang et al., 2023a) as baselines, together with YourTTS (Casanova et al., 2022) as the TTS baseline. We notice that SoundStorm (Borsos et al., 2023a) improves the multi-stage acoustic generation. However, SoundStorm takes duplicate semantic tokens as the condition, which is an inappro-
priate setting for other baselines since all the other models remove the consecutive repetitions, and duplicate semantic tokens would reduce the difficulty of acoustic generation. Also, duplicate semantic tokens would cause failures in the generation of semantic tokens (Lakhotia et al., 2021) that limits the application in speech generation (Kharitonov et al., 2023) and resynthesis for speech translation system (Lee et al., 2022a). Therefore, we do not take SoundStorm for comparison here.

heading: 4.1.3 Evaluation Metrics
text: The synthesized speech should align with the semantic input and match the voice of the prompt. Therefore, we are interested in the following metrics for speech language models: (1) word error rate (WER), (2) speaker similarity (SPK), and (3) speech quality (DNSMOS). We employ the HuBERT-Large (Hsu et al., 2021) model as the ASR model for English to calculate WER and Wav2Vec2-XLSR-53 (Baevski et al., 2020) for Chinese to calculate CER. We take the publicly available speaker verification model WavLMTDNN (Chen et al., 2022) to evaluate the speaker similarity between the prompt and the synthe-
sized speech. We use the MOS estimator DNSMOS (Reddy et al., 2021) to estimate the perceived audio quality of the generated samples. We compare DNSMOS with the examples provided on VALL-E’s demo page for fairness. The subjective evaluation MOS is not applicable due to other models are not open-sourced. Appendix A.3 lists all the evaluation tools.

heading: 4.2 Results and Analysis
text: LibriSpeech Evaluation. Table 1 and Table 2 summarize the results of different inference modes. Compared to the baseline models, GPST achieves the best results in terms of WER, SPK, and DNSMOS. In the semantic to acoustic mode, GPST
reaches the lowest WER score with only 33% parameters of AudioLM. The quality of semantic tokens is constrained due to the use of a toy model for text-to-semantic generation, resulting in a minor performance drop of GPST-TTS. We expect that more training data would further improve the TTS performance. We also notice a performance drop in GPST-Hi-Res, which indicates that Hi-Res speech generation, with more quantizers, is still a tough task. In the speaker identity transfer mode, GPST achieves the best scores in all the metrics, validating that GPST can better transfer the speaker identity while maintaining the spoken content. It is worth noting that GPST-Hi-Res gets better DNSMOS than GPST, largely because more quantizers can preserve more acoustic details.
Multilingual Evaluation. Table 3 shows the re-
sults of GPST on multi-lingual datasets. Although trained on a small dataset, GPST demonstrates its generalization ability in multi-lingual settings. Since the Aishell-2 Chinese dataset is noisy, the CER score is bad even for GroundTruth. However, GPST can still achieve a score close to the GroundTruth, which proves the robustness of the model. We also design a Zero-Shot Cross-Lingual Transfer for Multi-lingual settings. We adopt the model trained on English LibriLight only, while inference is conducted on Chinese Aishell-2 with Acoustic Continuation mode without any further training. GPST shows the performance close to the model especially trained on Chinese Aishell2, which demonstrates GSPT’s support for spoken multi-lingual tasks.
Effect of Model Architecture Settings. We conduct an ablation study on the number of layers for the global and local transformer. To match the parameters of every stage in AudioLM, which consists of a global transformer with 12 layers, we adjust the total parameters of the global transformer and local transformer in GSPT to be approximately equal. Since the parameters of one global transformer layer equals four local transformer layers, we adopt the setting of (12−x)×Ng+4×x×Nl. Table 4 shows that increasing the layer number of the local transformer helps GPST learn acoustic tokens better, further improving the performance of acoustic generation. However, the generation speed is slowed down a little bit as we increase local transformer layers.
On the Hi-Res Quality. We plot the melspectrograms of the same speech generated by GPST with 6kbps (8 quantizers) and 12kbps (16 quantizers) respectively in Figure 3. Generally,
richer harmonic energy in the high-frequency regions indicates higher speech quality. As observed, the speech generated by GPST with more quantizers exhibits more details in the mel-spectrogram.

heading: 5 Conclusion
text: We introduce GPST, a generative pre-trained speech language model that integrates semantic tokens and acoustic tokens within a hierarchical transformer architecture, allowing for a unified onestage generation process. GPST demonstrates its capability to generate coherent speech and speaker identity transfer with in-context learning. Furthermore, we show that GPST can generate Hi-Res speech and spoken multi-lingual speech as well.

heading: 6 Limitations
text: Our proposed GPST exhibits remarkable capabilities in the speech generation task, which is challenging for a single model. However, the GPST model cannot directly synthesize speeches with text input. Some models (Kharitonov et al., 2023) have proposed the techniques to enhance the text to semantic token generation model.

heading: 7 Ethics Statement
text: GPST may present new risks, such as the potential for malicious actors to impersonate public figures or commit fraud. To mitigate such risks, it is possible to watermark the generated speech that is invisible to humans but algorithmically detectable.

heading: Acknowledgements
text: This research was supported by the National Key Research and Development Program of China (Grant No. 2022YFB3103100), the National Natural Science Foundation of China (Grant No. 62276245).

heading: A Appendix
text: A.1 Unconditional Generation Cases We provide some transcription cases of unconditional generation in Table 5.
A.2 Implementation Details We leverage the XLSR v2 (Babu et al., 2022) model released by SeamlessM4T (Communication et al., 2023) to extract semantic tokens, resulting in a rate of 50 tokens per second. We remove the consecutive duplicate semantic tokens since such duplicates would cause generation failures (Lakhotia et al., 2021). We adopt the neural audio codec model EnCodec (Défossez et al., 2022) to extract acoustic tokens, which produce codes at 75 Hz. We choose 8 hierarchical quantizers as the default setting as VALL-E, leading to 75 × 8 = 600 tokens per second. We also test a larger bitrate setting for GPST-Hi-Res with 16 quantizers, which is not applicable to other baselines.
Each layer of the global transformer in GPST has 16 attention heads, an embedding size of 1024 with a feed-forward layer dimension of 4096. Each layer of the local transformer is smaller than the global transformer, with 8 attention heads, an embedding size of 512, and a feed-forward layer dimension of 2048. We set the probability of a local drop to 0.5 only for Hi-Res generation. The models are trained on LibriLight using 16 NVIDIA TESLA V100 32GB GPUs with a batch size of 64 for 1M steps, which takes about 1 week. The multi-lingual models are trained for 400K steps. We use the Adam optimizer with a learning rate of 0.0005 and an inverse square root learning rate decay schedule with 10K warm-up steps. To prevent over-fitting, we use label smoothing of 0.1 for training. All experiments are conducted using the FAIRSEQ toolkit (Ott et al., 2019).
A.3 Open-Sourced Tools ASR HuBERT-Large: https://github.com/ facebookresearch/fairseq/tree/main/ examples/hubert
ASR Wav2Vec2-Large-XLSR-53-Chinese-zhcn-gpt: https://huggingface.co/ydshieh/ wav2vec2-large-xlsr-53-chinese-zh-cn-gpt
Speaker verification WavLM-TDNN: https: //github.com/microsoft/UniSpeech/tree/ main/downstreams/speaker_verification
DNSMOS: https://github.com/microsoft/ DNS-Challenge/tree/master/DNSMOS
VALL-E demo page more samples: https://www.microsoft.com/en-us/ research/project/vall-e-x/vall-e
SeamlessM4T: https://github.com/ facebookresearch/seamless_communication
EnCodec: https://github.com/ facebookresearch/encodec

heading: Future Work
text: We adopt the setting of a large global module with a small local module to simulate potential applications that use LLMs as the global module, which we leave for future work. The learning objective is exactly factorized as
p(S,A) = p(S)p(A|S) = T1∏
t=1
p(st|s<t; θglobal)
D,T2∏
q,t=1
p(aqt |a≤D<t , a<qt , S; θglobal, θlocal)
(4)
The end-to-end learning process is within one model in one stage, which mitigates the error propagation issues that can arise in a multi-stage formulation. Global Transformer. The global transformer is an Ng layer decoder-only transformer with a causal mask. It has two types of tokens concatenated into a single sequence as input. The first type comprises semantic tokens, which can capture long-term consistency. The second type is the sum of the acoustic tokens obtained by RVQ
E(st) = Es(st) + PEg(t), for 1 ≤ t ≤ T1
E(at) =
D∑
q=1
Ea(a q t ) + PEg(t+ T1), for 1 ≤ t ≤ T2
ht = GlobalTransformer(s1,. , sT1 , a1,. , aT2), 1 ≤ t ≤ T1 + T2 (5)
where Es and Ea are embedding functions for semantic and acoustic tokens respectively. PEg is a positional embedding for the global transformer. We add special tokens at the first position and the segment boundary of the sequence to inform the model to switch the generation space. Local Transformer. The local transformer consists of Nl layers. Given the contextualized hidden states ht from the global transformer, the local transformer auto-regressively predicts D acoustic codes a1t ,. , a D t at position t
E(aqt ) = Ea(a q t ) + PEl(q), 1 ≤ q ≤ D at = LocalTransformer(ht, a1t ,. , a D t )
(6)
where PEl is a positional embedding for the local transformer, which is shared across position t. GPST is trained to minimize the negative loglikelihood:
L = T1∑
t=1
− log p(st|s<t; θglobal)
− T2∑
t=1
D∑
q=1
log p(aqt |a<qt , S; θglobal, θlocal) (7)
Local-drop. The number of residual quantizers increases when generating Hi-Res speech, which would cause high computational complexity. We propose a technique named local-drop to improve the training efficiency of GPST further. Since the local transformer only models individual stacks of acoustic tokens, it has an input shape of (Batch × T2, D). The dimension of acoustic sequence length
T2 is unfolded to the first batch dimension, which means the stack of codes is not attended by selfattention. We randomly drop some tokens a≤Dt to decrease the size of the first dimension