SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models
Authors: Xin Zhang
Dong Zhang
Shimin Li
Yaqian Zhou
Xipeng Qiu
Venue: ICLR 2024 poster (2024)
Keywords: speech
audio
multi-modal
large language model
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Current speech large language models build upon discrete speech representations,
which can be categorized into semantic tokens and acoustic tokens. However,
existing speech tokens are not specifically designed for speech language modeling. To assess the suitability of speech tokens for building speech language
models, we established the first benchmark, SLMTokBench. Our results indicate
that neither semantic nor acoustic tokens are ideal for this purpose. Therefore, we
propose SpeechTokenizer, a unified speech tokenizer for speech large language
models. SpeechTokenizer adopts the Encoder-Decoder architecture with residual
vector quantization (RVQ). Unifying semantic and acoustic tokens, SpeechTokenizer disentangles different aspects of speech information hierarchically across
different RVQ layers. Furthermore, We construct a Unified Speech Language
Model (USLM) leveraging SpeechTokenizer. Experiments show that SpeechTokenizer performs comparably to EnCodec in speech reconstruction and demonstrates
strong performance on the SLMTokBench benchmark. Also, USLM outperforms
VALL-E in zero-shot Text-to-Speech tasks. Code and models are available at
https://github.com/ZhangXInFD/SpeechTokenizer/.

Published conference paper at ICLR 2024
SPEECHTOKENIZER: UNIFIED SPEECH TOKENIZER FOR SPEECH LANGUAGE MODELS
Xin Zhang; Dong Zhang', Shimin Li, Yaqian Zhou! Xipeng Qiu
School Computer Science, Fudan University Shanghai Key Laboratory of Intelligent Information Processing, Fudan University Llin_zhang22 dongzhang22 fudan. edu. cn [sml ammli20, zhouyaqian, xpqiu) wfddanneeee edu.cn
http:://oouuttt......hii///////r io/
ABSTRACT
Current speech large language models build upon discrete speech representations, which can be categorized into semantic tokens and acoustic tokens. However, existing speech tokens are not specifically designed for speech language mod- eling. To assess the suitability of speech tokens for building speech language models, we established the first benchmark, SLMTokBench. Our results indicate that neither semantic nor acoustic tokens are ideal for this purpose. Therefore, we propose SpeechTokenizer, Tokenizer, unified speech tokenizer for speech large language models. SpeechTokeniee adopts the Encoder-Decoder architecture with residual vector quantization (RVQ). Unifying semantic and acoustic tokens, SoeechTok- enizer disentangles different aspects of speech information hierarchically across different RVQ layers. Furthermore, We construct Unified Speech Language Model (USLM) leveraging SpeechTokenize enizer performs comparably to EnCodec in speech reconstruction and demonstrates strong performance the SLMTToBBench benchmark. Also, USLM outperforms VALL-E in zero-shot Text-to-Speech tasks. Code and models are available at attps://gite com httpa//wwww.....aaaaaaaaeeeeeeea
Experiments show that Speech
INTRODUCTION
Large language models (OpenAl 2023; Touvron et al. 2023) have demonstrated remarkable per- formance on various natural language processing tasks. This has inspired numerous works build speech language models (Borsos et al. 2022), which have achieved significant breakthroughs across various speech processing tasks (Wang al. [023] Zhang et al., 2023] Rubenstein et al., [2023] Dong et al. 2023). A key commonality among these works is the utilization of discrete speech representations. Current discrete speech representations can be categorized into two types: semantic tokens and acoustic tokens (Borsos et al. 2022). Semantic tokens are typically from self-supervised pre-trained models with masked language modeling as training objective (Hsu et al.) 2021 Bacvski ct al. 2020 Chung al. 2021). Derived through k-means clustering representations from specific intermediate layer, semantic tokens are depicted as sequences with die-dimensional structure. Acoustic tokens be extracted from neural audio codecs with reconstruction training objec- tive (Zeghidour et al. 2021 Défossez et al.] (022). Utilizing residual vector quantization (RVQ) with hierarchical quantizers for discretization, acoustic tokens represented matrices consisting of two dimensions: timesteps and quantizers.
Building upon two speech tokens, there exist three modeling approaches for speech language models, as listed in Table i) Semantic language models are constructed using semantic tokens and employ
an external unit vocoder for speech synthesis. (Lakhotia al. 2021 Zhang al., 2023 Hassid
et al. 2023). While capturing semantically accurate content, their speech generation results in poor
"Equal contribution. Order is random
1 "ooresponniig author

---

Published conference paper at ICLR 2024
Acoustic tokens
Semantic tokens
Unified tokens
Decoder
NAR
concat
SoundStream Encodec
HuBERT W2VBERT
SpeechTokenizer
AR
(b) Speech Language
Figure 1: Left: Illustration of information composition of different discrete speech representations. Right: Illustration of unified speech language models. AR refers to autoregressive and NAR refers to non-autoregressive. Speech tokens represented colored circles and different colors represent different information.
Accurate Content
High-quality Speech
Single Tokenzier
Semantic LM Acoustic LM
Hierarchical LM USLM (ours)
Table 1: Comparision between different speech language models. Semantic LM refers to semantic language models. Acoustic LM refers to acoustic language models. Hierarchical LM refers to hierarchical speech language models. USLM refers to unified speech language model.
quality and a loss acoustic details. ii) Acoustic language models are built on acoustic tokens. Taking VALL-E (Wang et al., 2023) speech (TTS) capabilities, it still suffers from problems like inaccurate content, due to the complex information within acoustic tokens. iii) Mierarchical speech language models comprise semantic token language models and acoustic token language models, which capture content information and acoustic details respectively (Borsos et al.) 2022] Rubenstein et al., [2023] Dong et al. 2023). This structure shows promising results in both content and speech quality, but the multi- modeling approach more complex, leading several drawbacks such as error accumulation and slower processing speed. Additionally, there is significant information redundancy between semantic tokens and acoustic tokens, which introduces unnecessary modeling complexities. An ideal speech language model should not only accurately model content, but also generating diverse, high-quality speech, while maintaining an architecture of elegant simplicity. Correspondingly, ideal speech tokens should meet the following two key characteristics: Strong alignment with text; ii) Effective preservation of speech information.
example, despite achieving impressive zero-shot text-to-
However, existing speech tokens not explicitly designed for speech language modeling, and there has been no exploration into their suitability for building speech language models. To address this gap, build Speech Language Model Token Benchmark, assess the suitability of speech tokens for constructing speech language models. Our evaluation reveals that semantic tokens exhibit high alignment with text while losing some information in speech, such as timbre. Acoustic tokens excel in preserving speech information effectively but do not demonstrate a strong alignment with text. With these observations, we aim to build specialized speech tokens designed for speech language models by unifying semantic and acoustic tokens. Specifically, we can conduct information disentanglement in the RVQ structure of acoustic tokens, enabling the first RVQ quantizer to generate tokens containing content information, similar semantic tokens, while the subsequent quantizers complement the remaining paralinguistic information, as illustrated in Figure
With the above motivation, propose SpeechTokenizer unified speech tokenizer for speech large language models. SpeechTokenize adopts the Encoder-Decoe quantization. Unifying semantic and acoustic tokens, SpeechTokenize disentangles different aspects of speech information hicrarchically across different RVQ layers. By employing a semantic teacher to guide the first RVQ quantizer, the first layer tokens can effectively capture content information. With residual structure, the subsequent quantizers complement the remaining paralinguistic information.
architecture with residual vector
2

---

Published conference paper at ICLR 2024
Building upon SpeechTokenizer, we build Unified Speech Language Model consisting of autore- gressive and non-autoregressive models. Experimental results show that SpeechTokenizer performs comparably to EnCodec (Défossez et al. 20222 in speech reconstruction and demonstrates strong performance on the Senhh benchmark. The USLM notably outperforms VALL-E (Wang
et al. 2023 in zero-shot Text-to- tottttt--phhhhhp (TTS) tasks.
Our contributions include the following:
We propose SpecchTokenizer, which is specially designed for speech large language models and unify the semantic and acoustic tokens through disentangling different aspects of speech information hierarchically.
We establish SL.MTokBench, the first benckmark assess the suitability of speech tokens for constructing speech language models.
We construct unified speech language model based on Speech which outperforms VALL-E on zero-shot TTS task.
SLMTOKBENCH: SPEECH LANGUAGE MODEL TOKEN BENCHMARK
To build powerful speech language models, discrete speech representations should possess the fol- lowing two key characteristics: Strong alignment with text; ii) Effective preservation of speech information. Building on this premise, we establish speech Language Model Token Benchmark (SLM- TokBench) to assess the suitability of speech tokens for constructing speech language models.
2.1 TEXT ALIGNMENT EVALUATION
We evaluate the degree of text alignment by estimating the mutual information between speech tokens and text. For notation, X denotes discrete speech representations; Y denotes text; I(X: Y) denotes the mutual information; test dataset is denoted as 'D = {( yı) }:)} and denotes the downstream model. Through the derivation in Appendix A
estimate I(X;Yo Y) as:
N
1(x: Y)
Σ
Dog go ()s(ax))) - log go [v,l
N
i=
where (o(Y) (Y|)) is the variational distribution and can be parameterized by the downstream model 6.
The downstream model a vanilla 2-layer 1024-unit BLSTM optimized by CTC loss on characters and it takes speech tokens as inputs. Specifically, for each discrete representation, we first establish an embedding matrix, which can be either randomly initialized derived from the k-means centroid matrix vector quantization codebooks obtained during the discretization process. We use the embedding matrix to embed the discrete representations and obtain continuous representations, which then fed into the downstream models. We train the downstream model on LibriSpeech train-clean- 100 subset and use dev-clean subset for estimating mutual information. We also calculate the word error rate (WER) the test set. For downstream model training, configure the training setup with a batch size of 32, a learning rate of le-4, and a total of 200k global steps.
2.2 INFORMATION PRESERVATION EVALUATION
To evaluate the preservation of speech information in discrete speech representations, we convert speech tokens back to speech and evaluate resynthesized speech by automatic metrics on content and timbre. We train unit-HiFIGAN (Polyak et al. (202T) on LibriSpeech dataset to convert HuBERT units to waveform. Notably, to avoid interference from additional information, we don't 't supply any speaker information during training, For Encodec tokens, we used the Encodec decoder to directly produce the waveform. Content preservation is evaluated by computing the WER through transcribing the resynthesized speech using the Whisper en-medium model (Radford et al. 2023). Timbre preservation is evaluated by utilizing WayLM-TDNY (Chen et al., 2022) to calculate speaker similarity between the synthesized and groundtruth speech. We randomly sample 300 speech samples from LibriSpeech test set for evaluation.
3

---

Published conference paper at ICLR 2024
Discriminator
- -ir-- ---
-f 1f-1
Quantizer
Encoder
Decoder
S0A
Stucken
Semantic Distillaion
Teaeher
HuBERT
module for trining only cotatfou training ony
Figure 2: Illustration of SpeechTokenize Tokenizer framework.
2.3 COMPARING SEMANTIC & ACOUSTIC TOKENS
We HuBERT 1.9 units to represent semantic tokens and EnCodec codes represent acoustic tokens. As shown in Table semantic tokens achieve high mutual information with text but their resynthesized speech has low speaker similarity. Acoustic tokens achieve low WER and high speaker similarity for resynthesized speech but have low mutual information with text.
SPEECHTOKENIZER
3.1 MODEL. STRUCTURE
Our model is built on the framework of RVQ-GANs, following the same pattern Sound- StreamZZeghidour al., 202T) and EnCodec(Défossez et al. 2022). As depicted in Figure]] our model uses the convolutional-basel encoder-decoder network from EnCodec, which performs tem- poral downscaling with a chosen striding factor. Notably, we have substituted the two-layer LSTM, originally following the convolution blocks in the EnCodec encoder, with a two-layer BiLSTM to augment the semantic modeling ability. We conduct ablation studies model structure in Ap- pendix B We quantize the encoder outputs using Residual Vector Quantization (RVQ), method that can operate quantizes residuals following an initial quantization steps with distinct codebook. Further details of model structure can be found in Appendix During training, semantic teacher provides semantic representation to guide the residual quantization process.
3.2 SEMANTIC DISTILLATION
To achieve a hierarchical modeling of diverse information across different RVQ layers, we employ semantic guidance for the first quantizer, enabling it to capture content information. Leveraging a residual structure enables the subsequent quantizers to complement the remaining paralinguistic information.
We employ HuBERT (Hsu et al. 2021) as our semantic teacher in this study, as HuBERT is demon- strated to encompass substantial content information (Mohamed et al. 2022). We introduce two types of distillation: continuous representation distillation and labed prediction.
For continuous representation distillation, employ the 9th layer HuBERT representation the average representation across all HuBERT layers as semantic teachers. The training objective is to maximize the cosine similarity at the dimension level across all timesteps between the outputs of RVQ first layer and semantic teacher representations. Formally, the continuous distillation loss is

---

Published conference paper at ICLR 2024
defined as:
1 D
Σ log a (co) (())) d=1
Edistill
where QI and denote the quantized output of RVQ first layer and semantic teacher representation respectively. A denotes the projection matrix and D is the dimension of semantic teacher representa- tion. The superscript (:, d) signifies vector comprising values from all timesteps at dimension d. cos(-) represents cosine similarity and o-)) denotes sigmoid activation. This continuous distillation loss function deviates from the commonly employed approach, which calculates the loss based on the representations output by the student and teacher models the same timestep. comparative analysis of these two methodologies is provided in Appendix
For pseudo-label prediction, we adopt HuBERT units as the target label. The training objective is constructed as:
Cdistül Σ u² (og(SoftmaxAqq])) 1=1
where qf and uı' respectively denote the quantized output of the first VQ layer and the HuBERT unit timestep t. T denotes the number of time steps and A the projection matrix.
3.3 TRAINING OBJECTIVE
Our training approach includes both a reconstruction task and a semantic distillation task. In the reconstruction task, we employ GAN objective, optimizing combination of reconstruction term, discriminative loss term, and RVQ commitment loss. In the semantic distillation task, the training objective involves semantic distillation loss term. In the following, x represents an speech signal and & denotes the reconstructed signal by the network.
Reconstruction Loss The reconstruction loss comprises a time and a frequency domain loss. For time domain, minimize the L1 distance between and ê. ie. L, |xx x||1. For frequency domain, we linearly combine the L1 and L.2 losses over the mel-spectrogram using several time scales. Formally, L
(x) [[({(x)) (2)))]l,, where S, is 64-bins mel-spectrogram using a normalized STFT with window size 2' and hop length of 2' 2//4, 5,... 11 is the set of scales.
Discriminative Loss We use the same dicriminators as HiFi-Codes Yang et al. (023) that consist of three discriminators: A multi-scale STFT-based (MS-STFT) discriminator; multi-pe mulli-period discrimina- tor (MPD) and a multi-scale discriminator (MSD). Further details of discriminators can be found in Appendix D The adversarial loss is used to promote perceptual quality and it is defined as hinge loss over the logits of the discriminator, averaged over multiple discriminators and over time. Let K denote the number of discriminators, the adversarial loss for the generator Lp is constructed as follows, bg
k==1 max(1 Dx (x), 0). For the discriminators Lg is defined
CD
max Dx(x), 0) + max(1 + D2(2).0),
k=1
Additionally, feature matching loss for the generator is computed follow:
1 K
Deeat
KL
me
where the mean computed over all dimensions and the number of layers in discriminators.
RVQ Commitment Loss We add commitment loss L., between the pre-quantized value, and its quantized value, without gradient computed for the quantized value. RVQ commitment loss is defined as: L.p = "n i2; - zg - where z; and Zn denote current residual and nearest entry in the corresponding codebook respectively.
Generally, the generator is trained to optimize the following loss:
La XeL, + Xy + >yLb, + Afeathfeat Xun + Ndistall Ldistilly
where de, Ay, Ag, Afeat, Au and Adistall are hyper-paramen parameters used to balance each loss term.

---

Published conference paper at ICLR 2024
3.4 UNIFIED SPEECH LANGUAGE MODEL
As shown in Figure 1 we can build unified speech language model upon SpeechTokenizer. Con- sisting of autoregressive and non-autoregressive models, it can hierarchically model information in speech. The autoregressive (AR) model captures the content information by modeling tokens from the first RVQ quantizer. The non autorrgresssev (NAR) model complements paralinguistic information for the AR model by generating tokens from the subsequent quantizers conditioned on the first-layer tokens. We validate the effectiveness of unified speech language model on zero-shot TTS task.
The AR model built upon the first-layer tokens C1. Utilizing transformer decoder-only architecture 0 AR we approach this conversion as a casual language modeling task with the phoneme sequence serving as the prompt for the AR model. The training objective can formulated as
Can log II p(c OAR). t=
The NAR model produces tokens C2.8 from the subsequent quantizers. Its architecture resembles that of the AR model, comprising eight distinct acoustic embedding layers and output prediction layers. To control the characteristics of the speaker's voice, acoustic prompt Č is employed for timbre guidance. The model is conditioned phoneme sequence u, acoustic prompts C and tokens from previous quantizers, leading to the formulation of the training objective as follows
ANAR log II p(c C, C,
During inference, we convert text input to phoneme sequence and speech prompt to speech tokens. They concatenated to form the prompts for AR and NAR models. Conditioned on that, the AR model generates first-level tokens, while the NAR model iteratively produces tokens of subsequent levels. The tokens generated by the AR and NAR models are then concatenated to construct the speech token matrix. Finally, we use the SpeechTokenizer decoder to generate the waveform conditioned on the complete token matrix.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Datasets For Tokenizer training, use LibriSpeech (Panayotov et al. 2015) dataset. We randomly crop 3-second segment from the speech samples at each training iteration. For zero-shot TTS, we train AR and NAR models on the English subset of Multilingual LibriSpeech (Pratap et al. 2020) dataset, which contains 44K hours of transcribed speech data derived from LibriVor audiobooks. We select speech samples with durations ranging from 3 to seconds for training data. The sampling rate is 16KHz for all speech data.
we introduce the details about model structure section 3.1 and Ap- pendix D For zero-shot TTS experiments, AR model and NAR model are both 12-layer Transformer decoders with 16 attention heads, an attention dimension of 1024 and the FFN dimension of 4096.
Model For Speech
Training For Speech the model are trained learning rate of 4e-4 and batch size of 20 per GPU, For Unified Speech Language Model, both AR and NAR models The AR model is trained with batch size 7500 tokens per GPU, and the NAR model is trained with batch size of 5000 tokens per GPU.
A800 GPUS for 20 epochs with maximum
trained on 8 A800 GPUS for 500k steps with maximum learning rate of 5e-4.
Baselines We adopt EnCodec 24khz 24khz_6kpbs (hereinafter referred to as EnCodec) (Défossez et al. 2022) as the baseline for TokechTo and VALL-E (Wang al. 2023) the baseline system for zero-shot TTS. We train VALL-E under the same dataset and experimental setups as EnCodec.

---

Published conference paper at ICLR 2024
Objective WER,
Subjective MUSHRA
Tokenizer
VISQOL
Groundtruth
4.58
91.46
EnCodec
5.11
4.37
79.86
SpeechTokenizer
5.04
4.30
90.55
Table 2: Results of speech reconstruction
4.2 SPEECH RECONSTRUCTION EVALUATION
We randomly sample 300 speech samples from LibriSpeech test set for speech reconstruction evaluation. We take into account both subjective and objective evaluation metrics.
Objective Metries We use ViSQOL metries (Hines et al. 2012) to measure the speech quality. Additionally, we evaluate content accuracy through Word Error Rate (WER) by transcribing the speech utilizing the Whisper en-medium model (Radford et al. 2023).
Subjective Metrics We adopt a crowd-sourced methodology inpspired by MUSHRA protocol (Series) 2014), with hidden reference but no lowerpass- septtreed anchor, for subjective evaluation. We instruct evaluators to rate the perceptual quality of the given samples on a scale of I to 100.
4.3 UNIFIED SPEECH LANGUAGE MODEL EVALUATION
We conduct zero-shot TTS evaluation the VCTK dataset, which comprises 108 speakers. There no speaker overlap between the training data and VCTK dataset. For each speaker, we randomly selected 3s utterance as the prompts while the textual content of different utterance is used the input text.
Objective Metrics We evaluate the TTS systems with speaker similarity and WER. We evaluate the speaker similarity between the generated speech and the prompt speech. We calculate the similarity with the following steps: 1) utilize WavLMMTDN to calculate the speaker embedding for the generated speech and the prompt speech. 2) we calculate the cosine similarity between the normalized embeddings. We employ Whisper medium model transcribe the generated speech and calculate the WER.
Subjective Metries We determine the Mean Opinion Score (MOS) and Similarity Mean Opinion Score (SMOS) through human evaluations. MOS reflects the naturalness of speech, while SMOS assesses the degree of similarity to the original speaker's 's voice. We engaged 12 and native speakers as contributors for MOS and SMOS evaluations, respectively. MOS and SMOS both span from 1 to 5, with higher values signifying greater speech quality and voice similarity respectively.
4.4
MAIN RESULTS
Speech Reconstruction Table summarizes the results of speech reconstruction experiments. The SpkeceTokenizer achienves lower WER than Encodec, demonstrating its superior ability preserve content. Additionally, SpeechTokenizer attains comparable VISQOL score but higher MUSHRA score than EnCodec, which indicates its stronger capability in generating high-quality speech.
Performance on TokkhhBench Table 3 displays the performance of Speech Tokenizer on SLM- TokBench. Compared with EnCodec-RV RVQ-1, BhkeechTokenizer-B.
achieves higher mutual information between text and lower WER of downstream model. This suggests that SpeechTokent exhibits a stronger alignment with textual content. Meanwhile, the of resynthesized speech of Speech- Tokenizer RVQ-1 tokens achieves lower WER and speaker similarity, indicating its capability to retain more content-related information while disregarding timbre characteristies, similar to semantic tokens. The resynthesized speech of SpeechTokenize 1:8 tokens demonstrates low WER and high speaker similarity, illustrating SpeechTokenizer's competence in preserving comprehensive speech information, similar to acoustic tokens. Furthermore, the speaker similarity of resynthesized speech of SpeechTokemizer RVQ-1 tokens notably low, whereas that of SpeechTokenizer RVQ-1:8 tokens is considerably high. This observation implies that the tokens from subsequent layers compensate for the timbre information that is discarded by the first layer tokens.
7

---

Published conference paper at ICLR 2024
Text Alignment
Information Preservation
Tokenizer
Teacher
MIf
WER J
WER
SIMI
Groundtruth
4.58
1.0
HuBERT
KM500
31.2
9.88
16.26
0.77
EnCodec
RVQ-1
16.5
61.52
38.34
0.92
EnCodec
RVQ-1:8
23.6
30.91
5.11
0.98
Ablations
SpenchTokenizerr
RVQ-1
HuBERT avg
30.9
15.58
9.57
0.74
SpeechTokenizer
RVQ-1:8
HuBERT avg
29.7
16.03
5.04
0.97
Sokezzeo
RVQ-1
HuBERT L.9
32.9
12.68
14.17
0.73
SpenchTokenizer
RVQ-1:8
HuBERT L9
31.6
13.12
5.31
0.97
SokeceTo
RVQ-1
HuBERT units
24.2
34.13
20.02
0.72
SpeechTokenizer
RVQ-1:8
HuBERT units
25.1
30.71
5.84
0.95
Table 3: Results on SLMTokBench. MI and WER refer to mutual information and word error rate of the downstream model. WER and SIM refer word error rate and speaker similarity of resynthesized speech respectively. RVQ-n denotes the tokens of the n RVQ layer. RVQ-n:m denotes the tokens from the th layer the mh layer.
Objective WER
Subjective SMOS
Model
Tokenizer
STM
MOSt
Groundtruth
1.9
0.93
4.5
3.96
VALL-E
EnCodee
7.9
0.75
3.08
3.31
USLM
Speech Tokenizer
6.5
0.84
3.63
3.45
Table 4: Results of zero-shot TTS
Zero-shot TTS As shown Table 4 our USLM demonstrates lower WER than VALL-E. This result highlights that SpkechTokenizer can contribute to more precise modeling of content informa- tion. Additionally, the USLM demonstrates superior speaker similarity, implying that decoupled information structure is more conducive to modeling speaker-related information.
ANALYSIS
5.1 CHOICES OF SEMANTIC TEACHERS
As shown in Table semantic teachers, HuBERT L9 representations perform better than HuBERT units in both Text Alignment and Information Preservation, regardless of whether it's RVQ-1 or RVQ-1:8. The reason may be that discrete HuBERT units lose some content information compared to the continuous representations, thereby providing weaker semantic guidance to SpeechTokenizer.
When comparing HuBERT 1.9 representations with HuBERT average representations, we find that in terms of Text Alignment, the mutual information is higher when HuBERT 1.9 representations serve the teacher. This because HuBERT average representations contain some timbre information, while HuBERT L9 offers purer content information. On the other hand, HuBERT average shows better performance Information Preservation, reflected in lower WER. We speculate that this is due to a certain level of task conflict between semantic distillation and reconstruction, where the former aims to retain only content information while the later aims to preserve various aspects of speech. The presence of some timbre information in HuBERT average representations could to some extent alleviate this task conflict.
5.2 EFFECTIVENESS OF INFORMATION DISENTANGLEMER
To demonstrate that different speech information can be hierarchically modeled in SpeechToeenizer, conduct one-shot voice conversion (VC) experiment. This experiment aims to convert speech from any source speaker to an arbitrary target speaker using only few seconds of reference speech from the target speaker. To Tokezzzo for one-shot VC, the first step is transform the source speech and reference speech into token matrices. By concatenating the RVQ-1 tokens

---

Published conference paper at ICLR 2024
Source Reference
WERL
SIM
Groundtruth
0,4
0.93
RVQ-1
RVQ-2
2.6
0.72
RVQ-1
RVQ-2:4
11.7
0.80
RVQ-1
RVQ-2:8
35.4
0.82
Table 5: Results of one-shot voice conversion. Source and Reference refers to source token matrix and reference token matrix respectively.
DUM 2014
05 PAC--
RiD
Figure 3: Visualization of quantized output of different RVQ layers of SpeechTokenizer. The first layer denoted as RVQ-1, while the sum of the second layer to the eighth layer is denoted as RVQ-2:8.
of source token matrix with RVQ-2:8 tokens of the reference token matrix, and then passing this combined token matrix the decoder, we can achieve voice conversion. The lengths of the reference and source tokens may not align perfectly. To address this, we use truncation or circular padding ensure they share the same temporal length, thereby facilitating the concatenation process. We conduct experiments serve as the source speech. From the remaining 107 speakers, we individually selected one speech sample of different content to act the reference speech. We employed two metrics for evaluation: WER and speaker similarity.
VCTK dataset. We randomly selected one speech sample from a speaker to
Table reports the results of one-shot VC experiments. From the table, can that as the number of layers for reference tokens increases, speaker similarity also gradually increases. This suggests that more information from the reference speaker is being transferred over, proving that speaker information embedded in tokens from the second to the last layers. When the reference tokens are selected from the second the fourth layers, we achieve low WER and high speaker similarity, resulting in a satisfactory one-shot VC performance. This indicates that the information disentanglement successful.
We also visualize quantized outputs from different layers in FigureB Specifically, We randomly select five speakers from the VCTK dataset and pick 10 random speech samples per speaker. We extract quantized output of different RVQ layers of SpeechTokenizer. The first layer output is denoted as RVQ-1 representations, while the sum of the outputs from the second layer to the eighth layer is denoted as RVQ-2:8 representations. By performing mean pooling along the temporal dimension, each representation is converted into single vector. These vectors are then visualized in a 2D space using E-SNE, with speech samples from the same speaker represented in the same color. From the plot, it can be observed that the RVQ-1 representations for different speakers are scattered randomly without discernible pattern. In contrast, the RVQ-2:8 representations for the same speaker tend to cluster together, while being distinct from those of other speakers. This suggests that speaker-specinee information is contained from the second layer up the eighth layer.
6 RELATED WORK
Oure related work is put in Appendix

---

Published conference paper at ICLR 2024
7 CONCLUSION
In this study, we present SLMTo,Bench,, which assess the effects of various speech token kinds. Meanwhile, propose SpeechTokenizer, unify the discretization of both types of speech tokens to overcome the issue of employing several models to extract semantic and acoustic discrete tokens separately. Furthermore, We developed unified speech language model (USLM) based on Speech- Tokenizer, with better results regarding the generated speech's 's content accuracy and quality, The study unified speech tokenizer an essential part of the further development speech language model in terms of efficiency and quality.
REFERENCES
Ba, J. L.,, Kiros, R., and Hinton, G. E. Layer normalization, 2016.
Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self- learning of speech representations. Advances Neural Information Processing Systems, 33: 12449-12466 2020.
Bengio, Y., Léonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.33 3433..88..42222 2013.
Borsos, Z. Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Teboul, O., Grangier, D., Tagliasacchi, M., and Zeghidour, N. Audiolm: language modeling approach audio generation, 2022.
Casanova, E., Weber, J., Shulby, C. D., Junior, A. C., Gölge, E., and Ponti, M. A. YouTTTS: Towards zero-shot multi-speaker TTS and zero-shot voice conversion for everyone. In Chaudhuri, K., Jegelka, S., Song, L.,, Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2709-2720, PMLR, 17-23 Jul 2022. URL httpps://prcceedinggg html
mlr. www.ppesewv//eeec. wttes////11l//ccaavv2222.222
Chang. H.-J., Yang. S.-w., and Lee, H.-y. Distilhubert: Speech representation learning by layer-wise distillation hidden-unit bert. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7087-7091. IEFF, 2022.
Chen, S., Wang, C.,, Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y., Qian, Y., Wu, J., Zeng, M., Yu, X., and Wei, F. WavLM: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 1606))1505-5518. 2022. doi: 10. 10.1109//stsp22222.38883
URL
https: / do: orq 111.00011111FF1111111 3tt 3188113
Cheng, P., Hao, W., Dai, S. Liu, J., Gan, Z., and Carin, Club: A contrastive log-ratio upper bound of mutual information, 2020.
Chung, Y.A.,, Zhang, Y., Han, W., Chiu, C.-C., Qin, J., Pang, R., and Wu, Y. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training, 2021.
Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (elus), 2016.
Dong, Q., Huang, Z., Tian, Q., Xu, C., Ko, T., Zhao, Y., Feng, S., Li, T., Wang, K., Cheng, X., Yue, F., Bai, Y., Chen, X., Lu, L.,, Ma, Z., Wang. Y., Wang, M., and Wang, Y. Polyvoice: Language models for speech speech translation, 2023,
Défosse7, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression, 2022.
Hassid, M., Remez, T., Nguyen, T. A., Gat, I., Conneau, A., Kreuk, F., Copet, J., Defossez, A., Synnaeve, G., Dupoux, E., Schwartz, R., and Adi, Y. Textually pretrained speech language models, 2023.
10

---

Published conference paper at ICLR 2024
Hines, A., Skoglund, J., Kokaram, A., and Harte, N. Visgol: The virtual speech quality objective listener. In /WAENC 2012; International Workshop on Acoustic Signal Enhancement, pp. 1-4, 2012.
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Kharitonov, E., Vincent, D., Borsos, Z., Marinier, R., Girgin, S., Pietquin, O., Sharifi, M., Tagliasacchi. M., and Zeghidour, N. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision, 2023.
Kong, J., Kim, J., and Bae, J. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems,
2020.
Lakhotia, K., Kharitonov, E., Hsu, W.-N., Adi, Y., Polyak, A., Bolte, B., Nguyen, T.-A., Copet, J. Baevski, A., Mohamed, A., et al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguisties, 9:1336-1354. 2021.
Liu, H., Wang. T., Fu, R., Yi, J., Wen, 7., and Tao, J. Unifyspeech: A unified framework for zero-shot texe-to-speech and voice conversion, 2023.
Mohamed, A., yi Lee, H., Borgholt, L.., Havtorn, D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaloe, L.., Sainath, T. N., and Watanabe, S. Self-supervised speech representation learning: A review. JEEE Journal of Selected Topics in Signal Processing, 16(6)1119--1210. oct 2022. doi: 10.1100110000 1099/sssp.2022.2277055 URL httph::t//////tt //doi: tttp 2022. 3207050
OpenAl. Gpt-4 technical report, 2023.
Panayotov, V., Chen, G., Povey. D., and Khudanpur, S. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference Acousties. Speech and Signal Processing (ICASSP), pp. 5206-5210, 2015. doi: 10. 10.1009/CASS9:22555777988
Polyak, A., Adi, Y., Copet, J., Kharitonov, E., Lakhotia. K., Hsu, W.-N., Mohamed, A., and Dupoux, E. Speech resynthesis from discrete disentangled self-supervised representations, 2021.
Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., and Collobert, MI.: A large-scale multilingual dataset for speech research. In Interspeech 2020. ISCA, oct 2020. doi: 10.214337mnlespeecch. 2020-2826. URL /tt/lttttt///a// tttp:///.........eeeeeeeeeeeeee. 2ct.eeeehh.cc00-0-00
Qian, K., Zhang, Y., Chang, S., Yang, X., and Hasegawa-Johnson, M. Autove: Zero-shot voice style transfer with only autoencoder loss, 2019.
Qian, K., Zhang, Y., Chang, S., Hasegawa-Johnson M., and Cox, D. Unsupervised speech decompo- sition via triple information bottleneck. In International Conference on Machine Learning, pp. 7836-7846. PMLR, 2020.
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pp. 28492-28518 PMLR, 2023.
Rubenstein, P. K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., de Chaumont Quitry. F., Chen, P., Badawy, D. E., Han, W., Kharitonov, E., Muckenhimn, H. Padfield, D., Qin, J.. Rozenberg, D., Sainath, T., Schalkwyk, J.. Sharifi, M., Ramanovich, M. T., Tagliasacchi, M., Tudor, A., Velimirovié, M., Vincent, D., Yu, J., Wang, Y., Zayats, V., Zeghidour, N., Zhang, Y., Zhang, Z., Zilka, I.., and Frank, C. Audiopalm: A large language model that can speak and listen, 2023.
Salimans, T. and Kingma, D. P. Weight normalization: A simple reparameterization accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016.
11

---

Published conference paper at ICLR 2024
Series, B. Method for the subjective assessment of intermediate quality level of audio systems. International Telecommunication Union Radiocommunication Assembly, 2014.
Shi, Y., Bu, H., Xu, X., Zhang, S., and Li, M. Aishell-3: A multi-speaaer mandarin its corpus and the baselines, 2021.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Wang, C., Chen, S., Wu, Y., Zhang. Z., Zhou, L.., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., He, L.., Zhao, S., and Wei, F. Neural codec language models are zero-shot text to speech synthesizers, 2023.
Wu, D.-Y. and Lee, H.-y. One-shot voice conversion by vector quantization. In ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 7734-7738. IEEE, 2020.
Yang. D., Liu, S., Huang, R., Tian, J., Weng, C., and Zou, Hifi-codec: Group-residual vector quantization for high fidelity audio codec, 2023.
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec, 2021.
Zhang. D., S., Zhang, X., Zhan, J., Wang. P., Zhou, Y., and Qiu, Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities, 2023.
12

---

Published conference paper at ICLR 2024
A MUTUAL INFORMATION ESTIMATION
For notation, denotes discrete speech representations; denotes text; I(X; Y) denotes the mutual information; test dataset is denoted as D {(.,, w)}= and denotes the downstream model. measure of mutual information between variable X and Y can be formulated as:
P(X.Y) I(X;Y) = Jx log P(X)P(Y)
where P(X) and P(Y) are the marginal distributions of X and Y respectively, and P(X, Y) denotes the joint distribution of X and Y.
The variational contrastive log-ratio upper bound (vCLUB) (Cheng et 2020) of mutual information defined by:
I(X;Y) Pxxx,y) log to (Y]X)] Pep())(()) [log go (Y]X)]
where go (Y is the variational distribution to approximate the ground-truth probability P(Y|X) and can be parameterized by the downstream model 6.
With test dataset D. I(X; Y') has unbiased estimation as:
N
N
((x:Y)= = N
Dog (x,) - log go (y3 [x;)]
1=1 ==
13

---

Published conference paper at ICLR 2024
B MODEL STRUCTURE ABLATIONS
conducted an ablation study on whether to use LSTM or BiLSTM. In the table 6 it can be seen that the performance of BiLSTM on text alignment is better than that of LSTM, indicating that BiLSTM is better at capturing semantic information.
Text Alignment
Information Preservation
Model Structure
MI
WER
WER +
SIM
CNN+LSTM
RVQ-1
27.60
20.71
9.06
0.74
RVQ-1:8
28.61
20.38
5,44
0.97
CNN+BiLSTM
RVQ-1
30.9
15.58
9.57
0.74
RVQ-1:8
29.7
16.03
5.04
0.97
Table 6: Results of BiLSTM ablation experiment on SLMTokBench. We employ the average representation all HuBERT layers as semantic teachers in this experiment.
14

---

Published conference paper at ICLR 2024
C CONTINUOUS DISTILLATION LOSS ANALYSIS
extant literature, the commonly employed loss functions for continuous sequence distillation are typically computed along the temporal axis, with the objective of minimizing the difference between the student and teacher model outputs at each timestep. For instance, the loss function proposed in (Chang et al. 2022) aims maximize the cosine similarity between the student and teacher model representations at the same timestep while minimizing their L1 distance, thereby facilitating the transfer of knowledge from the teacher to the student model. To adapt this formula for our specific lask, we can modify the the loss function as follows:
Cusee = L11 + XCcos
T Σ t=1
|s' -
Aqill1-Aa (coo Aqi)))
where qi and s' respectively denote the quantized output of RVQ first layer and the D dimensional semantic teacher representation at timestep t. cos(-) is cosine similarity. T denotes the number of timesteps and A is the projection matrix. o-)) denotes sigmoid activation. À > 0 controls the contribution of the cosine layers. We refer to this loss function as "T-axis" to distinguish it from the "D-axis" loss function that propose in Section 3.2 The latter term is used to denote the loss function introduced in the aforementioned section. These designations are employed to differentiate between these two types of loss functions in this paper.
We investigated the impact of two distinct continuous distillation loss functions on the performance of OpeechTokenize on SLMMokBench. The results of this experiment are summarized in Table When compared to the performance of EnCodec on SeMTo,Bench, as presented in Table 3 employing the "T-axis" continuous distillation loss function significantly enhances SpeechTokenizer ' capability in text alignment. However, this improvement is somewhat inferior to that achieved by SpeechTokemize utilizing the "D-axis" loss function. In terms of Information Preservation, SpeechTokkenizer with the "D-axis" loss function also outperforms its "T-axis" counterpart. The experimental results demonstrate that the "D-axis" continuous distillation loss function yields superior distillation effects compared to the traditional "T-axis" loss function. We attribute this improvement to the "D-axis" loss function's strategy of calculating cosine similarity across each dimension, ensuring that the student model closely aligns with the teacher model on each feature dimension. This approach provides richer supervision signal, promoting the learning process of the student model by focusing not only on the overall output similarity but also the similarity within each dimension.
Text Alignment
Information Preservation
Ldistill
MI
WER
WER*
SIM
T-Axis
RVQ-1
26.65
21.10
10.75
0.76
RVQ-1:8
25.97
21.54
5.29
0.96
D-Axis
RVQ-1
30.9
15.58
9.57
0.74
RVQ-1:8
29.7
16.03
5.04
0.97
Table 7: Results continuous distillation loss ablation experiment on SLMTokBench. We employ the average representation across all HuBERT layers semantic teachers in this experiment.
15

---

Published conference paper at ICLR 2024
D DETAILS OF MODEL STRUCTURE AND DISCRIMINATORS
Encoder & Decoder Architecture The encoder is constructed as sequential scries of components: starting with a ID convolutional layer featuring channels and a kernel size of 7, followed by a set of B residual conventional blocks. Each block is composed of two dilated convolutions with dilation rate of (1, 1) and kernel size of (3. 1) and a skip-connection, followed by a strided convolutional nown-sampling layer, with kernel size K of the twice the stride R. Whenever down-sampling. the number of channels is doubled. Unlike in EnCodec that the convolution blocks are followed by two-layer LSTM, we use BiLSTM convolution layer with kernel size of 7 is used to set the dimensionality of embeddings to D. We use C 32, B 4 and (2, 4, 5) as strides. We use ELU (Clevert et al. (0166) as non-linear activation either layer normalization (Ba et al.) 2016) or weight normalization (Salimans & Kingma The decoder mirrors the encoder and uses transposed convolutions and LSTTM instead of stride convolutions and BiLSTM, with the strides in reverse order as in the encoder. The decoder outputs the final audio signal.
augment the semantic modeling ability. A final ID
Residual Vector Quantizer We use Residual Vector Quantizer (RVQ) to quantize the encoder output and follow the same training procedure as EnCodec. During training, code selected for each input is updated using exponential moving average with decay of 0.99, and codes which have not been assigned any imput vector for several batches are replaced with input vectors randomly sampled within current batch. Straight-through-- through-estimator (Bengio et al. 2013) is used to compute the gradient of encoder, e.g. as the quantization step was the identity function during the ward phase. Finally, commitment loss, consisting of the MSE between the input of the quantizer and its output, with gradient only computed with respect to its input, is added to the overall training loss.
Discriminator
The MS-STFT discriminator utilizes networks with identical structures that operate on multi-scaled complex-valued STFT, where the real and imaginary parts are concatenated. For each sub-network, it is composed of a 2D convolutional layer (using kernel size 3 with 32 channels), followed by 2D convolutions with increasing dilation rates in the time dimension (1, 2 and 4), and and stride (1, 1) provide the final prediction. For MSD and MPD, we follow the same settings as in HiFiGAN (Kong) jet al.) 2020) but adjust the channel number to align the discriminator's 's parameters more closely with that of MS-STFT.
stride of 2 over the frequency axis. final 2D convolution with kernel size
16

---

Published conference paper at ICLR 2024
E
RELATED WORK
Discrete Speech Representations There are two popular speech discrete representations: semantic tokens and acoustic tokens. Semantic tokens can be extracted from self-supervised learning of speech representations (Hsu et al.) 2021 Chung al., 202I) and encode high-level representations that correlate with coarse, symbolic features while paralinguistic information such as speaker identity and acoustic details are removed. Acoustic tokens
be extracted from neural audio codee (Zeghidour iet al. 2021 Défossez et al., 2022, Yang et al.) 2023) and provide high-fidelity reconstruction of the acoustic details. But they can not decouple different information of speech. Tokenizr unifies the two types of tokens, enabling both high-quality audio reconstruction and decomposition of different information of speech.
Spoken Generative Language Models Speech discrete representation based spoken generative lan- guage models have demonstrated remarkable performance on various speech processing tasks (Borsos
et al. 2022 Wang et al. 2023; Kharitonov et al., 2023; Zhang et al. (003).. AudioLM (Borsos al.
2022) proposes model speech based on audio codecs together with semantic codes, which can synthesize speech in a textlesss setting. VALL-E (Wang et al. (202)) leverages neural codec models to represent speech in discrete tokens from eight quantizers. VALL-E comprises of an autoregressive language model that converts phoenmes to acoustic tokens from the first quantizer and an non-autoreeressive language model to generate codes of the other seven quantizers. However, VALL-E suffers from problems that some words may be unclear, missed, or duplicated in speech synthesis due to the information gap between acoustic tokens and phoneme. To bridge the gap, SPEAR-TTS (Kharitonov al. 2023) uses semantic tokens as a bridge between text and acoustic tokens. It first generates semantic tokens from text and then produces acoustic tokens from semantic tokens. However, this multi-stage modeling approach is more complex and can lead to problems like error accumulation and slow inference speed. The first quantizer of semantic tokens, while the remaining seven quantizers produce acoustic tokens by modeling the paralinguistic information lost in the semantic tokens. SpeechTokeniee Tokenizer-basee VALL-E combines the advantages of VALL--E and SPEAR-TTS, where the autoregressive model can perform text-to- semantic tokens conversion, and the non-autoregressive model can achieve semantic-to-acoustic tokens conversion.
generates
Speech Representation Disentanglement Human speech can be roughly decomposed into three components: content, timbre, and prosody (Liu et al. 2023). Content represents the main information in the speech, which can be expressed using text or phonemes. Timbre represents the speaker's characteristics, while prosody encompasses intonation, stress, and rhythm of speech, reflecting how the speaker conveys the content information. Current Speech Representation Disentanglement (SRD) methods mostly separate speaker information from content information for voice conversion (Qian ctt al.] 2019 Casanova et al. 2022]. These approaches adopt parallel disentanglement strategy, where the speech is fed into parallel content and speaker encoders to obtain different representations (Qian al. 2020p. However, this strategy heavily relies on prior knowledge and introduces strong inductive biases. making the modeling process more complex and potentially overlooking certain speech information like prosody. Differently, VQVC (Wu & Lee, 2020) models the content embedding as a series of discrete codes and take the difference between quantize-before and quantize-after vector as the speaker embedding. Similarly, Speech Tokenizer utilizes a residual structure to perform serial decomposition of speech information and models different information as discrete tokens.
17

---

Published conference paper at ICLR 2024
F CODEBOOK ANALYSIS
We investigate whether the tokens learned by the first RVQ quantizer relate to phonetic information. Utilizing SpeechTokenizer or EnCodec, we derive speech tokens from the TIMIT training set and extract the RVQ-1 tokens, denoted 91. We then compute the conditional probability p(phonemel g1 based on the co-occurrence between phonemes and the codes. The alignments are constructed by selecting the phoneme that occurs most frequently in the receptive field for each ς1-
Figure visualizes the conditional probability p(phonemelg) q1) for both Speech and EnCodec. darker color block indicates higher p(phonemelg)) ). A more distinct contrast between the diagonal color band and its surrounding area signifies greater phoneme purity, which in turn suggests a more accurate mapping between the code and its corresponding phoneme. For Speech it's evident that in the codebook of RVQ-1 quantizer, many discrete codes seem to specialize in capturing specific phonetic sounds, indicating RVQ-1 quantizer can obtain good alignment between codes and labeled phonemes. However, for EnCodec, this phenomenon is not as obvious.
Additionally. Figure 4 also reveals that over 600 codes from the EnCodec RVQ-1 codebook have never been utilized, suggesting a suboptimal utilization rate of the codebook when EnCodec encodes speech. A lower utilization rate the codebook implies that more RVQ layers are required to ensure the quality of synthesized speech, consequently necessitating the generation of more codes during the construction of a spoken generative language model, resulting in greater space, time and computation power consumption.
We further evaluate the models using 2021). As shown in Table RVQ-1 tokens of SpeechTokenize achieve superior PNMI score to that of HuBERT units and sigmificantly outperforms EnCodec-RVOQ -VQ--- This suggests that the semantic distillation process in SpeechTokenize is effective, thereby explaining its enhanced text alignment performance.
Mutual Information (PNMI) Hsu et al.
Phonemo
(a) Speech Tikenizer
(bi Encodec
Figure 4: Visualization of the conditional probability P(phoneme/code) on TIMIT train set. The y-axis is the phoneme set and the x-axis is the codewords of the first RVQ layer sorted by the most correlated phoneme.
Tokenizer
PNMI
HuBERT
KM500
0.43
EnCodec
RVQ-1
0.28
SpeechTokenizer
RVQ-1
0.71
Table 8: PNMI of different discrete speech representation.
18

---

Published conference paper at ICLR 2024
G EXTENSION TO UNSEEN LANGUAGE
Since the paralinguistic information considered to language-agnostic, we attempted to apply Spkeizro directly to unseen languages. We choose German and Chinese. For German, select samples from the German subset of Multilingual LibriSpeech dataset for testing. For Chinese, select samples from the Aishell-3 dataset (Shi et al., 2021) for testing. We resynthesize speech from RVQ-1 and RVQ-1:8 tokens. Resynthesized speech are displayed in our demo website [] also analysis the melspectrogram of German speech and English speech in Appendix H
We
Results show that for languages either closely or distantly related to English, resynthesized speech from RVQ-1 tokens tend lose timbre and prosody information while maintaining clear content. The resynthesized speech generated from RVQ-1:8 tokens is very close to the grountruth. That suggests SpeechTokenizer can achieve hierarchical information disentanglement on unseen language, even though SpeechTokenizer Tokenizer is trained solely on English data. We believe that Speech Tokenizer may possess the ability to extract content from speech while disregarding language gepenndde features. This ability holds promise for the development of multilingual Speech Tookenizr.
19

---

Published conference paper at ICLR 2024
H
MELSPECTORGRAA ANALYSIS
We plot the melspectrogram of raw speech, resynthesized speech of EnCodec RVQ-1 I tokens, and resynthesized speech of Tokenizer RVQ-1 tokens. From the figure [ it's evident that the melspectrogram corresponding to EnCodec RVQ-1 largely retains the stripes and shapes in the raw melspectrogram. In contrast, the speech resynthesized from SpeechTokenizer RVQ-1 essentially loses all of the horizontal stripes, which indicates that timbre and prosody information has been diminished.
Raw speech
4230
2045
$12
Time
Bacyttnnsiae Speech of FaCader-RVO!
2045
1374
0.
Time
Resyntheslaed Speech SpeechTokerieee-F00
0192
2040
1074
Figure 5: Melspectorgram of raw speech, resynthesized speech of SpeechTokening and EnCodec RVQ-1 tokens.
20

---

Published conference paper at ICLR 2024
We alse plot melspectrogram of raw German speech and resynthesized German speech of Speech Tok- enizer RVQ-1 tokens. As shown in the Figure the same patterns observed in English speech are also present in German speech.
speech
0192
2040
522
Resynthesized Speech speechookeriarrr.v0
0192
2048
512
Figure 6: Melspectorgram of German speech and resynthesized speech of Speech Tokenizer RVQ-1 tokens.
21


Reviews:
[{"review_id": "z9yHptTNl7", "reviewer": "Reviewer_tD35", "summary": "This paper propose a new model to learn a speech tokenizer that can both reconstruct speech well and retain enough semantic information in speech. It simplifies previous work that need to use separate tokenizers for semantic and acoustic information. Since good tokenizers play an important role in speech language models, the results in this paper could benefit the advancement of research in speech language models in the LLM era.", "questions": "1. Is it possible to directly combine the training loss of hubert and soundstream/encodec instead of distillation to achieve unified speech tokenizer?\n2. Since soundstream/encodec is a small model, is it possible the model can learn good representations when the model is large enough?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: this is the first work trying to inject semantic information to acoustic tokens based on soundstream/encodec which has been successfully applied to speech language modeling, TTS, etc. Though the idea is kind of straight, this paper still deserves originality.\nQuality: Overall the solution is clearly motivated and reasonably implemented, and corresponding evaluations are comprehensive.\nClarity: the paper is easy to read, the results are easy to understand.\nSignificance: A good tokenizer is important for many downstream tasks, especially for speech generation tasks. This study shows it's possible to use a single model as tokenizer and achieve comparable or even better performance than previous studies using combination of different tokenizers encoding different aspects of speech", "weaknesses": "1. It's not quite clearly explained why the authors chose the first RVQ to inject semantic information. Why not the later ones or even for all RVQs\n2. It could be better if the authors can show some experiments using the new tokenizers for speech understanding tasks like recognition/speech translation with relatively large scale training data, and compare to Hubert/soundstream/encodec tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hDoRVdXJJD", "reviewer": "Reviewer_pbGo", "summary": "The paper addresses the limitations of current speech representations in large language models. It introduces a benchmark called SLMTokBench to assess the suitability of existing speech tokens for speech language modeling. The results indicate that neither semantic nor acoustic tokens are ideal for this purpose. To overcome this, the authors propose SpeechTokenizer, a unified speech tokenizer for speech large language models.\n\nSpeechTokenizer utilizes the Encoder-Decoder architecture with residual vector quantization (RVQ) and combines semantic and acoustic tokens. It disentangles different aspects of speech information hierarchically across RVQ layers, providing a more comprehensive representation of speech data. The authors also construct a Unified Speech Language Model (USLM) using SpeechTokenizer.\n\nExperimental results demonstrate that SpeechTokenizer performs comparably to EnCodec in speech reconstruction and shows strong performance on the SLMTokBench benchmark. Overall, the paper introduces SpeechTokenizer as a solution for improving speech language models by addressing the limitations of existing speech tokens.", "questions": "Why the authors not directly cite the number in VALL-E paper for evaluation? I don't think it is a good idea to re-train VALL-E on a smaller training set.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of the paper is strong. Speech research area needs speech specific fundamental innovation. Many papers simply borrow ideas from NLP and CV and test their performance on speech data. Exploring how to discretize continuous speech signals is an intriguing concept. The authors propose a simple and logical approach, which serves as a promising initial step for the field.\n\n\n2. SLMTOKBENCH offers valuable insights for evaluating current speech tokenization methods. It has the potential to become a fundamental benchmark for speech tokenization research. However, it still requires some improvements, as mentioned in the weaknesses . section.", "weaknesses": "1. My primary concern lies with the experimental section of the paper. Firstly, the authors trained SpeechTokenizer on the LibriSpeech set, while the EnCodec was trained on a mixed dataset of 10,000 hours. As a result, EnCodec should possess better generalization capabilities for speech reconstruction, particularly in the presence of noise. However, the authors only evaluated the performance on LibriSpeech, which is an in-domain test set for their method but falls outside the domain of EnCodec. This unfair experiment fails to demonstrate whether their method outperforms EnCodec. A fair experiment should involve testing the model on various test sets, especially those unseen during training.\n\nSLMTOKBENCH should encompass a range of speech environments, not limited to LibriSpeech alone. Audiobooks represent just one type of speech data, and we should consider incorporating a more diverse test set for a comprehensive evaluation.\n\n\n2.How did you arrive at the WER number of 7.9 in Table 4? I recall that the VALL-E paper reported a WER of around 3.X on LibriSpeech.\nAdditionally, other studies on LibriSpeech have achieved comparable or even lower WER values.\n\n3. In the context of the speech tokenizer, it is necessary to take into account additional baselines like SoundStream and the multi-band diffusion-based method, rather than solely comparing it with Encode.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b5whjdKc6A", "reviewer": "Reviewer_usJL", "summary": "This paper proposed SpeechTokenizer to learn unified speech tokenizer for language models.  Experiments demonstrate that the Unified Speech Language Model (USLM) leveraging SpeechTokenizer SpeechTokenizer shows comparable performance when compared with EnCodec in speech reconstruction and SLMTokBench benchmark. In zero-shot Text-to-Speech tasks, USLM achieves better performance than VALL-E.", "questions": "My questions are listed above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation of the paper is straightforward and push the boundary of speech language models to more unified representation for different tasks. \n2. Extensive experiments are done to validate the performance comparison.", "weaknesses": "1. The paper misses the comparison with Hierarchical speech language models as listed in the paper. if the comparison could be added, it would be more convincing.\n2. Although the motivation of the paper is a shining point for the paper, the technical contribution of the paper is limited as the framework is borrowed from existing work, e..g, the framework of RVQ-GANs. If authors could propose some modifications on the training strategy or the model architectures based on some observations from experiments, it would add more insightful points.\n\nIf my above concerns are resolved, I would consider increasing my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pnApPROPxO", "reviewer": "Reviewer_g1yJ", "summary": "The paper mainly makes two parts of contributions. The first one is a benchmark, SLMTokBench, which assesses the suitability of speech tokens for building speech language models. The second one, which is also the main contribution of the paper, is a unified speech tokenizer that combines both semantic and acoustic tokens. SpeechTokenizer performs comparably with pure acoustic token (EnCodec) in speech reconstruction, and its performance is much higher than SpeechTokenizer in VALL-E.", "questions": "Can the author provide some explainations why the improvement of USLM in zero-shot TTS is much larger than the improvements of SpeechTokenzier in speech reconstruction (Table 4 v.s. Table 2) ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a novel method to unify the semantic tokens and acoustic tokens. \n\n2. The experiments show that the unified tokenizer, SpeechTokenizer, large improves the acoustic tokens from EnCodec in the zero-shot TTS experiment in VALL-E. The improvements are significant in both objective metrics and subjective metrics. \n\n3. The proposed benchmark, SLMTokBench, is well motivated and reasonable.\n\n4. The paper is well written and easy to follow.", "weaknesses": "1. My first concern is about the necessity for a unified speech tokenizer. In the last several sentence of page 1, the authors argue that \"the multi-stage modeling approach is more complex, leading to several drawbacks such as error accumulation and slower processing speed\". Is there any evidence to support the claim? In addition, I may somewhat disagree with how the paper describe the characteristic of \"Semantic LM and Acoustic LM\" in Table 1. In my view, Semantic LM does not generate Accurate Content but it is speed is much faster. And Acoustic LM does generate Accurate Content. \n\n2. The second concern, which is also my major concern, is about whether the complicated training pipeline really leads to better performance in zero-shot TTS than Hierarchical LM. If using VALL-E with the semantic token for HuBERT and RVQ1:7 from EnCodec (It has the same bit rate as USLM), what is performance?\n\n3.  At last, I am doubtful whether the SpeechTokenizer only works for AR + NAR model like VALL-E. As VALL-E only uses the first speech token in the AR model, putting more information on the first token may be beneficial to its performance so that the improvement of USLM in zero-shot TTS is much larger than the improvements of SpeechTokenzier in speech reconstruction. I am not sure such large improvements will still exist in Hierarchical model like AudioLM. \n\nI understand that weakness 3 is hard to verify during the rebuttal. Therefore, If the authors can resolve my first two concern, I am willing to raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)