RepCodec: A Speech Representation Codec for Speech Tokenization (2024)

Abstract: With recent rapid growth of large language models (LLMs), discrete speech tokenization has played an important role for injecting speech into LLMs. However, this discretization gives rise to a loss of information, consequently impairing overall performance. To improve the performance of these discrete speech tokens, we present RepCodec, a novel speech representation codec for semantic speech tokenization. In contrast to audio codecs which reconstruct the raw audio, RepCodec learns a vector quantization codebook through reconstructing speech representations from speech encoders like HuBERT or data2vec. Together, the speech encoder, the codec encoder and the vector quantization codebook form a pipeline for converting speech waveforms into semantic tokens. The extensive experiments illustrate that RepCodec, by virtue of its enhanced information retention capacity, significantly outperforms the widely used k-means clustering approach in both speech understanding and generation. Furthermore, this superiority extends across various speech encoders and languages, affirming the robustness of RepCodec. We believe our method can facilitate large language modeling research on speech processing. Our code and models are released at https://github.com/mct10/AudioDec_ct.

heading: 1 Introduction
text: The significant achievements of large language models (LLMs) within the field of natural language processing have attracted considerable attention, as evidenced by notable works such as OpenAI (2023); Brown et al. (2020); Radford et al. (2019); Wei et al. (2021); Chowdhery et al. (2022). Bridging the realms of continuous speech and tokenbased language models necessitates a key technique known as speech tokenization, which discretizes an
∗Contribute equally † Corresponding author
audio signal into a finite set of tokens. By converting speech into discrete tokens, language models can predict the future semantic content and generate realistic speech with long-term consistency (Nguyen et al., 2022). As a result, a growing body of research has begun to incorporate speech tokenization into the realm of LLM. Noteworthy examples include AudioLM (Borsos et al., 2023), AudioPaLM (Rubenstein et al., 2023), Vall-E (Wang et al., 2023a), PolyVoice (Dong et al., 2023), and SpeechGPT (Zhang et al., 2023a).
Discrete speech tokens can be divided into two categories: semantic tokens and acoustic tokens. Acoustic tokens are produced by audio codecs (Zeghidour et al., 2022; Défossez et al., 2022; Zhang et al., 2023b), which aim to reconstruct the original audio so that it can be perceptually identical to listeners. However, attempting to preserve all information of the audios leads to high bitrates of acoustic tokens. The process not only imposes significant computational demands on the LLMs, but sometimes makes training infeasible with such lengthy sequences. For example, converting a 30-second audio segment into acoustic tokens needs 18,000 tokens. Therefore, current language modeling approaches often require substantial architectural adjustments to accommodate such long sequences (Borsos et al., 2023; Wang et al., 2023a).
Semantic tokens, on the other hand, aim at preserving only the semantic information of the audio, which allows much lower bitrates. If the task relies only on the content of the speech (e.g. speech recognition / translation), using semantic tokens should be a better choice. At present, k-means clustering on speech representations (Hsu et al., 2021) is the most prevalent technique of extracting semantic tokens. However, this method has two drawbacks. Firstly, it suffers from a loss of semantic information compared to the original speech representations (Lee et al., 2022b; Borsos et al., 2023). Secondly, not all sets of speech representations are
5777
suitable for clustering. Rubenstein et al. (2023) reported that the choice of speech encoder significantly affects the downstream task performance of speech tokenization. Thus, we are motivated to improve the method of extracting semantic tokens by addressing the above problems.
In this paper, we propose RepCodec, a general tokenization approach for representations that can be applied to speech to extract its semantic tokens. RepCodec leverages an end-to-end neural codec to preserve more information of the speech representations. RepCodec is composed of an encoder, a vector quantizer (VQ) and a decoder, and it is trained to reconstruct the input speech representations as much as possible. The speech encoder, codec encoder and the VQ codebook together constitute the speech tokenization pipeline, which can produce high-quality semantic tokens for downstream tasks with a low bitrate. We evaluate the RepCodec tokens in downstream tasks of speech understanding and generation. Specifically, we use a decoder-only ASR modelling task to evaluate the speech understanding of RepCodec tokens, and a unit-to-speech generation task to measure the quality of RepCodec tokens for speech generation. Our comprehensive experiments demonstrate that RepCodec significantly outperforms the dominant kmeans clustering approach. It is worth noting that supervised approach such as ASR and phoneme recognition can also be considered as forms of speech tokenization that convert speech into word tokens or phoneme tokens. However, it is essential to highlight the large amount of parallel data required for supervised training only exists for highresource languages. RepCodec, on the contrary, is an unsupervised speech tokenization method that can be applied to any languages. The contribution of our work can be summarized as follows:
1. We propose a novel framework, RepCodec, which applies compression techniques on representations to enhance the preservation of information within representations.
2. The experiments show that, by applying RepCodec to speech, semantic tokens exhibit an improved capacity for retaining information, and they surpass the prevailing k-means clustering approach in both speech understanding (4.5% v.s 2.8% word error rate (WER)) and generation (7.6% v.s 4.7% WER). In addition, further experiments demonstrate that RepCodec is a robust method that can be applied to various speech encoders and languages.
3. Our further analysis underscores that the quality of semantic tokens primarily relies on the information loss instead of their similarity to the phonemes. This finding serves as motivation for future advancements in semantic token refinement.

heading: 2 Related Work
text: Several lines of works are related to RepCodec, including self-supervised speech representation learning, speech tokenization and vector quantization. Self-supervised Speech Representation Learning. This line of research has recently gained huge success in the area of speech processing. Prevalent methods usually requires the model to predict the content of unseen regions (Hsu et al., 2021; Baevski et al., 2022; Chung et al., 2021; Chung and Glass, 2020) or to contrast the target unseen frame with randomly sampled ones (Baevski et al., 2020; Conneau et al., 2021). Specifically, HuBERT (Hsu et al., 2021) is a pioneering work to employ k-means for speech tokenization. The generated tokens serve as the training targets for the speech encoders. And they find that these tokens exhibit a strong correlation with the phonemes. Later on, Lee et al. (2022b); Meng et al. (2023) show that these semantic tokens can be used directly to perform downstream tasks like ASR or voice conversion. In addition to the self-supervised approaches that only use unlabeled speech data, many works train the speech encoder in an end-to-end manner (Gulati et al., 2020; Radford et al., 2023) with labeled speech data. Among them, Whisper (Radford et al., 2023) is currently the largest open-source model for speech recognition, which achieves the lowest overall WER across various domains. Speech Tokenization. Discrete speech tokens can be divided into two categories: Semantic tokens (Lee et al., 2022b; Hsu et al., 2021; Chung et al., 2021) and Acoustic tokens (Zeghidour et al., 2022; Défossez et al., 2022; Wu et al., 2023). Semantic tokens maintain the linguistic information of the speech and have high correlation with phonemes (Hsu et al., 2021). They are commonly generated by applying k-means clustering to pretrained speech encoders like HuBERT (Hsu et al., 2021) or data2vec (Baevski et al., 2022). Semantic tokens are widely used for downstream tasks. For example, Lee et al. (2022b); Polyak et al. (2021); Lee et al. (2022a) use them to train a unit-vocoder to generate raw speech. AudioLM (Borsos et al., 2023) inputs the semantic tokens of w2v-BERT to
represent semantic information of the audio. Zhang et al. (2023a) jointly trains a language model with semantic tokens to inject speech recognition ability to GPT-like models. Dong et al. (2023) also incorporates semantic tokens for speech to speech translation. However, the discretization step of kmeans discards plenty of information of the speech, resulting in degraded performance in downstream tasks (Lee et al., 2022b; Borsos et al., 2023).
In contrast to semantic tokens, acoustic tokens aim to preserve all the information of the audio. Soundstream (Zeghidour et al., 2022) and EnCodec (Défossez et al., 2022) use a neural audio codec with Residual Vector Quantizers (RVQ) to learn acoustic tokens that can be directly reconstructed into raw audios. As these acoustic tokens contain acoustic information of the audio, they can be used to perform more complicated tasks than semantic tokens. For example, VALL-E (Wang et al., 2023a), uses them for zero-shot text to speech (TTS), and AudioLM (Borsos et al., 2023) employs these tokens not only to produce realistic speech but also music. However, as acoustic tokens need to preserve a lot of information unrelated to semantics, their bitrates surpass the capacity of conventional language models.Consequently, handling these tokens requires specialized techniques (van den Oord et al., 2017; Borsos et al., 2023), making their practical utility challenging. Vector Quantization. Learning vector quantization is important for efficient coding of information. VQ-VAE (van den Oord et al., 2017) introduces vector quantization into VAE (Kingma and Welling, 2014) to reconstruct images by learning discrete codebooks. They propose a straight-through gradient method to allow gradient back-propagation through a non-differentiable quantization operation so that optimization of the network is feasible. Furthermore, recent advancements include
softmax quantization (Kankanahalli, 2018), exponential moving average (EMA) (Gârbacea et al., 2019), and Gumble-softmax (Yang et al., 2022) also work well for optimization of the VQ module. In addition to multiple VQ codebooks in van den Oord et al. (2017), Soundstream (Zeghidour et al., 2022) introduces a new RVQ method that is able to compress the raw audio with different bitrates.

heading: 3 Method
text: Despite the wide applications of semantic tokens in speech modeling (Borsos et al., 2023; Lee et al., 2022b), the discretization of the representations results in severe information loss. Consequently, downstream tasks, such as ASR or speech translation, suffer from a significant downgrade in performance. In AudioLM (Borsos et al., 2023), the WER is dramatically increased from 2.5% to 6.0% by using the discrete tokens of k-means from w2vBERT XL (Chung et al., 2021). The unit-vocoder in mHuBERT (Lee et al., 2022b) also relatively increases the WER of the generated audio by about 70%. These results all demonstrate that the severe information loss actually prevents the discretization of speech from obtaining SOTA performance.
The information loss motivates us to preserve more information during the discretization of the representations. To this end, we propose a novel method, RepCodec, to perform more efficient compression on the representations so that the semantic tokens can preserve more information and achieve better performance in downstream tasks.

heading: 3.1 Architecture of RepCodec
text: In order to achieve better compression of the representation, RepCodec uses a parametric network, which consists of 3 components (Figure 1): a codec encoder, a VQ module, and a codec decoder. The codec encoder takes as input the speech represen-
tations X = [x1, · · · ,xT ] ∈ RH×T and produces latent representations Z = [z1, · · · , zT ] ∈ RH×T . Here, H is the dimension of the speech representation and T is the length of the sequence. Z is then passed through the VQ module to be quantized into a sequence of discrete tokens s = s1 · · · sT with codebook E = [e1, · · · , eK ], where K is a predetermined number of clusters. The codec decoder utilizes these tokens E to reconstruct the original speech representations.
Encoder and Decoder. The architecture of the encoder and decoder follows Zeghidour et al. (2022); Wu et al. (2023), which achieves great success in compressing audio signals. As shown in Figure 2, the encoder consists of several 1D convolution layers with convolution applied to the time dimension of the input representation X. The encoder block contains residual path to allow better optimization of the network (He et al., 2016). The decoder has a similar design, which is also composed of several 1D convolution layers and residual paths. In our paper, we do not downsample or upsample in both the encoder and the decoder, and keep frequency of the representation the same as the input.
Vector Quantizer. Vector Quantizer aims to compress the latent representations Z to a series of discrete tokens s. It projects the latent z to its closest codebook ek and outputs ek to the decoder. We adopt two kinds of quantizer, regular VQ (van den Oord et al., 2017) and RVQ (Zeghidour et al., 2022). RVQ is a M -layer quantizer where each layer quantizes the residual of the previous layer, and it is effective for compressing the audio signals. When M = 1, RVQ is equivalent to VQ.

heading: 3.2 Training Objective
text: Our training objectives consist of a reconstruction loss on X, which aims to preserve as much input information as possible for downstream tasks, and a quantization loss to effectively train the VQ. Reconstruction loss lr. We minimize the squared ℓ2 distance between the input representations X and the output representations X̂. Formally,
lr = 1
HT ∥X− X̂∥2F (1)
where H is the hidden dimension of the representation and ∥ · ∥F denotes the Frobenius norm. Quantization loss lq. Following Défossez et al. (2022); Wu et al. (2023), we apply a quantization loss lq between the output of the encoder and the quantized value from VQ. Formally, given the latent representations Z = [z1, z2, · · · , zt] and the codebooks E = [z1, z2, · · · , zK ], we minimize
lq = 1
T
T∑
t=1
1
H
K∑
k=1
Ik(zt)∥zt − ek∥22 (2)
where Ik(zt) ∈ {0, 1} is binary indicator variables indicating which of the K clusters the data point zt is assigned to. Ik(zt) = 1 if zt is assigned to cluster k, and Ik(zt) = 0 otherwise. When RVQ is used, the quantization loss in Equation (2) is generalized as:
lq = M∑
i=1
1
T
T∑
t=1
1
H
K∑
k=1
Iik(zit)∥zit − eik∥22 (3)
where i ∈ [1,M ], denotes the ith quantizer of RVQ. And the superscript i in Iik, zit, eik denotes the indicator, input representation and codebook for ith
quantizer respectively. lq is only used for updating the encoder parameters. It makes the latent representation Z of the encoder suitable for the clustering of quantizer. The quantizer is updated by EMA described in Section 3.3. Overall, RepCodec is trained by a combination of the two losses,
l = λr · lr + λq · lq (4)

heading: 3.3 Optimization of Vector Quantizer
text: Both k-means (Lloyd, 1982) and VQ (Gray, 1984) are algorithms to discretize a high-dimensional vector into a discrete label. They optimize a common objective function: aiming to find the best clusters measured by ℓ2 in Equation (2) (Bishop and Nasrabadi, 2006). However, they adopt different kinds of optimization algorithms.
K-means adopts an EM algorithm (Bishop and Nasrabadi, 2006) to search for the best clusters. Despite its success in clustering raw input, its sharp changes hinder the back-propagation of gradient through the quantization module, which potentially results in training process instability. On the contrary, the optimization algorithms adopted in VQ, including Straight-through Gradient Method, Exponential Moving Average (EMA) and Gumblesoftmax, gradually change the quantization. This ensures a stable update of the encoder so that it can be trained end-to-end with other components of the model (van den Oord et al., 2017). We follow the optimization in Zeghidour et al. (2022) and use the EMA algorithm. Formally speaking, let {z1, . . . , zb} be the minibatch input, where b is the batch size, then the codebook entries ek are updated by EMA with factor 0 ≤ γ ≤ 1, where ñk and ẽk represents the moving average of the number and codebook of the k-th cluster. Denoting Ik(zj) as the indicator that j-th feature belongs to k-th cluster, we have
ñk = γñk + (1− γ) b∑
j=1
Ik(zj),
ẽk = γẽk + (1− γ) b∑
j=1
Ik(zj)zj
(5)

heading: 3.4 Downstream Tasks
text: To measure the performance of semantic tokens, we evaluate these tokens on downstream tasks of decoder-only ASR and unit-to-speech generation. These two sets of experiments simulate the audio
input and audio output of a language model respectively. And we measure the amount of semantic information captured by the tokens by WER. Speech Resynthesis and Voice Conversion. For each set of tokens, a unit-based HiFi-GAN vocoder is built to resynthesize speech. We follow Lee et al. (2022a); Polyak et al. (2021) for the training and inference of the vocoders. The vocoders are trained with a combination of the generator-discriminator loss and the mean square error (MSE) of each unit segment in logarithmic domain. Following the common practice to evaluate semantic tokens (Borsos et al., 2023; Lee et al., 2022b), the quality of tokens are measured by ASR-WER of the resynthesized speech with the Whisper large-v2 model (Radford et al., 2023). Additionally, the acoustic quality of the speech is evaluated by F0 error and MOS, which are presented in ??. Decoder-only ASR. Following the emergence of LLM, and a series of works that inject audio information into a decoder-only transformer, we evaluate the quality of RepCodec using a decoder-only transformer. Given a series of semantic audio tokens s = s1s2 · · · sT and its corresponding transcript y = y1y2 · · · ym, we form a sequence
p = s1s2 · · · sn 
 y1y2 · · · ym
where 
 is a special token indicating the start the transcription. As ASR is a sequence-to-sequence task, we find a transformer F that maximizes the conditional probability

heading: F∗ = argmax
text: F p(y|s) = argmax F
m∏
i=1
p(yi|y<i, s),
instead of full language modeling of p(s,y). We also compare the full language modeling of p(s,y) in Appendix C.2, which is much worse than p(y|s).

heading: 4 Experiments

heading: 4.1 Experiment Setups
text: Choices of Representation. We select the widelyused self-supervised pretrained models, HuBERT (Hsu et al., 2021) and data2vec (Baevski et al., 2022), as the speech encoder of RepCodec. In accordance with the common practices of selecting the layer for representations, we choose the output from the layer at about 2/3 of the total layers as the input representations of RepCodec. In addition, we include the most-powerful open-sourced representations, Whisper (Radford et al., 2023), in our
evaluation of RepCodec. As we find the representations from the top layer of Whisper encoder is most suitable for ASR in SUPERB (wen Yang et al., 2021), we use them as the input representations for RepCodec. Moreover, we extend our evaluation beyond single-layer representations, exploring whether the linear combinations of multiple layers are more suitable for the downstream tasks. We use SUPERB toolkit to find the best linear combination of representations for SUPERB ASR, and use it as the input representation for RepCodec.
Baselines. We compare RepCodec with several baselines, including k-means (Hsu et al., 2021), VQ and EnCodec (Défossez et al., 2022). Kmeans has been a predominant method in prior literature for semantic token extraction (Hsu et al., 2021; Lee et al., 2022b; Borsos et al., 2023). VQ directly takes the input speech representation X to the vector quantizer without either encoder or decoder. K-means and VQ are similar methods, sharing the same model and objective function, except that VQ uses the same optimization method as RepCodec. EnCodec is an open-source model similar to SoundStream (Zeghidour et al., 2022), which compresses the information directly from raw audio. We limit the bitrates of EnCodec for fair comparison among different methods.
In addition to the baselines, we also include a upper bound for decoder-only ASR, speech resynthesis and voice conversion respectively. For ASR, we replace the tokens with the original representations as the input of the decoder, preserving the complete information for ASR. Concerning speech resynthesis and voice conversion, ASR-WER of the original audio is reported as a benchmark.
Training Semantic Tokenizers. The detailed architecture and hyperparameters of training RepCodec is available at Appendix B. We employ a fixed cluster count of K = 1024 for all semantic tokenizers. And we use the train-clean-100 subset of the LibriSpeech corpus (Panayotov et al., 2015) to train our all semantic tokenizers. It ensures a fair comparison between k-means, VQ and RepCodec (previous implementation of k-means cannot use large amount of audio data due to memory constraint). For multilingual experiments, we further incorporate 100h subsets from MLS (Pratap et al., 2020) for French and Spanish.

heading: 4.2 Decoder-Only ASR
text: We convert the full 960h of LibriSpeech (Panayotov et al., 2015) speech into speech tokens and use them to train decoder-only ASR models. The decoder is a Base transformer with 12 layers, embedding dimension 768 and FFN dimension 3072. After training, the ASR models are then evaluated at the test-clean and dev-clean subsets of LibriSpeech. Detailed experimental setup are deferred to Appendix B. As shown in Table 1, RepCodec achieves much lower WER than both k-means and VQ across all representations. For single-layer representations, RepCodec is particularly effective for large speech encoders such as data2vec large and Whisper. For data2vec large, RepCodec improves WER by about 2% in absolute value, and achieves very close performance to the original representation. In case of both Whisper medium and Whisper large models, RepCodec improves the WER by more than 4% in absolute terms, relatively decreasing WER by nearly 50%.
For linear combination of representations, we observe that they are not as suitable as the single layer representations for clustering. Nevertheless, RepCodec still achieves large improvement in WER. Particularly, RepCodec can produce meaningful WER for Whisper representations, while both VQ and k-means cannot successfully cluster them. Although EnCodec (Défossez et al., 2022) preserves information beyond linguistic content of the speech, its performance is inferior to RepCodec in terms of semantic information. While EnCodec uses higher bitrates than RepCodec (1.5kps v.s 0.5kbps), RepCodec still achieves lower WER. It shows that RepCodec is more suitable for downstream tasks which rely on semantic information of speech. Scaling RepCodec. In Table 2a, we use all 960h data of LibriSpeech to train a larger RepCodec model (the architecture is deferred to Appendix A). The downstream decoder-only ASR shows that the model achieves even lower WER, validating the scaling ability of our method. In Appendix C.4, we also compared the scaling ability of RepCodec with k-means and report the WER on test-other subset of LibriSpeech. Residual Vector Quantization. In Table 2b, we present the performance of RepCodec with a 2- layer RVQ. It has higher bitrates and preserve more information of the speech representation. Therefore, RepCodec has further improvements in the downstream decoder-only ASR task.
Multilingual. We conduct experiments with representations from mHuBERT (Lee et al., 2022b) and train a unified speech tokenizer for three languages (English, French and Spanish). Then we train a unified decoder for ASR of all three languages. The WER is evaluated on the test-clean of LibriSpeech and the test sets of MLS. As shown in Table 2d, RepCodec outperforms k-means and VQ across all three languages, demonstrating that RepCodec can be applied to multiple languages.

heading: 4.3 Speech Resynthesis and Voice Conversion
text: We conduct the unit-to-speech resynthesis and voice conversion on two datasets: LJSpeech (Ito and Johnson, 2017) and VCTK (Veaux et al., 2017) . LJSpeech is a single speaker English TTS corpus comprising 13,100 speech utterances, equivalent to approximately 24 hours of audio. VCTK is a multispeaker English TTS corpus uttered by 109 speakers. It comprises around 43,800 speech utterances,
equivalent to approximately 44 hours of audio. We follow the data partition in Polyak et al. (2021), and split the data into training sets, validation sets and test sets. All the audios are downsampled to 16kHz and trained with a fixed 50k training steps. For the experiments of LJSpeech, we use the architecture and toolkit provided in Lee et al. (2022a) for the training and inference of the vocoders. For VCTK, we follow the model architecture and toolkit provided in Polyak et al. (2021). In this setup, the synthesized speech’s speaker characteristics are conditioned on speaker embeddings. In Table 3, "Speech Resynthesis" indicates that the speaker embedding utilized for resynthesizing the speech corresponds to the ground truth speaker, while the speaker embeddings of "Voice Conversion" were randomly selected from the seen speakers.
In Table 3, we report the ASR-WER of EnCodec, RepCodec and k-means. We only compare RepCodec with k-means, which is similar to VQ and
achieves lower WER in ASR. We only evaluate semantic tokens of single-layer representations, which are shown more suitable for downstream tasks in Table 1. When using Whisper to transcribe the speech, we turn off the temperature scheduling and use temperature 0 to remove the randomness of the evaluation. The WER difference between original audio and generated speech shows the quality of the semantic tokens. RepCodec reduces WER by more than 2% in absolute value for all these representations in both LJSpeech and VCTK, which is much more significant than the improvement in decoder-only ASR. For Whisper representations, RepCodec improves WERs by more than 30% in absolute terms. This performance is in stark contrast to the deteriorations of ASR-WER in Borsos et al. (2023) and Lee et al. (2022b). In those instances, the WER experienced a relative increase of approximately 70%. However, for RepCodec, this relative downgrade is significantly reduced to about 35%. In Table 4, we present the acoustic quality of the generated speech. Following Polyak et al. (2021), we use F0 Voicing Decision Error (VDE), F0 Frame Error (FFE), Speaker Similarity and Mean Opinion Scores (MOS) to measure the similarity and naturalness of the generates speech. The Speaker Similarity follows Wang et al. (2023b) and calculates the cosine similarity between the speaker embedding. Table 4 shows that the tokens of RepCodec does not produce downgraded quality of the speech. On the contrary, with better information preservation, RepCodec even improves the acoustic quality of the resynthesized speech, proving the superiority of the acoustic quality of RepCodec tokens in the speech generation tasks.

heading: 4.4 Analysis
text: Phone-Normalized Mutual Information (PNMI) versus Reconstruction Loss. Hsu et al. (2021) and Borsos et al. (2023) propose several methods to measure the quality of semantic tokens, includ-
ing PNMI and ABX error. These quantities measure the similarity between each phoneme and semantic token. Specifically, PNMI defined in Hsu et al. (2021) measures the correspondence of single phoneme and single semantic token. As presented in our observations, we find that such one-to-one correspondence cannot fully reflect the quality of semantic tokens.
In the left plots of Figure 3, we show the changes of PNMI, reconstruction loss and WER as the training step increases. The reconstruction loss is normalized by its ℓ2 norm. The reconstruction loss decreases as the training of RepCodec proceeds, and so does the WER of decoder-only ASR. However, PNMI also decreases for longer training steps. The right of Figure 3 shows a similar observation, where we plot the relationship of reconstruction loss v.s WER and PNMI v.s WER for k-means, VQ and RepCodec for different speech representations. Methods with higher PNMI do not result in lower WER in downstream tasks. In contrast, downstream tasks performance is positively correlated to the reconstruction loss of the clustering. These outcome underscores that higher PNMI does not necessarily correspond to reduced WER values. Instead, when we increasingly retain information from the speech representation (represented by decreasing lr), the semantic tokens have higher quality for downstream tasks, although these tokens get dissimilar to the phonemes.
It is worth noting that our findings do not contradict to the assertion made by Hsu et al. (2021), which suggests token sets with higher PNMI lead to better performance. In Hsu et al. (2021), the discretized tokens serve as training targets for the speech encoders, while our tokens serve as representations of the speech itself for downstream tasks. The difference in token usages leads to the diversion on the perspective of PNMI. Interpretability of RepCodec. While PNMI defined in Hsu et al. (2021) cannot reflect the quality of semantic tokens for downstream tasks, we believe the tokens of RepCodec still correspond to semantic tokens like phonemes. However, the
correspondence is not one-to-one between individual tokens and individual phonemes. Instead, a sequence of RepCodec tokens corresponds to a sequence of phonemes. To measure the correspondence between sequences of discrete tokens and sequences of phonemes, we extend PNMI for n-gram tokens. It is calculated by the following equation
PNMIn = I(sj : sj+n; zj : zj+n)
H(sj : sj+n) , (6)
where zj : zj+n denotes a sequence of semantic tokens with length n, starting at the j-th frame. sj : sj+n denotes a phoneme sequence of length n. I is mutual information and H is entropy. When n = 1, PNMI1 is the original PNMI defined in Hsu et al. (2021). As shown in Table 5, while PNMI1 of k-means is higher than RepCodec. However, for longer sequences of tokens, RepCodec has actually higher PNMIn than k-means. This implies that for longer sequences like a word or a sentence, RepCodec provides more deterministic information for the downstream decoder to learn the corresponding text. Therefore, the performance of RepCodec is higher than k-means. Number of Clusters. In Table 2c, we study how the performance of RepCodec for varying number of clusters K. With different K, RepCodec all outperforms k-means and VQ. Moreover, RepCodec is more robust against the changes of K than other two baselines. Even RepCodec with K = 512 has lower WER than k-means with K = 4096.

heading: 5 Conclusion
text: Interacting with LLMs through speech leads to an increased demand for speech tokenization. To this end, we propose RepCodec, a novel speech representation codec to convert continuous speech waveforms into discretized tokens. In contrast to previous methods, RepCodec employs a parametric network to preserve more semantic information of the speech representations. The extensive experiments demonstrate that semantic tokens extracted by RepCodec outperform the prevalent k-means algorithm in downstream tasks of both speech understanding and generation. Moreover, the experiments also demonstrate that RepCodec is a universal algorithm that can be applied to any speech encoders and to multiple languages.

heading: Limitations
text: While our method obviously outperforms the baselines, there is still a performance gap between using representations and discretized tokens. Future works may need to use more sophisticated codec architectures (e.g. Transformers) and objective functions (e.g. adversarial loss) to minimize the gap. In addition, because of the constraint of computational resources, we cannot scale our training data to larger dataset like GigaSpeech, and we only evaluate RepCodec on limited number of European languages like English, French and Spanish.

heading: A Detailed Architecture of RepCodec
text: In this section, we will introduce the architecture of RepCodec. We first provide the components of RepCodec and then show the specific architecture used for our method in Table 6. 1D Convolution Layer Conv1d(C, c, k, s). We denote Conv1d(C, c, k, s) as the non-causal 1D convolution layer with input channel C, output channel c, kernel size k and stride s without dilation. Residual Unit Res(C, k, s). We denote Res(C, k, s) as the residual unit used for both the encoder block and decoder block in Figure 2. Each Res(C, k, s) consists of two Conv1d(C, C, k, s) layers, with a residual path added across them. Encoder Block Enc(C, k, s). With the input having a channel of C, the encoder block Enc(C, k, s) consists of two Residual Units followed by a 1D convolution layer: Res(C, k, s) -> Res(C, k, s) -> Conv1d(C, C, k, s) Decoder Block Dec(C, k, s). With the input having a channel of C, the decoder block Dec(C, k, s) consists of a 1D convolution layer followed by two Residual Units: Conv1d(C, C, k, s) -> Res(C, k, s) ->

heading: Res(C, k, s)
text: Residual Vector Quantizer RVQ(M, K, C). RVQ(M, K, c) denotes a M -layer Residual Vector Quantizer (RVQ) with number of clusters K and codebook dimension C. When M = 1, RVQ is equivalent to VQ.

heading: B Details of the Experiments

heading: B.1 Training Speech Tokenizers.
text: K-means. We use the script from HuBERT1 to train the k-means model and perform k-means clustering, with all hyperparemeters unchanged. RepCodec. We train RepCodec for 200,000 steps. The batch size is 32 speech representations, each of which has 96 frames. We use Adam (Kingma and Ba, 2014) to optimize the model with a fixed learning rate 1×10−4 and β1 = 0.5, β2 = 0.9. We set λr = 45, λq = 1 and weight decay as 0 for all the experiments. VQ. We remove the encoder and decoder from RepCodec and train the model for 50,000 steps. Other hyperparameters are the same as RepCodec.
1https://github.com/facebookresearch/fairseq/ tree/main/examples/hubert/simple_kmeans

heading: B.2 Decoder-only ASR.
text: As shown in Figure 4, we use the Base Transformer decoder from fairseq (Ott et al., 2019) for decoderonly ASR. We fix the training steps to 100,000 for all the experiments. The decoder is optimized by Adam with β1 = 0.9, β2 = 0.999. The learning rate is warmed up by 5,000 steps to 1× 10−3 and then follow inverse square root decay to 0. We use cross entropy loss with smooth factor 0.1 to optimize the model, and we select the best checkpoint on the dev-clean set for further evaluation on the
test sets.
We use SentencePiece as the text tokenizer, and we train a new SentencePiece model on each dataset with all its transcripts. The vocabulary size is 5,000 for English dataset. For multilingual dataset in Table 2d, we jointly train the SentencePiece model of all three languages with a vocabulary size of 10,000. We implement beam search as the decoding strategy with size 5. For convenience, we do not add additional language model or length penalty.

heading: C Additional Experiments

heading: C.1 Addition Results of Decoder-Only ASR.
text: Table 7 and Table 10 show the WER of ASR on the dev sets. RepCodec still outperforms baselines by a large margin. It is worth noting the RepCodec achieves even lower WER on the layer 18 representation of data2vec large.

heading: C.2 Full Language Modelling v.s Conditional Language Modelling
text: Table 8 compares the WER on the test-clean and dev-clean subset of LibriSpeech when optimizing p(s,y) or p(y|s) for decoder-only ASR. For all three methods, full language modelling p(s,y) results in much higher WER than the conditional
language modelling p(y|s). Therefore, we choose to optimize p(y|s) for decoder-only ASR task.

heading: C.3 Ablation Study on λq and λr
text: As shown in Table 11, we change the weights of reconstruction loss λr to train the encoder of RepCodec and report its WER on the downstream task of ASR modeling with representation from the 18th layer of data2vec large model. The results show that RepCodec is robust to the changes of the weights of (λq, λr).

heading: C.4 Comparison of Scaling Ability of RepCodec and k-means
text: In Table 9, we present the results of downstream tasks using tokens generated by k-means and RepCodec that are both trained on full LibriSpeech 960h. We report WERs of the decoder-only ASR on test-clean and test-other subsets. When training on a larger amount of audio, RepCodec can achieve lower WERs than that trained on the trainclean-100 subset. However, we find that k-means is not very scalable because of its limited number of parameters. Even with larger training datasets, the WER of k-means doesn’t show improvement over data2vec features, and RepCodec outperforms k-means by a larger margin than the model trained on train-clean-100 subset.

heading: Future Work
text: Future works may need to use more sophisticated codec architectures (e. g. Transformers) and objective functions (e. g. adversarial loss) to minimize the gap. In addition, because of the constraint of computational resources, we cannot scale our training data to larger dataset like GigaSpeech, and we only evaluate RepCodec on limited number of European languages like English, French and Spanish