TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling

## Abstract

Abstract Recent efforts target spoken language models (SLMs) that not only listen but also speak for more natural human-LLM interaction.
Joint speech-text modeling is a promising direction to achieve this.
However, the effectiveness of recent speech tokens for joint modeling remains underexplored.
To address this, we introduce T ext- A ligned S peech T okenization and E mbedding (TASTE), a method that directly addresses the modality gap by aligning speech token with the corresponding text transcription during the tokenization stage. We propose a method that can achieve this through a attention-based aggregation mechanism and with speech reconstruction as the training objective. We conduct extensive experiments and show that TASTE can preserve essential paralinguistic information while dramatically reducing the token sequence length.
With TASTE, we perform straightforward joint spoken language modeling by using Low-Rank Adaptation on the pre-trained text LLM.
Experimental results show that TASTE-based SLMs perform comparable to previous work on SALMON and StoryCloze; while significantly outperform other pre-trained SLMs on speech continuation across subjective and objective evaluations.
To our knowledge, TASTE is the first end-to-end approach that utilizes a reconstruction objective to automatically learn a text-aligned speech tokenization and embedding suitable for spoken language modeling. Our demo, code, and model are available at https://mtkresearch.github.io/TASTE-SpokenLM.github.io .

## 1 Introduction

Figure: Figure 1: The concept overview. Conventional methods extract speech tokens solely from speech, inducing length-mismatch problem when conducting joint speech-text modeling. By taking dual modalities as input, we generate speech tokenization that is aligned with text, facilitating straightforward and effective joint modeling.
Refer to caption: x1.png

Learning a speech representation suitable for universal speech-processing tasks has long been a significant challenge [31, 48, 42].
Unlike text—which can be encoded discretely [39, 18, 19]—speech is a continuous waveform carrying layered information (acoustic, semantic, prosodic, etc.).
Recent neural self-supervised learning (SSL) methods move beyond filter-banks and MFCCs to encode raw audio into compact, high-dimensional embeddings that excel on discriminative tasks such as automatic speech recognition (ASR), emotion recognition, and speaker verification.
Despite these gains, learning representations for generative speech tasks remains an open and more complex problem that has begun to attract focused attention [45, 17, 44, 28].

Among the generative speech tasks, spoken language modeling (SLM) is an intriguing direction, aiming to create models that can not only listen but also speak.
Typically, building an SLM requires two stages: first, deriving speech tokenizations; second, training a language model on the speech tokens.
For the speech tokens, previous approaches either apply SSL-based representations following by discretization techniques [14, 21, 32, 11] or reuse units from neural codec models like EnCodec and SoundStream [5, 49, 20, 41].
Although autoregressive modeling with these speech tokens shows great potential in text-to-speech (TTS) [45, 47], previous SLMs that model only speech tokens [21, 32] have been shown to lack semantic fidelity [22].

To bridge this gap, one promising direction is to utilize text—which is rich in semantic—during spoken language modeling.
TWIST [11] shows that SLMs can benefit from initializing with text LLMs.
More recent work often conducts joint speech-text modeling on tokens of both modalities to facilitate the semantic coherence on the generated speech [33, 6, 46, 9].
Yet integrating text and speech tokens introduces a length-mismatch challenge, as speech token sequences are usually longer than their text counterparts.
Common remedies may include interleaving speech and text tokens [33] or inserting padding to synchronize sequence lengths between modalities [6, 46, 9].
However, these methods require either additional speech-text alignment or heuristic rules to enable joint modeling.

In this work, we introduce Text-Aligned Speech Tokenization and Embedding (TASTE), a special type of speech tokenization tailored for speech-text joint spoken language modeling.
By acknowledging that the length mismatch introduces additional complexity in joint modeling, we develop our speech token to be aligned with its corresponding text transcription tokens.
To achieve this, we first obtain the textual transcription of a speech with the ASR model; then we derive the speech token based on the transcription through a specialized cross-attention mechanism for speech reconstruction.
Note that the full process can be accomplished in an end-to-end manner, with no explicit speech-text alignment required.
Unlike previous speech tokens that are developed under a fixed stride with fixed down-sampling rate, our speech token has dynamic frequency as it is text-aligned. Figure [1](https://arxiv.org/html/2504.07053v2#S1.F1) shows an overall concept of TASTE, illustrating how our tokenization allows straightforward joint modeling.

To evaluate the effectiveness of TASTE, we first conduct extensive experiments on speech reconstruction.
Our results on LibriSpeech [34] show that TASTE not only resynthesizes speech in high quality, but also retains similarity to the original speech.
TASTE achieves high-end reconstruction at an extremely low bit rate ($\sim$150 bps); while the other comparable methods are often more than thousands of bps.
More intriguingly, we demonstrate that TASTE allows simple text-aligned speech editing.
By exchanging the partial text-aligned speech tokens from two different utterances with the same content, we demonstrate that the paralinguistic information such as duration and tone can be exchanged precisely following the words being exchanged, resulting in natural edited speech.

On the other hand, we demonstrate that TASTE successfully allows effective spoken language modeling.
We perform straightforward joint modeling with TASTE under Low-Rank Adaptation [15].
We first perform speech continuation experiments with 3-second speech prompts given.
The evaluation is three-fold.
We use GPT-4o for evaluating the semantic aspect; UTMOS [38] for the acoustic aspect; and the human listening test for the general evaluation.
Results show that our SLMs not only generates natural, meaningful speech continuations, but also outperforms the other 7B pre-trained SLMs across all the continuation evaluation aspects with 1.3B parameters.
We also evaluate our SLMs on two benchmarks, SALMON [25] and StoryCloze [11] and our results show that our SLMs achieve comparable performance compared to the other speech-text joint modeling methods.
Moreover, we show that our pretrained SLM can perform spoken question answering under few-shot scenario.

In summary, we derive TASTE, a text-aligned speech tokenization that allows effective joint speech-text spoken language modeling. By aligning the speech tokenization with its text counterpart during the tokenization stage, TASTE enables straightforward modeling. To our best knowledge, we are the first one to utilize the reconstruction objective to automatically derive a text-aligned speech tokenization and embedding that is suitable for joint speech-text spoken language modeling. Our demo is available at [https://mtkresearch.github.io/TASTE-SpokenLM.github.io](https://mtkresearch.github.io/TASTE-SpokenLM.github.io).

## 2 Method

We propose text-aligned speech tokenization and embedding (TASTE) to facilitate effective joint speech-text spoken language modeling.
Here, we first introduce how we derive our tokenization—TASTE—in Section [2.1](https://arxiv.org/html/2504.07053v2#S2.SS1), and then discuss how we use TASTE for spoken language modeling (§ [2.2](https://arxiv.org/html/2504.07053v2#S2.SS2)).

Figure: Figure 2: The overall framework of our text-aligned speech tokenization and embedding. The left side illustrate the process of obtaining the TASTE tokenization $\hat{\bm{z}}$, detailed in Section [2.1.1](https://arxiv.org/html/2504.07053v2#S2.SS1.SSS1); while the right side demonstrate how we reconstruct the speech with TASTE (Section [2.1.2](https://arxiv.org/html/2504.07053v2#S2.SS1.SSS2)). The training objective for our speech reconstruction is discussed in Section [2.1.3](https://arxiv.org/html/2504.07053v2#S2.SS1.SSS3).
Refer to caption: x2.png

### 2.1 Building TASTE

As depicted in Figure [2](https://arxiv.org/html/2504.07053v2#S2.F2), TASTE is comprised of the two main components: the text-aligned speech tokenizer (§ [2.1.1](https://arxiv.org/html/2504.07053v2#S2.SS1.SSS1)) that produces the text-aligned speech tokenization; and the speech decoder (§ [2.1.2](https://arxiv.org/html/2504.07053v2#S2.SS1.SSS2)) to reconstruct speech based on the text token and the TASTE speech token aligned with it.
The training objective of speech reconstruction is described in Section [2.1.3](https://arxiv.org/html/2504.07053v2#S2.SS1.SSS3).

#### 2.1.1 TASTE Speech Tokenizer

In TASTE, the speech tokenizer, denoted as $\mathrm{Tokenizer}(\cdot)$, is designed to generate the text-aligned speech tokenization and embedding with the speech-text pair $X=(\bm{u},\bm{v})$ taken as input, where $\bm{v}$ represents the textual transcription of the speech utterance $\bm{u}$, which can be easily obtained through an automatic speech recognition (ASR) system.
Recent developments in robust and efficient ASR ([35, 10]) allow us to focus on discussing how to derive the text-aligned speech token effectively by assuming that $\bm{v}$ is of sufficient quality. The TASTE speech tokenizer is composed of three major components: an encoder, an aggregator, and a quantizer.

The encoder $\mathrm{Encoder}(\cdot)$ contains $L$ layers of Transformer ([43]) encoder blocks and is used to extract high-dimensional speech representation.
We employ the pre-trained Whisper ASR encoder [35] as our speech encoder, and it is frozen during training.
For an input speech utterance $\bm{u}$, the encoder produces a sequence of hidden states from each layer $[\bm{h}^{(1)},\bm{h}^{(2)},\ldots,\bm{h}^{(L)}]$.
In our experiments, we retain the last hidden layer representation $\bm{h}^{(L)}$ and the shallow representation $\bm{h}^{(l)}$ from the first half of the hidden representations of the encoder for later usage, denoted as:

$$ ${\bm{h}^{(L)},\bm{h}^{(l)}}\ =\mathrm{Encoder}(\bm{u}),\quad\text{ where }1 \leq l\leq\bigl{\lfloor}\tfrac{L}{2}\bigr{\rfloor}.$ $$

Note that both of the hidden representations $\bm{h}^{(L)},\bm{h}^{(l)}\in\mathbb{R}^{T\times d_{h}}$ have their length denoted as $T$ and the hidden dimension indicated by $d_{h}$.

The hidden representations extracted from the encoder are then passed to the aggregator.
The aggregator is designed to obtain a more compressed speech representation $\bm{z}$ that is aligned in length with the text transcription $\bm{v}$.
Consider that $\bm{v}=[v_{1},v_{2},\ldots,v_{N}],v_{i}\in\mathbb{V}$ is a text token sequence with length $N$, the input and output of the aggregator can be denoted as:

$$ $\bm{z}=\mathrm{Aggregator}(\bm{v},\bm{h}^{(L)},\bm{h}^{(l)}),\text{ where }\bm {z}\in\mathbb{R}^{N\times d_{z}},\bm{v}\in\mathbb{V}^{N},\text{ and }\bm{h}^{( L)},\bm{h}^{(l)}\in\mathbb{R}^{T\times d_{h}}.$ $$

To make the speech representation $\bm{z}$ text-aligned, we conduct a simple yet effective attention mechanism based on the three inputs.
Consider that the original multi-head attention in [43] is denoted as $\mathrm{MultiHead}(Q,K,V)$, our first layer attention in the aggregator takes:

$$ ${Q}=\text{text transcription }\bm{v},\quad{K}=\text{encoder last hidden }\bm{h }^{(L)},\quad{V}=\text{encoder shallow hidden }\bm{h}^{(l)}.$ $$

By doing so, the length of our first multi-head attention output should follow the text transcription $\bm{v}$.
Note that the query of the following layers becomes the output from the previous layer.
In addition, intuitions of using the encoder’s last hidden representation as keys, and the shallow hidden representation as values can be described as follows:
1) In Transformer-based ASR models, the last hidden states often encode rich speech-text alignment cues; sometimes the cross-attention weight matrices can even be exploited as soft word-alignment maps [35, 10].
2) The shallow representation has been shown to support high-quality speech reconstruction even when the quantization is applied [7, 8].
Based on the above observations, we design our aggregator that can use the soft attention maps obtained from last encoder representations and the text transcriptions, to aggregate the shallow encoder representations that is beneficial for high-end speech reconstruction.

After getting the text-aligned representation, the quantizer $\mathrm{Quantizer}(\cdot)$ is adopted to discretize the text-aligned representation.
We use the residual vector quantization (RVQ) to allow coarse-to-fine quantization.
Given the text-aligned speech representation $\bm{z}$ and the quantizer containing $R$ residual vector quantization layers, we generate:

$$ $\bm{q},\hat{\bm{z}}=\mathrm{Quantizer}(\bm{z}),\text{ \qquad}\bm{q}=[\bm{q}^{( 1)},\bm{q}^{(2)},\ldots,\bm{q}^{(R)}],\quad\hat{\bm{z}}=\sum^{R}_{r=1}\hat{\bm {z}}^{(r)}$ (1) $$

where each $\bm{q}^{(r)}\in\mathbb{C}^{N}$ denotes the $r$-th layer code sequence with code set $\mathbb{C}$; and the quantized embedding $\hat{\bm{z}}$ to be the summation over each layer of the codebook vectors.
Note that both of the code sequence and the quantized speech embedding $\hat{\bm{z}}$ are text-aligned, with the lengths to be $N$.

#### 2.1.2 TASTE Speech Decoder

The speech decoder aims to perform speech reconstruction conditioned on the text token sequence and the text-aligned speech tokenization.
As shown in Figure [2](https://arxiv.org/html/2504.07053v2#S2.F2), the text and speech tokens are aligned in lengths and being fed into the speech decoder after weighted sum in an autoregressive manner.
The speech decoder is composed of the two components: the unit decoder and the unit-to-speech vocoder.

The unit decoder $\mathrm{UnitDecoder}(\cdot)$ is a Transformer-based decoder that takes the text token sequence $\bm{v}$ and the aligned speech embedding $\hat{\bm{z}}$ as input and predicts the speech unit $\bm{y}$ for reconstruction:

$$ $\bm{y}=\mathrm{UnitDecoder}(\hat{\bm{z}},\bm{v}).$ (2) $$

Note that the additional speaker embedding is also taken as input to facilitate global speaker voice control in our spoken language models [16].
After we generating the speech unit $\bm{y}$, we use a unit-to-speech vocoder to further transform the unit into the reconstructed speech.

#### 2.1.3 Training Objective

Similar to other reconstruction-based speech tokens [51, 24], we derive TASTE by training it for speech resynthesis.
To achieve this, we extract the speech unit $\bm{y}^{\text{target}}$ with length $T^{\prime}$ from the original speech $u$ as the target unit for our speech tokenizer and speech decoder.
Given the text transcription $\bm{v}$, the TASTE speech embedding $\hat{\bm{z}}$, and the unit from the original speech $\bm{y}^{\text{target}}$ as the target, the speech reconstruction through the tokenizer and the unit decoder parametrized by $\theta$ under the next prediction schema can be considered as minimizing the cross-entropy loss below:

$$ $\displaystyle\mathcal{L}_{\text{ce}}(\theta)=\frac{1}{|T^{\prime}|}\sum_{t=1}^ {T^{\prime}}-\text{log }p_{\theta}({y}^{\text{target}}_{t}\big{|}\hat{\bm{z}}, \bm{v};\bm{y}^{\text{target}}_{<t})$ (3) $$

On the other hand, we employ the quantization loss as well to tokenize the continuous representation $\bm{z}$ extracted from the encoder-aggregator.
Following prior works [5, 49], given that $\bm{z}^{(r)}$ is the $r$-th residual and $\hat{\bm{z}}^{(r)}$ indicates the $r$-th quantized residual, the the commitment loss is defined as:

$$ $\displaystyle\mathcal{L}_{\text{rvq}}(\theta)=\sum_{r=1}^{R}\|\bm{z}^{(r)}- \hat{\bm{z}}^{(r)}\|.$ (4) $$

By summation over both losses, we formulate the overall loss for training TASTE as:

$$ $\displaystyle\mathcal{L}_{\text{taste}}=\mathcal{L}_{\text{ce}}+\mathcal{L}_{ \text{rvq}}\text{.}$ (5) $$

Note that to allow gradient to back-propagate from the unit decoder through the tokenizer, the straight-through estimation technique is applied towards the quantization process during traning.

### 2.2 TASTE for Spoken Language Modeling

Next, we describe how we conduct effective spoken language modeling with TASTE.
Following previous work [11, 33], we perform pre-training on speech data.
The text transcription of the speech data is also used for joint speech-text pre-training of our text-aligned spoken language model (TASLM).
Since TASTE tokenization already aligns with the text token sequence, we can conduct a straightforward joint modeling, as illustrated in Figure [1](https://arxiv.org/html/2504.07053v2#S1.F1).
To demonstrate the robustness of TASTE, we perform two types of text-aligned spoken language modeling.
First, we build $\rm{TASLM}_{\text{token}}$ over our text-aligned speech token $\bm{q}$, discussed in Section [2.2.1](https://arxiv.org/html/2504.07053v2#S2.SS2.SSS1).
Then, we show how we build $\rm{TASLM}_{\text{emb}}$ with our text-aligned speech embedding $\hat{\bm{z}}$, detailed in Section [2.2.2](https://arxiv.org/html/2504.07053v2#S2.SS2.SSS2).

#### 2.2.1 Modeling TASTE Token

As our speech tokens derived from the RVQ quantizer contain $R$ layers of codes, we employ $R$ linear heads for multi-head prediction in our $\rm{TASLM}_{\text{token}}$.
Namely, the $\rm{TASLM}_{\text{token}}$ simultaneously predicts the next text token and the corresponding $R$ layers of speech tokens in each step.
The overall training objective follows the original next token prediction scheme, but with multiple predictions across modalities at each step.
Specifically, given the text transcription $\bm{v}$ and $R$ layers of quantized RVQ codes $\bm{q}$, the multi-head next-token prediction training objective can be formulated as:

$$ $\displaystyle\mathcal{L}_{\text{token}}(\phi)=\frac{1}{|N|}\sum_{i=1}^{N}\Bigl {(}-\text{log }p^{\text{text}}_{\phi}\bigl{(}{v}_{i}\big{|}\bm{v}_{<i},\bm{q}_ {<i}\bigr{)}+\sum_{r=1}^{R}-\text{log }p^{\text{(r)}}_{\phi}\big{(}{q}^{\text{ (r)}}_{i}\big{|}\bm{v}_{<i},\bm{q}_{<i}\bigr{)}\Bigr{)},$ (6) $$

with $\phi$ represents the parameter of the $\mathrm{TASLM}_{\text{token}}$, and $p^{(r)}$ is the $r$-th probability prediction for the $r$-th RVQ code.
As for inference, we directly sample the codes and the text simultaneously, and transform the codes into the corresponding embedding for the speech decoder to generate speech.

#### 2.2.2 Modeling TASTE Embedding

Besides the token code sets, recent progress on latent modeling [17, 28] motivates us to conduct experiments on modeling our text-aligned speech embedding.
Referencing MELLE [28], we employ a linear layer that predicts the mean vector $\mu_{i}$ and a log-magnitude variance vector $\text{log }\sigma^{2}_{i}$, where $i$ indicates the $i$-th frame of the sequence.
And the final predicted latent of frame $i$ is denoted as $e_{i}=\mu_{i}+\sigma_{i}\odot\epsilon$, where $\epsilon\sim\mathcal{N}(0,I)$.
Following MELLE, the straight-through estimator is applied to allow gradients to back-propagate properly during training.

To facilitate latent prediction, we apply the regularization loss and the Kullback-Leibler (KL) divergence loss druing training, which is described as follows:

$$ $\displaystyle\mathcal{L}_{\text{reg}}(\psi)=\|\bm{e}_{\psi}-\hat{\bm{z}}\|^{2} _{2},\quad\mathcal{L}_{\text{KL}}=\frac{1}{2}\sum^{N}_{i=1}\sum^{d_{z}}_{j=1} \bigl{(}\sigma_{i}[j]+(\mu_{i}[j]-\hat{z}_{i}[j])^{2})-1-\text{log }\sigma^{2} _{i}[j]\bigl{)},$ (7) $$

where $\psi$ indicates the parameter of $\mathrm{TASLM}_{\text{emb}}$, and $d_{z}$ is the dimension of our text-aligned embedding $\hat{\bm{z}}$.
The regularization loss $\mathcal{L}_{\text{reg}}$ is adopted to predict close latent towards the target embedding $\hat{\bm{z}}$.
The KL divergence loss calculates the KL divergence between the predicted latent distribution and the target distribution.
Following MELLE, we select the target distribution to be $\mathcal{N}({\hat{\bm{z}}}_{i},I)$.
This allows simplification of $\mathcal{L}_{\text{KL}}$, which can then be approximated with the predicted vectors $\mu_{i},\sigma_{i},\text{and the target embedding }\hat{\bm{z}}_{i}$.
Finally, the overall loss along with the text loss is described as:

$$ $\displaystyle\mathcal{L}_{\text{emb}}(\psi)=\lambda_{\text{reg}}\cdot\mathcal{ L}_{\text{reg}}+\lambda_{\text{KL}}\cdot\mathcal{L}_{\text{KL}}+\frac{1}{|N|} \sum_{i=1}^{N}-\text{log }p^{\text{text}}_{\psi}\bigl{(}{v}_{i}\big{|}\bm{v}_{ <i},\hat{\bm{z}}_{<i}\bigr{)},$ (8) $$

where $\lambda_{\text{reg}},\lambda_{\text{KL}}$ to be the weighted coefficients of the two losses, respectively.

## 3 Experiment Setup

### 3.1 Model Configuration

For our TASTE speech tokenizer, we initialize our encoder from Whisper [35].
Specifically, we use whisper-large-v3 for our initialization.
By doing so, we can reduce computational cost between obtaining the ASR transcription and extracting the TASTE tokenization with the TASTE encoder frozen during training.
On the other hand, we use the S3 token from CosyVoice [7] as the target unit for speech reconstruction.
Since their speech tokenization facilitates additional speaker embedding, we follow the same procedure to obtain one.
Adding speaker embedding allows global speaker voice control, which is a reasonable and useful scenario for spoken language models.
The unit-to-speech vocoder is comprised of a flow model [23, 27] and a HifiGAN.
We use the published pre-trained ones from  [7], and they are not involved in our training.
For the quantizer, we set the RVQ layer $R=4$, the codebook size $512$, and the codebook dimension to be $256$.
For the spoken language modeling, we follow previous work [11, 33, 6, 22] and initialize our spoken language model from a text LLM.
However, this introduces the vocabulary mismatch problem between the ASR and LLM.
We resolve this issue by using word-level TASTE tokenization and embedding, which is detailed in Appendix [A.2](https://arxiv.org/html/2504.07053v2#A1.SS2).
Moreover, we conduct Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning of our TASLMs.
We set the corresponding hyperparameters rank $r=64$ and $\alpha=128$.

### 3.2 Dataset

We use two datasets–Emilia and LibriTTS–as our training datasets.
Emilia [12] is an in-the-wild dataset where the speech is web-scaled and the transcriptions are pseudo-labeled.
We use only the English subset of this multi-lingual corpus, which is about 40,000 hours.
LibriTTS [50] is a reading-style corpus based on LibriSpeech [34].
We use all the training splits in LibriTTS for training, which is approximately 600 hours of speech.
In addition, the test-clean split in LibriSpeech is used for evaluation purposes for our TASTE tokenizer and TASLMs.

## 4 Result

We separate the evaluation into two phases: Section [4.1](https://arxiv.org/html/2504.07053v2#S4.SS1) shows the results regarding our TASTE tokenization; while Section [4.2](https://arxiv.org/html/2504.07053v2#S4.SS2) evaluates our TASLM across multiple aspects, including acoustic, semantic, and continuation.
For clarity, the metrics are introduced within each section.

### 4.1 Results of TASTE Tokenization

#### 4.1.1 Speech Reconstruction Evaluation

We first represent the speech reconstruction evaluation results.
For comprehensive evaluation, we use different metrics, including the reference-free metrics for quality assessment, and the reference-based metrics for evaluating the similarity between the reconstructed and the original speech.

##### Quality Assessment

We use ASR-WER, UTMOS [38], and DNS-MOS [37] as our metrics for evaluating the speech quality.
For ASR-WER, we use HuBERT-Large [14] as the ASR model to transcribe the speech, and then calculate the word-error rate (WER) on the transcription. (^1^11[https://huggingface.co/facebook/hubert-large-ls960-ft](https://huggingface.co/facebook/hubert-large-ls960-ft))
UTMOS and DNS-MOS are both neural-based MOS predictors.
While both evaluate the speech quality, the design purpose of DNS-MOS makes it more suitable for evaluation regarding the noise levels.

##### Similarity Assessment

For similarity, we measure ViSQOL, duration consistency (Drtn. Con.), speaker similarity (Spkr. Sim.) and the MUSHRA test as human evaluation.
We use ViSQOL [4]is a production-ready tool that predicts speech quality via spectro-temporal image similarity comparisons.
For the duration consistency, we first get the word-level alignment of the transcriptions of the original and the reconstructed speech using Montreal Forced Aligner [26]; then we calculate if the duration between each of the same words is matched under a preset tolerance window, which is set to 50 milliseconds.
For MUSHRA human listening test, we reference the original protocal [40] to instruct
evaluators to rate the similarity and quality on a scale of 1 to 100 with reference given.

Table: Table 1: The speech tokenization evaluation results on the test-clean split of LibriTTS. The evaluation is separated into the Quality and the Similarity assessments, as introduced in Section [4.1.1](https://arxiv.org/html/2504.07053v2#S4.SS1.SSS1). We use gray text to indicate the worst-performing methods in each metric.

##### Speech Reconstruction Results

The evaluation results of our speech reconstruction on LibriSpeech are shown in Table [1](https://arxiv.org/html/2504.07053v2#S4.T1).
We highlight that our TASTE speech tokenization has the lowest bitrate among all the other speech tokenization methods.
Note that since the speech tokenization is dynamic in frequency, we estimate our bitrate based on calculating the overall token count and the duration over the testing set.
Despite the low bitrate, we generally attain much better performance comparing with the worst performing methods (gray text in the table) across each metric.
Moreover, on the quality assessment, our MOS prediction scores are the second highest and even surpasses the ground truth, showcasing the reconstructed speech is of high quality.
Next, we focus on the results of the similarity assessment.
As for the duration consistency, we score the second-worst performance comparing with the other methods.
We attribute this to the fact that our tokenization compress the sequence in a very dynamic way.
Despite that, we still outperform the text-only method with a large margin, perform close towards other speech tokenization methods which all have a fixed down-sampling rate.
Lastly, our method attains the second-highest MUSHRA score (excluding the ground-truth anchor).
This highlights TASTE’s effectiveness: even without reproducing every microscopic detail, it still yields perceptually high-quality speech in human listening tests.
Overall, TASTE carries rich paralinguistic information, facilitating high-end speech reconstruction under an extremely low bitrate.

#### 4.1.2 TASTE for Text-Aligned Speech Editing

After comprehensive speech-reconstruction experiments, we show that TASTE can also perform text-aligned speech editing.
Suppose we have two utterances with the same transcript but different paralinguistic characteristics.
By exchanging their TASTE token sequences word by word, we ask whether the associated paralinguistic traits are transferred as well.
To make the effect easy to see, we choose utterances that differ mainly in speaking rate and focus on duration changes.
The overall text-aligned editing procedure is describe as follows:
1) Extract the TASTE tokens $\hat{\bm{z}}^{\text{orig}}$ for each source utterance 2) Swap the tokens at the desired text positions, resulting in edited TASTE tokens $\hat{\bm{z}}^{\text{edit}}$. 3) Decode the edited token sequence $\hat{\bm{z}}^{\text{edit}}$ back to speech.
In Figure [3](https://arxiv.org/html/2504.07053v2#S4.F3), we present the alignments obtained from MFA [26] of the original speech and the speech after editing, and compare them horizontally.
As shown in the figure, words whose tokens were swapped exhibit clear duration shifts, while the untouched words keep their original timing—evidence that TASTE enables precise, text-aligned manipulation.
Additional examples that target other paralinguistic dimensions are provided on our demo page.

Figure: Figure 3: An illustration of TASTE for text-aligned speech editing. On the left shows the process of our text-aligned speech editing. We first extract the TASTE tokens; swap the tokens partially; and then decode the edited TASTE tokens into edited speech. On the right shows an example visualization. Only the durations of the words with exchanged TASTE tokens show significant difference.
Refer to caption: x3.png

### 4.2 Evaluating Text-Aligned Spoken Language Modeling

To provide a comprehensive evaluation of our text-aligned spoken language modeling (TASLM), we first compare our pre-trained SLM with other methods through speech continuation and likelihood-based benchmarks in Section [4.2.1](https://arxiv.org/html/2504.07053v2#S4.SS2.SSS1).
Then, to further investigate the understanding capabilities, we conduct evaluation with spoken question answering in Section [4.2.2](https://arxiv.org/html/2504.07053v2#S4.SS2.SSS2).

Table: Table 2: Pretrained SLM speech continuation and likelihood-based next-speech selection results. The superscripts at the bottom of the table indicate the base models used by each SLM, indicated by superscripts. Cascade models refer to the pipeline with ASR (whisper-large-v3), text continuation by LMs, and TTS (CosyVoice). This comparison evaluates SLMs and cascade models in continuation evaluation. As shown in the table, TASLM tends to preserve the semantic capabilities of LMs.

#### 4.2.1 Comparing TASLM with Pretrained SLMs

##### Speech Continuation Evaluation

A typical way to evaluate the pre-trained SLM is by performing conditional generation.
Following previous work [11, 22], we use the 3-second prompt speech from the LibriSpeech test-clean.
To evaluate the quality of the continuations generated by the SLMs, we employ GPT-4o to assign MOS scores regarding the transcribed speech continuation using ASR, focusing on the semantic coherence of the continuation.
In addition, we compute UTMOS as for evaluating the speech quality and naturalness.
Last but not least, we conduct human listening test, in which each evaluator is asked to give a MOS score regarding the overall performance of the generated speech continuation.
The details of the instructions for GPT-4o and human are in Appendix.

##### Likelihood-Based Evaluation

Following previous work [11, 33, 22], we also evaluate our SLMs through likelihood-based benchmarks, where the accuracy score is based on whether the model chooses the correct continuation from the two given speech utterances based on its output likelihoods.
We adopt two established benchmarks SALMON [25] and spoken StoryCloze [11, 29], which covers the acoustic aspect and the semantic aspect, respectively.
Since both benchmarks contain multiple tasks,
we report the average accuracy across these tasks within each benchmark for simplicity.
The detailed results are in Appendix [A.5.1](https://arxiv.org/html/2504.07053v2#A1.SS5.SSS1) for the interested readers.
We also report the mean of the SALMON and StoryCloze as an overall assessment for both aspects.

##### Results

The results of TASLM comparing to other pre-trained SLM are in Table [2](https://arxiv.org/html/2504.07053v2#S4.T2).
First, we highlight that our TASLMs have achieved significantly higher scores on speech continuation across human and machine evaluations; and good performance on the likelihood-based benchmarks.
Note that our base language model contains only 1.3 billion parameters, showing the effectiveness of using TASTE for joint modeling.
Compared to the cascade method that has the same base model (first row), our $\rm{TASLM}_{\text{emb}}$ scores comparable on GPT-4o but better on human MOS.
This indicates that our generated speech is more natural than the cascade one that utilizes TTS for synthesis.
Next, our TASLM is the only SLM that not only maintains but even surpasses the performance of its corresponding text-base model.
Moreover, we demonstrate that directly using the S3 token for joint modeling following [46] does not yield performance comparable in any aspect.
The result further strengthen the intuition behind TASTE (mitigating the length-mismatch during tokenization stage facilitates effective joint spoken language modeling).

**Table 3: Evaluation of spoken question answering. Performance across modalities is compared row-wise, where T denotes text and A denotes audio.**
| Method | Mode | Web Q. | LLaMA-Q. |
| --- | --- | --- | --- |
| Mini-Omni 0.5B(T$\xrightarrow{}$T) | T | 21.3 | 39.0 |
| Mini-Omni 0.5B [46] | T+A | 4.5 | 11.6 |
| \hdashlineHelium 7B (text) | T | 32.3 | 75.0 |
| Moshi 7B [6] | T+A | 26.6 | 62.3 |
| \hdashlineLLaMA3.1-8B-Instruct | T | 60.4 | 71.7 |
| Llama-Omni-8B [9] | T+A | 35.5 | 67.3 |
| \hdashlineLLaMA3.2-1B^† | T | 24.0 | 51.0 |
| TASLM 1B (embed.)^† | T+A | 27.1 | 57.6 |
| ^†We apply few-shot learning to facilitate question answering. |  |  |  |

#### 4.2.2 TASLM for Spoken Question Answering

Following [6], we conduct evaluation on spoken question answering to investigate the understanding ability of our TASLM.
For this experiment, we use the $\rm{TASLM}_{\text{emb}}$ for simplicity.
We compare our pre-trained only SLM with other instruction-finetuned joint SLMs such as Mini-Omni [46], Moshi [6], and Llama-Omni [9]. We use two spoken question answering benchmarks, Web Questions [2] and LLaMA-Questions [30], following [30]. We report the accuracy of answer containment.
For fairness, we report not only the performance of the speech-text joint SLMs, but also the base text LLM they used if applicable.
Our results indicate that our TASLM is the only method that does not degrade the corresponding text base LLM.
We attribute the phenomenon to the effectiveness of our TASTE tokenization for joint speech-text modeling.

## 5 Conclusion

In this work, we propose Text-Aligned Speech Tokenization and Embedding (TASTE), to facilitate joint speech-text spoken language modeling.
By aggregating proper encoder representation through the specialized cross-attention mechanism and taking the ASR model as initialization, we make the speech tokenization text-aligned in an end-to-end manner with no explicit word alignment required.
We conduct extensive evaluation on our TASTE tokenizer.
Our results show that TASTE allows high quality speech reconstruction at an extremely low bitrate.
With our text-aligned speech tokenization and embedding, joint speech-text modeling becomes straightforward and effective.
Our experimental results indicate that TASTE enables turning a text LLM into a spoken one with the simple parameter-efficient finetuning technique applied.

##### Limitation

Several limitations of our current work point to promising avenues for future research.
First, neither our TASTE tokenization nor the text-aligned SLM has been optimized for time efficiency; developing a low-latency, streaming variant remains future work.
Second, we have evaluated TASTE only on English data—its portability to other languages deserves thorough investigation.
Third, although our pretrained SLM generates high-quality continuations, it does not yet support robust turn-taking or instruction-following behavior, both of which are essential for truly interactive systems.

## References

- Baevski et al. [2020]
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.
wav2vec 2.0: A framework for self-supervised learning of speech representations.
In *Advances in Neural Information Processing Systems*, 2020.
- Berant et al. [2013]
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang.
Semantic parsing on freebase from question-answer pairs.
In *Proceedings of the 2013 conference on empirical methods in natural language processing*, 2013.
- Chen and Rudnicky [2022]
Li-Wei Chen and Alexander Rudnicky.
Fine-grained style control in transformer-based text-to-speech synthesis.
In *ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2022.
- Chinen et al. [2020]
Michael Chinen, Felicia SC Lim, Jan Skoglund, Nikita Gureev, Feargus O’Gorman, and Andrew Hines.
Visqol v3: An open source production ready objective speech and audio metric.
In *2020 twelfth international conference on quality of multimedia experience (QoMEX)*, 2020.
- Défossez et al. [2023]
Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi.
High fidelity neural audio compression.
*Transactions on Machine Learning Research*, 2023.
- Défossez et al. [2024]
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour.
Moshi: a speech-text foundation model for real-time dialogue.
*arXiv preprint arXiv:2410.00037*, 2024.
- Du et al. [2024a]
Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al.
Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens.
*arXiv preprint arXiv:2407.05407*, 2024a.
- Du et al. [2024b]
Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al.
Cosyvoice 2: Scalable streaming speech synthesis with large language models.
*CoRR*, 2024b.
- Fang et al. [2024]
Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng.
Llama-omni: Seamless speech interaction with large language models.
*CoRR*, 2024.
- Gandhi et al. [2023]
Sanchit Gandhi, Patrick von Platen, and Alexander M Rush.
Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling.
*arXiv preprint arXiv:2311.00430*, 2023.
- Hassid et al. [2023]
Michael Hassid, Tal Remez, Tu Anh Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Defossez, Gabriel Synnaeve, Emmanuel Dupoux, et al.
Textually pretrained speech language models.
*Advances in Neural Information Processing Systems*, 2023.
- He et al. [2024]
Haorui He, Zengqiang Shang, Chaoren Wang, Xuyuan Li, Yicheng Gu, Hua Hua, Liwei Liu, Chen Yang, Jiaqi Li, Peiyang Shi, et al.
Emilia: An extensive, multilingual, and diverse speech dataset for large-scale speech generation.
In *2024 IEEE Spoken Language Technology Workshop (SLT)*, 2024.
- Hsu et al. [2024]
Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen.
Liger kernel: Efficient triton kernels for llm training.
*arXiv preprint arXiv:2410.10989*, 2024.
- Hsu et al. [2021]
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed.
Hubert: Self-supervised speech representation learning by masked prediction of hidden units.
*IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 2021.
- Hu et al. [2021]
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models.
*arXiv preprint arXiv:2106.09685*, 2021.
- Ju et al. [2024]
Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al.
Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models.
*International Conference on Machine Learning*, 2024.
- Kim et al. [2024]
Jaehyeon Kim, Keon Lee, Seungjun Chung, and Jaewoong Cho.
Clam-tts: Improving neural codec language model for zero-shot text-to-speech.
*ICLR*, 2024.
- Kudo [2018]
Taku Kudo.
Subword regularization: Improving neural network translation models with multiple subword candidates.
*Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, 2018.
- Kudo and Richardson [2018]
Taku Kudo and John Richardson.
Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.
*Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, 2018.
- Kumar et al. [2023]
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar.
High-fidelity audio compression with improved rvqgan.
*Advances in Neural Information Processing Systems*, 2023.
- Lakhotia et al. [2021]
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al.
On generative spoken language modeling from raw audio.
*Transactions of the Association for Computational Linguistics*, 2021.
- Lin et al. [2024]
Guan-Ting Lin, Prashanth Gurunath Shivakumar, Aditya Gourav, Yile Gu, Ankur Gandhe, Hung-yi Lee, and Ivan Bulyko.
Align-slm: Textless spoken language models with reinforcement learning from ai feedback.
*arXiv preprint arXiv:2411.01834*, 2024.
- Lipman et al. [2022]
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le.
Flow matching for generative modeling.
*The Eleventh International Conference on Learning Representations*, 2022.
- Liu et al. [2025]
Alexander H Liu, Sang-gil Lee, Chao-Han Huck Yang, Yuan Gong, Yu-Chiang Frank Wang, James R Glass, Rafael Valle, and Bryan Catanzaro.
Uniwav: Towards unified pre-training for speech representation learning and generation.
*The Thirteenth International Conference on Learning Representations*, 2025.
- Maimon et al. [2024]
Gallil Maimon, Amit Roth, and Yossi Adi.
Salmon: A suite for acoustic language model evaluation.
*arXiv preprint arXiv:2409.07437*, 2024.
- McAuliffe et al. [2017]
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger.
Montreal forced aligner: Trainable text-speech alignment using kaldi.
In *Interspeech 2017*, 2017.
- Mehta et al. [2022]
Shivam Mehta, Ambika Kirkland, Harm Lameris, Jonas Beskow, Éva Székely, and Gustav Eje Henter.
Overflow: Putting flows on top of neural transducers for better tts.
*Interspeech 2023*, 2022.
- Meng et al. [2024]
Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, et al.
Autoregressive speech synthesis without vector quantization.
*CoRR*, 2024.
- Mostafazadeh et al. [2016]
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen.
A corpus and evaluation framework for deeper understanding of commonsense stories.
*Proceedings of NAACL-HLT*, 2016.
- Nachmani et al. [2024]
Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich.
Spoken question answering and speech continuation using spectrogram-powered llm.
In *The Twelfth International Conference on Learning Representations*, 2024.
- Nguyen et al. [2020]
Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux.
The zero resource speech benchmark 2021: Metrics and baselines for unsupervised spoken language modeling.
*NeuRIPS Workshop on Self-Supervised Learning for Speech and Audio Processing*, 2020.
- Nguyen et al. [2023]
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abdelrahman Mohamed, et al.
Generative spoken dialogue language modeling.
*Transactions of the Association for Computational Linguistics*, 2023.
- Nguyen et al. [2025]
Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R Costa-Jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, et al.
Spirit-lm: Interleaved spoken and written language model.
*Transactions of the Association for Computational Linguistics*, 2025.
- Panayotov et al. [2015]
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur.
Librispeech: An asr corpus based on public domain audio books.
In *2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2015.
- Radford et al. [2023]
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.
Robust speech recognition via large-scale weak supervision.
In *International conference on machine learning*, 2023.
- Rasley et al. [2020]
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.
Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.
In *Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery and data mining*, 2020.
- Reddy et al. [2021]
Chandan KA Reddy, Vishak Gopal, and Ross Cutler.
Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors.
In *ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, 2021.
- Saeki et al. [2022]
Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari.
Utmos: Utokyo-sarulab system for voicemos challenge 2022.
*Interspeech 2022*, 2022.
- Sennrich et al. [2016]
Rico Sennrich, Barry Haddow, and Alexandra Birch.
Neural machine translation of rare words with subword units.
*Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, 2016.
- Series [2014]
B Series.
Method for the subjective assessment of intermediate quality level of audio systems.
*International Telecommunication Union Radiocommunication Assembly*, 2014.
- Siuzdak et al. [2024]
Hubert Siuzdak, Florian Grötschla, and Luca A Lanzendörfer.
Snac: Multi-scale neural audio codec.
*Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation*, 2024.
- Tsai et al. [2022]
Hsiang-Sheng Tsai, Heng-Jui Chang, Wen-Chin Huang, Zili Huang, Kushal Lakhotia, Shu-wen Yang, Shuyan Dong, Andy T Liu, Cheng-I Jeff Lai, Jiatong Shi, et al.
Superb-sg: Enhanced speech processing universal performance benchmark for semantic and generative capabilities.
*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics*, 2022.
- Vaswani et al. [2017]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
*Advances in neural information processing systems*, 2017.
- Vyas et al. [2023]
Apoorv Vyas, Bowen Shi, Matthew Le, Andros Tjandra, Yi-Chiao Wu, Baishan Guo, Jiemin Zhang, Xinyue Zhang, Robert Adkins, William Ngan, et al.
Audiobox: Unified audio generation with natural language prompts.
*arXiv preprint arXiv:2312.15821*, 2023.
- Wang et al. [2023]
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al.
Neural codec language models are zero-shot text to speech synthesizers.
*arXiv preprint arXiv:2301.02111*, 2023.
- Xie and Wu [2024]
Zhifei Xie and Changqiao Wu.
Mini-omni: Language models can hear, talk while thinking in streaming.
*arXiv preprint arXiv:2408.16725*, 2024.
- Xin et al. [2024]
Detai Xin, Xu Tan, Kai Shen, Zeqian Ju, Dongchao Yang, Yuancheng Wang, Shinnosuke Takamichi, Hiroshi Saruwatari, Shujie Liu, Jinyu Li, et al.
Rall-e: Robust codec language modeling with chain-of-thought prompting for text-to-speech synthesis.
*arXiv preprint arXiv:2404.03204*, 2024.
- Yang et al. [2021]
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al.
Superb: Speech processing universal performance benchmark.
*Interspeech 2021*, 2021.
- Zeghidour et al. [2021]
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi.
Soundstream: An end-to-end neural audio codec.
*IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 2021.
- Zen et al. [2019]
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu.
Libritts: A corpus derived from librispeech for text-to-speech.
*Interspeech 2019*, 2019.
- Zhang et al. [2024]
Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu.
Speechtokenizer: Unified speech tokenizer for speech large language models.
*ICLR*, 2024.

## Appendix A Technical Appendices and Supplementary Material

### A.1 Related Work

Recent SLMs often require speech tokenization to conduct language modeling with the next prediction objective as the text LLMs. Unlike text, the speech signal is continuous and lengthy, making it difficult to derive proper speech tokenization for spoken language modeling. Common approaches may utilize self-supervised learned (SSL) speech models followed by quantization techniques to extract speech tokens [1, 14, 21, 11, 33]. In addition, audio or speech codec models have also been used for tokenization in recent SLMs [49, 5, 6, 51]. These models are designed for resynthesis, where the speech decoders are jointly learned with the encoders, making them easy to use for spoken language modeling.

With speech tokenization, GSLM [21, 32] first demonstrates the possibility of building an SLM that can generate speech. TWIST [11] further shows that SLM can benefit from initialization with the text-pretrained LLM. With regard to the huge success of text-only LLMs, recent work shifts the focus towards joint speech-text modeling [11, 6, 46]. Challenged by the modality gap between speech and text tokens, different techniques are introduced to facilitate joint modeling. Spirit LM [33] adopts an interleaving strategy; moshi [6] trains its own tokenizer with a reduced token frequency. Moreover, different patterns and strategies such as delayed or sequential generation are introduced for joint modeling, aiming for more reasonable and coherent speech outputs [46].

Despite the increasing demand of joint speech-text modeling [33, 6, 46], we do not find any work discussing the effectiveness of current speech tokenization for it. Moreover, the speech token is often derived with speech or audio-only data (^2^22An exception is CosyVoice [7]. We discuss it in Section [2](https://arxiv.org/html/2504.07053v2#S2) since it is related to our method.). Nonetheless, we observe that recent work is trying to mitigate the modality gap by reducing frequency speech token or conducting additional training stage for text-speech alignment. This motivates us to design a speech tokenization that is directly aligned with its text counterpart, tackling the mismatch issue during the tokenization stage.

In the main text, we have mentioned that we utilize a specialized mechanism based on attention to extract and aggregate the encoder representations.
We clarify that the text-speech cross-attention mechanism has also been used for fine-grained control of text-to-speech synthesis (TTS).
More specifically,  Chen and Rudnicky [3] propose content-style cross-attention to indicate their text-speech cross-attention mechanism that enables style transfer in TTS.
Although both utilize specialized text-speech cross-attention mechanism, the design choices and problem formulations are completely different.
We attribute of our main novelty to inventing a text-aligned speech tokenization and embedding for joint spoken language modeling, and the text-speech cross attention mechanism is considered and shown to be a clean, effective, and straightforward way of achieving it.

### A.2 Tackling the Vocabulary Mismatch

The vocabulary mismatch problem lies in the fact that the vocabulary sets are different between the ASR and the LLM, and TASTE is aligned with the text transcription tokens from ASR.
Consider that given a text transcription $\bm{v}$ and the vocabulary sets of ASR and LLM denoted as $\mathbb{V}^{\text{asr}}$ and $\mathbb{V}^{\text{llm}}$, the ASR tokenized sequence $\bm{v}^{\text{asr}}=[v^{\text{asr}}_{1},v^{\text{asr}}_{2},\ldots,v^{\text{asr
}}_{N}],v^{\text{asr}}_{i}\in\mathbb{V}^{\text{asr}}$ and the LLM tokenized sequence $\bm{v}^{\text{llm}}=[v^{\text{llm}}_{1},v^{\text{llm}}_{2},\ldots,v^{\text{llm
}}_{M}],v^{\text{llm}}_{i}\in\mathbb{V}^{\text{llm}}$ can be different in terms of token ids and sequence lengths.
Since the TASTE token and embedding are aligned with $\bm{v}^{\text{asr}}$, we need to derive a method to align them with $\bm{v}^{\text{llm}}$ for text-aligned speech-text modeling.
Notice that $\bm{v}^{\text{asr}}$ and $\bm{v}^{\text{llm}}$ both represent $\bm{v}$, we propose to mitigate the issue through word-level grouping, averaging, and aligning, detailed in Algorithm [1](https://arxiv.org/html/2504.07053v2#alg1).
By crafting TASTE speech tokenization into the word level, we are able to align it with the text tokens of the LLM, denoted as $\tilde{\bm{q}},\tilde{\bm{z}}$.
In practice, we also adopt the word-level averaging technique during the TASTE tokenization training phase, ensuring that the word-level TASTE tokenization facilitates high-quality reconstruction.

Figure: Algorithm 1 Aligning TASTE with LLM Tokenization via Word-Level Techniques

### A.3 Training Details

We separate the training process into the two phases: deriving TASTE tokenization and conducting spoken language modeling with TASTE.
In the tokenization phase, only the $\mathrm{Aggregator}$, $\mathrm{Quantizer}$, and the $\mathrm{UnitDecoder}$ is trainable.
We use the $\mathrm{Adam}$ optimizer and the learning rate is set to 0.0016.
The batch size is set to 160 seconds on each of the 8 NVIDIA A6000 GPUs we used.
Note that in the first 2 epochs the quantization is not applied. From the beginning of the third epoch, quantization is applied and the $\mathrm{Quantizer}$ starts to be updated.
We train the TASTE tokenizer for 5 epochs, which takes about 2 days for learning, with the learning rate gradually decayed.

As for the spoken language modeling training phase, we use the $\mathrm{AdamW}$ optimizer, the $\mathrm{Consine}$ scheduler with the learning rate set to 1e-5.
We use 8 Nvidia A6000 GPUs for training.
The total batch size summation over the GPUs is set to 768 samples with the gradient accumulation steps set to 2.
To reduce the memory overhead and the computational cost, we employ bfloat16 mixed precision during training.
Tools such as DeepSpeed [36] and Liger Kernel [13] are also applied to speed up the fine-tuning process.

### A.4 Evaluation Details

#### A.4.1 Human Evaluation

We conduct human listening tests through Amazon Mechanical Turk.
In each experiment, we randomly select the same 20 samples from each method; and for each sample we collect more than 10 evaluation scores across different human evaluators.

##### MUSHRA

In Table [1](https://arxiv.org/html/2504.07053v2#S4.T1), we have shown our result of the MUSRHA human listening test [40].
Following [51], we conduct the evaluation with a hidden reference but without a lowerpass-filtered anchor.
We instruct evaluators to rate the perceptual quality of the given samples with respect to the ground truth on a scale of 1 to 100.

##### Speech Continuation MOS

In Table [2](https://arxiv.org/html/2504.07053v2#S4.T2), we mention that we have conducted the human listening test to evaluate the overall performance of the speech continuations.
Here, we present the instruction for human speech continuation MOS evaluation as follows:

#### A.4.2 GPT-4o for MOS Evaluation

As introduced in Section [4.2.1](https://arxiv.org/html/2504.07053v2#S4.SS2.SSS1), we use GPT-4o to assign MOS scores to the speech continuation results.
Here, we describe the detailed procedure.
First, whisper-large-v3 is applied to transcribe the generated speech.
Then, given the transcription, the text content from the prompt audio, and the instruction template, GPT-4o can produce a score between 1 and 5. The instruction template is provided below:

### A.5 Additional Results

#### A.5.1 Details on SALMON and StoryCloze

Our detailed results on SALMON and StoryCloze are reported in Table [A.5.1](https://arxiv.org/html/2504.07053v2#A1.SS5.SSS1).
The introductions of the two benchmarks—SALMON and StoryCloze—are described below.

**Table 4: The evaluation results on SALMON and StoryCloze of different SLMs, and BG means background. We report likelihood-based accuracy on SALMON (acoustic aspect) and StoryCloze (semantic aspect). The baseline (S3 token) is conducted by joint speech-text modeling with the S3 token as speech tokenization.**
| Method | Mode | Web Q. | LLaMA-Q. |
| --- | --- | --- | --- |
| Mini-Omni 0.5B(T$\xrightarrow{}$T) | T | 21.3$\pm$0.9 | 39.0$\pm$2.8 |
| Mini-Omni 0.5B [46] | T+A | 4.5$\pm$0.5 | 11.6$\pm$1.8 |
| \hdashlineHelium 7B (text) | T | 32.3$\pm$1.0 | 75.0$\pm$2.5 |
| Moshi 7B [6] | T+A | 26.6$\pm$1.0 | 62.3$\pm$2.8 |
| \hdashlineLLaMA3.1-8B-Instruct | T | 60.4$\pm$1.1 | 71.7$\pm$2.6 |
| Llama-Omni-8B [9] | T+A | 35.5$\pm$1.1 | 67.3$\pm$2.7 |
| \hdashlineLLaMA3.2-1B^† | T | 24.0$\pm$0.9 | 51.0$\pm$2.9 |
| TASLM 1B (embed.)^† | T+A | 27.1$\pm$1.0 | 57.6$\pm$2.9 |
| ^†We apply few-shot learning to facilitate question answering. |  |  |  |

##### SALMON for Acoustic Evaluation

SALMON offers a comprehensive set of metrics designed to evaluate SLMs in multiple dimensions.
In summary, each test sample consists of a positive sample and a negative sample.
The negative sample differs from the positive sample by having some segments altered.
These alterations include changes in speaker, gender, environment (e.g., room acoustics), or sentiment in the middle of the utterance.
The SLM serves as an anomoly detector that aims to distinguish between the pairs of positive and negative samples.
The distinction is based on the likelihood score given by each SLM, which is then evaluated with the overall precision between the ground truth and the prediction.

##### StoryCloze for Semantic Evaluation

To evaluate the SLMs’ ability to comprehend semantic coherence and logical reasoning, we employ the spoken version of StoryCloze test (sSC) and the Topic StoryCloze test (tSC) assembled by ([11]).
Assessment of narrative understanding involves presenting a four-sentence story setup, followed by two possible endings.
These tasks require the model to select the most appropriate conclusion, thereby testing its grasp of causal and temporal relationships within a narrative.
Similarly to SALMON, we measure the accuracy of the distinctions based on the likelihood scores.

#### A.5.2 Report of Standard Deviations

We report the standard deviations of our tables in the main text to allow further investigation.

Table: Table 5: Results with standard deviations of Table [1](https://arxiv.org/html/2504.07053v2#S4.T1)

Table: Table 6: Results with standard deviations of Table [2](https://arxiv.org/html/2504.07053v2#S4.T2).

**Table 7: Results with standard deviations of Table [4.2.1](https://arxiv.org/html/2504.07053v2#S4.SS2.SSS1.Px3).**
| Method | Mode | Web Q. | LLaMA-Q. |
| --- | --- | --- | --- |
| Mini-Omni 0.5B(T$\xrightarrow{}$T) | T | 21.3$\pm$0.9 | 39.0$\pm$2.8 |
| Mini-Omni 0.5B [46] | T+A | 4.5$\pm$0.5 | 11.6$\pm$1.8 |
| \hdashlineHelium 7B (text) | T | 32.3$\pm$1.0 | 75.0$\pm$2.5 |
| Moshi 7B [6] | T+A | 26.6$\pm$1.0 | 62.3$\pm$2.8 |
| \hdashlineLLaMA3.1-8B-Instruct | T | 60.4$\pm$1.1 | 71.7$\pm$2.6 |
| Llama-Omni-8B [9] | T+A | 35.5$\pm$1.1 | 67.3$\pm$2.7 |
| \hdashlineLLaMA3.2-1B^† | T | 24.0$\pm$0.9 | 51.0$\pm$2.9 |
| TASLM 1B (embed.)^† | T+A | 27.1$\pm$1.0 | 57.6$\pm$2.9 |
| ^†We apply few-shot learning to facilitate question answering. |  |  |  |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - As the paper emphasizes that the goal for the speech tokens is to learn paralinguistic information, is there any sort of text encoding done to learn semantic information from the language? 
- It is unclear to me how the decoding is done from TASLM output. Since the decoder requires both text and speech, how is it achieved?
soundness: 2
strengths: - The paper proposes an interesting way to formulate tokenizers by using multimodal speech text tokens to provide complimentary information. This can help in significantly reducing the number of tokens needed to model speech.
- It is very interesting to see the behavior of this model in text-aligned speech editing. The model seems to be able to take paralingual information from one speech sequence and transfer it to another containing the same content. It would be interesting to pursue on this further and understand whether this type of editing can be done across very different voice types, like belonging to different genders etc.
summary: The paper proposes a novel tokenizer, called TASTE, specifically designed for text audio generative modeling. The motivation is that speech tokens generally have much longer length for the same content compared to their text counterpart. The proposed approach generates speech tokens of equal length as textual tokens. Technically, this is achieved by first passing the audio through a pre-trained speech encoder (Whisper). From this speech encoder, both last layer and shallow hidden states are extracted and fed into an attention based aggregator as key and value respectively. The queries for this aggregator are initialized by a text transcription of the speech. Since the queries are based on text, the final speech tokens match in length to the text tokens. The paper provides an intuition that these speech tokens focus on learning paralinguistic information as they are supposed to be used along with the corresponding text tokens. For decoding, a speech decoder uses both the text and speech tokens and decodes them back to speech using a vocoder. The tokenizer is trained on a speech resynthesis task wiht a cross entropy loss to predict the speech unit and commitment losses on the residual codebooks.

Apart from the tokenizer, the paper also proposes a speech language model which integrates the TASTE tokenizer, called TASLM. There are two variants - one which trains on the discrete codebook indices similar to a traditional language model and one which directly operates on the continuous latent embeddings.

Experiments are done by training on Emilia and LibriTTS datasets and evaluating on LibriTTS test split. The paper evaluates the tokenizer for speech reconstruction, speech editing and  TASLM on speech continuation, likelihood-based next-speech selection and spoken question answering.
weaknesses: - The proposed TASTE tokenizer tends to learn text aligned speech embeddings. One limitation of such an approach would be for sounds that occur commonly in speech but may not have a textual counterpart. For example, if a person laughs, it is unclear how the corresponding textual token would be retrieved. The approach may _collapse_ the laughing related information to the neighboring speech tokens if they have textual counterparts.
- The paper does not evaluate on the sWUGGGY and sBLIMP ZeroSpeech 2021 benchmark which is standard in literature [1,2]. These benchmark test the lexical and grammatical knowledge of the model to understand how well model can identify real words vs phoenetically similar non-words. I believe this might be a limitation of this tokenization approach as it needs a text transcription of the speech input.
- The paper does not explain its underperforming benchmarks. For example, In table 1, the proposed method has significantly worse WER compared to other tokenizers but the paper does not dive deeper into it. Instead, the text focuses on how WER is better than text-only baseline (343-344). Similarly, for table 2, it would be good to understand what factors contribute to the underperformance on SALMON benchmark. In table 8, this underperformance seems to happen for most of the experiments in SALMON and some discussion around it would be helpful.
- While the paper cites [2], it is unclear why it is not compared against the experiments of the paper. 
- Related work section is missing from the main paper. While it is present in the appendix, it should be a part of the main paper. 

References:
1. Spirit LM: Interleaved Spoken and Written Language Model, ACL 2025
2. Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback, ACL 2025

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. Please see the Weakness section for points where additional analysis, clarification, or experimentation would strengthen the paper.
2. Can the authors provide clearer evidence that the observed improvements come from genuine text-speech alignment rather than architectural or bitrate differences? Specifically, can they (i) compare with stronger recent baselines under matched bitrates, (ii) justify the representational choice of K = h(L), V = h(l) with empirical analysis, and (iii) evaluate robustness under noisy, multilingual or imperfect ASR to confirm practical applicability?
soundness: 2
strengths: 1. Using dynamic token frequency for better alignment with text is an interesting and well-motivated approach. It directly targets the speech-text mismatch that often limits speech tokenizer alignment.


2. Deriving speech tokens based on transcription-guided cross-attention is an interesting design choice that offers an interpretable mapping between continuous speech features and discrete text-aligned tokens.


3. The text-aligned editing demonstration is interesting. Swapping word-level TASTE tokens to transfer duration and other attributes in a controlled way is well-motivated and could inspire future work on speech style transfer or voice cloning.
summary: The paper presents TASTE, a text-aligned speech tokenization approach that uses transcription-guided cross-attention and RVQ to better align speech and text representations. The goal is to address the speech-text alignment mismatch common in spoken language modeling. While the motivation is clear and the design is coherent, the method largely builds upon existing codec frameworks with a text-conditioned aggregation step. Experimental results are competitive but not stronger than prior models such as SpeechTokenizer, Mimi, and EnCodec, and the evaluation omits more recent baselines. Overall, the contribution is conceptually sound but represents a modest advancement without clear empirical evidence of improved alignment and performance.
weaknesses: 1. The core idea of using text to guide lower-frequency speech tokens is sensible, but similar strategies have appeared before. Prior works have reduced speech token rates or trained tokenizers with auxiliary text objectives/distillation. The paper should better position TASTE relative to SpeechTokenizer, TWIST, DM-Codec, TadiCodec, and other text included tokenization approaches. It should clearly delineate what is new beyond the addition of a cross-attention aggregator and RVQ. The claim that TASTE is the “first end-to-end reconstruction objective to learn joint tokenization” is not fully substantiated because the distinction from prior work remains unclear.

2. The experimental evaluation is constrained. In Table 1, TASTE is compared only against SpeechTokenizer, Mimi, and EnCodec. While these are recognized baselines, they are relatively older and weaker compared to recent models such as WavTokenizer, DM-Codec, FACodec, and BigCodec, which have shown superior reconstruction performance. Similarly, Table 2 compares only to TWIST and SpiritLM, and should include newer SLMs approaches (Cosyvoice2, SparkTTS, and Llasa). Without such comparisons, the reported results lack convincing evidence of improvement. 

3. TASTE does not outperform existing baselines (e.g., SpeechTokenizer, and EnCodec) across key metrics (WER, ViSQOL, Duration Consistency, Speaker Similarity, and MUSHRA). This raises questions about the practical benefit of the more complex and resource-intensive TASTE training pipeline. The paper repeatedly attributes its lower performance to “lower bitrate,” but this argument is unconvincing without matched-bitrate experiments. The authors should directly compare TASTE with low-bitrate variants of established codecs (e.g., DAC, WavTokenizer, XCodec2, BigCodec). Without such controlled comparisons, the main claim of improved speech tokenization remains weak.

4. The design choice of setting the aggregator's keys as last-layer encoder features (K = h(L)) and values as shallow-layer features (V = h(l)) is based on questionable representational assumption and not supported with experiments. The paper assumes h(L) captures alignment cues while h(l) preserves acoustic detail, but offers no empirical analysis (e.g., attention visualizations or layer ablations) to support this. It is unclear whether the observed effects are from meaningful alignment learning or coincidental correlations. 

5. The ASR dependence of TASTE limits general applicability. The model requires accurate transcriptions to derive aligned tokens, yet there is no analysis of robustness under realistic noisy ASR or low-resource conditions. Without such tests, the practical viability of TASTE is uncertain. Moreover, The evaluation is also restricted to English corpora (LibriSpeech, Emilia). This ASR dependence also limit TASTE's applicability to spontaneous or multilingual speech. Given that TASTE explicitly depends on text-speech alignment, it is unclear whether it generalizes to languages with more complex non-phonemic orthography.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: - How exactly is the attention map used in the alignment process, and how sensitive is performance to layer selection for keys and values?
- What is the computational overhead of training the alignment module compared to standard speech tokenizers?
- Could the model be extended to use unpaired speech in a semi-supervised manner to improve robustness?
- How does TASTE handle inter-word silences or pauses during alignment and embedding generation? Are such regions ignored, or do they get represented implicitly in the embeddings?
- What is the rationale for using the last hidden layer as key and shallow hidden layers as value? Have other layer combinations been tested, and how sensitive is performance to this choice?
soundness: 3
strengths: The paper tackles an important problem in spoken language modeling: how to represent speech and text within a shared embedding space for joint generation. The proposed method is conceptually simple yet effective, combining attention-based feature aggregation with an end-to-end reconstruction objective to achieve text-aligned speech embeddings. The learned representations capture both linguistic and paralinguistic information in a compact form and integrate smoothly with pre-trained text LMs through lightweight adaptation. The experimental results are consistent across multiple tasks, and the qualitative analyses illustrate interpretable alignment behavior. The paper is clearly written and provides a strong empirical demonstration of how modality alignment can improve spoken language modeling.
summary: This paper introduces TASTE (Text-Aligned Speech Tokenization and Embedding), a method for learning text-aligned speech representations that directly bridge the modality gap between text and speech. The approach aggregates speech features using an attention-based mechanism guided by text transcriptions and is trained end-to-end with a speech reconstruction objective. This alignment produces compact and semantically consistent speech embeddings that preserve paralinguistic information while reducing sequence length. By adapting a pre-trained text language model with low-rank adaptation, the resulting joint model can process and generate speech coherently without additional alignment heuristics. Experiments show that TASTE improves speech continuation and next-speech prediction performance compared to existing spoken language models, demonstrating its effectiveness for unified text-speech modeling.
weaknesses: While the paper is technically sound, several aspects of the design and analysis remain underexplored. The attention-based aggregation is central to the method, yet its behavior and learned alignment patterns are not clearly analyzed, making it difficult to understand how alignment emerges during training. The paper lacks ablations isolating the effects of key design factors such as the chosen encoder layers, the reconstruction objective, and the LoRA adaptation on the joint model’s performance. Moreover, the evaluation focuses mainly on speech continuation and next-speech prediction, without examining general spoken understanding tasks or perceptual quality metrics that would better validate the claim of improved paralinguistic preservation. These gaps limit the interpretability and completeness of the study.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Please detail the decoder fusion: where are the gates/weights applied between $v$ and $\hat{z}$, and how are they trained and stabilized?

- Can you report out-of-domain continuation and SALMON/StoryCloze on non-LibriVox corpora, and include dedup checks between LibriTTS and LibriSpeech?

- What are latency/memory costs for tokenization/decoding, and can TASTE stream in real time?
soundness: 3
strengths: - The key contribution is to align speech tokens to the text sequence during tokenization and show that this enables simpler, stronger joint modeling at low bitrate. The idea is clean and the empirical gains on continuation with a small LM are practically meaningful for SLM research.

-  TASLM (1.3B + LoRA) outperforms prior 7B SLMs on continuation (GPT-4o, UTMOS, Human) and is competitive on SALMON/StoryCloze. The LoRA-only fine-tuning and word-level handling of ASR/LLM vocab mismatch increase practical relevance.

- ASR robustness ablations (GT vs ASR transcripts; Whisper vs Parakeet) show limited sensitivity, and the word-swap editing demo nicely evidences alignment of paralinguistics to lexical units.
summary: The paper introduces TASTE, a text-aligned speech tokenization and embedding method aimed at simplifying joint text–speech modeling. Given a speech–text pair, a frozen Whisper encoder provides shallow and last-layer features, an attention-based aggregator uses text tokens as queries over these features to produce a text-length speech representation, which is discretized with RVQ, and finally, a decoder reconstructs speech from the text and the aligned speech embedding for a reconstruction objective.
weaknesses: - While key idea of the proposed method is clean, novelty is rather moderate. Text-speech cross-attention has been used for fine-grained TTS control, and recent joint SLMs alleviate length mismatch via interleaving (e.g., Spirit-LM) or lower-frequency tokenizers (e.g., moshi).

- Key "semantic MOS" relies on GPT-4o over ASR transcripts. Despite ablations, this double-black-box can amplify mistakes, such as prosody -> ASR error -> GPT score. Human semantic judgments or text-only LM plausibility on ASR outputs would be a stronger counterpoint, and code-switch/noisy-ASR conditions would probe robustness more realistically.

- The autogressive "weighted sum" between text and aligned speech in the unit decoder is under-specified. I would suggest the authors include equations/ablation to clarify.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the Weaknesses.
soundness: 3
strengths: 1. This work proposes a novel highly compressible speech tokenizer named TASTE that can reduce the modality gap between text and audio pairs. This is practically useful to reduce the length of the token sequence for a given audio with little loss of reconstruction quality. 

2. TASTE helps finetuning LLMs to SLMs via Low-Rank Adaptation (LoRA). This is practically useful to speed up the development of SLMs or omni-modal LLMs.
summary: This work proposes TASTE, a speech tokenizer that enables the alignment with semantic text tokens. It can generate highly compressed audio tokens via minimization of the speech reconstruction loss. The experiments are performed to show that TASTE successfully reconstructs the audio and helps LLMs easily finetuned to be SLMs (Spoken Language Models).
weaknesses: 1. One of the most practical uses of audio tokenizers may be training of SLMs, as this work experimented. 
- Thus, it may need to be studied whether the proposed tokenizers can lead to SOTA performance of SLMs. 
- For example, TASLM’s performance can be compared with that of SOTA SLM models such as LLaMA-Omni 2. 

2. The generalization of this tokenizer should be discussed. 
- What if the domains of text and audio corpora are changed? Do we need to train again? 
- Since the proposed tokenizers are forced to learn the joint representation of both “semantics” and “phonetics”, the learned tokenizer may be much more variable according to the training sets than the one learned only focusing on “phonetics”.


Decision: Accept (Poster)