A Variational Framework for Improving Naturalness in Generative Spoken Language Models
Authors: Li-Wei Chen
Takuya Higuchi
Zakaria Aldeneh
Ahmed Hussen Abdelaziz
Alexander Rudnicky
Venue: ICML 2025 poster (2025)
Keywords: Generative Spoken Language Modeling;Speech Language Model
Primary area: applications->language_speech_and_dialog
TL;DR: A variational autoencoder combined with a token-based speech language model to improve the naturalness of speech synthesis.

Abstract: The success of large language models in text processing has inspired their adaptation to speech modeling.
However, since speech is continuous and complex, it is often discretized for autoregressive modeling.
Speech tokens derived from self-supervised models (known as semantic tokens) typically focus on the linguistic aspects of speech but neglect prosodic information.
As a result, models trained on these tokens can generate speech with reduced naturalness.
Existing approaches try to fix this by adding pitch features to the semantic tokens.
However, pitch alone cannot fully represent the range of paralinguistic attributes, and selecting the right features requires careful hand-engineering.
To overcome this, we propose an end-to-end variational approach that automatically learns to encode these continuous speech attributes to enhance the semantic tokens.
Our approach eliminates the need for manual extraction and selection of paralinguistic features.
Moreover, it produces preferred speech continuations according to human raters.
Code, samples and models are available at https://github.com/b04901014/vae-gslm.

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
Li-Wei Chen
Takuya Higuchi? Zakaria Aldeneh
Ahmed Hussen Abdelaziz Alexander Rudnicky
Abstract
and complex nature. As result, previous works (Lakhotia et al., 2021; Borsos et al., 2023; Maiti et al., 2024) tokenized speech into simpler discrete units to enable the application of language modeling techniques originally developed for text. However, these semantic tokens are typically derived by performing k-means clustering on features extracted from self-supervised pre-trained speech models, such as HuBERT (Hsu et al., 2021). We use the term semantic to- kens to distinguish them from acoustic tokens (Borsos et al., 2023), which capture general acoustic information. These models primarily capture the linguistic aspects of speech, such as phonetic information, while often overlooking par- alinguistic features, such as prosody (Weston et al.. 2021). As result, training an autoregressive model solely with such semantic tokens restricts the model's ability to fully capture and represent the diverse information encoded in speech.
The success of large language models in text pro- cessing has inspired their adaptation to speech modeling. However, since speech is continuous and complex, it often discretized for autore- gressive modeling. Speech tokens derived from self- supervised models (known as semantic to- kens) typically focus the linguistic aspects of speech but neglect prosodic information. As a result, models trained on these tokens can gen- crate speech with reduced naturalness. Existing approaches try fix this by adding pitch features to the semantic tokens. However, pitch alone cannot fully represent the range of paralinguistic attributes, and selecting the right features requires careful hand-engineering. To overcome this, we propose an end-to-end variational approach that automatically learns to encode these continuous speech attributes to enhance the semantic tokens. Our approach eliminates the need for manual ex- traction and selection of paralinguistic features. Moreover, it produces preferred speech continua- tions according to human raters. Code, samples and models are available at https: //t::///iiee //github. com com/b0490 901 1011114 4/ve/avvvlllnn
To address the aforementioned limitation, Kharitonov et al. (2022) augmented the tokens with extracted fundamental frequency (Fo, or pitch) enable prosody-aware modeling. However, augmenting semantic tokens with manually de- fined paralinguistic attributes can be inherently suboptimal. First, pitch alone cannot capture the full range of paralin- guistic information encoded in speech. For instance, energy- related (e.g., loudness, cero-crossing-ratt raae)) and spectral- related (c.g.. frelufrey cepstral coefficients) features are also important paralinguistic features (Schuller et al., 2009; 2013; Eyben al., 2015). Furthermore, training correct pitch tracker introduces additional complexity (Kim et al., 2018).
1. Introduction
Large language models (LLMs) have achieved tremendous success in text processing (OpenAI, 2024), offering new ways to interact with machines. This progress has motivated efforts to extend their capabilities to speech to enable more natural spoken interactions with machines. However, model- ing speech presents unique challenges due its continuous
Instead of relying on hand-engineered paralinguistic fea- tures, we propose an approach to learning these features di- rectly from the input signal, within an autoregressive frame- work. These learned features are optimized to both: reconstruct the input speech, and 2) enhance the autoregres- sive modeling process. Our approach allows the learned features to complement semantic tokens, removing the need for pre-extracted paralinguistic features as required in pre- vious methods. As result, our method generates more natural-sounding speech compared to baseline models while maintaining comparable meaningfulness of the syntheses.
Language Technology Institute, Carnegic Mel- Correspondence to: Li-
lon Wei Chen <liweiche www.s.cc.eeee
University "Apple.
Takuya Higuchi Zakaria Aldeneh <zaldench @ apaleehhappl..ccm apple.com2 comm,, Ahmed Hussen Abdelaziz <ahussenabde-
<takuya higuchi@apple app.e.com>,
laziz @ @pize. com,> Alexander Rudnicky <air @ereess.. edu..
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
2. Preliminaries
where DKz is the Kullback-Leibler (KL) divergence be- tween two distributions, and (Z) is fixed prior distri- bution (usually a Gaussian). In Equation 1, OELBO is known as the evidence lower bound (ELBO), which pro- vides a lower bound for log po()) since (g X) (Z X)) is always nonnegative. Therefore, instead of directly optimizing Ex[log Ex| pa (X)] the VAE maximizes the tractable lower bound Ex(OEEBB), Here, we refer to the learned continuous latent Z from the VAB as the variational features.
In this work, we work on mel-spectrogram, and consider vocoding, the act of turning mel-spectrogram back to raw waveform, as a problem that has already been addressed. We denote the mel-spectrogram as X ( I ))= where d, represents the number of filter-banks, T is the total number of time frames in the spectrogram, and is the frame at time t. We use X to denote the sub-sequence (= and define 1 = 0. Our goal is to model p(X) using a generative approach.
3. Proposed F Framework
Token-based Speech Language Model We describe the general framework of speech language models that rely on the use of semantic tokens, as seen in works like Lakho- tia et al. (2021); Borsos al. (2023); Maiti et al. (2024). This approach consists of three components: a speech tok- enizer, an autoregressive model, and decoder. The speech tokenizer maps X to sequence of discrete semantic to- kens 2d = [=f E N=) where N₂- = (1,2....,k) k}, and k is the vocabulary size of the semantic tokens. We use p(
Figure I provides an overview of our proposed framework. This section is organized as follows: Section 3.1 introduces our setup that combines a VAE with an autoregressive model for the latent variables. Section 3.2 describes how in- tegrate semantic tokens into the framework. Section 3.3 discusses how to balance the different loss terms that arise in our setup. Section 3.4 describes the use of normalizing flows to improve the expressive power of the autoregressive prior. Finally, Section 3.5 introduces the diffusion decoder and the utterance encoder used in the framework.
X) to denote the implicit distribution of the pre- trained speech tokenizer. The autoregressive model, param- eterized by vb, models the probability of token sequences z as Pa ) I (=f 11. Finally, the decoder, parameterized by 0, is trained to convert Zd back to X by modeling Pe [X] ) However, this framework is limited to semantic tokens Zd, which primarily capture linguistic in- formation and ignore paralinguistic information. As a result, the decoder 0 may struggle with accurate reconstruction, and the autoregressive model can have difficulty incorpo- rating paralinguistic information. To address this limitation, we propose to incorporate the variational autoencoder frame- work to learn continuous features to complement Z
3.1. VAE with an Autoregressive Prior
Our method starts by modeling the prior of the VAE, which is typically a fixed Gaussian distribution, with trainable autoregressive model pio (Z") = II (y 1i..-)). We refer to this framework as VAE with autoregressive prior. We note that VAE with an autoregressive prior has been ex- plored in previous works (Vahdat & Kautz, 2020; Zhu et al.. 2020) within the computer vision domain. Additionally, Sun et al. (2020) also applied similar framework for TTS, but with prior and posterior distributions optimized separately instead of jointly. Here, we adopt the VAE framework with autoregressive prior for speech continuation and further integrate it with discrete token-based models to enhance the naturalness of the synthesis. We use diagonal Gaussian distribution to model the variational posterior, where the statistics are predicted by a neural network:
Variational Autoencoder (VAE) Latent variable models introduce unobserved latent variables Z = (zf E TR y that influence the observed variable X. di is the dimen- sion of each 2f. and is a syper-parameter chosen prior to training. In VAE, the likelihood of the observed data given the latent variable, po PX Z2), is modeled by neu- ral decoder, parameterized by 0. The variational posterior, (Z" X), is modeled by neural encoder, parameterized by Φ. Using this modeling setup, the log-likelihood of the data, log popPo (X), can be written as:
(0(29 X) = (X),
(2)
E g (2) Po X 2 1Krl99(22 X) (zz)))
OELBO
Since each zद is conditionally independent given X, we can
(1) express the posterior as: (Z X) = I (z X). With this decomposition, and the parameterized autoregres- sive prior, the OELBO in Equation can be further derived2
+ +DKL(g (Z (Z X)),
'Speech tokenizers can operate mel-spectrograms or directly on raw waveforms.
"See Appendix A.1 for detailed derivation.
2

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
O rrcc
VAE with an Autoregressive Prior
Ze
Encoder Φ
L
Variational Features
Zi:t-1
Autoregressive Model yp
Decoder θ
X Speech
Z1:t111
Speech Tokenizer
zd
Ld
z
Semantic Tokens
(Not trainable)
Token-based Speech Language Model
Figure 1. Overview of our proposed approach. Our method integrates the token-based speech language model (outlined in Scction 2, represented by the lower shaded region) with variational autoencoder (VAE with autoregressive prior, shown in the upper shaded region). This setup allows the model to learn variational features Z that complement the pre-extracted semantic speech tokens Z In our proposed joint setup. the variational features Z are trained reconstruct speech X alongside z (by maximizing Orea): facilitate the prediction of the next speech token z! (by minimizing ( (by minimizing (
) 3) support the sequential prediction of the variational features themselves
into:
ing the conditional independence af and 2f given the past generations. We further discuss this modeling assumption in Appendix This allows us to re-write" OELBO from Equation
O OELBO = Ey (2 (
(log (X] zn)
Orea
Ezg...
[DLi x))lp(<=f Zin- 1))]
OELBO
(4)
(2ox)) /0
E 7
By maximizing OELBO, we maximize the first term, the reconstruction objective Oree. and minimize the second term, the variational feature prediction loss Li.- We note that training model to maximize Equation 3 is feasible without incorporating discrete semantic tokens Zd. This token-free approach is also depicted as the upper shaded region in Figure (VAE with an Autoregressive Prior), and its properties are further explored Section 5.
T
Ezz (00 X)Ilp(=( Z1a-1))]
Ez logpo Zia-
3.2. Incorporating the Semantic Tokens with VAE
From Equation 4, our training objective O ELBO consists of three terms: Oreas L and Ld Orec is the reconstruc- tion objective. Maximizing Orec trains the decoder 0 to reconstruct X from both Ze and Z while encouraging the encoder & to generate Z with helpful information to re- construct X. Ci is the variational featturr feature prediction loss. Minimizing Ci trains the autoregressive model i to predict the next variational feature of and encourages the encoder & to generate Z that is easier for to model. Lg the semantic token prediction loss, which trains the autoregres-
We now integrate the semantic tokens 2d with the VAE with an autoregressive prior. Using these tokens, the model no longer needs to encode as much phonetic information as in z" allowing Z focus on other attributes of continuous speech. To this end, we introduce joint latent variable Z (z+ E dR No), where is the concatenation of o and of. Since z' and Z are conditional independent given X, we can express the new variational posterior as: X). Then, we model (2 Z1=-1) = pe (2 Z (= ZI assum-
(Z X) = (Z XZ
See Appendix A.2 for detailed derivation.
3

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
sive model i to predict the next semantic token given the previous Zd and %
3.5. Other Components
We describe the modeling of the our decoder pe (X Z) and the utterance encoder designed capture static informa- tion. While these components are not the main focus of our study, they help ensure fair comparison between different methods. We use these components for all methods in our experiments and focus on how changing the inputs to the autoregressive model affects performance.
3.3. Balancing the loss terms
In Equation 4, the terms Orees Cig and Ci can work against each other. For instance, the encoder ø optimizes both Oree and L.- Maximizing Ore encourages the variational features Z" to encode more information about X, while minimizing Li regularize Ze to be simpler for the autore- gressive model i to predict. Similarly, optimizing C and Cig with the autoregressive model vi is a multi-task learning scenario, where v learns to predict two different objectives given the same input. Moreover, these terms may operate on different scales due to how the losses are computed, ne- cessitating balancing mechanism. As result, inspired by 3-VAE (Higgins et al., 2017), we introduce two scalars: |3 and y, to balance the loss terms as follows:
Diffusion Decoder We model the decoder po (X Denoising Diffusion Probabilistic Model (DDPM) (Ho et al., 2020). We choose DDPM due to its flexibility in modeling complex distributions. We condition the diffusion process on Z. For baok-propagation through the encoder & we use the reparameterization trick (Kingma & Welling, 2019) to sample from ( X), and combine with embedded semantic tokens Z The outcome is then concatenated with each intermediate layer of the diffusion decoder for conditional diffusion. We train all diffusion decoders with 1000 DDPM steps. Note that proposed approach is not limited to a specific decoder. Although we opted for cettuiion-based decoder for ease of training, our method is compatible with various decoding strategies. There are no constraints on the type of decoder used to parameterize po (X Z Z%).
Z) with
OELBO Oree 3 (Ciu + Eki
(5)
Here, a larger favors a simple p(Z"), while a smaller 3 encourages the variational features Z to encode more in- formation about X. Larger y encourages the autoregressive model & to prioritize accurate predictions of zd over Z In practice, we employ linear warm-up strategy for 8, in- creasing it from zero to its final value during the early stages of training. This approach, inspired by prior works on text generation (Bowman et al., 2016; Fu et al., 2019), helps mitigate posterior collapse. Empirically, we find that this strategy allows for higher values of 3 without causing Li to collapse zero.
Utterance Encoder Static features, such as speaker infor- mation and recording environments, often vary little across given utterance. In our current modeling approach, this static information would be redundantly encoded at each time step. To address this issue, we introduce an additional utterance-level feature encoder that encourages Z to focus on time-varying signals. Specifically, we randomly segment a portion of the mel-spectrogram X and feed it to the ut- terance encoder to produce an utterance-level embedding. This embedding is then concatenated with Z before being provided to the diffusion decoder. The utterance encoder is trained end-to-end with the entire system.
3.4. Time-wise Normalizing Flow
We employ a lightweight normalizing flow (Rezende & Mo- hamed, 2015) that is shared across time to improve the pressive power of the autoregressive prior (2f Zza-1). Specifically, an invertible flow network f maps each to a point in the Gaussian distribution, and sampling can be realized by running the network in reverse. By using the change of variables, we can write:
Pe (= -==-1)=
(6)
4. Experimental Setup
N(fo(z)) (ZI- (((1))))) det (zf
4.1. Datasets
Ozf
We use two datasets in our experiments: Lib- riSpeech (Panayotov et al., 2015) and Libri-light (Kahn et al., 2020), consisting of audiobooks narrated in English. LibriSpeech contains 960 hours of speech, while Libri-light contains 60k hours of speech. For semantic token extraction, we follow Hassid et al. (2023); Maiti et al. (2024) and use tokens derived from HuBERT representations (Hsu et al., 2021). We use the official HuBERT checkpoints,
where Plg dy og are modeled by autoregressive neural net- works (i.e., transformer). We choose affine coupling lay- ers (Dinh et al., 2017) as the backbone of our normalizing flow due to their simple implementation and efficient com- putation. We note that similar approaches using normalizing flows to enhance prior distributions have also been observed in Kim al. (2021; 2020) for text-to-speech.
4

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
pre-trained on LibriSpeech' and Libri-light'. We run k-means clustering with & 200 the output of the last transformer layer of HuBERT using 10% of data randomly sampled from the training set. We pick k = 200 after testing values from (50, 200, 1000) and choosing the one that produced the best language modeling performance The result is also consistent with Maiti et al. (2024). More details on the choice of l are provided in Appendix F.
similar to Chen al. (2023); Défossez et al. (2024). We include this baseline compare with recent methods (Dé- fossez et al., 2024) that integrate acoustic tokens into the autoregressive generation process.
Variational speech modeling approach (Proposed) This is our proposed approach introduced in Section 3. In this approach, we learn to extract variational features that sup- plement the semantic tokens while jointly training the au- toregressive model. The learned variational features are used by both the autoregressive model and the decoder. This approach eliminates the need for the selection and extraction of paralinguistic features based hand-made engincering. Additionally, we set our latent dimension d² = 4. While we observed performance improvements with larger de we opted for a smaller value to ensure a fairer comparison, as it results in less variation in parameter size. Our additional experiments on the latent dimension di is in Appendix E.
4.2. Methods
We compare our proposed approach to methods that use only semantic tokens in the autoregressive model, as well as methods that use semantic tokens with added pitch features in the autoregressive model. To ensure fair comparison, we fix the autoregressive model architecture to be the same for all methods, varying only the input and output layers. We also use the same configuration for the diffusion de- coder and utterance encoder across all methods. 6 For the neural vocoder (i.e., mapping the mee-spectrogram back to waveform), we train HiFi-GAN (Kong et al., 2020) LibriSpeech and use for all of the methods. We leave the detailed configuration of model architectures in Appendix B. Below, we provide further details on the three approaches.
For inference, we use temperature- based sampling similar to Lakhotia et al. (2021). Specifically, we set the temper- ature to 0.85 for both semantic tokens Zd and continuous variational features Z For variational features, the tem- perature is the scalar multiplied to the standard deviation of the normal distribution in Equation 6 before sampling, as done Kim al. (2020). For the diffusion decoder, we use denoising diffusion implicit models (DDIM) from Song et al. (2021) with 7) 0.5 and 100 diffusion steps. Training details are provided in Appendix C.
Token-LM We adopt the token-based speech language model (described in Section 2) our baseline, represent- ing approaches such as Lakhotia et al. (2021): Borsos et al. (2023); Maiti et al. (2024), which apply only discrete se- mantic tokens to the autoregressive model.
Token-LM Pitch In this baseline approach, we aug- ment the semantic tokens of token-based speech language model (described in Section 2) with log pitch features he- fore passing them into the autoregressive model. The pitch features are extracted using CREPE (Kim et al., 2018). Ad- ditionally, we introduce pitch regression task alongside the standard next-token prediction task, optimizing it with L1 loss. This method incorporates hand-engineered paralin- guistic features, similar to the approach used by Kharitonov et al. (2022).
4.3. Evaluation Metrics
We evaluate the comparison methods on both reconstruction and speech continuation. The reconstruction metrics, intro- duced in Section 4.3.1, involve only the encoder-decoder pair and indicate how much information is preserved in the extracted representations. The remaining metrics fo- cus on speech continuation, which our primary objective, where the performance of the autoregressive model is also assessed.
4.3.1. OBJECTIVE METRICS
Token-LM Acoustic In this comparison method, we augment semantic tokens with acoustic tokens (Borsos et al., 2023; Défossez et al., 2023). Specifically, we train a residual vector quantization (RVQ) autoencoder to discretize speech into four levels of acoustic tokens. At each transformer time step, the model first predicts the semantic token, followed by the acoustic tokens, which are autoregressively generated over the code levels using an additional transformer layer,
Reconstruction Metries We use Fo-RMSE, mel-ceptral distortion (MCD), and character error rate (CER) to measure the quality of the reconstructed signal. Fo-RMSE measures the root mean squared difference between the pitch contour of the ground-trutt signal and the reconstructed one. We use CREPE (Kim et al., 2018) to extract pitch and only consider the voiced parts of the signal when computing the difference. MCD measures the Euclidean distance be- tween the 23 mel-cepstral coefficients (MCEPs) extracted from the ground-trui -tuth and reconstructed signals. For calcu- lating CER, we use a pre-trained Whisper (Radford et al.,
hhtts://hugginngfac.cc httn:://hhgiiggeeeccoccchoohooth wwwfffichhaoo/hhhrrrr-eeee----ee
https:/hhggingffcccoooiiiii www.fucchhookhhube--aagggg-lll
'xceept for the Token-LM + Acoustic Tokens method, which uses the RVQ decoder directly.
5

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
2023) automatic speech recognition model.? 7 We use the dev-clea and dev-other subsets of LibriSpeech for evaluating reconstruction. To ensure deterministic results, instead of sampling each 2f from (= X), we directly use the Gaussian mean M [x () from Equation 2. In prac- tice, we observed that the stochastic noise of (2(2f) X) has little effect on the reconstructed syntheses.
Table I. Results of speech reconstruction evaluation (Fo-RMSE, MCD, CER) for the models discussed in Section 4.2. The evalua- tion metrics are detailed in Section 4.3. All models were trained on the Libri-light dataset.
Method
Fo- --RSSS))))
MCD(J)
CER(L)
Ground-truth
n/a
n/a
2.35
Token-LM
43.90
7.55
10.19
+ Pitch
25.46
6.90
6.59
ZeroSpeech Metrics We adopt the commonly-used met- rics (Borsos al., 2023; Hassid al., 2023; Maiti et al., 2024) from the ZeroSpeech challenge (Nguyen et al., 2020): sWUGGY and sBLIMP to measure language capability objectively. For these two metrics, speech utterances are given in positive-negative pairs, with each model scoring both utterances. The model's accuracy is the percentage of instances where the positive example receives a higher score than the negative one. sWuggy measures if the model scores real word higher than a phonetically similar non- word (e.g., "brick" v.s. "blick"). sBLIMP measures if model scores a grammatically correct sentence higher than a similar but incorrect one (e.g., "the dogs sleep" vs. "the dog sleep"). Both metrics use text-to-speech to generate the examples. In line with Borsos et al. (2023), we eval- uate sWUGGY using only words existing in LibriSpeech (referred as the "in-vocab" version). We use the test split for evaluation. See Appendix G for detailed description how we estimate the scores for the methods.
+ Acoustic
15.05
2.58
3.73
Proposed
16.56
5.43
4.35
Reconstruction Quality. First, the results in Table 1 show that compared to Token-LM and Token-LM + Pitch, our proposed approach improves the reconstruction of the orig- inal signal. These findings highlight three key points: 1) discrete semantic tokens alone are insufficient to capture all the components necessary for faithful reconstruction, 2) incorporating only pitch information is not enough, and 3) the learned variational features Ze in our approach effec- tively complement the discrete semantic tokens Zd, lead- ing to better reconstruction of the speech signal. On the other hand, our proposed method achieves slightly lower reconstruction quality than Token-LM + Acoustic. Since the variational features are continuous, they should be able to encode more information than four levels of acoustic tokens. Therefore, our results suggest that the information encoded in the variational features is effectively regularized by the autoregressive losses: Cy and Cop
4.3.2. SUBJECTIVE METRICS
We use subjective human evaluations to assess the natu- ralness and meaningfulness of the generated speech. We randomly sampled 100 utterances from the LibriSpeech dev-clean and dev-other subsets, cropping the first three seconds use prompts. Each audio sample was rated by seven annotators. For naturalness, annotators rated how human-likk the generated speech sounded on a five- point Likert scale, where one corresponds to "Very unnat- ural" and five to "Very natural." For meaningfulness, they rated the grammar and content of the speech on a five-point Likert scale, where one corresponds "Very Poor" and five to "Excellent." Additional details on the subjective evaluations are provided in Appendix D.
Speech continuation of our approach is more natural compared to the speech generated from the baselines. The subjective evaluation of speech continuation, measured by the mean opinion score of naturalness (N-MOS) in Ta- ble 2, shows that the syntheses produced by our proposed approach have significantly higher naturalness compared to all baselines. This finding further supports our hypothe- sis that the variational features Ze learned by our approach improve the quality of the synthesis. While Token-LM + Acoustic achieves the best reconstruction in Table 1, the autoregressive model struggles to effectively process the additional information encoded in the RVQ tokens, result- ing in significantly lower speech continuation performance, as shown in Table 2. Additionally, Table 2 compares the number of parameters between different methods. The re- sult indicates that the overhead of the proposed method is relatively small (< 1% of the total parameters). while still achieving noticeably better performance.
5. Experimental Results
5.1. Main Results
Tables and 2 present the results for the three methods de- scribed in Section 4.2. Table reports objective metrics for speech reconstruction, while Table provides both objective and subjective results for speech continuation. We discuss our observations below.
Speech generated using our proposed approach achieves subjective meaningfulness (as measured by M-MOS) comparable to the baselines. The results in Table in-
6

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
Table Results of speech continuation evaluation for the models discussed in Section 4.2. The evaluation metrics are detailed in Section 4.3. M-MOS refers to the meaningfulness mean opinion score. N-MOS refers to the naturalness mean opinion score. Both M-MOS and N-MOS are evaluated on speech continuation are presented along with 95% confidence intervals. All models were trained on the Libri-light dataset. ## Param." refers to the number of parameters used during inference. 'M' stands for million.
Method
# Param.
sWUGGY(1)
sBLIMP(t)
M-MOS(1)
N-MOS(1)
Ground-truth
n/a
n/a
n/a
3.94 + 0.08
3.89 + 0.09
Token-LM
219M
61.75
58.31
3,24 + 0.09
3,19 + 0.11
Token-I.M + Pitch
219M
60.75
56.92
3.29 + 0.09
3.08 + 0.10
Token-LM + Acoustic
226M
56.23
52.03
2.75 + 0.09
3.03 = 0.10
Proposed
221M
60.48
56.56
3,45 + 0.09
3.60 + 0.10
dicate that our proposed approach produces syntheses that are comparable to or better than baselines, as reflected by its higher meaningfulness mean opinion score (M-MOS). However, all compared methods show lower SWUGGY and sBLIMP scores than To.en-I.M. This outcome is expected, as the model must predict additional acoustic information beyond semantic tokens, which primarily encode linguistic content. Consequently, given fixed model parameter bud- get, language modeling performance naturally declines as the model allocates capacity model acoustic information. This effect is also evident in the low M-MOS of Token-LM Acoustic, where the acoustic tokens may capture excessively detailed information, such as recording noise, which does not contribute meaningfully to synthesis.
SWUGGY and sBLIMP metrics, performance decreases as 8 increases. This finding aligns with our discussion in Section 3.3, where we discussed how lower , values encourage better reconstruction, but make it harder for the autoregressive model to effectively model Z
Varying Table 4 shows that increasing 7 leads to worse pitch reconstruction, as measured by Fo-RMSE, but im- proves CER. This result indicates that 7 governs the type of information captured in the variational feature Z Wiit a higher the system prioritizes the prediction of semantic to- kens. Therefore, the variational feature Z is encouraged to encode more phonetic information, resulting in lower CER and MCD. In contrast, lower 7 encourages Z to focus more on encoding pitch-related information, as indicated by the lower FM-E Then, we analyze subjective measures and observe that both M-MOS and N-MOS favor a lower y, We attribute the performance decline to the increased diffi- culty in autoregressive generation of Z By increasing the weight of C the model sacrifices its focus on minimizing Ci which in turn compromises its ability to model Z"
However, one may question why the trend in the SWUGGY and sBLIMP scores does not align with the M-MOS evalua- tion. We analyze the ASR transcriptions from the compared methods and observe that the transcriptions of Token-LM do have higher meaningfulness than those of other approaches, consistent with the trend of the SWUGGY and sBLIMP scores. However, after listening to the audio samples, we found that the natural prosody of our proposed method sig- nificantly improves intelligibility. Although Whisper ASR can still transcribe speech of unnatural prosody generated by To,en-LM, human raters often needed multiple passes to fully comprehend the linguistic content. In practical applica- tions, interactive dialogue systems must generate speech that users can easily understand in a single pass. The M-MOS score serves as an indicator of the suitability of a system in this regard.
5.3. Removing the Semantic Tokens
Here, we evaluate the utility of the semantic tokens in our proposed approach by training a model that uses only varia- tional features Z This removal corresponds only train- ing a VAE with an autoregressive prior with Equation 3 without the use of discrete semantic tokens.
Table 3 shows the impact of removing the discrete seman- tic tokens from our proposed approach, which is denoted as Proposed ( (-tokens). We find that excluding semantic tokens leads slight improvement in the SWUGGY met- ric compared including them. However, this exclusion significantly worsens the CER, indicating poorer phonetic reconstruction. These results suggest that without discrete semantic tokens, our approach struggles to effectively en- code abstract phonetic information in the variational fea- tures (Z") but still performs well on s sWUGGY, possibly by leveraging other cues. One possible explanation is that the
5.2. Impact of Loss-balancing Parameters
Here, we study the effect of varying the loss-balancing hyper-parameters: B and 7, which are described in Sec- tion 3.3.
Varying 3 Table 3 shows that, for reconstruction metrics (Fo-RMSE, MCD, CER), lower values of 3 result in smaller errors, indicating better reconstruction. However, for the
7

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
Table Results showing the impact of varying the parameter (as described Section 3.3) and the effect of removing phonetic tokens from our proposed approach on both language modeling and speech reconstruction performance. The parameter (as described in Section 3.3) for the proposed methods fixed to 0.5. All models here were trained on the LibriSpeech dataset for lower computation cost.
Method
B
sBLIMP(t) SWUGGY(1)
Fo-RMSE(L)
MCD(L)
CER(L)
0.03
65.56
51.12
16.76
5.19
5.06
Proposed
0.04
65.96
51.40
16.88
5.53
5.43
0.05
66.46
51.77
17.20
5.75
5.45
Proposed (-tokens)
0.04
69.33
51.85
17.47
5.48
13.02
Table 4. Results showing the impact of varying the parameter (as described in Section 3.3) in our proposed approach on both language modeling and speech reconstruction performance. The 3 parameter (as descrihed Section 3.3) is fixed to 0.04. M-MOS denotes the meaningfulness mean opinion score, and N-MOS denotes the naturalness mean opinion score, both presented with 95% confidence intervals. All models were trained on the Libri-light dataset.
7
SWUGGY()) sBLIMP(1) Fo FO-RMSE(L)
MCD(()
CER(L)
M-MOS(()
N-MOS(1)
0.5
60.48
59.88
16.56
5,43
4.35
3,45 + 0.09
3.60 + 0.10
1.0
59.41
59.12
17.06
5.36
4.05
3.31 0.09
3,46 0.10
2.0
58.19
58.19
17.41
5.21
3.75
3.07 + 0.09
3,26 =
adapted to this new set of semantic tokens. For our experi- ments, we used the official SpeechTokenizer checkpoint 8.
Table Results of speech continuation evaluation for comparison on different semantic token extraction methods detailed in Sec- tion 5.4. M-MOS and N-MOS refer to the meaningfulness and naturalness mean opinion score, presented along with 95% confi- dence intervals. All models were trained on the Libri-light dataset.
As shown in Table 5, our approach achieved supe- rior naturalness and meaningfulness scores compared to SpeechTokenizer-LM. This verifies that our framework effec- tively enhances various approaches to extracting semantic tokens.
Method
M-MOS(1)
N-MOS(t)
Ground Truth
3.87 + 0.08
3.97 = 0.08
SpeechIooennier--IM
3.26 + 0.09
3.33 0.10
Proposed
3.68 + 0.09
3.61 + 0.10
Flexibility with Different Decoders Additionally, for both SpeechTokenizer and our proposed method, we did not adopt the diffusion decoder mentioned in Section 3.5. Instead, we predicted the remaining RVQ tokens from the semantic tokens (or semantic tokens and variational features for our approach) and leveraged the pre-trained Speech- Tokenizer decoder for speech reconstruction. As noted in Section 3.5, our training framework is adaptable and is not tied a specific decoder type. We adopted diffuusion-based decoder for simplified training and fair comparisons in our previous work. The empirical results in Table 5 further vali- date this flexibility, as our model still achieves high human evaluation MOS scores with a different decoder.
synthesized non-existent words in sWUGGY, being out-of- domain for the text-to-speech system, may exhibit subtle prosodic irregularities that our model is able detect. On the other hand, the best reconstruction results are obtained when semantic tokens are included, removing them leads to worse reconstruction metrics.
5.4. Generalization to Different Semantic Tokens
In Section 5.1, we demonstrated the effectiveness of our proposed approach using semantic tokens derived from Hu- BERT representations. Here, we investigate its performance with an alternative approach to extracting semantic tokens, SpeeccTokenizer (Zhang et al., 2024). SpeechTokenize quantizes speech using Residual Vector Quantization (RVQ), which optimizes for reconstruction. However, its first-level RVQ tokens additionally minimizing distillation loss with HuBERT representations to encode content. We replace the semantic tokens in Token-LM with the first-level RVQ tokens from SpeechTokenizer, naming this new baseline SpeeccKToeniier--L.M Our proposed method was similarly
6. Related Work
Emerging speech language models typically use discrete semantic tokens for autoregressive modeling. These tokens are often obtained by k-means clustering of features ex- tracted from self-supervised pre-trained models (Hsu et al., 2021; Chen et al., 2022). For instance, Lakhotia al. (2021) used semantic tokens for generative spoken language mod-
hhup:://ithaaacom////nnggnnnnnnf Tokenizer/
8

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
eling (GSLM). Subsequently, Kharitonov et al. (2022) en- hanced this approach by incorporating pitch information alongside semantic tokens joint inputs to the autoregres- sive model. Our proposed approach improves upon this line of research by using a variational autoencoder to automat- ically learn paralinguistic speech attributes in conjunction with the autoregressive model. Borsos et al. (2023) proposed a two-stage approach for the decoder that used acoustic to- kens (Zeghidour et al., 2022; Défossez. al., 2023). This type of framework is also widely used in text-to-speech sys- tems (Chen et al., 2025; 2024). In contrast, our approach focuses on the joint modeling of linguistic and paralinguistic features by enhancing the inputs to the autoregressive model rather than improving the decoder.
8. Limitations and Future Work
Our results indicate that the performance of our proposed approach is sensitive the choice of and 7. Future work will explore automated methods for tuning these hyper-parameters. Additionally, evaluation is limited to English datasets, and it remains unclear if the approach generalizes to languages with different prosodic patterns. Future work will extend training and evaluation to additional languages to assess cross- cross-lingual applicabil- ity. Finally, our model has a relatively small number of parameters and is trained on a smaller dataset compared to existing frameworks (Hassid et al., 2023; Rubenstein et al., 2023). We plan to scale up both the model and the training data to examine whether our findings hold with increased computational resources and larger datasets.
Recently, line of research has emerged focusing on im- proving speech language models through the integration of text-based models. Hassid et al. (2023) initialized their speech language model using pre-trained text-based large language model (LLM). Similarly, Rubenstein et al. (2023); Maiti et al. (2024) expanded the vocabulary of pre-trained text-based LLMs by integrating the semantic tokens. Build- ing on this, Yang al. (2024); Du al. (2024) further explored multi-task training involving text-conditioned gen- erative speech tasks, combining text and audio within single LLM. We note that our proposed approach takes different direction but can still be integrated with these ap- proaches. For example, one could initialize the transformer in our autoregressive model using parameters from a text- based LLM. Recent works (Défossez al., 2024) incorporate discrete acoustic tokens directly into autoregressive modeling. How- ever, these approaches often require complex designs, such as delay patterns and text-based pretraining. In Section 5, we demonstrate that directly incorporating acoustic tokens to autoregressive modeling significantly affects the genera- tion of linguistic content, while our method does not.
A Acknowledgments
This work was supported by grant from Apple. Any views, opinions, findings, and conclusions recommendations ex- pressed in this material are those of the authors and should not be interpreted as reflecting the views, policies or po- sition, either expressed or implied, of Apple. We thank Professor Shinji Watanabe for his valuable feedback. We also thank the reviewers and the area chair for their insight- ful comments and suggestions.
Impact Statement
We proposed an approach that improves the naturalness of speech language models without compromising their language proficiency, which can be leveraged by existing paradigms in this literature. While a model that generates more natural speech can enhance the user experience in conversational agents, it can also be exploited for harmful purposes, such as creating fake videos or making spam phone calls.
7. Conclusion
References
In this work, we proposed an approach that combines a variational autoencoder with existing token-based based speech language models. We conducted experiments to evaluate its effectiveness in terms of language capability and syn- thesis naturalness. Empirical evaluations suggest that our proposed approach, in contrast with other recent techniques, is capable of producing synthesis with better subjective meaningfulness and naturalness. Additionally, we examined the effects of the weights of different loss terms, ,3 and ?, on performance. Our findings indicate that 3 governs the amount of information encoded from the mel-spectrogram into the variational feature, whereas 7 controls the type of information encoded within the variational feature.
Adigwe. A., Tits, N., Haddad, K. E., Ostadabbas, S., and Dutoit, T. The emotional voices database: Towards controlling the emotion dimension in voice generation systems, 2018. URL 0806.09514.
abs/
Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grang- ier, D., Tagliasacchi, M., and Zeghidour, N. Audi- olm: A language modeling approach to audio gen- eration. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2523-222 2533, 2023. doi: 10.1109/AASLP.2223..328
9

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
Bowman, S. R., Vilnis, L.., Vinyals, O., Dai, A., Joze- fowicz, R., and Bengio, S. Generating sentences from continuous space. In Riezler, S. and Goldberg, Y. (eds.). Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pp. 10-21, Berlin, Germany, August 2016. Association for Compu- tational Linguistics. doi: 10.18663/W1/K11-10000 URL petts://aclantee wttp://ooooooooooooo1--1-1
Défossez, A., Mazaré, L.,, Orsini, M., Royer, A., Pérez, P., Jégou, H., Grave, E., and Zeghidour, N. Moshi: a speech- text foundation model for real-time dialogue, 2024. URL attps://axxiv.orgg// www. 2 00037.
Eyben, F., Scherer, K. R., Schuller, B. W., Sundberg. J., André, E., Busso, C., Devillers, L. Y., Epps, J., Laukka, P., Narayanan, S. S., et al. The Geneva minimalistic acoustic parameter set (GeMAPS) for voice research and affective computing. IEEE transactions on affective computing, 7 19:0000202, 2015.
Chen, L.-W., Watanabe, S., and Rudnicky, A. A vec- tor quantized approach for text to speech synthesis on real-world spontaneous speech. Proceedings of the AAAI Conference on Artificial Intelligence. 37(11): 12644-12652, Jun. 2023. doi: 2..6099aaai.v37il1 26488. URL https:://oott aaai. org/ index. php/ php/
Fu, H., Li, C.,, Liu, X., Gao, J., Celikyilmaz, A., and Carin, L. Cyclical annealing schedule: A simple approach to mitigating KL vanishing. In Burstein, J., Doran, C., and Solorio. T. (eds.). Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies Volume (Long and Short Papers), pp. 240-250, Min- neapolis, Minnesota, June 2019. Association for Compu- tational Linguistics. doi: 10.18653/v1/1199 1021. URL org N00g/NN191110
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J.. Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y. Qian, Y., Wu, J., Zeng, M., Yu, X., and Wei, F. Wavlm: Large-scale self-supervised pre-trate training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6))1505-1518, 2022. doi: 10.1 /L110///TSSP2222.3388888
Hassid, M., Remez, T., Nguyen, T. A., Gat, I., Conneau, A., Kreuk, F., Copet, J., Defossez, A., Synnaeve, G., Dupoux, E., Schwartz, R., and Adi, Y. Textually pretrained speech language models. In Oh, A., Naumann, T., Globerson, A.. Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. Curran Associates, Inc., 2023.
Chen, S., Liu, S., Zhou, L., Liu, Y., Tan, X., Li, J., Zhao, S., Qian, Y., and Wei, F. VALL-E 2: Neural codec language models are human parity zero-shot text to speech syn- thesizers, 2024. URL https: //tp://aaavvvv/aa///a//eeeeaaavl org/ abs/ 2406. 05370.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (GELUs). In International Conference Learning Representations, 2017. URL https: / net / www.rmmddBBBoM@M.
Chen, S., Wang, C., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., He, L.,, Zhao, S., and Wei, F. Neural codec language models are zero-shot text speech synthesizers. IEEE Transactions on Audio, Speech and Language Processing, 33:705-718, 2025. doi: 10.1 1199/TAALLPR..222..55500000
Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta- VAE: Learning basic visual concepts with a con- strained variational framework. In International Confer- ence on Learning Representations, 2017. URL https: / //openreveew net/ forum? 2immmmddddyy2ssSS
Défossez, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression. Transactions Machine Learning Research, 2023. ISSN 2835- 8856. URL https::///// /ttp:://oppeeeeeeeeee net/ Eorum? id=iwCd828rR2. Featured Certification, Repro- ducibility Certification.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851. Curran Associates, Inc., 2020. URL httpps://proccednggg... neurips. cc/ paper fwwes filess/pape:/222//ELL oes5cffece54arro@0fffaaallllll1 pdf.
Dinh, L., Sohl-Dickstein, J., and Bengio, S. Density estimation using real NVP. In International Confer- ence on Learning Representations, 2017. URL https: // //openreview. net/ forum? id=HkpbdH99
Du, Z.., Wang, J., Chen, Q. Chu, Y., Gao, Z., Li, Z., Hu, K., Zhou, X., Xu, J., Ma, Z., Wang, W., Zheng, S., Zhou, C., Yan, Z., and Zhang, S. LauraGPT: Listen, attend, understand, and regenerate audio with GPT, 2024. URL https://arxx org/ abs///b//21111 04673.
Hsu, W.N., Bolte, B., Tsai, Y.H.. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. HuBERT: Self- supervised speech representation learning by masked pre- diction of hidden units. IEFF/ACM Transactions on Au-
10

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
dio, Speech, and Language Processing, 29:3451-3460, 2021. doi: 10.1099/AASLP..001332229
Lakhotia, K., Kharitonov, E., Hsu, W.-N., Adi, Y., Polyak, A., Bolte, B., Nguyen, T.-A., Copet, J., Baevski, A., Mohamed, A., and Dupoux, E. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336- 1354, 2021. doi: 10.1162/tac_a_0000 Latiaaaaa_a0a0 a_00400. URL https: //aclanthology owg/ooog/../// /ttp//22...aa1117....
Kahn, J., Riviere, M., Zheng, W., Kharitonov, E., Xu, Q., Mazaré, P.-E., Karadayi. J.. Liptchinsky, V., Collobert, R., Fuegen, C., et al. Libri-light: A benchmark for ASR with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673. IEEE, 2020.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019. URL ht.ps:: //tp:////epeeeeeeeeeeeeee net /
Kharitonov, E., Lee, A., Polyak, A., Adi, Y., Copet, J., Lakhotia, K., Nguyen, T. A., Riviere, M., Mohamed, A., Dupoux, E., and Hsu, W.-N. Text-free prosody-aware generative spoken language modeling. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 8666- 8681, Dublin, Ireland, May 2022. Association for Com- putational Linguistics. doi: 10. 10.18663//1122022aalHooro 593. URL https: httpss///acaanttoooooooo//////// acl- mell-onng.o 593.
6RiC
Maiti, S., Peng, Y., Choi, S., Jung, J.-W., Chang, X., and Watanabe, S. Voxtlm: Unified decoder-only models for consolidating speech recognition, synthesis and speech, text continuation tasks. In ICASSP 2024 2024 IEEE International Conference on Acousties, Speech and Sig- nal Processing (ICASSP), pp. 13326-13330, 2024. doi: 10. (U009//CAASSP444852222...0004
Nguyen, T. A., de Seyssel, M., Rozé, P., Rivière, M., Kharitonov, E., Baevski, A., Dunbar, E., and Dupoux, E. The zero resource speech benchmark 2021: Metries and baselines for unsupervised spoken language model- ing, 2020. URL https: //tp:v//////vv///ae arg/abb///......... 11588.
Kim, J., Kim, S., Kong, J., and Yoon, S. Glow-tts: A generative flow for text-to-speech via monotonic align- ment search. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Infor- mation Processing Systems, volume 33, pp. 8067-8077. Curran Associates, Inc., 2020.
OpenAL. GPT-4 technical report, 2024. URL https: /laaxiv..rg/abs////0 08774.
Kim, J., Kong, J., and Son, J. Conditional variational au- toencoder with adversarial learning for end-to-end text- to-speech. In Meila, M. and Zhang, T. (eds.), Pro- ceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5530-5540. PMLR, 18-24 Jul 2021. URL https: /ptp/ceeepoggsssceeeeepppcc...se mlr, press/ v139 1L39/kim22ft html.
Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 9206-5210 2015.
doi: 10. 10.1109/CAASSPP2115.77777
Perez, E., Strub, F., de Vries, H., Dumoulin, V., and Courville, A. FiLM: Visual reasoning with a general conditioning layer. Proceedings of the AAAI Conference Artificial Intelligence, 32(1), Apr. 2018. doi: 10.1609/ aaai aaai.v32i1.11677 URI. hRt.tttss://ojs.eee index. httph///waa/attttll//eeee/e/////
Kim, J. W., Salamon, J., Li, P., and Bello, J. P. Crepe: A convolutional representation for pitch estimation. In 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 161-165, 2018. doi: 10.1109//ASSPP.211....08 8.8461299.....
.rgg
Press, O., Smith, N., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapo- lation. In International Conference on Learning Represen- tations, 2022. URL https: woweeeveeieeeeeeeeeeeeeeeeeeeeee ew .nt// forum? Rim.middddss@PPCCW
Kingma, D. P. and Welling, M. An introduction varia- tional autoencoders. Foundations and Trends@ in Ma- chine Learning, 12(4)3307-32,, 2019. ISSN 1935-8245. doi: 10.1561/22000000556 URL http: /oii. org/10.1 15611..1611111 /220000000
Radford, A., Kim, W., Xu, T., Brockman, G., Mcleavey, C., and Sutskever, I. Robust speech recognition via large- scale weak supervision. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.). Proceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 28492-288 2892-2818. 18. PMLR, 23-29 Jul
Kong, J., Kim, J., and Bae, J. Hifi-gam Generative ad- versarial networks for efficient and high fidelity speech synthesis, In Larochelle, H., Ranzato, M.. Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Infor- mation Processing Systems, volume 33, pp. 17032-17033. Curran Associates, Inc., 2020.

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
press/ 2023. htRLttttt:///oooooo ngs mlr. .tt..ggg...ssss/////.... v202/radford23a... html.
Sun, G., Zhang. Y., Weiss, R. Cao, Y., Zen, H., Rosenberg, A., Ramabhadran, B., and Wu, Y. Generating diverse and natural text-to-speech samples using quantized fine- grained vae and autoregressive prosody prior. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6699-6703, 2020. doi: 10.1 00.1009/CASPP4477222000000
Reyes-Gonzälez, H. and Torre, R. Testing the bound- aries: Normalizing flows for higher dimensional data sets. Journal of Physics: Conference Series, 2438 (1):012155, 02 2023. Copyright Published under li- cence by IOP Publishing Ltd. This work is published un- der http://creativecoomnns. http///ecooooooooooooiiieeeee/// org/licenses/by/3.00 (the "Li- cense"). Notwithstanding the ProQuest Terms and Con- ditions, you may use this content in accordance with the terms of the License; Last updated- 2023-11-28.
Ulyanov, D., Vedaldi, A., and Lempitsky, V. Instance normalization: The missing ingredient for fast styliza- tion, 2017. URL https:: penttts://arxxtt //tp///avvvvv//////////////aawvv http//awwwv.....aa//aaa/aa woo.orgg//aao. httg//////a/ 1607 08022.
Rezende, D. and Mohamed, S. Variational inference with normalizing flows. In Bach, F. and Blei, D. (eds.), Pro- ceedings of the 32nd International Conference on Ma- chine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1530-1538, Lille, France, 07- 09 Jul 2015. PMLR. URL https:h///// /pwoceedingg.... mlr. pwwps/eesseeeee. / vv7 wewendeleeedeeeeeee html.
Vahdat, A. and Kautz, J. Nvac: A deep hierarchical vari- ational autoencoder. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19667-19679. Curran Associates, Inc., 2020.
Weston, J., Lenain, R., Meepegama, U., and Fristed, Learning de- de-identified representations of prosody from raw audio. In Meila, M. and Zhang, T. (eds.), Pro- ceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 11134-11145. PMLR, 18-24 Jul /
Ronneberger, O., Fischer, P., and Brox, T. U-Net: Convolu- tional networks for biomedical image segmentation. In Navab, N., Hornegger, J., Wells, W. M., and Frangi, A. (eds.), Medicul Image Computing and Computer-Assisted Intervention - MICCAI 2015, pp. 234-241, Cham, 2015. Springer International Publishing. ISBN 978-3-319- 2457444.
2021. Utt. http://oocedddeoooo iig.. ml press/ v139 weeston2l..htm.
Xiong, R., Yang, Y., Hc. D., Zheng, K., Zheng, S., Xing. C., Zhang, H., Lan, Y., Wang, L., and Liu, T. On layer normalization in the transformer archi- tecture. In III, H. D. and Singh, A. (eds.), Proceed- ings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learn- ing Research, pp. 10524-10533, PMLR, 13-18 Jul 2020. URL ht:p:: http:htttsp//porceeeeeeeeeeeeeee http///prcceeencceeeeeee mww. press/ v119/x 0tl19/xing20bb.oo html.
Rubenstein, P. K., Asawaroengchai, C., Nguyen. D. D., Bapna, A., Borsos, Z., de Chaumont Quitry, F., Chen, P., Badawy, D. E., Han, W., Kharitonov, E., Mucken- hirn, H., Padfield, D., Qin, J., Rozenberg, D., Sainath, T., Schalkwyk. J., Sharifi, M., Ramanovich, M. T., Tagliasac- chi, M., Tudor, A., Velimirovié, M., Vincent, D., Yu, Wang, Y., Zayats, V., Zeghidour, N., Zhang, Y., Zhang, Z., Zilka, L., and Frank, C. AudioPaLMM: A large lan- guage model that can speak and listen, 2023. URL https:/
Yamagishi, J., Veaux, C., and MacDonald, K. CSTR VCTK Corpus:
/ttp///aavvvvv//.a///aa/e..e.aaa org/ abs 2obs/2305.00 12925.
English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92), 2019. URL https:// doi. Noi.org// /tlg/110.00 10. 7488/4888//8 d6/2
Schuller, B., Steidl, S., and Batliner, A. The interspeech 2009 emotion challenge. In Interspeech 2009, pp. 312- 315, 2009. doi: 10.214 10.2447/meerspeechh000-00
Yang, D., Tian, J., Tan, X., Huang, R., Liu, S., Guo, H., Chang. X., Shi, J., Zhao, S., Bian, J., Zhao, Z., Wu, X., and Meng, H. M. UniAudio: Towards univer- sal audio generation with large language models. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp. F. (eds.), Pro- ceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 56422-56447. PMLR, 21-27 Jul 2024. URL http://pccceeengg...ee/// mlr press// 1235/yang22 html.
Schuller, B., Steidl, S., Batliner, A., Vinciarelli, A., Scherer, K., Ringeval, F., Chetouani, M., Weninger, F., Eyben, F., Marchi, E., Mortillaro, M., Salamin, H., Polychroniou, A., Valente, F., and Kim, S. The interspeech 2013 compu- tational paralinguistics challenge: social signals, conflict, emotion, autism. In Interspeech 2013, pp. 148-152, 2013. doi: 10.21 / 2013-56.
Song. J., Meng, C., and Ermon, S. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021. URL https: http:p:://ppreeee /pppreveeeee.eee net/ forum? www.mmmddsttlliiarrrr....
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural
12

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
audio codec. IEEE/ACM Transactions Audio, Speech, and Language Processing, 30:495-507, 2022. doi: 10.
1109TTASLP.22221312299999
Zhang. and Sennrich, R. Root mean square layer normal- ization. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Zhang, X., Zhang, D., Li, S., Zhou, Y., and Qiu, X. Speech- Tokenizer: Unified speech tokenizer for speech language models. In International Conference on Learning Rep- resentations, 2024. URL https: / http///eeeeee.eeeeeeeeeee (oworummid=AFgSSS
Zhu, Y., Min, M. R., Kadav, A., and Graf, H. S3vae: Self- supervised sequential vae for representation disentangle- ment and data generation. In JEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
13

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
A. Mathematical Derivations
A.1. Equation 3
For notation simplicity, we drop the superseript c of Z' into Z in this proof.
With the parameterized prior, the modeling distribution of X now also depends on th:
Pe.ú (X) -f p (Z)dZ,
Pe.v X, Z)
(X Z Z)p ZZ)
P8.v (Z X)
Pa. (X)
Pa. (X)
Following a similar proof in Kingma & Welling (2019):
Proof.
log Po.v (X) = (z (2)x
log po, ab (X)
pa. [X, Z)
(2)ag(2\xx)
log
Pa.v Z
x)
pa X, Z) [Z X)
(z~as(Zix)
log
P0.v (Z X) [Z X)
(X,
( X)
(2)go(2)X)
log
(Z
log
(Z X)
po ZZ X)
Po.w Z)
E2~go(2/x)
log
DKL(qs (Z
x)]lpa,00
X)).
2o Z X)
Therefore,
Pe.v (X, Z)
ELBO (2
log
(Z (Z X)
PX Z)px
(Z)1g(Zi
log
(
X)
Pio Pro(Z) (Z)
Ezzag
[log Po
X
[i)] + Ezzg (Z(X)
log
4 (Z X
(2)q,
plog X Z)] DKI
x)
(Z)).
With (Z = II (z( X). and (Z) IT Pv (z)
ZTa-1):
yo (Z | X)
(Z X))p((Z)) EZ~ge (2|x)
log
Ps (Z)
go (z (Ze X)
12
log
I 1 Pe (zt
Zi:t-1) 1)
T
qo (x
(2 X)
Ez Ez-as(Z2
log
(2e Z
Ez, [Dxcloo(21 x)llp((F Zı=-1))]
where Z IT 20(2) X).
14

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
Time-wise Flow
Time-wise Flow Block (Reverse)
(T. 4
(T.
Block (Forward)
(T, 512) +
Convolution Layer
Channel Concat
Channel Concat
Zz dz/2)
(I. dz/2)
(I. /2)
(I. (2))
RELU
Swap
(T, 2048)
Blocks
Last Layer Output
Linear
GELU
(1, dz/2)
(1. (1.d/2)
Convolution Layer (I, 512)
LaverNorm Lieear
Linear
Last Layer Output
GELU
14.0
FILM
Instance Norm
LayerNorm
Convolution Layer (I, 512)
(T. d2)
222
Linear
Swap
(I. d,/2)
(I. (2,
(T. d2)
(T. d2/2)
Input Projection Input (T, 80)
Channel Split
Channel Split
Input (T,
Input d2)
(a) Encoder Architecture
(b) Time-wise Normalizing Flow Architecture
Figure 2. (a) Residual block architecture the encoder Φ) (b) Model architecture for the time-wise normalization flow introduced in Section 3.4.
A.2. Equation 4
Proof. Since Orea is straightforward derive from Equation (decompose Z into Z and Zd), here we show how Li and C are derived from the ( (Z) X II (Z)) in Equation
With (Z X) (Z
(Z
X) and (z Z1a-11 p=( Z1:-1)po(2f Z1a-1)::
DKL(g0
Z
X ppp (Z))
(Z Z X)
Ez
log
Pei (Z)
4o lZ
X x)p(( 2 X)
Ez
log
1r=1 Pat (2,
Z1:--1)
g (Z-
(Z
I))p(Zz a
X)
Ez
log
(2f
Z1:--1 (2)
Z Z
Lo (Z c X)
p( d X
Ëz log
+ Ezz log
(2f Z
[T= Pio (2
Zi
Ez
[DKL(9s(2}
Z1=-1))]
Ez,,. blog p (
Z1:-1)]
Ez [log p( z
x)]
Since Ez|log d
X does not depends on any parameters, it can be dropped during optimization.
B. Model Architectures
Encoder qo X) We use different number of residual blocks for the encoder. We use a kernel size of 7: the hidden dimensions used for all models are in Figure 2 (a). The architecture of the residual block is illustrated in Figure (a). Finally, after 3 residual blocks, we apply another instance normalization, followed by separate linear heads to output the mean and log- log-variance of Equation 2. We used the same size encoder for experiments with LibriSpeech and Libri-light. Instance Norm refers to instance normalization (Ulyanov et al., 2017).
Autoregressive Transformer We follow the typical implementation of transformers with Post-LN (Xiong et al., 2020). We use RMSNorm (Zhang Sennrich, 2019) and GELU activation (Hendrycks & Gimpel, 2017). We use ALiBi (Press
15

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
Table Model configuration of the autoregressive transformer for training I LibriSpeech and Lihri-light respectively. This configuration is shared for all comparing methods. feed-forwatt size' refers to the width of the worward linear layer.
Dataset
# of layers
# of heads
hidden size
hord-forward size
LibriSpeech Libri-light
4
8
512
2048
16
16
1024
4096
et al., 2022) for relative positional encoding. We used different model sizes for the LibriSpeech and Libri-light experiments, with the configuration summarized in Table The same configuration is shared for all comparison methods.
Time-wise Normalizing Flow The architecture of our time-wise normalizing flow illustrated in Figure 2 (b). Here, y and the mean and standard deviation that will be multiplied and added to the input. This part mainly follows the implementation of Dinh et al. (2017). The "Last Layer Output" in Figure 2 (b) refers to the output of the last transformer layer. "FiLM" refers to FiLM conditioning (Perez et al., 2018) 1). "Swap" refers to the swapping of the two inputs in their channel order. We used 4 flow blocks for all experiments.
Diffusion Decoder For our diffusion decoder 8, we apply the same residual block as in Figure 2 (a). However, here we have additional skip connections between the output of residual blocks following the commonly-used U-Net architec- ture (Ronneberger et al., 2015). We encode the current diffusion step with sinusoidal positional encoding, linear project and add it to each time frame of the output of the first convolution layer each of the residual blocks. For both data sets, we used six residual blocks, with the same hidden dimensions and kernel size as the encoder Φb
Utterance Encoder The utterance encoder consists of blocks, where each block sequentially includes convolution with stride and kernel size 4, followed by instance normalization (Ulyanov al., 2017) and RELU activation. The hidden size of the convolution layer is: 128, 256, 512. Afterward, simple time-averaging is applied to the output generate an utterance-level embedding.
C. Training Details
For model training, we use the AdamW optimizer (Loshchilov & Hutter, 2019) with 331 0.9, B2 = 0.98. We used weight decay of 0.01 for LibriSpeech models and 0.1 for the Libri-light models. We trained the models with mixed precision. For Libri-light models, we use 2 L40S GPUs with gradient accumulation of step size 2. This makes the effective batch size 192. We trained for 600k update steps. We warm /3 from 0 to the final value in the first 30k update steps. It takes about 14 days to train the Libri-light models.
For LibriSpeech models, we discovered that methods involving discrete tokens suffer from early overfitting (but not in Libri- light). Therefore, we train these models (including our proposed approach) to only 100k steps. For the diffusion decoder of Token-LM and To pitch, we separately train them to 500k steps, where we observe marginal improvement of loss functions between epochs. For pure variational approaches, we train to 400k steps as we did not observe overfitting. We used the same effective batch size on the 2 L40S GPUs but without gradient accumulation. For LibriSpeech models, warm 3 from 0 to the final value in the first 20k update steps. It takes about 2 days to train for the 400k step models and less than I day for the 100k step models. For both models, we use an initial learning rate of 5e - 4 and apply cosine learning rate decay to 5e 5.
For the input to the utterance encoder, we randomly cropped the segment to be between 2 and 4 seconds. For diffusion model. we use LI loss to predict the diffusion noise and apply the cosine schedule for the diffusion noise variance.
D. Subjective Evaluation
We use crowd-sourcing for subjective human evaluation on speech meaningfulness and naturalness. The recruited raters speak English and were paid at least the minimum wage. We sample 100 prompts from LibriSpeech development subsets, crop the first 3 seconds, and feed to each model to produce a 10-second continuation (total 13 seconds). The same 100 prompts are used across all methods for fair comparison. Since we not train our model to predict the end of speech, we
16

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
Please listen to the computer--eneratted speech sample below and rate how well its grammar and content convey meaningful information. Focus avaluating the grammar and the content, not the naturalness quality of the speech.
How meaningful this speech sample terms grammnr and content?
Very
Poor
Figure 3. A screenshot of the Meaningfulness (M-MOS) assessment task, the crowd-sourced rater sees
listen the computer generated sample below how (l.e,, human-like) sounds
(very unnatural) (very
0:00
0:11
>>
How natural (le., human-soundingl this speech sample?
Very unnatural
Somewhat unnatural
Neither natural nor unnatural
Somewhat natural
Very natural
Figure 4. screenshot of the Naturalness (N-MOS) assessment task, as the crowd-soureed rater sees it.
Table Performance varying latent dimension di proposed approach (without speech tokens). Models trained LibriSpeech.
d
SWUGGY(1) sBLIMP(1)
FO-RMSE(L)
MCD(()
CER(J)
4
69.33
51.85
17.47
5.48
13.02
16
73.49
51.69
16.68
5.37
7.80
64
73.25
50.91
17.37
5.35
7.79
observed that some synthesis ends earlier than 13 seconds. We use pre-trained voice activity detection from pyannote" to post-process the samples, removing trailing silences and non-speech that might affect evaluation.
In Figures 3 and 4, we provide screenshots what raters see during the evaluation. Raters are presented with a spoken utterance and are instructed rate its naturalness or meaningfulness on five-point Likert scale, where corresponds very unnatural or meaningless and corresponds very natural or meaningful.
E. Dimension of the latent variable de
Table 7 presents our results of increasing the latent dimension dç. We perform the sweep in the variational approach without semantic tokens for simplicity. From Table 7, we observe that increasing the latent dimension from to 16 results in uniform improvements across the measures. However, further increasing the dimension from 16 leads to marginal degradation. We speculate that this performance plateau may arise from the difficulty normalizing flows face when modeling higher-dimensional distributions (Reyes-González & Torre, 2023).
F. Discrete token vocabulary size
Table 8 shows our evaluation results on semantic token models (Token-LM) trained with varying k. Here, & refers to number of clusters for the k-means clustering on obtaining the discrete token, which equal to the vocabulary size the discrete tokens. Our result is consistent with (Maiti al., 2024), which shows that & 200 obtains the best sWUGGY score.
17

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
Table 8. Comparison model trained on different number of discrete tokens k. Models are trained on LibriSpeech.
k
sWUGGY(1)
sBLIMP(1)
Fo Fo-RMSE(
MCD(J)
CER(L)
59.63
52.49
41.11
6.49
11.87
200
67.32
52.46
35.41
6.23
5.40
1000
65.11
50.99
32.60
5.99
4.48
Table Performance speech emotion recognition models trained on different features. The features are extracted from models pre-trained on Libri-light using our proposed method.
Method
Emotion Recognition (ACC, %)
Tokens
57.46 + 1.59
Variational Features
91.57 + 0.35
Tokens Variational Features
92.74 + 0.37
Reconstruction metrics indicate that & = 200 provides a significant improvement over = 50, whereas increasing 200 1000 produces only a marginal gain. Interestingly, having larger seems to negatively impact sBLIMP. We speculate that the small vocabulary size (k = 50) is adequate distinguish word-level changes in sentences, but insufficient to detect subtle phonetic variations within words.
G. Scoring SWUGGY and sBLIMP
Token-LM To obtain the scores for sWUGGY and sBLIMP for semantic token only models, we follow (Borsos et al., 2023) and use the log-likelihood returned by the model normalized by the sequence length.
Token-LM + Pitch, Token-LM + Acoustic Tokens, Proposed Methods For methods that have additional inputs other than the discrete tokens, we only use the model's likeliood the discrete tokens. We do not use the likeliiiddoodd of the 7 assume that the discrete tokens zd should contain all the information needed for sWUGGY and sBLIMP. In practice, we do observe that including the log-likelihood of the Ze slightly lowers the score for our proposed method.
Since there are discrete tokens involved in Proposed token, we directly use the log-likelihood of Z The likelihood can be estimated using Equation 6.
Proposed token
For Proposed and Proposed w.o. token, to ensure a deterministic outcome, we again use no (x, from Equation 2 directly as Z' instead sampling Z from (Z X).
H. Side Experiments on inspecting learned features
Speech Emotion Recognition We evaluate speech emotion recognition on the EmoV-DB (Adigwe al., 2018) dataset. We follow a 9:1 split on training and testing for the dataset. The dataset contains five emotion categories: amused, angry, neutral, disgust, and sleepiness. We train a classifier with the same structure to predict emotion categories based on different features. The experiments are repeated 20 times to report the mean and 95% confidence interval. From Table 9, we can observe that the variational features alone obtain significantly better performance compared to tokens, showcasing its capability of capturing paralinguistic information. Combining both tokens and variational features gives slight improvement over using variational features alone.
Speaker Identification For speaker identification, evaluate the performance on the VCTK (Yamagishi al., 2019) dataset, which consists of read English sentences, with 400 sentences each from 110 speakers. We again follow 9:1 train-test split and repeat each run 20 times to report the mean and 95% confidence interval. We also evaluate our embedding of utterance, which is designed to capture static utterance-level information (see Section 3.5). From Table 10, we can see that using tokens only results in poor speaker identification accuracy. With variational features, the classifier obtains improved accuracy. We attribute this improvement to the fact that speaking styles can be captured in the variational features to classify speakers. On the other hand, the utterance embedding outperforms the other features in this task. These results support our
18

---

A Variational Framework for Improving Naturalness in Generative Spoken Language Models
Table 10. Performance of speaker identification models trained different features. The features extracted from models pre-trained on Libri-ligh light using our proposed method.
Method
Speaker Identification (ACC, %)
Tokens
7.08 + 0.40
Variational Features
63.41 + 0.43
Tokens + Variational Features
63.13 + 0.45
Utterance Embedding
94.06 + 0.32
claim that the utterance encoder encodes global speaker information while variational features capture local paralinguistic attributes.
I. Conditional independence assumption of 2 and 2f
In general, Ze and Zd are not independent, since the language content can imply the paralinguistic information, and vice versa. However, our modeling assumes only conditional independence. Specifically, the past generations Z are first passed through the autoregressive transformer d to produce the intermediate representation o₂ Trons pransfrrmee Then, separate heads predict zf and zt based or. This framework assumes that the transformer former ( (Z1:-1)). learn o, such that of and of become conditionally independent given o₁. Given the transformer's 's modeling capacity, believe it can extract shared information (or) between of and z from Z while delegating the distinct information to their respective heads.
19


Reviews:
[{"review_id": "IUU164H7jr", "reviewer": "Reviewer_enQh", "summary": "The authors propose a variational approach that directly encodes desired information from raw audio inputs, addressing the challenge of preserving prosody when modeling discrete speech codes primarily focused on phonetic content (such as HuBERT tokens). Their variational approach shows improved performance compared to baselines such as Token-LM (phoneme-based tokens only), Token-LM+acoustic (phoneme and acoustic tokens), and Token-LM+acoustic baseline.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate both the recognition of our formulation's correctness and the critical feedback from the reviewer. We address the feedback below.\n\n## Computational Complexity\n>The model relies on a diffusion decoder, which increases computational complexity compared to predicting discrete acoustic codecs, making it unsuitable for real-time speech generation applications.\n\nWe would like to clarify several points:\n - **Decoder flexibility**: Our variational approach isn't tied to a specific decoder. We chose diffusion for easier training, but our method works with various decoding strategies, including discrete token prediction. There is no constraint on the decoder $\\theta$ used to parameterize $p_\\theta(\\mathbf{X}\\mid\\mathbf{Z})$. To validate this, we conducted additional experiments replacing the diffusion decoder with a token decoder which converts semantic tokens to SpeechTokenizer tokens [1], then used a pre-trained SpeechTokenizer decoder to generate speech. [Table 1](https://anonymous.4open.science/api/repo/icml-rebuttal-BDD8/file/ICML-rebuttal-2.pdf) shows that this variant (*Proposed + Token Decoder*) achieves similar or better performance compared to the diffusion approach.\n - **Fair comparison**: We used the same diffusion decoder architecture across all comparing methods. We ensure a fair comparison where only the modeling approach varies, isolating the impact of our variational method.\n - **Research focus**: Our primary contribution is on improving prosodic naturalness with the variational approach, not on optimizing for real-time applications. The improvement in naturalness MOS validates the effectiveness of our method.\n - **Reasonable RTF**: On a single NVIDIA L40S GPU with batch size 1, our current implementation achieves an RTF of 0.68 (<1). While this is not our focus, it shows that this approach remains practical. Importantly, the diffusion decoder only operates once after the AR model completes its generation, so the additional latency introduced is not as significant as it might initially appear.\n\nWe will make adjustments to Sections 3 and 4 to emphasize these points and avoid confusion.\n>Can the mel-spectrogram be decoded only after all continuous features have been generated?\n\nYes. Our current implementation with diffusion decoder and HiFi-GAN vocoder is not streamable. However, as mentioned above, our framework is compatible with any type of decoder, including streamable ones.\n\n## Use of Re-implemented Models\n>The baselines used are re-implemented versions of existing models rather than direct comparisons with the original models.\n\nWe re-implemented the baseline models following standard scientific methodology to ensure a controlled and fair comparison. This decision was made for several critical reasons:\n\n- **Architectural consistency**: We needed to maintain consistent architecture and model size across all methods. Our re-implementations use identical decoder and autoregressive model architectures, differing only in the specific modeling approach being evaluated. This isolates the impact of our variational method, which is the primary contribution of our work.\n- **Data consistency**: We ensured all models were trained on exactly the same data. Off-the-shelf implementations typically differ in training datasets, preprocessing pipelines, and model sizes, introducing confounding variables that would make it impossible to attribute performance differences specifically to our methodological innovation.\n\n## Comparison with More Recent Approaches\n>If the model were compared to more recent approaches using more compressed acoustic tokens such as WavTokenizer, it would likely provide better insight into prosody than an LLM trained only on SSL-based semantic tokens.\n\nWe include the Token-LM + Acoustic baseline, which employs acoustic tokens from residual vector quantization (RVQ), capturing information beyond what semantic tokens.\nTo compare with recent approaches on acoustic tokenization, we have included an additional comparison with SpeechTokenizer in [Table 1](https://anonymous.4open.science/api/repo/icml-rebuttal-BDD8/file/ICML-rebuttal-2.pdf) (details in our response to Reviewer XtBM). This expanded analysis further validates the advantages of our variational method across different tokenization techniques.\n\n## Token-LM Baseline with Limited Clusters\n>Token-LM models very small discrete tokens with only 200 clusters, the results are not sufficiently comprehensive.\n\nWe specifically chose k=200 for our HuBERT baseline after conducting a sweep across different values (k=50, 200, 1000), finding that k=200 performed the best for language modeling, as detailed in Appendix F. Additionally, the new added SpeechTokenizer baseline has 1024 clusters, which our model still outperforms.\n\nThank you for your valuable feedback which has helped us articulate our contributions more clearly.\n## References\n\n[1] X. Zhang et al., SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models, ICLR, 2024", "claims_and_evidence": "The comparisons in this work mainly focus on Token-LM models derived from self-supervised models like HuBERT, which do not use a reconstruction loss. However, if the model were compared to more recent approaches using more compressed acoustic tokens such as WavTokenizer, it would likely provide better insight into prosody than an LLM trained only on SSL-based semantic tokens. This weakens the motivation behind the proposed approach.", "methods_and_evaluation_criteria": "Not really. The use of a VAE to encode information absent in SSL-based discrete semantic tokens is reasonable. However, the model employs a diffusion decoder to reconstruct continuous representations, requiring 100 iterative steps for generation. This significantly increases computational complexity compared to predicting discrete acoustic codecs, making it unsuitable for real-time speech generation applications.", "theoretical_claims": "The VAE formulation is correct, and I have reviewed the loss derivation provided in the appendix.", "experimental_designs_or_analyses": "In terms of experimental design, the baselines used are re-implemented versions of existing models rather than direct comparisons with the original models. As a result, the validity of the reported evaluation results is somewhat diminished.", "relation_to_broader_scientific_literature": "While this approach could serve as an alternative to acoustic codecs, its reliance on a separate diffusion model for continuous feature generation increases computational cost significantly. Consequently, it is unlikely to be practical for real-time generation or streaming scenarios.", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "Weakneeses\n\nThe authors compare their approach only with re-implemented models rather than conducting direct comparisons with existing baseline models. Since the baseline is based on Token-LM, which models very small discrete tokens with only 200 clusters, the results are not sufficiently comprehensive.\n\nMethodologically, as mentioned above, it is difficult to determine what advantages this approach has over LLM-based models that predict acoustic tokens with delay patterns. Additionally, the reliance on a diffusion decoder raises concerns about whether this method can be used in streaming scenarios.", "other_comments_or_suggestions": ".", "questions_for_authors": "What is the real-time factor (RTF) of the model's generation speed? \n\nCan the mel-spectrogram be decoded only after all continuous features have been generated?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a4xchDPgmf", "reviewer": "Reviewer_XtBM", "summary": "This paper proposes a variational framework to enhance the naturalness of generative spoken language models by jointly learning continuous paralinguistic features and discrete semantic tokens. Traditional token-based models often neglect prosodic information, leading to unnatural speech. The authors address this by integrating a variational autoencoder (VAE) with an autoregressive prior to automatically encode continuous speech attributes (e.g., pitch, energy) alongside pre-extracted semantic tokens.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their constructive feedback. We address your concerns comprehensively below. We present the Tables of new experiment results [here](https://anonymous.4open.science/api/repo/icml-rebuttal-BDD8/file/ICML-rebuttal-1.pdf).\n\n\n\n\n## Comparison with Additional Baselines\n> Some work such as SpeechTokenizer which uses RVQ-based methods and distills semantic tokens in the first layer can also address the issues: present the paralinguistic features while modeling the semantic features. The author should discuss them.\n\n\n> The author should compare it with SpeechTokenizer; 2) the generative model is also not well designed. I suggest using AR to predict the first semantic tokens and NAR to predict the residual acoustic features which use semantic tokens as conditions.\n\n\nWe appreciate your points about comparing our approach with additional baselines. We acknowledge that our comparison could be more comprehensive and have implemented a SpeechTokenizer-based approach as suggested.\n\n\n**SpeechTokenizer Baseline**: We used the official SpeechTokenizer checkpoint for the encoder and decoder components (semantic and acoustic tokens ↔ speech). But we trained our own NAR decoder for predicting acoustic tokens from semantic tokens, as the original implementation requires text input (it was originally designed for TTS). We also re-trained the autoregressive model for semantic token prediction due to token set differences (1024 vs 200 tokens).\n\n\n\n\nAs shown in [Table 2](https://anonymous.4open.science/api/repo/icml-rebuttal-BDD8/file/ICML-rebuttal-1.pdf), our approach still achieves superior naturalness and meaningfulness scores compared to the new established SpeechTokenizer baseline. Perceptual listening revealed that the SpeechTokenizer approach indeed improves upon Token-LM in prosody patterns. However, it introduces more audio quality artifacts compared to Token-LM which uses a diffusion-based decoder to convert semantic tokens to speech. These artifacts offset the benefits of better prosody naturalness in the human evaluation scores. Note that we re-evaluate all methods in the human evaluation for fair comparison.\n\n\nWe will revise Sections 5 and 6 to incorporate these new results.\n\n\n## Roles of VAE and Comparison with Existing Works\n\n\n> Why such VAE is necessary? What benefits can it bring compared with existing works?\n\n\nOur approach offers distinct advantages over existing methods:\n1. **Compared to SpeechTokenizer and Token-LM**: In these baselines, the AR model accesses only semantic tokens. Our AR model additionally accesses variational features encoding prosody, better utilizing the stronger AR model for improved naturalness.\n2. **Compared to Token-LM+Acoustic**: The acoustic (RVQ) tokens from this baseline is extracted without supervision from the AR model, while our approach jointly optimizes variational features for both reconstruction and AR prediction, where the encoder receives training signal from the AR model.\n3. **Compared to Token-LM+Pitch**: Our approach doesn't require hand-engineered features, as it learns prosodic information in an unsupervised fashion.\n\n\nThe VAE framework enables joint learning of variational features optimized for both reconstruction and AR prediction, yielding superior naturalness and meaningfulness of the syntheses. We'll revise Section 6 to emphasize these advantages.\n\n\n\n\n## Additional Metrics\n\n\n> Why use CER instead of WER? \n\n\nWe originally chose CER following prior work [1], which evaluates pronunciation errors more accurately. As requested, we've added WER measurements in [Table 1](https://anonymous.4open.science/api/repo/icml-rebuttal-BDD8/file/ICML-rebuttal-1.pdf). The WER results follow the same trend as CER.\n\n\n> The speaker similarity metric is not reported.\n\n\nIn [Table 1](https://anonymous.4open.science/api/repo/icml-rebuttal-BDD8/file/ICML-rebuttal-1.pdf), we add speaker similarity metrics, which is calculated using the cosine similarity of the speaker embeddings extracted from a pre-trained speaker verification model. Our method achieves a speaker similarity score slightly lower compared to the baseline approaches (0.42 v.s. 0.45). However, as the reviewer can verify from our audio samples, the perceptual difference in speaker identity is minimal, while naturalness and intelligibility improvements are noticeable. This represents a reasonable trade-off for most speech applications.\n\n\nWe believe these additions and clarifications address the reviewer's concerns and strengthen our paper.\n\n\n## References\n\n\n[1] K. Lakhotia, et al., On Generative Spoken Language Modeling from Raw Audio, Transactions of the Association for Computational Linguistics, 2021.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The method is well-written and easy to follow. \n\nSome questions:\n\n1.  This paper proposes VAE with autoregressive prior, to learning acoustic features to enhance naturalness in semantic token based LLM. However, some work such as SpeechTokenizer which uses RVQ-based methods and distills semantic tokens in the first layer can also address the issues: present the paralinguistic features while modeling the semantic features. The author should discuss them.\n\n[1] SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models\n\n2. Why use CER instead of WER? The speaker similarity metric is not reported.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "The comparison between the proposed and baseline methods is not sufficient. Generally, this work models the residual acoustic features to minimize the information loss of semantic token based LLM generation. \n\nGenerally, there is some existing (not limited) serial of works: 1)  some works such as SpeechTokenizer [1] directly distill semantic tokens in the first layer of RVQ and use AR to generate the semantic tokens and NAR model to predict the rest. This is different from the setting in \"Token-LM acoustic\" since the acoustic tokens are learned without semantic token supervision, and the generative process is not well-designed. 2) some works only use AR/NAR methods to generate the semantic tokens and model the acoustic features by an additional diffusion model. It can also model the paralinguistic features to enhance the naturalness. These works include but not limited to MaskGCT[2], CosyVoice[3], tortoise-tts[4]\n\n[1] SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models\n[2] MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformer\n[3] Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens\n[4] tortoise-tts: Better speech synthesis through scaling.\n\nMy questions are:\n* 1. The author should compare with more baselines, such as SpeechTokenizer, MaskGCT, et. al.\n* 2. The author should discuss more with existing works. why such VAE is necessary? What benefits can it bring compared with existing works?\n* 3. The \"Token-LM acoustic\" shows good reconstruction results but weak generation performance. But I think it comes from: 1) the tokenizer used is not reasonable, at least, the author should compare it with SpeechTokenizer; 2) the generative model is also not well designed. I suggest using AR to predict the first semantic tokens and NAR to predict the residual acoustic features which use semantic tokens as conditions.", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The strengths and weakness are discussed in the previous sections.", "other_comments_or_suggestions": "No.", "questions_for_authors": "The questions are discussed in the previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vsPhvAJ2x8", "reviewer": "Reviewer_Pvan", "summary": "This paper automatically learns continuous speech attributes (such as pitch, energy, spectrum) through VAE, and jointly model them with semantic tokens to improve the naturalness and language fidelity of generated speech. Experiments show that this method significantly outperforms the baseline model in subjective naturalness scores and does not require manual design of rhythmic features.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough review and acknowledging the strengths of our paper, including the balance between innovation and practicability, the automatic learning of continuous prosodic features, and our rigorous experimental design.\n\nBelow, we address the feedback from the reviewer:\n\n## End-to-End Design Novelty\n\n> This \"end-to-end\" design idea does not seem to be new in the field of speech synthesis.\n\nWe thank the reviewer for this observation. End-to-end is indeed the current trend for building speech synthesis models. Our specific contribution lies in making the prosodic features end-to-end learnable with the speech language model through the integration with VAE. This approach enables automatic learning of continuous prosodic features without manual feature engineering, which we believe represents a meaningful advancement in the field.\n\n## Hyperparameter Sensitivity\n\n> The performance of the model is highly dependent on the tuning of β and γ, which may limit the robustness of the method under different data distributions.\n\nThis is an insightful observation. We agree that hyperparameter sensitivity is an important consideration for most work in this field. In Section 8: Limitations and Future Work, we have acknowledged this challenge and identified exploring automated methods for hyperparameter tuning as an important direction for future research. The current work serves as a first and crucial step in demonstrating that the proposed method effectively improves the prosodic naturalness of speech language models.\n\n## Cross-Lingual Adaptability\n\n> The experiment only verifies the English dataset, while the prosodic patterns of different languages are quite different, and it is unclear whether the model needs to be retrained or adjusted hyperparameters.\n\nThis is an excellent point that highlights an important direction for future research. We hypothesize that our approach of learning continuous prosodic features automatically rather than using hand-designed features offers inherent advantages for cross-lingual adaptation. Unlike rule-based systems that require language-specific expertise, our VAE framework can potentially discover and model language-specific prosodic patterns without human supervision.\n\nWe will add this discussion to the limitations section and highlight it as a direction for future work. Our approach, which learns continuous representations rather than relying on predefined features, may be well-suited to adapt to the diverse prosodic patterns across languages, though this remains to be verified through additional research.\n\n\nWe thank the reviewer for the valuable suggestions, which will help direct our ongoing research efforts.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.\nThe ELBO decomposition and KL divergence derivation are mainly checked to comply with the standard theory of variational autoencoder and are correct.", "experimental_designs_or_analyses": "Yes.\nThe experimental design verifies the effectiveness of the proposed variational method by comparing the three baseline methods of Token-LM, Token-LM+Pitch and Token-LM+Acoustic, and analyzes its regulatory effect on information encoding and generation quality through hyperparameter β and γ ablation studies.", "relation_to_broader_scientific_literature": "This paper continues the idea of [1] to improve the naturalness of generation by combining modeling language and prosodic information, but abandons the hand-designed fundamental frequency features and moves towards end-to-end learning.\n\nIn contrast to the discrete acoustic token method of [2], the advantages of continuous latent variables in prosodic modeling are verified, and the information loss caused by multi-stage discretization is avoided.\n\n[1] Kharitonov, E., Lee, A., Polyak, A., Adi, Y., Copet, J., Lakhotia, K., Nguyen, T. A., Riviere, M., Mohamed, A., Dupoux, E., and Hsu, W.-N. Text-free prosody-aware generative spoken language modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8666– 8681, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long. 593.\n\n[2] Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grangier, D., Tagliasacchi, M., and Zeghidour, N. Audiolm: A language modeling approach to audio generation.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "In this paper, a good balance is struck between innovation and practicability, and the continuous prosodic features are automatically learned by the VAE, which not only avoids the cumbersome process of manually designing prosodic features in the traditional method, but also improves the naturalness of the generated speech by jointly modeling semantic and prosodic information. \n\nThe experimental design is also rigorous, and the validity of the method is verified by multi-dimensional indicators (reconstruction quality, language modeling ability, subjective scoring), and the subjective evaluation adopts standardized processes, which enhances the credibility of the results.\n\nBut this \"end-to-end\" design idea does not seem to be new in the field of speech synthesis.\nAnd the performance of the model is highly dependent on the tuning of β and γ, which may limit the robustness of the method under different data distributions, as wel as the experiment only verifies the English dataset, while the prosodic patterns of different languages are quite different, and it is unclear whether the model needs to be retrained or adjusted hyperparameters.\n\nBut overall, the paper proposes a valuable solution to the problem of naturalness of speech generation, so I give a weak acceptance.", "other_comments_or_suggestions": "No.", "questions_for_authors": "see other strengths and weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J02mwPFH3V", "reviewer": "Reviewer_odWp", "summary": "This paper proposes a variational approach to speech-language modelling in contrast to traditional auto-regressive models. The aim is to capture information other than semantics.\n\n## update after rebuttal\nI checked the results in Appendix H and I think the results are interesting. Why not add those results into the main paper (since we as reviewers are not required to see Appendices)? I increase my score to 4 to acknowledge this effort.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for recognizing both the robustness of our experimental design and the novelty of our contribution. Below, we address each concern raised:\n\n## Emotion and Speaker Recognition\n\n> My main concern is how this method is useful for more practical downstream tasks such as ASR, emotion or speaker recognition, to reflect that capturing the additional (mainly paralinguistic) information is useful. I strongly encourage the authors to conduct at least 2 of the above practical tasks using the variational speech LM and compare it to token-based speech LM to see any potential benefits of using variational methods.\n\nWe appreciate this valuable feedback. We would like to direct the reviewer to Appendix H, where we present our comprehensive evaluation results on speech emotion recognition and speaker recognition tasks. In Tables 8 and 9, our results suggest that the variational features encode speaking styles and prosodic patterns there are useful for both tasks. We kindly direct the reviewer to Appendix H for more detailed experimental setup, results and analysis. Additionally, we would like to note that our main objective remains to improve naturalness of speech synthesis, and the side experiments provide additional evidence that our method is capable of achieving that.\n\n\n## Additional References\n\n> G. Sun et al. \"Generating diverse and natural text-to-speech samples using a quantized fine-grained vae and autoregressive prosody prior\", In Proc. ICASSP. 2020.\nThis should be discussed in section 3.1. This paper is the first to apply a trainable auto-regressive prior in speech synthesis.\n\nThank you for this important suggestion. We will incorporate this reference in Section 3.1 and properly acknowledge its contribution.\n\n## Figure Presentation\n\n> Actually, Fig. 1 can be improved by unifying the font.\n\nWe appreciate this attention and will unify the font in Fig. 1.\n\n## Dataset Selection\n\n> The experiment was conducted using LibriSpeech and Libri-light, which are datasets with quite small variabilities other than semantic information. I believe the variability remains in the speaker representation space, which is not explicitly reflected in the experimental design.\n\nWe thank the reviewer for their valuable feedback. We acknowledge that LibriSpeech and Libri-light are not considered highly expressive. We evaluate on these datasets since they are standard benchmarks in the literature [1, 2, 3]. Importantly, our proposed approach improves synthesis naturalness even on these less expressive datasets. This suggests that our method would likely yield even greater improvements on more expressive datasets, where natural expressive speech synthesis presents additional challenges.\n\nWe hope these responses adequately address the reviewer's concerns and help in the evaluation of our work.\n\n## References\n\n[1] E. Kharitonov, et al., Text-free prosody-aware generative spoken language modeling, in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8666–8681, Dublin, Ireland, May 2022.\n\n[2] Z. Borsos, et al., AudioLM: A language modeling approach to audio generation, IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2523–2533, 2023.\n\n[3] K. Lakhotia, et al., On Generative Spoken Language Modeling from Raw Audio, Transactions of the Association for Computational Linguistics, 2021.", "claims_and_evidence": "Yes. I can find evidence in section 5 supporting the authors claim that the variational method improves the naturalness of synthesised speech.", "methods_and_evaluation_criteria": "Evaluation is comprehensive.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "The experimental design is sound and valid. My main concern is how this method is useful for more practical downstream tasks such as ASR, emotion or speaker recognition, to reflect that capturing the additional (mainly paralinguistic) information is useful. I strongly encourage the authors to conduct at least 2 of the above practical tasks using the variational speech LM and compare it to token-based speech LM to see any potential benefits of using variational methods.", "relation_to_broader_scientific_literature": "This is to my best knowledge the first work to incorporate variational framework in speech language models.", "essential_references_not_discussed": "G. Sun et al. \"Generating diverse and natural text-to-speech samples using a quantized fine-grained vae and autoregressive prosody prior\", In Proc. ICASSP. 2020.\n\nThis should be discussed in section 3.1. This paper is the first to apply a trainable auto-regressive prior in speech synthesis.", "other_strengths_and_weaknesses": "The experiment was conducted using LibriSpeech and Libri-light, which are datasets with quite small variabilities other than semantic information. I believe the variability remains in the speaker representation space, which is not explicitly reflected in the experimental design.", "other_comments_or_suggestions": "Actually, Fig. 1 can be improved by unifying the font.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a VAE approach to capture prosodic information atop semantic speech tokens, in contrast to discrete, possibly designed acoustic features (pitch like in pGSLM, or acoustic tokens like Encodec). This is a replacement with learned continuous representations (R3: the end-to-end part is not novel, and designing has traded with having to tune beta-VAE parameters; the scheme and use-case is, however). Generations from the resulting speech LMs match baseline zero-shot metrics while being more meaningful and notably more natural per subjective metrics. Though not linked in the body (I suggest the authors do), the Appendix showed improvements on emotion recognition and speaker ID, evidence of general effectiveness (led R4 to raise score to 4).

R1 raised valid concerns, such as the cost of diffusion (though authors clarify it is still better than real-time, plus streaming is not the goal). Authors note to R1 that the VAE method is not decoder-specific, and show the SpeechTokenizer decoder also works (a start to addressing R2’s request for more baselines). However, the improvements from that plus use of reimplemented models (to isolate the variational scheme), suggest their baseline may be weak. 

Reviewers generally agree the method is sound and well-presented. Though R1 raised key concerns, they seem to be mitigated and the other three support acceptance, with one solid accept. The work’s results validate a novel alternate approach to current discrete acoustic tokenizers in speech generation, with promising results on three tasks. To more convincingly improve current methods, authors should operate on stronger baselines (theirs have a relatively large gap w/ ground truth MOS) and reduce the tuning effort needed (but their honesty here is appreciated). I suggest Weak Accept.

Note: R1’s final update was not visible to authors, so I share here:

```python
The authors emphasize architectural and data consistency across experiments to ensure fair comparison, demonstrating that their proposed variational approach outperforms ablated versions that use only tokens, pitch, or acoustic information. However, the paper lacks comparison against strong external spoken language model baselines that have been carefully optimized, which significantly limits the claim of superiority. This absence of external benchmarks is a notable weakness.

Additionally, to support decoder flexibility, the authors replace the diffusion-based decoder with a tokenizer-based one, showing improved results (https://anonymous.4open.science/api/repo/icml-rebuttal-BDD8/file/ICML-rebuttal-2.pdf). However, this suggests that the diffusion decoder used throughout the paper may not be optimal. As noted in the initial review, diffusion decoders are generally not ideal for streaming applications, which reduces the practical value of the current setup. A stronger decoder choice—especially one that aligns better with practical use cases like streaming—would have strengthened the paper’s overall contribution.

While the paper aims to demonstrate the value of the variational approach, the lack of external baseline comparisons and concerns around the practical utility of the chosen decoder leave the impact of the proposed method unclear. Given these limitations, I lean toward a score between Reject (1) and Weak Reject (2).
```
Decision: Accept (poster)