Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space
Authors: Zhengrui Ma
Yang Feng
Chenze Shao
Fandong Meng
Jie Zhou
Min zhang
Venue: NeurIPS 2025 poster ()
Keywords: speech language modeling
autoregressive modeling
speech synthesis
Primary area: applications

Abstract: We introduce \emph{SLED}, an alternative approach to speech language modeling by encoding speech waveforms into sequences of continuous latent representations and modeling them autoregressively using an energy distance objective. The energy distance offers an analytical measure of the distributional gap by contrasting simulated and target samples, enabling efficient training to capture the underlying continuous autoregressive distribution. By bypassing reliance on residual vector quantization, SLED avoids discretization errors and eliminates the need for the complicated hierarchical architectures common in existing speech language models. It simplifies the overall modeling pipeline while preserving the richness of speech information and maintaining inference efficiency. Empirical results demonstrate that SLED achieves strong performance in both zero-shot and streaming speech synthesis, showing its potential for broader applications in general-purpose speech language models. Demos and code are available at \url{https://github.com/ictnlp/SLED-TTS}.

Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space
Zhengrui Ma 12,,3 Yang Feng 1.2 Chenze Shao 3, Fandong Meng's 3, Jie Zhou's 2, Min Zhang
Key Laboratory of Intelligent Information Processing
Institute of Computing Technology. Chinese Academy Sciences
2 University of Chinese Academy of Sciences
Pattern Recognition Center, WeChat AI, Tencent Inc
School of Future Science and Engineering, Soochow University
*; Corresponding author (mazhengrui21b fengyang) @ict. ac on
Abstract
We introduce SLED, an alternative approach to speech language modeling by en- coding speech waveforms into sequences of continuous latent representations and modeling them autoregressively using an energy distance objective. The energy dis- tance offers analytical measure of the distributional gap by contrasting simulated and target samples, enabling efficient training to capture the underlying continuous autoregressive distribution. By bypassing reliance residual vector quantization, SLED avoids discretization errors and eliminates the need for the complicated hier- archical architectures common in existing speech language models. It simplifies the overall modeling pipeline while preserving the richness of speech information and maintaining inference efficiency. Empirical results demonstrate that SLED achieves strong performance in both zero-shot and streaming speech synthesis, showing its potential for broader applications in general purposs speech language models. Demos and code are available at /tttps:://// //gi thub. com com/ictnlp/SLEII
Introduction
Text language modeling has achieved tremendous success in recent years. Works such as the GPT series [2 [50] have demonstrated that by modeling text sequences an autoregressive manner and pretraining large corpora, language models acquire impressive in-context learning abilities and can perform nearly any downstream natural language processing task. This remarkable potential of autoregressive language modeling has inspired researchers explore whether speech audio, which also possesses a linear structure, can be modeled in a similar fashion.
Unlike text sequences composed of discrete tokens from finite vocabulary, speech audio is often represented as a lengthy sequence of sampling points usually stored as integers or floating-por ppint values within range. This fundamental difference poses significant challenges for autoregressive speech modeling in a manner analogous to text. Text language models typically rely on a finite vocabulary to acquire categorical distribution each step, which is essential for stable training via cross-entropy loss and efficient ancestral sampling during inference. To adapt autoregressive modeling speech, researchers have explored discretizing speech into sequences of discrete tokens using external quantization modules. Examples include discretization on SSL--larned features [24] in discretization ASR- supevveed features in 7
83 31 and reconstruction-orienned 81 38 However, discretization inevitably creates information bottleneck, potentially losing the rich details in the waveform and thus degrading the reconstruction quality derived from the discretized tokens, One popular approach to mitigating this information loss residual vector quantization [RVQ; 70 82 which converts raw waveforms into
33
85 49)
codec 82
86,
80 51 25 in 72
88 10,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

45
kg
Local Model
NAR Model
Generative Model
0g
45
AR Model
Global AR Model
Global AR Model
81
22
02
200
822
20
J22
Figure Different speech language modeling approaches. Left: VALL-E-style hierarchical architecture for RVQ token sequences. Middle: RQ-Transformer-styll hierarchical architecture for RVQ token sequences. Right: Architecture for continuous token sequences.
multi-stream sequences of discrete tokens. However, RVQ introduces additional complexity, requiring hierarchical autoregressive architectures to effectively model multi-stream sequences [72 77.
As an alternative to discretizing speech, researchers are starting to explore whether speech language modeling can be performed in continuous latent space 46 69 39]. This approach offers several clear advantages. Firstly, eliminates the need for discretization, enabling speech modeling in a latent space with minimal information loss. Secondly, it removes the reliance on hierarchical architecture required for modeling multi-stream sequences in RVQ. significantly reducing modeling complexity. Similar to discrete autoregressive modeling, continuous autoregressive modeling requires capturing per-step distribution, but within continuous space. Unlike the discrete case, where categorical distribution can be constructed using the softmax function, continuous space modeling relies conditional probabilistic generative module to fulfill this role [68. This module should be conditioned on features extracted by autoregressive network, which captures the dependencies between the current step and the preceding ones. This raises an important question: How should we construct such generative module for speech language modeling in continuous latent space? Ideally, want the conditional generative module to function like the softmax the discrete case it should be lightweight, expressive, training-ste stable and inference-efficient. In speech language modeling, sampling efficiency becomes particularly important as the latent sequences of speech are typically lengthy, and each step in autoregressive generation involves sampling process.
In this research, we introduce an approach to speech language modeling in continuous latent space using energy distance (SLED). We model the per-step continuous distribution with lightweight im- plicit conditional generative module that takes features encoded with autoregressive dependencies and random noise inputs 337 600. To measure the discrepancy between the underlying data distribution and the model's 's distribution, we employ a specialized form of maximum mean discrepancy [MMD; 17). known generalized energy distance [GED; 42 By selecting appropriate distance function, the resulting metric becomes a strictly proper scoring rule [13] 59], enabling efficient training of the model capture the autoregressive continuous distribution. Our approach eliminates the need for iterative sampling at each autoregressive step [37 69 or post-AR refinement [46], significantly improving efficiency modeling lengthy speech sequences while maintaining sufficient capacity. We demonstrate the effectiveness of our method in zero-shot and streaming speech synthesis, showing its strong potential for application in general- purpooe speech language models.
2 Maximum Mean Discrepancy and Generalized Energy Distance
Integral probability metrics [4]] are types of distance functions between probability distributions over R". Let JF be class of real-valued functions R", integral probability metric is defined as:
Dr g) sup [E [m=) [xx) [f
(1)
feF

---

where p and can be model and data distributions. Due to the richness of function class F, it is often not practical estimate Dr(p. q) with finite samples. Maximum mean discrepancy [MMD; 17 solves this problem by restricting the function class F to be the unit ball in reproducing kernel Hilbert space H:
MMD(p, 4) sup [Bx) [f()) E -yy-
[I #11
Assuming real-valued function k(x,, y) is symmetric and positive definite, the kernel & defines a reproducing kernel Hilbert space H. such that every critic function f H can be expressed as [(x) = (f, kxx ()),, 7) This notion can be extended to the embedding of probability distribution
(S= Up) for all f E H. Gretton et al. [17] shows that, if the condition for the existence of Up is satisfied, Mp [kepp())(((x)))) -)] and MMD can be expressed as the distance between mean embeddings:
by defining Hp E H such that (x))(()) [a)))
MMD (p. q) = 11pp M4 11 = IE m, [k(2, x') + k( y²) - 2k(x
(3)
where x, xe' and y, y' are independent samples from p and q.
Müller 47 has shown that D, is a pseudometric for any choice of F, which satisfies all the properties of a metriddexcepp that there can be (p 9) 0 when is not identical to q. However, MMD(p. 9) is a metric when the mean embedding Up injective [17]. Kernlss that satisfy this condition are referred to as characteristic kernels [14]. By selecting characteristic kernel function k(x,, y). MMD(p, q) becomes a strictly proper scoring rule [5] and can be effectively used train
implicit probabilistic generative models.
In alternative view, Lyons 22 defines generalized energy distance (GED) between two probability distributions on metric spaces (IR". d) as:
GED g) Eu.e'-p [2d(x, y) d(x, x²) - d y')l w.y'~u
(4)
Several works 15 [2) have pointed out that if the distance function d is of negative type condi- tionally negative definite, then GED* (p. q) 2 O, and it forms pseudometric between distributions. Actually, GED is special case MMD. Sejdinovic et al. [57 shows that any nondegenerate kernel l on I defines valid semimetric d of negative type by
d(x; 3) = k(x, x) + k(y y) 2k(x, y),
(5)
and any semimetric d of negative type can be generated by at least one of its induced kernels:
k(x y) = d(I, x) + d(y z) - 2d(xx y),
(6)
where z can be any point on R GED,; is equivalent to the MMD3 associated to a kernel & that generates d (57]. Therefore, one can alternatively use GED as criterion for training implicit generative models, as long as the distance function d is chosen appropriately.
3 Approach
3.1 Language Modeling in Continuous Latent Space
Prior approaches to speech language modeling have commonly depended external quantization module, which discretizes speech waveform into sequence of tokens from a finite vocabulary for autoregressive modeling in manner analogous to text. The generated speech token sequence be resynthesized into a waveform using either the decoder of the quantization model or an externally explore encoding speech waveforms into sequences of continuous representation and performing autoregressive modeling within this continuous latent space. Given audio sample æ E RTT, we encode it into a sequence of continuous representation: = Enc(æ) h E RT xn. where the duration the audio, and f and fa denote the sample rates of the audio waveform and the latent sequence. Typically, f ranges from hundreds of thousands
trained vocoder [53 63) In this research,
metric satisfies the following four properties: nen-negativity. symmetry, triangle inequality and identity of indiscernibles.
The definitions of nondegenerate kernels and semimetrics can be found in Sejdinovic et al. 57
3

---

to several hundred thousand, while fn ranges from few to tens. The model aims to capture the distribution of the continuous vector h,, conditioned on all previously generated vectors hegr
T'n -1
(hh) - II p(hr) t=
In the discrete domain, autoregressive models can use softmax function to model the distribution of each step over the vocabulary space. However, autoregressive modeling in the continuous domain is more complex. It requires modeling the distribution in the IR= space for each step, conditioned on the outputs from previous steps:
2, = (h h ~ Pg (y z1; の)
where io can be any autoregressive network (e.g., decoder-only Transformer 71 and g can be any conditional generative module on IR".
3.2 Per-token Generative Modeling via Energy Distance
As discussed earlier, we perform autoregressive modeling in the continuous latent space using conditional generative module per step. The model is responsible for modeling the distribution of the current step h, based on the condition representation 2,, which incorporates autoregressive dependencies. Ideally, any conditional generative model can serve as g. However, in the context of speech language modeling, . needs meet certain requirements. Modeling Capability: Considering the diversity of speech audio, 9 requires a strong capacity to capture the distribution at each step. Analytically tractable distributions, such as Gaussian distributions, may not have sufficient capability, Sampling Efficiency: Speech audio has high sampling rate. Even though this rate is significantly reduced after encoding into the continuous latent space, the sequences remain lengthy. Given that each step of autoregressive decoding requires sampling operation from g. we aim for g to have high sampling efficiency, akin to categorical sampling over vocabulary. If we model g using methods such as diffusion [23] which require multiple iterations during sampling, it could significantly increase the latency of autoregressive decoding. Training Stability: Similar to training discrete autoregressive models using cross-entropy loss, we aim to train both the conditional generative module g and the main autoregressive network simultaneously using a simple and stable training algorithm. Under such constraints, the GAN-style training paradigm [16] may not be appropriate.
Based on the above analysis, we construct g as lightweight multi-layer perceptron that takes the condition vector and noise vector as inputs, mapping them to the continuous latent space of speech, "h:
h = 0(2,, c;
This implicitly defines a per-step distribution Py py (he over R". The sampling process involves simply sampling a noise vector e and passing it through g along with the condition 2,. the network g. we use the AdaLN module [52] to integrate noise into the hidden state of the condition vector 2; to introduce randomness. Rather than directly learning dimension- wise scale and shift parameters in the layer normalization module for zt, we treat them as random perturbations for sampling [60]. The scale and shift values are predicted by applying a linear transformation to the input noise 6,.
To train the implicit conditional generative module and the autoregressive network & simultaneously, we use a specialized form of maximum mean discrepancy, namely energy distance, as the metric between the distributions of the data and model. The training is performed by minimizing the energy distance between the simulated and the ground truth latent representation per step. with respect the parameters of both and s/. Since the data distribution remains fixed during optimization, terms in Eq. that do not depend on the model parameters can be discarded, reducing the training loss to:
LGED En. [2d(h,, h;) ddhh, h(),,
(10)
where h² is the target latent waveform representation, and h., hú are independent samples from Po (he zz)). As discussed in Sec. [2] LAED is a strictly proper scoring rule, provided that the kernel
function corresponding to the distance function dl is characteristic. [15] 66 suggests that d(x;, y) =
|| - y satisfies the condition when 3 E (0, 2). In our experiments, we set 3 = 1:
LGED E,En, n; [2
hillb |/, hill2
(11)

---

It is worth noting that Eq. 11 is very similar to the root mean squared error, with the addition of the repulsive term Eillh; - TTl This repulsive term is crucial for making the loss a proper learning objective. In contrast, RMSE is only regression loss, which does not well capture the differences between the distributions. Using RMSE loss as the objective can cause the model to fail to fit the data distribution [88.. We demonstrate this in Sec. 6.1
SLED performs autoregressive modeling in the continuous latent space, thus it cannot determine when to stop by predicting the EOS token. To address this issue, we adopt the approach from [ and introduce binary classification head to decide whether the output should terminate. We directly project the output of the autoregressive network to a scalar, followed by sigmoid function, and interpret the result as the probability of whether the output should stop at this step.
3.3 Classifier-free Guidance
Continuous latent generation exhibits higher noise levels than its discrete counterpart. To improve the generation quality and the alignment with prompts, adopt the classifier- free guidance technique [CFG: 22 66) during inference. At each step of autoregressive generation, we perform an additional forward pass through vi, during which the prompt text is masked out, obtain z/.. We then linearly interpolate with z and use the result as input to the per-token generative module g:
2. 2 + A(z z,), h g(2 cis e; (),
(12)
where X is the hyperparameter to control the strength of the guidance and it reverts to naïve conditional generation when 1.0. During training. we randomly mask out the text prompt with a probability of 0.1. The EOT token is preserved during masking, as it also serves as the token indicating the beginning of the speech.
3.4 Streaming Inference
One appealing feature of our approach is that is a purely autoregressive model and does not require any post-processing module refine the outputs at earlier positions after the entire autoregressive generation is complete. This desirable property allows our autoregressive modeling approach to support streaming generation 43 78
44]. Here, streaming generation has two levels of meaning: 1) After each autoregressive step generates a latent vector, it can immediately be used for waveform synthesis. This capability relies on the model's autoregressive generation not requiring any post- processing steps, and the decoder that converts latent vectors to waveforms supporting streaming. 2) Building on this, we aim for the model to begin generating even before the full prompt text provided. This incremental generation feature is particularly useful to reduce the response latency of a GPT-4o-like speech interaction system 122 following text LLM.
76
where the model serves as a TTS module
Terget
Offline
Input
Target
EDO
Streaming
nOT
BOT
Text
Speech
Figure 2: Illustration of our streaming inference mechanism. Text and speech tokens are interleaved based on predefined ratio, and the loss is computed only at positions where targets are shown.
Modern audio codecs typically have streaming decoder 19, so the first level of streaming generation naturally supported. To achieve incremental speech synthesis, we alter the order of the text and speech positions within the sequence during autoregressive modeling [8] As illustrated in Figure 2 we interleave text and speech positions in an n m ratio. This design allows the generation of m speech vectors for every n text tokens received, enabling incremental text-to-speech synthesis. An EOT token is used to indicate the end of the input text stream. Afterwards, the speech vectors are generated in the standard autoregressive way until stop decision is made by the binary classification head.
5

---

We train the model to generate target vectors and predict whether to stop only at the input positions that require generation during inference. We do not require the model to predict any filling tokens when the next step is text token according to the interleaving ratio. We also adopt the classifier- free guidance outlined in See. 3.3 in streaming inference. The model performs unconditional speech generation when the text stream requires masking, with EDT serving as the beginning.
4 Related Work
Given great success of autoregressive language modeling approach in text generation, researchers have started exploring whether speech sequences can be modeled in a similar way. These studies begin with speech synthesis foundational task
72 eventually extending to the development of general- 33
was the first to propose discretizing self-supervised representations of speech as semantic tokens and to perform autoregressive speech generation. The method of learning semantic tokens has since evolved to quantize ASR-super representations 7 Due to information loss, waveforms reconstructed from coarse semantic tokens are often unsatisfactory, necessitating modeling with acoustic tokens produced by reconstruction- 51 25]. Since the acoustic tokens generated by codecs often consist of multiple sequences from residual quantization, a hierarchical architecture is required for autoregressive speech modeling. [72) utilize an autoregressive model to generate the sequence of tokens from the first codebook, followed by non-auloregressive model to generate tokens from the residual codebooks. To mitigate the latency introduced by post-AR refinemen, [77 88 apply the RQ- OQ-Transformer (35, which uses smaller nested Transformer to model tokens from multiple codebooks within a single autoregressive step. Nonetheless, modeling multi-stream acoustic tokens introduces additional complexity. Inspired by advancements in image generation [68] 37 researchers have begun exploring performing speech language modeling in the continuous latent space. [46] use mel-spectrograms as the latent representation and model the per-step distribution with regression loss. However, using a regression loss fails to capture the underlying distribution, requiring a post-AR refinement network and handcrafted repetition loss. [69] adopt per-token diffusion loss [37], which requires iterative sampling for each autoregressive step. leading to significant inference latency. [73] models each step via flow matching [40], which also encounters similar problems. To this end, [26
purpose speech language models [48]
85
TO
38 31
3
oriented audio codecs (82) 32
86
80
proposes a patch-based continuous autoregressive framework that couples a language model for inter- inter-patch prediction with DiT [52 for intra-patch generation, thereby reducing the number of
sampling steps. 87 (39 model the data at each autoregressive step with a Gaussian distribution or
GMM and employ VAE to obtain the target distribution parameters for each step. These approaches impose overly strong constraints on the latent space, limiting the model's expressive capacity.
Experiments
5.1
Training Datasets
We train SLED on the large-scale Libriheavy [29 dataset. LibriHeavy contains approximately 50,000 hours of speech from 6,736 speakers, deriving from audiobooks from the LibriVox project. A BPE tokenizer 58 with a vocabulary size of 16,384 is applied for text.
5.2 Experimental Settings
Continuous Representation We employ Encodec p to extract continuous latent representations. For each frame, we sum the multi-stream token embeddings- drawn from eight codebooks to form its continuous vector, ensuring minimal information loss. Although a plain autoencoder can produce continuous latents, we find that enforcing quantization-regulaiieee regularized latent space substantially benefits downstream language modeling. This finding is in line with the standard practice in latent diffusion models (56). Considering the representative discrete modeling approach VALL-E [72] also employs Encodec as its tokenizer, this setup also enables a fair comparison between continuous and discrete autoregressive modeling while keeping the latent encoder consistent. The resulting continuous vector sequences have a sampling rate 75Hz and a latent dimensionality of 128.
KL-
Model Configuratioss The autoregressive network incorporates 12 Transformer layers 71 Instead of using standard Transformer layers, employ LLaMA [67] layers, which apply pre-normalization

---

Table I: Performance comparison 3s Prefix Prompt and Reference Utterance Prompt zero-shot speech synthesis tasks. WER-C C (%) represents the results using the Conformer- Conforme--Transducer,, whereas WER-H (%) indicates the results with the HuBERT-Large. *: WER obtained by Whisper.
3s Prefix as Prompt
Reference Utterance Prompt
System
#Params
WER-C
WER-H
SIM
WER-C
WER-H H
SIM
Ground Truth Ground Truth (Encodec)
1.78
2.15
0.668
1.78
2.15
0.778
1.79
2.30
0.606
1.79
2.30
0.738
Traditional TTS
MaskGCT 75
1.0B
2.63
0.687
F5-TTS G
0.3B
2,42*
0.66
MegaTTS 24
0.3B
1.82
0.78
Discrete-LMe LM----M-bbbas-- Approaches
VALL-R 72
0.4B
3.8
0.508
5.9
0.580
VALL-E
0.4B
1.6
2.32
0.529
1.5
2.44
0.678
ELLA-V
0.4B
2.10
2.91
0.340
7.15
8.90
0.331
VALL-E R 20
0.4B
1.58
2.32
0.397
3.18
3.97
0.395
Llasa 81
8B
1.49
0.740
Continuous LM based Approaches
CLaM-TTS 30
2.36
0.513
5.11
0.538
MELLE 46
0.2B
1.47
1.98
0.539
1.47
2.10
0.664
FELLE
73
0.2B
1.53
2.27
0.539
2.20
2.89
0.654
SLED
0.2B
1.59
1.99
0.515
1.51
1.97
0.664
Table 2: Performance comparison of Offline and Streaming Inference.
Table 3: Performance comparison using Energy Distance vs. Mean Squared Error as the objective in 3s Prefix as Prompt experiments.
Mode
m
WER-C DNSMOS
GT
1.79
3.89
Objective
WER-C
Offline
3.58
with Prompt
1.51
3.61
Root Mean Squared Error
40.60
Streaming 5:20
2.18
3.59
Energy Distance
1.59
Streaming 5:45
2.20
3.54
using RMSNorm [4], SwiGLU activation function [62], and rotary positional embeddings (65]. Each layer has 16 attention heads and embedding dimension of 1,024. We set the FFN hidden dimension to 2,752 to match the number of parameters of a standard Transformer with an FFN hidden dimension of 4,096. The dropout rate is set at 0.1. The input latent continuous vectors projected to the embedding dimensionality using a linear layer, followed by layer normalization to ensure stability. The lightweight conditional generative module consists of six residual blocks, each featuring an AdaLN module [52] to modulate the hidden states with noise, followed by two-layer MLP and residual connections. The hidden dimensionality of this generative module is set to 1024, and linear layer is used at the top to project the output back to the target dimensionality. The default value of in classifier-fres guidance set to 2.0.
Training Details We train the model with a batch size of 512 for 300,000 steps using BF16. We optimize the model with AdamW [41). configured with learning rate of 5c-4, weight decay of 0.01, x 10 The learning rate follows a linear decay, warming up to its peak value during the first 32,000 steps. A maximum gradient norm clip of 1.0 is applied.
31 = 0.9, 3g = 0.999 and
5.3 Evaluation Settings
We evaluate zero-shot speech synthesis performance using LibriSpeech test-clean set, ensuring that none of the test speakers are included in the training data. Following 772 46), we use samples with durations between and 10 seconds, resulting in 2.2-hour subset comprising 1,234 samples and 40 unique speakers. Since the LibriSpeech consists of case-insensitive text without punctuation, while

---

ORGNNSS
12
(a) WER-C and SIM against CFG (b) WER-C and SIM against CFG
(c) WER-C and DNSMOS against
in 3s Prefix as Prompt.
in Ref Utterance as Prompt
CFG in Streaming Inference.
Figure Evaluation of classifier-fres guidance effects across generation settings.
the expected input for models trained on LibriHeavy is unnormalized, we use the test-clean set of LibriSpeech-PC 445 to evaluate SLED. The LibriSpeech-PCr is nearly identical to LibriSpeech, but its text has been restored include capitalization and punctuation The evaluation is conducted under two settings: 1) 3s Prefix as Prompt: The model is provided with the text transcription and the first 3 seconds of the utterance as the prompt, and expected to perform speech continuation. 2) Reference Utterance as Prompt: The model is given reference utterance from the same speaker along with its transcription as the prompt. Using the text the target utterance, the model expected to synthesize speech that retains the characteristics of the reference speaker. For this experiment, we follow the configuration outlined in [5]. We first filter the samples in LibriSpeech test-clean set based on length and order them by sample ID. For each speaker, the i-th speech sample is synthesized using the (i 1) sample as the prompt, while the first sample is synthesized using the last sample from the same speaker the prompt. For Offtine and Streaming Inference experiments, follow previous data settings and no prompt speech is provided. We primarily use the following metrics to assess the intelligibility and in-context learning capabilitty
Word Error Rate (WER) To evaluate the intelligibility of the synthesized speech, we perform speech recognition on the generated audio and calculate the WER against the target text. We utilize Eefformer-Transduceer
two ASR models:
19] and CTC-ASR-tume HuBERT-Large 24
Speaker Similarity (SIM) We evaluate the in-context learning capability by measuring speaker similarity between the reference and the generated speech. We extract speaker embeddings using WavLM-TDNN and compute their cosine similarity.
Predicted Mean Opinion Score (DNSMOS) We evaluate the overall quality of the generated speech using the DNSMOS score (5), which ranges from to 5, with higher values indicating better quality. We use model trained with ground truth human ratings obtained using ITU-T P.808.
5.4 Main Results
Zero-shot TTS Table I presents the results of zero-shot TTS. In both the speech continuation and reference utterance prompting tasks. SLED achieves a word accuracy surpassing the ground truth (1.59/1.51 vs. 1.78 in WER-C), demonstrating its modeling ability. Interestingly, we found that using reference utterance as a prompt results more accurate synthesis compared using the prefix, showing the ability of in-context learning. Comparisons of SLED performance across different data scales are provided App- A We found that 1000 hours of speech is sufficient to achieve most of the generation and in in-context learning capabilities. Further scaling the training data to 50,000 hours enhances these abilities even more. Notably, comparison between SLED and VALL-E highlights key differences in autoregressive modeling approaches. Both models use Encodec as the latent encoder, but SLED performs modeling in continuous domain, whereas VALL-E operates the discrete domain. VALL--E-like hierarchical models require an extra non-autoregressive local model (>/ 155M) to predict residual tokens. In contrast, SLED relies on a lightweight generative module (=35M) for continuous vector generation, achieving better parameter efficiency. Despite this, SLED outperforms VALL-E on all metrics. These results show the superiority of our continuous speech language modeling approach. Compared with other continuous approaches, SLED achieves
'The test-clean set of Librispeech-Pe excludes a small number of lualquality samples (203 out of 2,620) from the original Librispeech test-clean set, thus the test subset after applying the same protocol contains only 1, 1,154 samples.

---

better or similar quality while enjoys the advantage in generation efficiency. MELLE incorporates an NAR refinement, whereas SLED is purely autoregressive and can generate in a streaming manner. FELLE must perform multi-step integration iterations along a predefined ODE path at each decoding step, while SLED only requires single forward pass through its generative module. Moreover, at roughly the same parameter scale, current speech language modeling approaches still exhibit significant gap in voice cloning compared with traditional TTS models. However, Llasa [81] shows that scaling up can dramatically strengthen discrete autoregressive speech models, which inspires us to further explore the scaling capabilities of continuous autoregressive modeling approaches.
Streaming Inference
Table presents comparison of accuracy (WER-C) and perceived quality (DNSMOS) between speech produced via streaming inference and that generated offline. We vary the interleaving ratio between text and speech positions to analyze the impact of streaming policies. In our experiments, we compare two configurations: one generates 20 speech vectors (0.27 seconds of speech) for every 5 subwords received, while the other generates 45 speech vectors (0.6 seconds of speech) per subwords. The latter configuration aligns with [8] which guarantees seamless streaming generation when serving as the speech synthesis module of cascaded speech-LLM system. We find that the two interleaving ratio configurations yield comparable results in both word accuracy and perceived quality. Moreover, the synthesized quality in streaming mode closely matches that of offline synthesis (3.59/3.54 3.54 vs. 3.58 in DNSMOS), despite a slight drop in word accuracy (2. 18/12/2.20 vs. 1.67 in WER-C), demonstrating the effectiveness of SLED for streaming tasks.
6 Analysis
6.1 Importance of Energy Distance
the learning objective of SLED (Eq. 11] closely resembles root mean squared error. By removing the repulsive term Ei h(ll)), the energy distance becomes equivalent the RMSE, with random noise acting as a perturbation (which is the L2 part of the regression loss in [46)). Despite their apparent similarity, these two objectives exhibit distinct statistical properties. Energy distance measures the distributional difference, guiding the model capture the underlying data distribution. In contrast, RMSE is only regression loss. The importance of the repulsive term demonstrated in Table 3 Removing this term results in a model failure, leading to a significantly worse word error rate. Listening to the generated samples, we observed that the model trained without the repulsive term failed to generate speech in most cases.
As discussed in Section 3.2
This makes us curious why MELLE [46] achieves strong performance using a regression loss. We note that 46) introduces a flux loss suppress repetition, which measures the variation of the generated spectral sequence over time by taking the norm of the Mel-spectrogram differences between adjacent frames. [46] reports that generation quality degrades severely without it. Although [46] frames the flux loss as a heuristic designed to curb repetition, we argue that its real efficacy stems from approximating the repulsive term in the energy distance. Because consecutive frames speech differ only slightly. they can be approximately regarded as two samples from the same time step. Under this view, MELLE's flux loss causes its training objective to approximate the energy distance, thereby enabling it to capture distributional differences to some extent.
6.2 Effects of Classifier-free Guidance
We investigate the impact of classifier-free guidance (CFG) across generation settings. As shown in Figure 3 without CFG (A= = 1.0 in Eq. [2). the synthesized speech exhibits poor word accuracy, with particularly high WER-C of 6.01 in streaming inference. However, upon applying CFG, the WER-C values decrease dramatically. A X value of 1.5 quickly results in a substantial improvement, bringing the WER-C down to approximately 2.0 across all settings. Further tuning of À leads even better word accuracy. However, in terms of timbre cloning and perceived speech quality, a moderate CFG setting (A = 1.5) yields the most improvements. Further increasing À leads to performance degradation. Notably, speech quality in streaming inference declines to level worse than that generated without CFG at = 2.5. This experiment demonstrates that classifier-free guidance significantly improves alignment between the text and synthesized speech while enhancing speech

---

quality only with moderate À value. We recommend setting À = 2.0 by default to achieve balance between word accuracy and speech quality across all generation settings.
6.3 Efficiency Analysis
We view SLED and DíTAR [26] as complementary approaches to more efficient continuous autore- gressive speech modeling. DiTAR reduces the number of sequential steps via ceni-auttreerrssivve framework, yielding time complexity between fully autoregressive and fully non-autoreerressive. whereas SLED accelerates sampling in the continuous latent space at each autoregressive step. We compare the computational efficiency of the two methods in Table 4 All the metrics are evaluated by inferring 10 seconds of audio.
Table 4: Comparison of decoding efficiency between SLED and DiTAR (2)..
Model
#Params
#Transformer Layer
RTF (bs=1)
FLOPs
SLED
0.2B
12 6+3646
0.8
280G
DÍTAR (patch size 4)
0.6B
0.66
2750G
Although DiTAR's patching scheme reduces the number of sequential steps, the approach is hierarchi- cal. Each step in DÍTAR bears heavy generation workload and therefore relies relatively large local decoder (DiT), in contrast to SLED's lightweight MLPs for per-step generation. This burden can erode the nominal benefits of patching and, by iterating within a local DiT, may also compromise global consistency; DiTAR paper reports that the local-DiT module must incorporate historical-pati dependencies generate correctly. increasing complexity and narrowing the method's computational advantage. Empirically, SLED achieves comparable RTF to DÍTAR while using nearly 10x fewer FLOPs and roughly one-third the parameters.
Conclusion and Limitation
In this work, we propose method for speech language modeling in continuous latent space using energy distance. This approach avoids the complex hierarchical architecture of traditional discrete models, while also preserving rich information in speech. This work has validated the effectiveness of this modeling approach in the speech synthesis task. In the future, we plan to extend to general- purpose speech language models. Moreover, the latent encoder employed in the current work was originally designed solely for the audio codec; additionally training a continuous latent encoder specifically for this method should further enhance its performance.
Acknowledgement
We gratefully acknowledge all the reviewers for their valuable comments and suggestions. This work was supported by the Natural Science Foundation of Beijing, China (Grant No. L257006).
References
[1] Z. Borsos. R. Marinier, Vincent. Kharitonov. Pietquin, Sharifi, D. Roblek. O. Teboul, D. Grangier, M. Tagliasacchi, and Zeghidour. Audiolm: a language modeling approach to audio
generation, 2023. URL https:tts:///rrat //t::////xxivvvv/a http:///aaaiivoorr/aa///222 003143
[2] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, F. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models few-shot learners, 2020.
URL RRLL/ttps://arxxt //tarvv/..iiv//// .oeeiiiiior////oooo absaabas///0// 2005 14165
[3] Q Chen, Y. Chen, Y. Chen, M. Chen, Y. Chen, C. Deng, Z. Du R. Gao, C. Gao, Z. Gao, Y. Li, X. Ly, Liu, H. Luo, B. Ma, Ni, X. Shi, Tang. H. Wang H. Wang. W. Wang, Y. Wang, Y. Xu, Yu. Z. Yan, Y. Yang. B. Yang. X. Yang, G. Yang. T. Zhao, Q Zhang. S. Zhang. N. Zhao, Zhang, C. Zhang, and J. Zhou. Minmo: A multimodal large language model for seamless voice interaction, 2025. URL org/ 20bs/2 06282
https: //apxii////vvv...// /arxivivv../////////...////a
10

---

[4] S. Chen, C. Wang. Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian. Y. Qian, J. Wu, M. Zeng. X. Yu, and F. Wei. Wavlm: Large-scale self- supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics Signal Processing, (6(6):1505-155 1518, Oct. 2022. ISSN 1941-0484. doi: 00.1109/ssss.2222.318883
URL
http://dx. www./xxxxoo..... org/10. 1109/JSTSPPPP 2022.02222222 3188113
[5] S. Chen, S. Liu, I.. Zhou, Liu, X. Tan, J. Li, S. Zhao, Y. Qian, and F. Wei. Vall-e 2: Neural codec language models are human parity zero-shot text to speech synthesizers, 2024. URL httts://arxix: org/ prg/abs/2406... 2406. 05370
[6] Y. Chen, Z. Niu, Z. Ma, K. Deng, C. Wang. J. Zhao, K. Yu, and X. Chen. F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching, 2024. URL https: org/ abs/2410
[7] Z. Du, Q. Chen, S. Zhang, K. Hu, H. Lu, Y. Yang, H. Hu, S. Zheng, Y. Gu, Z. Ma, Z. Gao, and Z. Yan. Cosyvoice: A scalable multilingual zero acro-shot text-to-speech synthesizer based on supervised semantic tokens, 2024. htRpsttps:// //axxiv. org/ harg/bs///0 abs/2407 05407
[8] Z. Du, Y. Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y. Yang, C. Gao, H. Wang, F. Yu, H. Liu, Z. Sheng, Y. Gu, C. Deng, W. Wang, S. Zhang. Z. Yan, and J. Zhou. Cosyvoice 2: Scalable streaming speech synthesis with large language models, 2024.
org/
[9] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi. High fidelity neural audio compression, 2022. URL.
sttge://arzvv.rr//aaa/2000 13438
[10] A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour. Moshi: speech-text foundation model for real-time dialogue, 2024. URL httpa://////a orgoorg///aoo wtg//b//////a////oo aba/2410 00037
[11] S. E. Eskimez, X. Wang, M. Thakker, C. Li, C.HH. Tsai, Z. Xiao, H. Yang, Z. Zhu, M. Tang, X. Tan, Y. Liu, S. Zhao, and N. Kanda. E2 tts: Embarrassingly casy fully non-autoregressive zero-shot tts, 2024, URL https://arxiv org/ abt///////222... 18009
[12] Q. Fang, S. Guo, Y. Zhou, Z. Ma, S. Zhang, and Y. Feng. Llama-omni: Seamless speech interaction with large language models, 2025. wtt.ttttt:///aaaataate http://wwxvvrrg///aaa//// ha////bb/2440 http:/24990066665 06666
[13] Q. Fang, Y. Zhou, S. Guo, Zhang, and Y. Feng. Llama-omni2: LIm-hased real-time spoken chatbot with autoregressive streaming speech synthesis, 2025. wtt.lattpp::/arzvvv.rrrg///// 2a/.//5555.000 02625
[14] K. Fukumizu, A. Gretton, X.
Sun, and B. Schölkopf. Kernel measures of conditional dependence. In Platt, Koller, Singer, and Roweis, editors, Advances Neural Information Prucessing Systems. volume 20. Curran Associates, Inc., 2007. URL /ttpps://proceeedngg www.eeddddnnnnnnnnsseee..em..... wwwpcpappp.alleeeeeeeeeeeeeeeeee /Tt///////////077733337777333110
Paper pdf
[15] T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477)33993377 2007. doi: 10.
[16] Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, Ozair, Courville, and
Y. Bengio. Generative adversarial networks, 2014. URL RRRl.ittts://ariiii org/ /a////baa//4/////////////////1// .406.2661
[17] A. Gretton, K. M. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723-77,, 2012. (ttp::/:: otg/papers/v13. pretton12a.. html
[18] A. A. Gritsenko, T. Salimans, R. van den Berg, J. Snoek, and N. Kalchbrenner. A spectral energy distance for parallel speech synthesis, 2020. URL httts:://tattt:t org/ abaaaaa////0 2008. 01160
[19] A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y. Zhang. J. Yu. W. Han, S. Wang. Z. Zhang, Y. Wu, and R. Pang. Conformer: Convolution-augmented transformer for speech recognition, 2020. URL https!
//arxiv.orz httg///oorr/aeo abw/2005.... 008100
[20] B. Han, L. Zhou, S. Liu, S. Chen, L. Meng, Y. Qian, Y. Liu, S. Zhao, J. Li, and F. Wei. Vall-e r: Robust and efficient zero-shot text-lo-speech synthesis via monotonic alignment, 2024. URL https //rxiv. org/ aba. /aab//240...m 07855
[21] J. Helel, B. Haddow, and A. Birch. Non aonreevvreeeessive machine translation: It's not as fast as it seems. 2022. URL https://arxiv. http://aariiv.or//////20000/ org/abs/2205 01966
[22] J. Ho and T. Salimans. Classifier free diffusion guidance, 2022. URL https://arxiv, wwtpsiii.orr////oo abs///bs22222 2207 12598
11

---

[23] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models, 2020. URL htttps:/aara org/aba/222 11239
[24] W.NN. Hsu, B. Bolte. Y.H. H. Tsai, K. Lakhotia, R. Salakhutdinov. and A. Mohamed. Hubert: Self- supervised speech representation learning by masked prediction of hidden units, 2021. URL https //arxiv. .rgg aa////aa////0 07447
[25] S. Ji, Z. Jiang. W. Wang. Y. Chen, M. Fang. J. Zuo, Q. Yang, X. Cheng, Z. Wang, R. Li, Z. Zhang. X. Yang. R. Huang, Y. Jiang, Q Chen, S. Zheng, and Z. Zhao. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling, 2025, httptsss://aaxx org/ hts//bb//22 at////0s/22...... 16532
[26] D.. Jia, Z. Chen, J. Chen, C. Du, Wu, J. Cong, X. Zhuang, C. Li, Z. Wei, Y. Wang, and Y. Wang. Ditar: Diffusion transformer autoregressive modeling for speech generation, 2025. URI. httptttttttttti...... abs/2502. 2393000930
[27] Z. Jiang, Y. Ren, R. Li, S. Ji. B. Zhang, Z. Ye, C. Zhang, B. Jionghao, X. Yang, J. Zuo, Y. Zhang, R. Liu, X. Yin, and Zhao. Megatts 3: Sparse alignment enhanced latent diffusion transformer for zero-shot speech synthesis, 2025. httl.hhtpa:/// aarxivv org/ aba/2502.... 18924
[28] Z. Ju, Y. Wang. K. Shen, X. Tan, D. Xin, D. Yang, Liu, Y. Leng, K. Song. S. Tang, Wu, T. Qin, X.-Y. Li, W. Ye, S. Zhang, J. Bian, L. He. J. Li, and S. Zhao. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models, 2024, http::tt/////i
03100
[29] W. Kang, X. Yang, 7. Yao, F. Kuang. Y. Yang, I.. Guo, I. Lin, and D. Povey. Libriheavy: a 50,000 hours corpus with punctuation casing and context. In ICASSP 2024-2024 IEEF International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10991-10995. IEFE, 2024.
[30] J. Kim, K. Lee, S. Chung, and J. Cho. Clam-tts: Improving neural codec language model for zero-shot text-to-speech, 2024. URL http:://axx/x org/ abs/2402/ 02781
[31] KimiTeam, D. Ding, 7. Ju, Y. Leng, S. Liu, T. Liu, 7. Shang, K. Shen, W. Song, X. Tan, H. Tang,
7. Wang, C. Wei, Y. Xin, X. Xu, J. Yu, Y. Zhang, x. Zhou, Y. Charles, Chen, Y. Chen, Y. Du, W. He, Z. Hu, G. Lai, Q. Li, Y. Liu, W. Sun, J. Wang. Y. Wang, Y. Wu, Y. Wu, D. Yang. H. Yang. Y. Yang. Z. Yang, A. Yin. R. Yuan, Y. Zhang, and Z. Zhou. Kimi-audio technical report, 2025. URL
httpa: //arxiv////xvv.. www.xiww.or//bb//50000 18425
[32] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar. improved rvggan, 2023. URL http::t/ttaaaattiieieea org aba/2306/602////// 2306 06546
audio compression with
[33] K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y. Adi, A. Polyak. B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski,
A. Mohamed, and E. Dupoux. Generative spoken language modeling from raw audio, 2021. URL https://arxiv. org/ abs 2202. 01192
[34] M. Lee, A. Vyas, Shi, B. Karrer, I., Sari, R. Moritz, M. Williamson, V. Manohar, Adi, J. Mahadeokar, and W.-N. Hsu. Voicebox: Text-guided multilingual universal speech generation at scale, 2023. URL. https://arxiv.org atgg/bss//3//./// 15687
[35] D. Lee, C. Kim, S. Kim, M. Cho, and W.S. Han. Autoregressive image generation using residual quantization, 2022. URL.tttss://axxv org/ h2tp//bbs/20...../ 01941
[36] K. Lee, D. W. Kim, J. Kim, S. Chung, and J. Cho. Ditto-tts: Diffusion transformers for scalable text-to- speech without dopain-ppecific factors, 2025. URL https://t///aaa //axxiv org/ wwtpab///./...aeeeee
[37] T. Li, Y. Tian, H. Li, M. Deng, and K. He. Autoregressive image generation without vector quantization, 2024. URL URL httttttst///aiiii orgo/o///// abtpaabs/...../////// 220041111133
[38] T. Li, J. Liu, T. Zhang, Y. Fang, D. Pan, M. Wang, Z. Liang, Z. Li, M. Lin, G. Dong, J. Xu, H. Sun, Z. Zhou, and W. Chen. Baichuan- A unified framework for end-to-end speech interaction, 2025. URL /tttts::///aatt
org/ absp////.2222... 17239
[39] W. Lin and C. He. Continuous autoregressive modeling with stochastic monotonic alignment for speech synthesis, 2025. URL
[40] Y. Lipman. R. T. Q. Chen, H. Ben-Hlamu, M. Nickel, and M. Le. Flow matching for generative modeling, 2023. URL https: https://arxix: http://axxxv...ga/b//////// 2xg/045/2210.02777
[41] 1. Loshchilov and F. Hutter. Decoupled weight decay regularization, 2019. URL htttps://arrtt org/ abs/ 1tt///111..010000
12

---

[42] R. Lyons. Distance covariance in metric spaces. The Annaly of Probubility, 41(5), Sept. 2013, ISSN 0091-1798. doi: 22114/122a0p803. URL httt:://dd /dx. doi www..oogw......ooooooo www.g/....111//1110 1211/2211222222 ADP803
[43] Z. Ma. S. Zhang. S. Guo, C. Shao, M. Zhang. and Y. Feng. Non-autoregressive streaming transformer for simultaneous translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5177-5190. Association for Computational Linguisties, 2023, URL (RRptt / /aclanthologyy org/ 2023. emmlp- malpmmma....1. 314/
[44] Z. Ma, Y. Feng, and M. Zhang. Overcoming non connooootnnyctyy in transducer- based streaming generation. In Proceedings of the 42nd International Conference on Machine Learning, 2025. URL https://arxiv
org/ abs/ / 2411 17170
[45] A. Meister, M. Novikov, N. Karpov, F. Bakhturina, V. Lavrukhin, and B. Ginsburg. Librispeech-pe Benchmark for evaluation of punctuation and capitalization capabilities of end-to-end asr models. arXiv preprint arXiv:2310.02943, 2023.
[46] L. Meng. L. Zhou, S. Liu, Chen, B. Han, Hu, Y. Liu, Li, Zhao, Wu, H. Meng, and Wei. Autoregressive speech synthesis without vector quantization, 2024. URL, Sttt.ttpt://areet wraxxiv.oog/aee / 2407 085551
[47] A. Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability, 29(2):229-44) 1997. doi: 10.2307/1428011
[48] T. A. Nguyen, E.
Kharitonov, J. Copet, Y. Adi, W.-N. Hsu, A. Elkahky, P. Tomasello, R. Algayres, B. Sagot, A. Mohamed, and F. Dupoux. Generative spoken dialogue language modeling, 2022. URL. https://arxiv orgooo//////.eeeeeeeooee hwwpg/aba/2/0 /203. 16502
[49] T. A. Nguyen, B. Muller, B. Yu, M. R. Costa-jussa, M. Elbayad, S. Popuri, C. Ropers, P.-A. Duquenne, R. Algayres, R. Mavlyutov, I. Gat, Williamson, Synnaeve, Pino, B. Sagot, and Dupoux. Spirit Im: Interleaved spoken and written language model, 2024. 202444UU/attt///aaaa//o////bbbb
[50] OpenAL. Gpt-4 technical report, 2024, URL
[51] D. Parker, Smirnov, J. Pons, C. Carr, Z. Zukowski, Z. Evans, and X. Liu. Scaling transformers for low-bitrate high-quality speech coding. 2024. |tL//ttss://axx/ atg/abs/2411. 19842
[52] Peehles and Xie. Scalable diffusion models with transformers, 2023. URI, CTRI.tttps://// /tttt/tt////atiii/ /ttpa////aavvv.va. /ariivovvvg.////////// abs/2212.0977
[53] A. Polyak, Y. Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux. Speech resynthesis from discrete disentangled self-supervised representations, 2021. URL hattps://arae org/ abs/2104.00m (035/ 00355
[54] C. K. Reddy, V. Gopal, and R. Cutler. Dnsmos: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 6493-6497. IEEE, 2021.
[55] C. Reddy, V. Gopal, and R. Cutler. Dnsmos p.835: non-intrusivv perceptual objective speech quality metric to evaluate noise suppressors. ICASSP 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEFF, 2022.
[56] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusson models, 2022. URL https:///rxivy
htrg//bs/21110 10752
[57] D. Sejdinovic, B. Sriperumbudur, A. Gretton, and K. Fukumizu. Equivalence of distance-based and rkhs-basee statistics in hypothesis testing. The Annals of Statistics, 41(5), Oct. 2013. ISSN 0090-5364. doi: 10.1214/13-aos 140. U doi. org/ 10. 1111/121111111111111111111111111
[58] R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units, 2016. URL https//ps:://// :/arxvvv////x...../ org/ abs/ 11508007900
[59] C. Shao, Meng, Y. Liu, and J. Zhou. Language generation with strictly proper scoring rules. In Proceedings of the 4 International Conference on Machine Learning, volume 235 of Proceedings of Machine Leurning Research, pages 44484-44488 PMLR, 21-27 Jul 2024. URL
mlr. wwwpppeess//ae watpassvvv3/// w2wp/////2.saaa..ccc shao24c. html
[60] C. Shao, F. Meng, and J. Zhou. Continuous visual autoregressive generation via score maximization, 2025.
URL https: h/appi://///xiii//// org/ abs/a//.......///.. 07812
13

---

[61] S. Shapiro and M. B. Wilk. An analysis of variance test for normality (complete samples). Biometrika, 52(3-4) 1965.
[62] N. Shazeer. Glu variants improve transformer, 2020. URL wwtphhtttt///aatt .rg. www.:ogg.aaaa........ wwwaaaaa....e..eee 20020/2202002
[63] H. Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high- quality audio synthesis, 2024. URL htl.lattpse//aazii.orr///g
00814
[64] Y. Song, Z. Chen, X. Wang, Z. Ma, and X. Chen. Ella-v: Stable neural codec language modeling with alignment-guien guided sequence reordering, 2024, URL httppe//aarxv..org//baaa
[65] J. Su, Y. Lu, S. Pan, A. Murtadha. B. Wen, and Y. Liu. Roformer: Enhanced transformer with rotary position embedding. 2023. URL /ttpss/:///aax org/
[66] G. J. Székely and M. L. Rizzo. Energy statistics: A class of statistics based on distances. Journal of statistical planning and inference, 143(8):129--1222 2013.
[67] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B, Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models, 2023. URL httttstttt//xii/ org/ /ttp..//....11.32
[68] M. Tschannen, C. Eastwood. and F. Mentzer. Givt: Generative infinit- transformers, 2024. URL Utt/tttp::// httts:t//aaaiiii http://wwwwwww..aaaaaa///a/oaa/a
221/ 02116
[69] A. Turetzky, N. Shabtay, S. Shechtman, H. Aronowitz, D. Haws, R. Hoory, and A. Dekel. Continuous speech synthesis using per-token latent diffusion, 2024. htt.bttpp///axiii org/ abs/2410 24100.60048
[70] A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. 2017. URL https://arxiv org/ aa/aabss//1// 11111.11.... 1711.00937777
[71] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need, 2023. L
https: : /axxiv.iivv...i http://iiiiirraaaaaaa/ ab//////b/////////////////////// 03762
[72] C. Wang, S. Chen, Y. Wu, Z. Zhang, Zhou, Liu, Chen, Liu, H. Wang. Li. He, S. Zhao, and F. Wei. Neural codec language models are zero-shot text to speech synthesizers, 2023. URL
actps://arxil: org/ abs/ abs//3011 22111020111111111111111011111111
[73] H. Wang, S. Liu, Meng, Li, Y. Yang, Zhao, Sun, Y. Liu, H. Sun, J. Zhou, Lu, and Y. Qin. Felle: Autoregressive speech synthesis with token-wise coarse-to toniiee flow matching, 2025. URL
https: : //axxiv.//xii..//aa org/ / 11128
[74] X. Wang. Y. Li, C. Fu, Y. Shen, L. Xie, K. Li, X. Sun, and L. Ma. Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen 1lm, 2024.
org/ aa//aba/22444 2411. 00774
[75] Y. Wang. H. Zhan, L. Liu, Zeng, H. Guo, Zheng, Q. Zhang, X. Zhang. S. Zhang, and Wu. Maskget: Zero-shot spxcchh with masked generative codec transformer, 2024. URL httphttps:/// peattts://arx/n org/ abs/2409.0000
[76] Xu, 7. Guo, He, Hu, Hc, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, B. Zhang, X. Wang, Chu, and J. Lin. Qwen2.5-omm technical report, 2025. URL/https: http:stt:://ariii //ta:s/////xxi
org/ abs/ 200///50022222 20215
[77] D. Yang, J. Tian, X. Tan, R. Huang, Liu, Chang, Shi, S. Zhao, Bian, Z. Zhao, X. Wu, and H. Meng. Uniaudio: An audio foundation model toward universal audio generation, 2024. URL sttps://arxiv..om org/ abs/ 2310. 2ttp///110000000
[78] Y. Yang. Z. Ma. Liu, Li, H. Wang, Meng, H. Sun, Y. Liang, R. Xu, Hu, Y. Lu, R. Zhao, and X. Chen. Intericaved speech-tex text language models are simple streaming text to speech synthesizers, 2024.
URL htt/:ttps:/// //appp:://rxiiis org/ aasp//bbs/22..../// 2412. 16102
[79] Y. Yang, Liu, Li. Hu, H. Wu, H. Wang, J. Yu, Meng, H. Sun, Liu, Y. Lu, K. Yu, and X. Chen. Pseudo-autorrgreesive neural codec language models for efficient zero-shot text-to-speech synthesis, 2025.
URL httphttts///xxii org/ abs/2504. 2504. 10352
[80] 7. Ye, P. Sun, Lei, Lin, X. Tan, 7. Dai, Q. Kong, J. Chen, J. Pan, Q. Liu, Y. Guo, and W. Xue. Codec does matter: Exploring the semantic shortcoming of codec for audio language model, 2024. URI. attps://arxx: org/ /208 17175
[81] Z. Ye, x. Zhu, C.-M. Chan, X. Wang. X. Tan, J. Lei. Y. Peng. H. Liu, Y. Jin, Z. Dai, H. Lin, J. Chen, X. Du, L. Xue, Y. Chen, Z. Li. L. Xie, Q Kong, Y. Guo, and W. Xue. Llasa: Scaling train time and inference-timg compute for Ilama- basa- speech synthesis, 2025. URL wtttttsss//arate www./bs/////////ooooooooooe 2502. 04128
14

---

[82] N. Zeghidour. A. Luebs, A. Omran, J. Skoglund. and M. Tagliasacchi. Soundstream: An end-to-end neural audio codec, 2021. /ttp://///////iitt...
abs/2107 03312
[83] A. Zeng, Z. Du, M. Liu, I.. Zhang, shengmin jiang, Y. Dong, and J. Tang. Scaling speech-text pre-training with synthetic interleaved data. In The Thirteenth International Conference Learning Representations, 2025. URL httpps:
wwwpeeeee//eeeeeeeee net formm? id= 3t3kkddssyyyr
[84] B. Zhang and R. Sennrich. Root square layer normalization, 2019. URL https://arxii..orrr abs/1910.0744
[85] D. Zhang. S. Li, X. Zhang. J. Zhan, P. Wang, Y. Zhou, and X. Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities, 2023. URL URRlhttts://arxii..oom abs 2305
11000
[86] X. Zhang, Zhang. S. Li, Zhou, and Qiu. Speechtokenizer: Unified speech tokenizer for speech large language models, 2024. Uttpttpps: :/tps::///xxivi org////ivoorroee abs /ttps//23...... 16692
[87] X. Zhu, W. Tian, Xie, Autoregressive speech synthesis with next-distribution prediction, 2024. URL https://arxiv..o aba/ 16846
[88] Zhu, D. Su, Xu, and Yu. Generative pre-trained speech language model with efficient hierarchical transformer, 2024. URL
00970
15

---

A
Results at Different Data Scales
Table 5: Comparison of model performance when trained on LibriSpeech versus LibriHeavy.
3s Prefix as Prompt
Reference Utterance as Prompt
System
WER-C
WER-H
SIM
WER-C
WER-H
SIM
Ground Truth
1.78
2.15
0.668
1.78
2.15
0.778
Ground Truth (Encodec)
1.79
2.30
0.606
1.79
2.30
0.738
Models trained on the 1,000-kour LibriSpeech
SLED
1.68
2.28
0.490
2.27
2.88
0.616
Models trained on the 50, 00.000r Librilleavy
SLED
1.59
1.99
0,515
1.51
1.97
0.664
B Why not Gaussian?
Sec. 3.2 claim Gaussian distribution does not have enough capacity model the per-token continuous distribution in latent space. We have designed experiments to demonstrate this. Specifically, using a trained SLED (which we assume has successfully captured the underlying data distribution). we sampled 100 latent vectors from the first speech position and conducted Shapiro- Hoapiro-Wilk Wilk test [61] to examine whether each dimension follows Gaussian distribution. The result showed that 0 of 128 dimensions passed the normality test (p 0.05), indicating that none of dimensions individually follow normal distribution.
C More Analysis of Inference Efficiency
Table Inference latency and RTF of generating a 10s speech. *: We use teacher-forcing-stye AR
forward pass to estimate FLOPs in AR module.
Model
AR Latency
Gen. Module Latency
NAR Latency
RTF
FLOPs#
VALL-E
6.91 s 0.16 s (Softmax)
0.94 s
0.8
8 x 222.7G
SLED
6.82 s
1.23 (MLP)
0.8
280.05 G
Table 7: RTF of SLED for different batch sizes.
Batch Size
SLED's RTF
0.8
16
0.055
64
0.027
As shown, SLED and discrete hierarchical models like VALL-E exhibit significant difference in RTF, However, SLED largely reduces FLOPS compared with hierarchical models and supports streaming generation. The additional time spent by the MI.P heads in continuous generation is offset by the efficiency gains from avoiding the decoding process required by local NAR models. Combined with its RTF being less than 1. SLED is well-suited for supporting real time applications.
Moreover, while SLED's RTF may appear higher than NAR TTS models such 66. argue that comparing RTF between and iterative NAR with the batch size fixed
somewhat misleading. f'at,, prior studies on NAR generation 21 have long advocated for evaluating AR and NAR models under larger batch sizes up to the capacity supported by the GPU-or cquivalently, under the same computational budget (FLOPs). In fact, we observe that increasing the batch size significantly reduces SLED's RTF.
D Broader Impact
The speech language modeling method presented in this paper enables voice cloning, and the experimental results demonstrate that voice cloning is feasible to some extent. However, the findings indicate, there remains
16

---

a noticeable gap between the cloned voice and the original speaker's voice. As such, speaker verification model be used to detect potential voice forgeries.
17

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We made clear claims in the abstract and introduction sections.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Section Z
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" scction their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well rell-specifications asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
18

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide all details of our experimental settings in Section 5.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the of cosed-source models, it may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, described supplemental material?
the data and code, with sufficient instructions
Answer: [Yes]
Justification:
In the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cclppps..cc/ppee cc/public/ quides/CodeSubmn seiobboliio for more details.
While we encourage the release of code and data, we understand that this might not be possihle, so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: :/ :/nniss//.///////////e:::eee cc/public/ qu..des//ooeeSubbllssssoool..... for more details.
19

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: We don't provide the experiments for statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
CL, if the hypothesis of Normality of errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
In Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
20

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform. every respect, with NeurIPS Code Ethics http:p/s//eeei.......baa//////aa
Answer: [Yes]
Justification:
The research conducted in the paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from Code of Ethics.
The authors should make sure to preserve anonymity (e-g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations. (e.g.. deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considcrations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should deseribe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
21

---

12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected"'
Answer: [Yes]
Justification:
They are mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ppaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged reach out asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowd sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection. curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowd sourcing nor research with human subjects.
Guidelines:
22

---

The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
recognize that procedures for this may vary significantly between institutions and Iocations, and we expect authors adhere to the NeurIPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this rescarch? Note that if the LLM is used only for writing, editing, or formatting purposes and does impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
The core method development this research does not involve LLMs as any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components.
Please refer to our LLM policy htttps///eeriisssc///oottrrre//p cc//oo/oreenccs//222222222 should should not be described.
LLM) for what
23


Reviews:
[{"review_id": "XYD5Yb9CJa", "reviewer": "Reviewer_GWrt", "summary": "This paper proposes SLED, a method for modeling speech auto-regressively in a continuous space using generalized energy distance as the training objective. By operating on continuous speech features rather than discrete tokens, SLED preserves finer-grained details and offers greater computational efficiency compared to multi-stream discrete representations. Experiments on LibriSpeech demonstrate that SLED achieves competitive performance with baseline methods in synthesis quality, and ablation studies highlight the importance of the repulsive term in the energy distance formulation.", "questions": "1. How is the reference utterance used as a prompt within the SLED architecture? \n2. Have the authors considered fine-tuning from a pre-trained language model? Would initializing from a pre-trained LLM improve convergence or synthesis quality?", "limitations": "See Strengths And Weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength\n\n1. The proposed method offers an elegant approach to modeling sequences of continuous speech features in an auto-regressive manner.\n2. Experiments on LibriSpeech demonstrate that SLED achieves competitive or superior synthesis quality using purely auto-regressive generation on continuous features.\n3. The model supports streaming text-to-speech (TTS) with only a slight degradation in output quality.\n\n# Weakness\n\n1. SLED relies on a pre-trained codec model to produce continuous embeddings. It would be helpful to compare this with alternatives such as mel-spectrograms or self-supervised features like WavLM.\n2. While the paper claims improved efficiency over multi-stream discrete token modeling, it lacks a thorough evaluation of computational cost. Additionally, classifier-free guidance introduces extra overhead that should be accounted for.\n3. The streaming policy assumes a fixed text-to-speech ratio, which may not generalize well to varied speaking rates. Moreover, the paper should clarify the exact latency—i.e., the delay between receiving a text token and producing its corresponding audio.", "was_revised": true, "final_justification": "My primary concern was the reliance on Encodec features; however, the authors have demonstrated that SLED also works with VAE-based features, confirming that the method is indeed generalizable. I still have a minor reservation about its robustness to varying speaking rates, but this is not a significant issue.\n\nOverall, I will maintain my positive score.", "rebuttal": "***Thank you for your thoughtful review. We would like to address your concerns as follows.***\n\n> *SLED relies on a pre-trained codec model to produce continuous embeddings. It would be helpful to compare this with alternatives such as mel-spectrograms or self-supervised features like WavLM.*\n\nThank you for your suggestions. We additionally experimented with training a VAE as a continuous tokenizer to extract latent speech representations. As VAE has no discrete bottleneck, it can encode speech into continuous vectors at a lower frame rate, thereby reducing the number of autoregressive steps in the downstream language model. We see this as another advantage of continuous modeling. We trained a VAE with compression ratios of 2 → 4 → 5 → 5 → 8, mapping 24 kHz audio to 15 Hz, 128d vectors. We then train another SLED on top of this tokenizer.\n\n**L**: 3s Prefix as Prompt **R**: Reference Utterance as Prompt\n|WER-C|SIM|WER-C|SIM|\n|-|-|-|-|\n|1.56 | 0.551|1.53|0.676|\n\nWe find, even under the lower frame rate setting, the model trained on the VAE‑based continuous tokenizer achieves comparable word accuracy while delivering better voice cloning similarity. These findings further highlight the feasibility of speech language modeling in a continuous latent space and also show the importance of tailoring the tokenizer to continuous autoregressive models.\n\n> *While the paper claims improved efficiency over multi-stream discrete token modeling, it lacks a thorough evaluation of computational cost. Additionally, classifier-free guidance introduces extra overhead that should be accounted for.*\n\nSorry for the confusion. We actually include an efficiency analysis in Appendix E; for your convenience, we reproduce the table here:\n\n*Table 5. Inference latency and RTF of generating a 10 s speech  \n\\* We use teacher‑forcing‑style AR forward pass to estimate FLOPs in the AR module.*\n\n| Model | AR Latency | Gen. Module Latency | NAR Latency | RTF | FLOPs\\* |\n|-------|-----------:|--------------------:|------------:|----:|--------:|\n| VALL‑E | 6.91 s | 0.16 s (Softmax) | 0.94 s | 0.8 | 8 × 222.7 G |\n| SLED (No CFG) | 6.82 s | 1.23 s (MLP) | — | 0.8 | 280.05 G |\n\n\nThis table compares the computational efficiency of continuous and discrete modeling approaches when both use the same tokenizer (Encodec). the SLED figures shown are obtained without CFG. When CFG is enabled, the computational load (FLOPs) of the global AR component doubles (while latency is almost unaffected), whereas the MLP component remains unchanged. We recognize that the additional computational overhead introduced by CFG remains an open issue. In future work, we intend to investigate guidance‑free approaches for energy distance-based continuous autoregressive modeling to alleviate this overhead.\n\n> *The streaming policy assumes a fixed text-to-speech ratio, which may not generalize well to varied speaking rates. Moreover, the paper should clarify the exact latency—i.e., the delay between receiving a text token and producing its corresponding audio.*\n\nIn our opinion, a fixed text‑to‑speech ratio does not hurt the model’s ability to generalize to different speaking rates, because the audio at a given position does not have to contain words that exactly match the current text segment. The streaming synthesis section of the demo at the anonymous URL showcases samples generated at various speaking rates.\n\nWe also measured the latency between receiving the first text chunk (first five tokens) and emitting its corresponding 0.6‑second audio segment. The delay is about 0.3–0.4 s, though it can vary with hardware. To ensure full reproducibility, we have included the complete training and inference code in the supplementary materials and will release it publicly.\n\n> *How is the reference utterance used as a prompt within the SLED architecture?*\n\nWe prepend the model input with \"prompt text + input text + prompt speech\", enabling the model to generate speech whose timbre closely matches that of the prompt speech.\n\n> *Have the authors considered fine-tuning from a pre-trained language model? Would initializing from a pre-trained LLM improve convergence or synthesis quality?*\n\nWe have experimented with initializing from Qwen2-0.5B but observed no clear improvement. In our view, TTS largely hinges on the model’s ability to generate speech, which gains little from a text-only pre-trained model. However, more complex speech generation tasks (e.g., InstructTTS) may require stronger text understanding, in which case initializing from a pre‑trained LLM could be more beneficial.\n\n***We hope these clarifications give you a clearer understanding of our work. Thank you again for your efforts in reviewing.***"}, {"review_id": "SsSbRCHozm", "reviewer": "Reviewer_BhHp", "summary": "This paper introduces SLED, a novel approach to speech language modeling that operates in a continuous latent space using energy distance. By avoiding discretization via residual vector quantization (RVQ) and complex hierarchical architectures, SLED simplifies the modeling pipeline while preserving speech information richness. It employs a lightweight implicit conditional generative module trained with generalized energy distance (GED), a form of maximum mean discrepancy (MMD), enabling efficient autoregressive modeling. Experimental results demonstrate strong performance in zero-shot and streaming speech synthesis, outperforming discrete counterparts like VALL-E in parameter efficiency and quality metrics (e.g., WER, SIM, DNSMOS)", "questions": "1. Are there any experimental results using more Encodec RVQ layers?\n2. How does the model’s performance scale with larger parameter sizes  and extended training data beyond 50,000 hours (like in Emilia)?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n* SLED eliminates discretization errors and information loss inherent in RVQ-based methods, leveraging continuous representations for more faithful speech reconstruction.\n* By eschewing hierarchical structures required for multi-stream RVQ tokens, SLED reduces complexity while maintaining efficiency in both training and inference.\n* The use of GED as a strictly proper scoring rule enables stable training, and the absence of iterative sampling or post-AR refinement enhances efficiency for long speech sequences.\n\nWeakness\n* The main drawback is that the selected baselines exhibit weak performance.", "was_revised": true, "final_justification": "All my concerns are resolved", "rebuttal": "***Thank you for your acknowledgement of this work. We would like to address your remaining concerns as follows.***\n\n> *Are there any experimental results using more Encodec RVQ layers?*\n\nIn our initial experiments with Encodec, we constructed the latent representation by summing the embeddings from first eight VQ layers. The goal was to align with VALL‑E’s setup and remove any confounding effects from the tokenizer itself.\nIn later experiments, to further showcase the benefits of modeling in a continuous latent space, we explored two modifications to the tokenizer:\n\n**1.Increase the number of Encodec RVQ layers used to build the continuous latent representation.**\n\n**2.Retrain a continuous tokenizer based on a VAE architecture, which contains no discrete bottleneck at all.**\n\nIn the first experiment, we evaluated configurations with $n_q=16$ and $n_q=32$. The $n_q=16$ setup performed on par with $n_q=8$, but training collapsed at $n_q=32$ and the model failed to generate intelligible speech. We attribute this collapse to the excessive number of codebooks: **when too many are combined, the RVQ‑VAE effectively degenerates into a plain auto‑encoder, and the summed embeddings no longer carries any meaningful structural priors. Such an unstructured latent space hampers continuous‑space modelling.** In contrast, a moderate number of codebooks achieves a useful compromise between information preservation and latent‑space regularisation—functionally similar to the KL term in a VAE—which in turn supports autoregressive modelling in the continuous latent space.\n\nIn the second experiment, we experimented with training a VAE as a continuous tokenizer to extract latent representations. As VAE has no discrete bottleneck, it can encode speech into continuous vectors at a lower frame rate, thereby reducing the number of autoregressive steps in the downstream language model. We see this as another advantage of continuous modeling. We trained a VAE with compression ratios of 2 → 4 → 5 → 5 → 8, mapping 24 kHz audio to 15 Hz, 128d vectors. We then train another SLED on top of this tokenizer.\n\n**L**: 3s Prefix as Prompt **R**: Reference Utterance as Prompt\n|WER-C|SIM|WER-C|SIM|\n|-|-|-|-|\n|1.56 | 0.551|1.53|0.676|\n\nWe find, even under the lower frame rate setting, the model trained on the VAE‑based continuous tokenizer achieves comparable word accuracy while delivering better voice cloning similarity. These findings further highlight the feasibility of speech language modeling in a continuous latent space and also show the importance of tailoring the tokenizer to continuous autoregressive models.\n\n> *How does the model’s performance scale with larger parameter sizes and extended training data beyond 50,000 hours (like in Emilia)?*\n\nCurrently we have scaled the model to 0.5 billion parameters by initializing the AR backbone from Qwen2‑0.5B, but we observed no clear improvement. More complex speech‑generation tasks (e.g., InstructTTS) may demand stronger text understanding, in which case initialization from a pre‑trained LLM could be more advantageous. We plan to investigate the scaling laws of continuous speech‑language models in future work.\n\n***We hope these clarifications give you a clearer understanding of our work. Thank you again for your efforts in reviewing.***"}, {"review_id": "vpOccltr2j", "reviewer": "Reviewer_pbK8", "summary": "This paper introduces SLED, which is a language modeling-based speech synthesis approach based on continuous representations instead of discrete. From an architectural point of view, the SLED model is simple enough: it only requires a speech VAE as continuous encoder, a decoder only transformer, and a very lightweight generative head in AdaLN structure. The algorithm of SLED is also easy to understand: it is essentially an MSE loss between the generated latent and ground truth latent, plus a \"repulsive\" term that intuitively encourages randomness between two independently generated latents. This optimization target is derived from energy modeling theory. The SLED model is trained and evaluated using zero-shot TTS task, and compared to other TTS baselines in different genres. The experiments show that SLED achieves comparable performance to strong baselines but with a small parameter count and high efficiency. The authors also implemented classifier free guidance and streaming inference as bonus functions.", "questions": "* Why do you think the speaker similarity of SLED is relatively low?\n* Continuous autoregressive generation using diffusion is suffering from sampling efficiency if the generation is frame-by-frame, but if this is  done chunk-wise instead, the sampling efficiency could not be a big problem. There are some explorations into this approach, e.g. DiTAR (https://arxiv.org/pdf/2502.03930). How do you compare SLED with those?\n* Why is the classifier free guidance mechanism designed in such way? Interpolating in the z-space does not seem very intuitive to me.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\nThe paper is theoretically solid, and the idea of exploring energy modeling in autoregressive TTS has not been researched with depth. The architecture and algorithmic design of SLED is also simple enough for others to follow. The simplicity in design also benefits parameter efficiency and inference latency, since only a small generative head upon an LLM backbone is enough. The writing of this paper is rigorous and professional.\n\n### Weaknesses\nFrom the final loss form, the loss still looks like an MSE for regression, plus an additional \"repulsive\" term. Although this looks doubtful, the authors manage to explain the importance of this repulsive term by theory and experiments. However, this brings another concern (and also mentioned in the paper) that SLED does not differ from MELL-E fundamentally, because MELL-E also introduces the penalty for adjacent frames (perhaps empirically). From the comparison in Table 1 we can also find MELL-E actually outperforms SLED in 5 out of 6 metrics. The authors also mention that \"MELLE incorporates an NAR refinement, whereas SLED is purely autoregressive and can generate in a streaming manner\" as an advantage of SLED over MELLE. So, how will MELLE perform without this NAR refinement compared to SLED?\n\nIn other words, I think the difference and advantage of SLED over MELLE need to be further illustrated and studied. Hence I tend to give borderline reject to this paper, but would be happy to raise it if the questions are properly addressed.", "was_revised": true, "final_justification": "In my original review, I raised two key questions:\n* What is the fundamental difference between SLED and MELLE, since MELLE also incorporates a similar loss?\n* What is the advantage of SLED over DiTAR, as the latter is a hybrid approach between AR and NAR?\n\nIn the rebuttal, the authors resolved these questions by:\n* Clarifying that SLED is a theory-grounded solution instead of empirical method. With the energy modeling theory behind, the model behavior is more explainable, and more inspirations can be drawn for future work.\n\n* Comparing the performance of SLED and DiTAR under the same setup, showing that it can reach better performance under 1/10 FLOPs of DiTAR, because of the very lightweight prediction head.\n\nTherefore I'd like to raise the final score to 4 (borderline accept).", "rebuttal": "***Thank you for your thoughtful review. We would like to address your concerns as follows.***\n\n>*Further illustrate the difference and advantage of SLED over MELLE.* \n\na. **The main contribution of SLED**: Our main contribution lies in **presenting a concise and theory‑grounded framework to train continuous speech language model.** \n$$\nL_t = \\mathrm{E}_{h_t,h_t'}[d(h_t.h_t^*)-d(h_t,h_t')]\n$$\nwhere $d$ is any conditionally negative definite distance function [1,2] (e.g. Gaussian Kernel).\nThis framework is grounded in the theory of maximum mean discrepancy, which quantifies the distributional discrepancy. **Optimizing this loss guarantees the model distribution to be closer to the data distribution.**\n\nb.  **We train SLED with a specific instance of a conditionally negative‑definite distance function, $d(x,y) = \\lVert x - y\\rVert_2^{\\beta}, 0<\\beta<2$. This choice also lets us clarify which heuristic loss designs proposed in earlier work are genuinely beneficial and which ones turn out to be unnecessary.** Specifically, we link the framework’s repulsive term \n\n$L_{repulsive}^{t} = - \\mathrm{E}_{h_t,h_t'}[d(h_t,h_t')]$ \n\nto the flux loss used in MELLE [3], \n\n$L_{flux}^{t}= -\\lVert y_{t} - y_{t-1}^{*}\\rVert_1$. \n\nThe MELLE paper observes that their model often enters a degenerate loop of repetitive predictions, yielding poor audio quality. Therefore, MELLE heuristically forces the current frame’s prediction to stay away from the previous frame’s target. **We argue that its actuall effectiveness stems from the fact that the flux loss approximates the repulsive term in energy distance to some degreee, which makes their training closer to capture distributional difference, much like our framework, instead of pure regression.**\n\nc. **By exploiting the theoretical link between energy distance and maximum mean discrepancy, we obtain some insights that can guide future research.** The distance function should be conditionally negative definite. For the standard $L_n$ metrics, the only one that satisfies the required condition is $d(x,y) = \\lVert x - y\\rVert_2^{\\beta}, 0<\\beta<2$. This points to using an L2 distance rather than an L1 distance, and to taking the L2 distance in its square‑root form (β=1) instead of the squared form.\n\n[1] Strictly proper scoring rules, prediction, and estimation.  \n[2] Energy statistics: A class of statistics based on distances.  \n[3] Autoregressive Speech Synthesis without Vector Quantization.  \n\n\n\n\n> *Why do you think the speaker similarity of SLED is relatively low?*\n\nWe attribute the slightly lower speaker similarity scores of SLED reported in the manuscript to our use of EnCodec as the latent encoder. In our initial choice of tokenizer, to keep tokenizer effects out of the discrete‑vs‑continuous comparison, we adopted the same tokenizer setup as VALL‑E, employing the same 8‑VQ-layer EnCodec tokenizer.\n\nHowever, we additionally experimented with training a VAE as a tokenizer to extract continuous latent representations. As VAE has no discrete bottleneck, it can encode speech into continuous vectors at a lower frame rate, thereby reducing the number of autoregressive steps in the downstream language model. We see this as another advantage of continuous modeling. We trained a VAE with compression ratios of 2 → 4 → 5 → 5 → 8, mapping 24 kHz audio to 15 Hz, 128d vectors. We then train another SLED on top of this tokenizer.\n\n**L**: 3s Prefix as Prompt **R**: Reference Utterance as Prompt\n|WER-C|SIM|WER-C|SIM|\n|-|-|-|-|\n|1.56 | 0.551|1.53|0.676|\n\nWe find the model trained on the VAE‑based continuous tokenizer achieves better voice cloning similarity even under the lower frame rate setting. These findings show the importance of tailoring the tokenizer to continuous autoregressive models.\n\n\n> *Continuous autoregressive generation using diffusion is suffering from sampling efficiency if the generation is frame-by-frame, but if this is done chunk-wise instead, the sampling efficiency could not be a big problem. There are some explorations into this approach, e.g. DiTAR (https://arxiv.org/pdf/2502.03930). How do you compare SLED with those?*\n\nWe see SLED and DiTAR as two distinct routes to improve the efficiency of continuous autoregressive speech synthesis. DiTAR takes a **semi‑autoregressive** approach: by trimming the number of sequential autoregressive steps, it gives a time complexity that sits between fully AR and fully NAR. In contrast, SLED tackles the problem by **boosting the efficiency of each individual autoregressive step**. We compare the perfomrance and efficiency of both methods trained on 50k hrs Libriheavy/LibriLight in the following table: (In this table, we report the results of SLED trained on Encodec's latent representation to keep consistent with our manuscript.)\n\n|     | WER | SIM | #Params | #Transformer Layer | RTF under Batch Size 1 | FLOPs|\n|-|-|-|-|-|-|-|\n|SLED | 1.51 | 0.664 | 0.2B | 12 | 0.8 | 280G |\n|DiTAR (patch size 4)| 1.78 | 0.64 | 0.6B |6+36+6 | 0.66 | 2750G |\n\nIn our opinions, the SLED architecture may offer the following advantages over DiTAR:\n\na. DiTAR is actually a hierachical model. Although patching cuts down the number of autoregressive steps, the per‑step generation workload in the DiTAR model is heavy, therefore each step relies on a relatively large local decoder instead of lightweight MLP in SLED. This will offset some of the advantages brought by the patch. Moreover, iterating within a local DiT model can make it prone to losing global information. In fact, Table 5 of the DiTAR paper notes that the local DiT model must incorporate historical‑patch dependencies to generate correctly; this further increases the model’s complexity and negates the computational advantages gained from patching. As a result, DiTAR attains an RTF comparable to SLED, but it incurs nearly ten times the FLOPs (though DiTAR is 3 times larger than SLED).\n\nb. From the point of reducing autoregressive steps, we think a more promising route is to train a continuous tokenizer that compresses the frame rate, thereby reducing the number of steps the autoregressive model must take. Our experiments with a VAE-based continuous tokenizer show that a well‑designed tokenizer that lowers the frame rate can preserve comparable word‑level accuracy while even boosting cloning similarity. In our view, pairing this strategy with SLED’s modeling scheme is the more efficient choice.\n\n\n\n> *Why is the classifier free guidance mechanism designed in such way? Interpolating in the z-space does not seem very intuitive to me.*\n\nWe devised the CFG approach in SLED following the rationale below.\nIn a probabilistic model, CFG is essentially doing the following: $p^{\\text{sample}}(x \\mid c) \\propto p(x \\mid c) \\left[\\frac{p(x \\mid c)}{p(x)} \\right]^{s}$. Let $l=logp$, then we have the interpolating rule $l^{sample}\n=l^{c} + s \\bigl(l^{c} - l^{u}\\bigr)$. In a discrete AR model, $l$ represents the model’s logits, which is the hidden states before the softmax is applied. In SLED, the lightweight MLP plays a role analogous to the softmax layer in discrete models (Both transforming an autoregressive model’s encoded hidden states into a probability distribution). Therefore, we treat the hidden states $z$ in SLED like the logits $l$ in a discrete AR model and apply the same interpolation strategy during CFG generation.\n\n***We hope these clarifications give you a clearer understanding of our work. Thank you again for your efforts in reviewing.***"}, {"review_id": "8cnb2C4RIe", "reviewer": "Reviewer_Fkcq", "summary": "This paper introduces SLED, a speech language model that performs autoregressive modeling in continuous latent space using a lightweight MLP and an energy distance loss. By avoiding discretization and the hierarchical complexity associated with RVQ-based approaches like VALL-E, the method aims to simplify the modeling pipeline while preserving efficiency.\nThe authors claim that their approach enables efficient, streaming generation and achieves competitive results in zero-shot speech synthesis, particularly in terms of intelligibility and speaker similarity.", "questions": "Have you tried using codec without quantization bottleneck?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe paper is easy to follow.\n\nWeaknesses:\nI have several concerns:\n1. The use of continuous representations and autoregressive modeling is not new. The core novelty appears to be the addition of a repulsive term to the loss function, intended to better match the target distribution. However, the model does not explicitly learn a per-token distribution, making it less expressive than approaches based on flow matching, diffusion, or token-wise GMMs. In this sense, the method feels more like a variant of Tacotron than a fundamentally new approach.\n2. While the authors advocate for continuous modeling, they still rely on a codec (e.g., Encodec) trained with discrete quantization. This undermines the claim that their method avoids quantization-based bottlenecks.\n3. The demo page lacks baseline audio samples, making it difficult for reviewers to meaningfully compare the quality of SLED against prior work.\n4. The paper appears to conflate autoregressive generation with streaming capability. In practice, autoregressive models do not inherently support streaming unless there is a mechanism to determine when to consume input incrementally and when to pause for additional context—something not clearly addressed in the current design.", "was_revised": true, "final_justification": "I am still concerned about the absence of baseline audio samples. However, since NeurIPS does not permit updating the content, I will let this pass. I have therefore increased my score to marginal accept.", "rebuttal": "***Thank you for your thoughtful review. We believe there may have been some misunderstandings, and we hope the following clarifications will help resolve them.***\n\n> *However, the model does not explicitly learn a per-token distribution, making it less expressive than approaches based on flow matching, diffusion, or token-wise GMMs. In this sense, the method feels more like a variant of Tacotron than a fundamentally new approach.*\n\n**Actually, our method explicitly learns a per‑token probability distribution. The repulsive term is derived from energy statistics [1, 2], which provide a principled measure of *distributional discrepancy* between model and data. This is the key difference from prior regression‑based systems such as Tacotron.** As detailed in Section 2, the generalized energy distance (Eq.4) is a special case of Maximum Mean Discrepancy (Eq.2) when the distance function is conditionally negative definite [1] (e.g. Gaussian Kernel or $d(x,y) = \\lVert x - y\\rVert_2^{\\beta}, 0<\\beta<2$). **MMD supplies a closed‑form metric for continuous distributions, analogous to the KL divergence for discrete ones. Therefore, minimizing the energy distance guides the model distribution toward the data distribution in much the same way that cross‑entropy minimization does in the discrete setting.** Moreover, modeling continuous distributions with the energy distance is theoretically more expressive than using analytically tractable distributions such as Gaussians or GMMs, because it imposes no prior assumptions on the latent‑space distribution. We elaborate on this in Appendix D.\n\n[1] Strictly proper scoring rules, prediction, and estimation.  \n[2] Energy statistics: A class of statistics based on distances.  \n\n> *1. While the authors advocate for continuous modeling, they still rely on a codec (e.g., Encodec) trained with discrete quantization. This undermines the claim that their method avoids quantization-based bottlenecks.*\n*2. Have you tried using codec without quantization bottleneck?*\n\n**We additionally experimented with training a VAE as a continuous tokenizer to extract latent speech representations.** As VAE has no discrete bottleneck, it can encode speech into continuous vectors at a lower frame rate, thereby reducing the number of autoregressive steps in the downstream language model. We see this as another advantage of continuous modeling. We trained a VAE with compression ratios of 2 → 4 → 5 → 5 → 8, mapping 24 kHz audio to 15 Hz, 128d vectors. We then train another SLED on top of this tokenizer.\n\n**L**: 3s Prefix as Prompt **R**: Reference Utterance as Prompt\n|WER-C|SIM|WER-C|SIM|\n|-|-|-|-|\n|1.56 | 0.551|1.53|0.676|\n\nWe find, even under the lower frame rate setting, the model trained on the VAE‑based continuous tokenizer achieves comparable word accuracy while delivering better voice cloning similarity. **These findings further highlight the feasibility of speech language modeling in a continuous latent space and also show the importance of tailoring the tokenizer to continuous autoregressive models.**\n\n> *The demo page lacks baseline audio samples, making it difficult for reviewers to meaningfully compare the quality of SLED against prior work.*\n\nThank you for your suggestion; we will improve the way we create the demo.\n\n\n> *The paper appears to conflate autoregressive generation with streaming capability. In practice, autoregressive models do not inherently support streaming unless there is a mechanism to determine when to consume input incrementally and when to pause for additional context—something not clearly addressed in the current design.*\n\nWe believe there is some misunderstanding. **In Section 3.4, we actually explain our streaming mechanism**: because the text‑to‑speech task exhibits strong monotonic alignment, we design the mechanism as a fixed‑ratio text–speech interleaving. **We also specify the exact interleaving ratios we used (5:45 and 5:20) when presenting the streaming generation results in Table 2.**\n\n***We hope these clarifications give you a clearer understanding of our work. Thank you again for your efforts in reviewing.***"}]

Author rebuttal:

Meta-review: This paper presents SLED, an autoregressive speech synthesis method operating on continuous latent representations and trained with generalized energy distance (GED). SLED preserves fine-grained speech details, supports streaming (monotonic!) TTS, and achieves competitive zero-shot and genre-diverse synthesis with high efficiency compared to baselines like VALL-E and DiTAR. Strengths, as highlithted by the reviewers, include theoretical grounding, simplicity, parameter efficiency, and clear exposition. During the discussion period, the authors addressed concerns about similarity to MELL-E and show superior FLOPs efficiency over DiTAR. Weaknesses are minor, including reliance on pre-trained encoders, limited large-scale evaluation, and questions about robustness to varying speaking rates (but this is due to the choice of fixed streaming policy, which isn't the main point of this paper). Overall, SLED offers a meaningful methodological contribution with strong empirical support.
Decision: Accept (poster)