Title: Efficient Speech Language Modeling via Energy Distance in Continuous Latent Space
Venue: NeurIPS 2025
Accepted: yes
Keywords: speech language modeling

autoregressive modeling

speech synthesis
Abstract: We introduce \emph{SLED}, an alternative approach to speech language modeling by encoding speech waveforms into sequences of continuous latent representations and modeling them autoregressively using an energy distance objective. The energy distance offers an analytical measure of the distributional gap by contrasting simulated and target samples, enabling efficient training to capture the underlying continuous autoregressive distribution. By bypassing reliance on residual vector quantization, SLED avoids discretization errors and eliminates the need for the complicated hierarchical architectures common in existing speech language models. It simplifies the overall modeling pipeline while preserving the richness of speech information and maintaining inference efficiency. Empirical results demonstrate that SLED achieves strong performance in both zero-shot and streaming speech synthesis, showing its potential for broader applications in general-purpose speech language models. Demos and code are available at \url{https://github.com/ictnlp/SLED-TTS}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SLED, a speech language model that performs autoregressive modeling in continuous latent space using a lightweight MLP and an energy distance loss. By avoiding discretization and the hierarchical complexity associated with RVQ-based approaches like VALL-E, the method aims to simplify the modeling pipeline while preserving efficiency.\nThe authors claim that their approach enables efficient, streaming generation and achieves competitive results in zero-shot speech synthesis, particularly in terms of intelligibility and speaker similarity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper is easy to follow.\n\nWeaknesses:\nI have several concerns:\n1. The use of continuous representations and autoregressive modeling is not new. The core novelty appears to be the addition of a repulsive term to the loss function, intended to better match the target distribution. However, the model does not explicitly learn a per-token distribution, making it less expressive than approaches based on flow matching, diffusion, or token-wise GMMs. In this sense, the method feels more like a variant of Tacotron than a fundamentally new approach.\n2. While the authors advocate for continuous modeling, they still rely on a codec (e.g., Encodec) trained with discrete quantization. This undermines the claim that their method avoids quantization-based bottlenecks.\n3. The demo page lacks baseline audio samples, making it difficult for reviewers to meaningfully compare the quality of SLED against prior work.\n4. The paper appears to conflate autoregressive generation with streaming capability. In practice, autoregressive models do not inherently support streaming unless there is a mechanism to determine when to consume input incrementally and when to pause for additional context—something not clearly addressed in the current design.\n\nQUESTIONS:\nHave you tried using codec without quantization bottleneck?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SLED, which is a language modeling-based speech synthesis approach based on continuous representations instead of discrete. From an architectural point of view, the SLED model is simple enough: it only requires a speech VAE as continuous encoder, a decoder only transformer, and a very lightweight generative head in AdaLN structure. The algorithm of SLED is also easy to understand: it is essentially an MSE loss between the generated latent and ground truth latent, plus a \"repulsive\" term that intuitively encourages randomness between two independently generated latents. This optimization target is derived from energy modeling theory. The SLED model is trained and evaluated using zero-shot TTS task, and compared to other TTS baselines in different genres. The experiments show that SLED achieves comparable performance to strong baselines but with a small parameter count and high efficiency. The authors also implemented classifier free guidance and streaming inference as bonus functions.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\nThe paper is theoretically solid, and the idea of exploring energy modeling in autoregressive TTS has not been researched with depth. The architecture and algorithmic design of SLED is also simple enough for others to follow. The simplicity in design also benefits parameter efficiency and inference latency, since only a small generative head upon an LLM backbone is enough. The writing of this paper is rigorous and professional.\n\n### Weaknesses\nFrom the final loss form, the loss still looks like an MSE for regression, plus an additional \"repulsive\" term. Although this looks doubtful, the authors manage to explain the importance of this repulsive term by theory and experiments. However, this brings another concern (and also mentioned in the paper) that SLED does not differ from MELL-E fundamentally, because MELL-E also introduces the penalty for adjacent frames (perhaps empirically). From the comparison in Table 1 we can also find MELL-E actually outperforms SLED in 5 out of 6 metrics. The authors also mention that \"MELLE incorporates an NAR refinement, whereas SLED is purely autoregressive and can generate in a streaming manner\" as an advantage of SLED over MELLE. So, how will MELLE perform without this NAR refinement compared to SLED?\n\nIn other words, I think the difference and advantage of SLED over MELLE need to be further illustrated and studied. Hence I tend to give borderline reject to this paper, but would be happy to raise it if the questions are properly addressed.\n\nQUESTIONS:\n* Why do you think the speaker similarity of SLED is relatively low?\n* Continuous autoregressive generation using diffusion is suffering from sampling efficiency if the generation is frame-by-frame, but if this is  done chunk-wise instead, the sampling efficiency could not be a big problem. There are some explorations into this approach, e.g. DiTAR (https://arxiv.org/pdf/2502.03930). How do you compare SLED with those?\n* Why is the classifier free guidance mechanism designed in such way? Interpolating in the z-space does not seem very intuitive to me.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper introduces SLED, a novel approach to speech language modeling that operates in a continuous latent space using energy distance. By avoiding discretization via residual vector quantization (RVQ) and complex hierarchical architectures, SLED simplifies the modeling pipeline while preserving speech information richness. It employs a lightweight implicit conditional generative module trained with generalized energy distance (GED), a form of maximum mean discrepancy (MMD), enabling efficient autoregressive modeling. Experimental results demonstrate strong performance in zero-shot and streaming speech synthesis, outperforming discrete counterparts like VALL-E in parameter efficiency and quality metrics (e.g., WER, SIM, DNSMOS)

STRENGTHS AND WEAKNESSES:
Strengths
* SLED eliminates discretization errors and information loss inherent in RVQ-based methods, leveraging continuous representations for more faithful speech reconstruction.
* By eschewing hierarchical structures required for multi-stream RVQ tokens, SLED reduces complexity while maintaining efficiency in both training and inference.
* The use of GED as a strictly proper scoring rule enables stable training, and the absence of iterative sampling or post-AR refinement enhances efficiency for long speech sequences.

Weakness
* The main drawback is that the selected baselines exhibit weak performance.

QUESTIONS:
1. Are there any experimental results using more Encodec RVQ layers?
2. How does the model’s performance scale with larger parameter sizes  and extended training data beyond 50,000 hours (like in Emilia)?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SLED, a method for modeling speech auto-regressively in a continuous space using generalized energy distance as the training objective. By operating on continuous speech features rather than discrete tokens, SLED preserves finer-grained details and offers greater computational efficiency compared to multi-stream discrete representations. Experiments on LibriSpeech demonstrate that SLED achieves competitive performance with baseline methods in synthesis quality, and ablation studies highlight the importance of the repulsive term in the energy distance formulation.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n\n1. The proposed method offers an elegant approach to modeling sequences of continuous speech features in an auto-regressive manner.\n2. Experiments on LibriSpeech demonstrate that SLED achieves competitive or superior synthesis quality using purely auto-regressive generation on continuous features.\n3. The model supports streaming text-to-speech (TTS) with only a slight degradation in output quality.\n\n# Weakness\n\n1. SLED relies on a pre-trained codec model to produce continuous embeddings. It would be helpful to compare this with alternatives such as mel-spectrograms or self-supervised features like WavLM.\n2. While the paper claims improved efficiency over multi-stream discrete token modeling, it lacks a thorough evaluation of computational cost. Additionally, classifier-free guidance introduces extra overhead that should be accounted for.\n3. The streaming policy assumes a fixed text-to-speech ratio, which may not generalize well to varied speaking rates. Moreover, the paper should clarify the exact latency—i.e., the delay between receiving a text token and producing its corresponding audio.\n\nQUESTIONS:\n1. How is the reference utterance used as a prompt within the SLED architecture? \n2. Have the authors considered fine-tuning from a pre-trained language model? Would initializing from a pre-trained LLM improve convergence or synthesis quality?\n\nLIMITATIONS:\nSee Strengths And Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}