Efficient Speech Language Modeling via Energy Distance in Continuous
  Latent Space
Authors: name: Zhengrui Ma, name: Yang Feng, name: Chenze Shao, name: Fandong Meng, name: Jie Zhou, name: Min Zhang

We introduce SLED, an alternative approach to speech language modeling by
encoding speech waveforms into sequences of continuous latent representations
and modeling them autoregressively using an energy distance objective. The
energy distance offers an analytical measure of the distributional gap by
contrasting simulated and target samples, enabling efficient training to
capture the underlying continuous autoregressive distribution. By bypassing
reliance on residual vector quantization, SLED avoids discretization errors and
eliminates the need for the complicated hierarchical architectures common in
existing speech language models. It simplifies the overall modeling pipeline
while preserving the richness of speech information and maintaining inference
efficiency. Empirical results demonstrate that SLED achieves strong performance
in both zero-shot and streaming speech synthesis, showing its potential for
broader applications in general-purpose speech language models.