Continuous Audio Language Models
Authors: Simon Rouard
Manu Orsini
Axel Roebel
Neil Zeghidour
Alexandre Défossez
Venue: ICLR 2026 Poster (2026)
Keywords: audio language model
speech
music
consistency models
continuous modeling
streaming
Primary area: generative models

Abstract: Audio Language Models (ALM) have emerged as the dominant paradigm for speech and music generation by representing audio as sequences of discrete tokens. Yet, unlike text tokens, which are invertible, audio tokens are extracted from lossy codecs with a limited bitrate. As a consequence, increasing audio quality requires generating more tokens, which imposes a trade-off between fidelity and computational cost. We address this issue by studying Continuous Audio Language Models (CALM). These models instantiate a large Transformer backbone that produces a contextual embedding at every timestep. This sequential information then conditions an MLP that generates the next continuous frame of an audio VAE through consistency modeling. By avoiding lossy compression, CALM achieves higher quality at lower computational cost than their discrete counterpart. Experiments on speech and music demonstrate improved efficiency and fidelity over state-of-the-art discrete audio language models, facilitating lightweight, high-quality audio generation.
Samples are available at iclr-continuous-audio-language-models.github.io. Finally, we release Pocket TTS, an open-source 100M-parameter text-to-speech model that can run faster than real time on a laptop CPU: github.com/kyutai-labs/pocket-tts.

Published as a conference paper at ICLR 2026
CONTINUOUS AUDIO LANGUAGE MODELS
Simon Rouard
Manu Orsini Kyutai
Axel Roebel
Kyutai UMR STMS
UMR STMS
manu@kyutai.osg
IRCAM-CNRS Sorbonne Univ.
IRCAM-CNRS Sorbonne Univ.
simon@kyutai.cm
Neil Zeghidour
Alexandre Défossez Kyutai
Kyutai
a al elex@kutaai.or
ABSTRACT
Audio Language Models (ALM) have emerged as the dominant paradigm for speech and music generation by representing audio as sequences of discrete tokens. Yet, unlike text tokens, which are invertible, audio tokens are extracted from lossy codecs with a limited bitrate. As a consequence, increasing audio quality requires generating more tokens, which imposes a trade-off between fidelity and computational cost. We address this issue by studying Continuous Audio Language Models (CALM). These models instantiate a large Transformer backbone that produces contextual embedding at every timestep. This sequential information then conditions an MLP that generates the next continuous frame of an audio VAE through consistency modeling. By avoiding lossy compression, CALM achieves higher quality lower computational cost than their discrete counterpart. Experiments on speech and music demonstrate improved efficiency and fidelity over state-of-the-arr discrete audio language models, facilitating lightweight, high-quality audio generation. Samples are available at jicl-cconinnoou-audiee language-modehe github.oo Finally, we release Pocket TTS, an open-source 100M-parameter text-to-speech model that can run faster than real time on laptop CPU: guthub.com/yuuullabbpp....
INTRODUCTION
Using classification over finite vocabulary as the training objective for autoregressive sequence models is an effective approach for naturally discrete modalities such as text, where large-scale scale Transformer-based (Vaswani et al. (017) language models such as LLaMa (Touvron et al.) 2023) and HEPT-4Open.Ae (2024) have achieved impressive results. To extend this powerful framework to continuous domains such as image, audio, video. previous work has mostly relied on discretizing signals using lossy compression algorithms (van den Oord ct al. al. (0188). such that they become akin to text. In particular, neural audio codecs (Zeghidour et al. 2021 Défossez et al. 2024a) have provided discrete representations of audio that are compact enough to allow for high- high-quality speech (Borsos) et al. 2023 Wang et al., 2023) and music (Agostinellii et al. 2023 Copet et al. (023) generation with autoregressive models. In this context, a Residual Vector Quantizer (RVQ) (Zeghidour et al. [022 transforms audio frame into coarse-to-fine hierarchy of tokens. As quantization inevitably intro- duces a perceptual quality loss, generating high-fideiity audio requires increasing the bitrate of audio tokens, which amounts to using deeper hierarchies of RVQ tokens. A consequence of growing the size of the token matrix (along time and token depth) is an additional computational load for the generative model, as the strong dependencies between tokens of the same frame([eeerciir et al. 2024) prevent fully parallel generation. The naive approach of flattening the token hierarchy (Borsos et al. 2023) being prohibitively expensive, Coopet et al, (0233) introduces a delay pattern that conjugates the com- putational efficiency of parallel generation with better modeling of inter-token dependencies, Lee et al. (2022) and Yang et al. (2023) furthermore introduce a smaller RQ- Transformer model that is
"Equal Contribution

---

Published as a conference paper at ICLR 2026
autoregressive along the depth axis, and Défossez et al. (2024b) combines this approach with the delay pattern. While these methods currently power state-of oh-the-ar generative models for audio (Dé fossez al. 2024b Labiausse et al. 2025), the trade-off imposed by residual quantization between quality and computation remains too constraining for generating high-quality audio on edge devices.
This motivates an alternative strategy: autoregressive modeling of continuous latents without quantization. Standard variational autoencoders (VAEs) are easier to train, are not affected by issues such codebook collapse, and can reconstruct audio at higher fidelity for the same latent dimensionality.. Pioneering work in the vision domain that autoregressively models continuous sequences includes GIVT (Tschannnn et al., (224) and MAR (Li et al., 2024). followed by some larger models in the image domain (Fan et al., 2025] Gu et al. 2025) and attempts in the audio domain (Turetzky et al., 2024; Pasini et al. 2024b Jia et al. 2025 wen Yang et al. 2025). MAR, the authors model the pok-token probability distribution with diffusion model (in the form of small MLP) conditioned on latent variable modeled by an autoregressive transformer backbone. SALAD Turetzky et al. 2024) and DiTAR (Jia et al. 2025) adapt MAR-style diffusion heads for text-to-speech modeling, achieving better audio quality than discrete baselines. However, these works are limited to text-to- to-specch on small-scale and domain-spectio
datasets, leaving open the question of how well they can adapt to more complex tasks such as speech continuation (without text supervision) and richer audio domains such as music. We apply CALM to 4 tasks which are speech and music continuation as well as text-to-speech and text-to-music.
We propose Continuous Audio Language Models (CALM) that predict sequences in the latent space of a VAE, bypassing the need for quantization. While we build on the MAR architecture where a transformer backbone uses (x+ ,..., x*=1) to predict an intermediate latent z", which then conditions a head (diffusion model) that models p(x" ), we find that without further improvements,
it fails to generate rich audio content and is slow to sample from. To overcome this, we introduce several key innovations:
Improving quality and stability: To mitigate error accumulation during inference, we follow Pasini et al. (2024b) and, during training, inject noise into the long-term context (x*, ...., ....,..,) Addi- tionally, we introduce short-context transformer that summarizes recent clean latents. providing the sampling head with both coarse long-range context and fine-grained local information. 2. Diffusion- to-Consistency replacement: We replace the diffusion model with a continuous consistency model (Lu & Song. 2025) during training, significantly accelerating inference without compromising sample quality. This change reduces the inference time of the sampler head by a factor of up to x20 in our music experiments and x x12 in our speech experiments compared to an RQ-Transformer head. 3. Gaussian Temperature sampling: Temperature control is crucial for high-quality speech generation, yet consistency models lack formal mechanism for temperature sampling. We present heuristic that approximates temperature sampling in the consistency setup. 4. Head batch multiplier: Sam- pling multiple noise levels at training time for the same latent highly accelerates training for a small cost. 5. Latent Classifier Free Guidance: we apply Classifier Free Guidance to the latent variable conditioning the consistency head for the conditioned CALM. 6. Latent Distillation: once that we have chosen a latent CFG coefficient, we can distill the CFG computation of the backbone into student backbone while keeping the same sampling head (MLP), hence dividing the batch size by 2 at inference time. This distillation can also be applied to a much smaller student backbone.
Finally, by using these innovations, we introduce Pocket TTS which is a 100M-parameter text-to- speech model that can run faster than real-time on a laptop CPU, We detail the results of Pocket TTS in See G and in the following technical report: kruaaioorppoce----sscchhih-----p
2 RELATED WORK
Autoregressive audio language models. Early autoregressive audio models operated on raw waveforms such as WaveNet (van den Oord et al. 2016), learned discrete codes via VQ-VAE, such as in Jukebox (Dhariwal et al. 2020) or continuous word-sized audio tokens (Algayres et al. 2003). The advent of neural audio codecs (Zeghidour et al.) 202 Défossez et al.) 2024a) enabled high-quality vector-quantizn tokenizers for general audio. These, in turn, powered Miscrete-toker audio LMs: AudioLM (Borsos et al. 2023) for unconditioned audio generation while AudioGen (Kreuk et al.) 2023). MusicLM (Agostinelli et al. 2023), and MusicGen (Copet et al. 2023) apply similar methods for text-to-audio and text-to-music generation. In speech, autoregressive modeling of RVQ codes
2

---

Published as a conference paper at ICLR 2026
-I.1..--1 11111-11-1-11111111
Consistency head
VAE Decoder
Comeaterccy head
Cornesberry head
Consalency head
Danasienccy head
Consistency MLP
Mices
Foar
Causal backbone transformer
Mix
Causal short context transformer
VAE Encoder
X-1
I. 1110111-1--1111
Figure 1: Overview of our model. During training, latent vectors x noised to encourage the backbone Transformer to focus on coarse structure. The consistency head is consistency model conditioned on the latent variable zuny produced by the backbone, as well as a short-term context vector Znzen computed from a short-context Transformer applied to the most recent clean latent tokens.
has been used for sext-to-speech generation (Wang al. 2023 Kharitonov al. (023) as well as for spoken dialogue (Défossez et al. (2024b) or translation (Labiausse al.) 2025 However, all of these systems rely on lossy quantization, which inevitably degrades audio fidelity unless a significant compute budget is spent to generate deep hierarchy of RVQ tokens. This is unlike CALM, which predicts continuous embeddings in one pass, providing a better quality-computation trade-off.
Continuous autoregressive models. In GIVT, Tschannen et al. (2024) use transformer to autoregressively model the latent space of VAE trained on images, parameterize a Gaussian Mixture Model, and train it with cross-entropy loss. The authors of MAR (Li et al., 2024) obtain better results by replacing the Gaussian mixture model with a diffusion-mode enabling the approximation of more diverse distributions. In MAR, large transformer backbone predicts a continuous embedding z given (x',..., x*11), which then conditions a small MLP diffusion network that models the probability distribution p(x" [z") of the next latent. This eliminates the need for discrete tokenizers, but at the cost of slow sampling: MAR typically needs hundreds of denoising steps per token. Hang et al. (025) aims speed up MAR by replacing the diffusion head with shortcut head (Frans et al. 2025) for few-step sampling. Shortcut models combine diffusion loss and self-consistency loss in order to accelerate the diffusion process. They reduce the number of diffusion steps from 100 to 8 with similar image quality. Remarkably, CALM achieves a quality comparable the best discrete models with only one step of consistency modeling.
In audio, the approach of MAR has been adapted for the task of Text-to-Speech (TTS) synthesis. SALAD (Tretzky et al., 2024) introduces a zero-shot TTS model that operates on continuous speech representations using a per-token latent diffusion process. By leveraging semantic tokens for contex- tual information and determining synthesis stopping points, SALAD achieves improved intelligibility and audio quality without relying quantization. Similarly, DITAR (Jia al.) 2025 presents patch- based autoregressive framework combining a language model with a diffusion transformer for speech generation. This approach models dependencies between aggregated local patches of continuous tokens, using a causal language model to produce embeddings, which, along with previous patches, serve as inputs
bidirectional diffusion transformer that predicts the next patch. The authors observe that providing local context through patching was determinant to improve their model, which corroborates what we observe with the introduction of short context transformer into our model. In Pasini et al. 2024b), the authors apply the MAR framework to music generation using relatively small dataset comprising 20,000 single-instrument stems, training their model on 10-second excerpts on top of their continuous compression model Music2Latent (Pasini et al.) 2024a). They introduce a method for noise augmentation of the data that allows the model to avoid error accumulation. How- ever, notice that scaling makes their approach struggle to maintain high-quality generation on longer sequences. To address these challenges, propose novel strategies that enhance generation quality while also improving in-
complex and diverse dataset consisting of full musical pieces
3

---

Published as a conference paper at ICLR 2026
ference efficiency. Finally, Music2Latent2 (Pasini et al. 2025) explores combining autoregressive and consistency modeling but for compression. IMPACT (Huang et al. 2025) explores MAR decoding for text-to-audio generation and sets new standard for short latency models on the AudioCaps (Kim et al. 2019) benchmark. More recently, MingUni-Audio (Yan et al., 2025) shows that continuous speech language models can scale 20B Mixture of Experts models with 3B active parameters.
Some other autoregressive speech models work in the continuous domain thanks to spectral rep- resentations such MELLE (Meng al. 2024) for TTS and Flow-Omni (Yuan et al. 2024 for speech-to-speech conversation.
BACKGROUND
Let W f be a monophonic waveform of seconds sampled frame rate fs- Our goal is to model W either in the discrete latent space of a RVQ-based codec or in the latent space of VAE. Let fr be the frame rate of the codec or VAE.
Notations:
In the case of the discrete modeling, W is represented by the sequence of discrete tokens (g* where & E (1.....5)) 5} indexes time and E (1..... K} indexes codebook depth and S = fr Each token qt E {1, Nk is drawn from a finite vocabulary.
In the case of continuous modeling. W is represented by sequence (x' ...,,,,'') and x E RC where C' is the latent dimension of the VAE.
x>) with S fr d
3.1 AUTOREGRESSIVE MODELING WITH RESIDUAL VECTOR QUANTIZATION BASED CODECS.
Autoregressive modeling of discrete tokens from RVQ-based codecs (Zeghidour et al. 202 I
Défossez et al. 2024a) is a prevalent method for figh-fidelity audio generation (Copet et al. 2023
Agostinelli al. 2023 Borsos et al., 2023 Kreuk et al., (023). Given a sequence of discrete tokens
(q"x) "E KEE early models like Borsos et al. (003); Agostinelli et al. (2023) flattened
the multi-level sequence, increasing its length by factor K and resulting high computational
costs due to the quadratic complexity of Transformer self-attention. MusicGen (Copet et al.) (023)
mitigates this by using delay pattern independently sample each of the K RVQ levels, adding only K 1 tokens to the sequence but introducing a fixed latency of K - 1 frames, which is problematic for real-time applications. RQ- Transformer (Lee al., 2022) addresses this by using a sampler transformer head that models the RVQ at a given time step, enabling low-latency generation.
Denoting q (g' "q."""...,,"".......... )". the Backbone Transformer Te encodes the history of previous timesteps to produce context vector z", and a RQ- (Q-Transformer Je autoregressively decodes the residual-wis components of the token stack at each time step. The context vector is then given by z" = To(q .a.......!!! ") and the logits ga. for predicting the k-th codebook token are computed k1. 1) ""..
as gs = Lin(z") and ga.k I+ (z',q*)
for k > 1.
These logits are trained using cross-entropy loss over discrete tokens - Σ log (q** -e=0 " <k<))..
LcE
This approach enables efficient parallel modeling of RVQ sequences, allowing all codebooks corresponding to single timestep to be generated simultaneously without introducing any delay. However, a key limitation lies in the of the RQ- Rr-Transformer. which is computationally intensive; as its resource requirements scale with the number RVQ, or even with the square the number of RVQ. if the attention dominates the computation cost.
3.2 CONSISTENCY MODELS
Flow Matching and Probability Flow ODE. Let Pdata be data distribution over Rd. Given Xy ~ Pdata diffusion (Ho et al.) 2020; Song et al. 2021) and flow matching (Lipman et al. 2023) models define forward noising process that gradually perturbs samples from Pdata through the noising process x, 0xX0 + oie, where e ~ (0, and t E [0. a, o, are pre- defined functions such that is decreasing with og 1,0r and is increasing with
4

---

Published as a conference paper at ICLR 2026
00 0, OT 1. In Flow Matching, neural network F is trained to minimize the loss (FM(O) Ex Paam (0,D), (~20(0,1) (w(() [x (a/xa + ofe))l
Once trained, sample generation is performed by solving a deterministic ordinary differential equation known the probability flow ODE (PF-ODE), which defines a continuous path from noise to data. In the context of Flow Matching, the PF-ODE is dx. (x () with XT ~ (O(0T) I).
Continuous- Consistency Models. (Song et al. 2023 is neural network fø( 1) trained to map noisy input x, directly to the corresponding clean data Xo in single step, by approximating the sampling trajectory of the probability flow ODE (PF-ODE) starting from x,. To ensure correct behavior, fe must satisfy the boundary condition fx 0) x. thus leading to the common parameterization fx(x,, coefficients satisfy Caip (0) 1 and Cour (0) to fulfiil the boundary condition.
(t)x, + Cour (t) (X,, t), where F& is a neural network and the
By using = 2 and a = cos(t), oz sin(t). Lu & Song (2025) derive the following continuous-time consistency loss where was (t) is adaptive weighting function:
(e (1)
df (xr, (t)
LcM(0, x) Ex.,t
Fà (xx,t) - FØ- (xx, t) cos(t)
(1)
D
dt
Lagrangian Self-Distillation. We also explore a new 1-step flow-matching method named Latent Self-Distillation (LSD) that has been introduced in [Boff et al. (2025). In this paper, the authors unify most 1-step methods into 2 categories and develops third one (LSD) which appears to be more stable at training. See Sec. A the equations.
3.3 AUTOREGRESSIVE MODELING OF CONTINUOUS LATENTS VIA DIFFUSION.
Li et al. 2024 propose MAR, a method for autoregressive modeling over a sequence (x!:.... x x5) of continuous latent vectors extracted from pretrained VAE, thereby eliminating the need for discrete quantization. As the discrete case, a Backbone Transformer Te maps the context to an embedding: z' T xx
Then, a diffusion process parameterized by a neural network eg is trained on each x" with the loss Lo Φ) = Σ Be-NN0,,), ||e (xix,, [t]]]] where x noisy version of x* at diffusion timestep t: x a,x* + o,E with e (O, /) and a, and Ø, are predefined schedules for all t E [0. 1]. In practice, an MLP significantly smaller than the backbone transformer estimates Eg. This replaces the categorical prediction used discrete models (done by the (Q-Tnssffrrmer) with denoising task in the continuous domain (done by the MLP). This method enables flexible and differentiable modeling continuous signals without requiring to perform quantization on the latent space which can lead to several issues such as codebook collapse, balancing quantization losses and training instabilities. A key limitation of this approach is that sample quality depends on the number of diffusion steps at inference, raising the question of whether it can surpass the RQ- Transffforrmr under similar computational constraints.
4 METHOD
4.1 OUR VAE-GAN
Most autoregressive audio models are built upon RVQ-GAN architectures (Zeghidour et al. 202 1 Défossez et al., 2024a| Kumar et al. 2023 Guo et al. 2025). Following the approach of Evans al. (2024), we instead adopt VAE-GAN framework, replacing the RVQ bottleneck with VAE bottleneck to regularize the latent space and enforce a Gaussian prior. Our VAE is fully causal and draws from the architecture of Mimi (Défossez al. 2024b). ) using Transformers in addition to convolutions in the encoder and decoder, which have been shown to improve performance.
While training the model with adversarial losses and VAE regularization without any reconstruction losses improves the quality of the model for speech, it degrades the reconstruction quality for music. Semantic distillation is performed for the speech VAE similarly as in Mimi, with WayLM (Chen
5

---

Published as a conference paper at ICLR 2026
al. 2021b) as teacher. There is no semantic distillation for the music model and we let this for future work as semantic content is harder to define for music. The loss is:
CVAE (x, X, R) + Lad
Cta + AKL CKL + Adisun
(2)
where & and L the temporal and frequential reconstruction losses, Lady is the adversarial loss, freat is the feature matching loss, fKL is the KI. regularization applied to the VAE bottleneck, and Laisan is the WavL.M distillation loss applied for the speech VAE.
4.2 CONTINUOUS AUDIO LANGUAGE MODEL (CALM) ARCHITECTURE
Let (x',.... x5) denote the sequence of continuous latent vectors produced by VAE encoder. As illustrated in Fig- [ our model comprises three main components. The motivations behind these design choices are described in Sec. 4.3 and the ablation study in Tab, 6 justiiiss these choices.
1. Causal Backbone Transformer with Noise Injection We build the MAR framework (Li et al. 2024) by employing a causal Transformer Tong.01 to capture long- teg-term dependencies. However, during preliminary experiments, we realized that music generation models with the MAR framework were generating poor quality audio and diverging quickly at inference because they were not robust to error accumulation, Pasini al. (2024b) introduced noise augmentation trick training time in order to tackle this problem. Given sequence (x x² x x)), they sample k; ~ U(0, 1) and E, (0, I) for every s E (I...., .....,)) S} and use a noised imput to the backbone x kgE +(-kk, )x" for every Early experiments showed that preserving the variance of x improved quality, so that we use instead X = Vk,c, cs + /1 kyx". We don't perform any noise augmentation at inference time. Thus, Z'ong Tong. 8a (x inference, but as shown in Tab. insufficient alone for high-quality music generation.
x² Noise injection prevents error accumulation during
2. Short-Contex, Transformer To supply local, high-resolution context to the denoising head, we introduce lightweight causal Transformer that attends the ! previous clean latents (we use K = 10, 0.4s of music): zahort = To (x*-K x This short-context embedding zeoo supplies fine-grand fine-grained information potentially lost through noise injection in the backbone. We show in Sec. E.3 that the value of K is not decisive hyperparameter but Tab. T indicaaes that the
short-context transformer is crucial to good quality generation.
3. Consistency-Model Head the sum of the long-term and short-term features, Z Zinng + sbont At inference time, for 1-step generation, the next latent X is sampled through: (O, 1 fx c, = 1,
Finally, small MLP-based consistency model fg is conditioned
In addition to consistency, we experiment with the TrigFlow (Lu & Song 2025> formulation of flow-matching for the MI.P Although Trig yields marginally higher fidelity, its inference cost makes it impractical for real-time use. While Tab. shows it, this tradeoff is studied Sec. F.4
Together, these three components form continuous autoregressive model that (i) leverages noise-robust long-term modeling, (ii) preserves local detail via short-context conditioning, and (iii) achieves rapid, high-fidelity latent sampling through consistency modeling.
The training objective for one sequence (x!...., x5 is defined by:
9
(e))(()
dfa(xi,t, Z²)
6c
Σ
Bis
Fo (xx,,, (xi,t,z") cos(t)
w. (t)
D
dt
(3)
where Z Znong + Shoo T Thong (xc x011) + T T2
(x*-K 7
x²11),
t ~ [0, 1) (0, I) and x; cos(t)x* + sin(t)e. All the parameters (6, d. e)) of the transformer backbone ot. the short-context transformer Tshon,0. sa, the consistency MLP fo and the adaptive weighting function we are jointly trained together with this consistency loss similarly as the backbone and the RO-Transfomme are trained through cross-entropy loss in the discrete case.

---

Published as a conference paper at ICLR 2026
4.3 COMBINING NOISY LONG-TERM CONTEXT AND CLEAN SHORT-TERM CONTEXT
In preliminary experiments, music generation models trained with the MAR framework were diverging quickly during inference because they were not robust to error accumulations. Applying noise injection during training slightly improves model stability but often reduces detail and instrument diversity, typically preserving only rhythmic elements, with audio fading into silence after 10-15 seconds. Since Pasini et al. (2024b) targets short, single-instrument clips, it's unsurprising the method performs best that constrained setting. We hypothesize that the added noise inhibits the backbone transformer from encoding fine-grained information into Ziong limiting the MLP's s ability to reconstruct detailed audio. However, combining this with short-context transformer computing shon yields the best results (Tab. ( likely because the clean short-tern context restores local detail needed to model the distribution of the next x*.
4.4 HEAD BATCH MULTIPLIER
Training is bottlenecked by the cost of generating the conditioning variable Ziong via the large causal transformer. To address this, we introduce the Head Batch Multiplier, which amortizes this cost by reusing zigny multiple times per training step. Specifically, for each input sequence, we compute Bongg once and use it across N loss computations, each with independently sampled noise levels t and e. This improves efficiency and stabilizes training by averaging the loss multiple samples. Tab. 6 and Fig. 3 show faster convergence and better final performance at comparable training cost.
4.5 GAUSSIAN TEMPERATURE SAMPLING
Sampling strategies, such as temperature sampling, have a significant impact on generation quality in the discrete setup, particularly for speech. To replicate this behavior in the continuous domain, we introduce a sampling heuristic that results in comparable gains. Similarly to the GAN noise truncation trick presented in [Brock et al. (2018), we sample more from the high probability zone of the Gaussian to trade diversity for fidelity.
While the GAN truncation trick truncates the Gaussian noise such that values outside of a certain range are redrawn, we chose to reduce the variance of the Gaussian noise instead. This is mathematically equivalent to applying temperature 7 to the Gaussian if we change the standard deviation vT. This makes temperature values between the discrete and continuous setups somewhat comparable, and we found that using a temperature of .8 for speech continuation was bringing good results in both setups. The effects of gaussian temperature are further discussed in Section
4.6 LATENT CLASSIFIER FREE GUIDANCE
Classifier Free Guidance (CFG) (Ho & Salimans 2022 is known to improve the generation quality of conditioned generative models. can be applied for diffusion and flow matching models on the sampling trajectory as well as on the logits of autoregressive language models (Kreuk et al. 2023). Since CFG cannot be applied on the trajectory of 1-step consistency models we decide to apply the CFG on the outputs of the Backbone and Short-Context transformers. Formally, given C a conditioning and a the CFG coefficient, we compute for every 8 of the sequence Z Zg + aZ Zz) and then we generate x* with the consistency head conditioned on Z We call this method Latent CFG, as it operates on the latent variable Z instead of the model output. It has been introduced in the vidco-to-audio model SoundReactor (Saito et al. 2025).
4.7 LATENT DISTILLATION
Once teacher model has been trained and the desired classifier free guidance (CFG) coefficient has been selected for inference, we distill the CFG-guided teacher into a student model to avoid the need to double the batch size during inference when using CFG. To this end, we distill only the backbone transformer and directly copy the teacher's 's MLP head into the student.
The distillation objective for the student backbone is & loss between the latent representation Z produced by the student backbone and the CFG-guided latent representation ZCFG produced by the teacher. Additionally, the student backbone transformer may contain fewer layers than the teacher.
7

---

Published as a conference paper at ICLR 2026
In practice for Pocket TTS, distill text-to-speech model with 24 transformer layers into student model with only 6 layers, using a latent CFG coefficient of a = 1.5. See Sec. | more details.
5 EXPERIMENTS AND RESULTS
5.1 SPEECH CONTINUATION
VAE: Our VAE is based on Mimi (Défossez et al. 2024b) but enforces gaussian inner latents instead of a categorical distribution. Like in Mimi, to enforce semanticity of the representations, we distill WavLM into the inner latent representation with a cosine similarity loss. Unlike Mimi, which applies this loss only the first codebook, we extend it to the entire latent representation.
Table 1: Speech compression models. Our VAE is on par with the VQ-VAE VAE on acoustic quality (MOSNeT)) and outperforms it semantic discriminability (ABX), PESQ Rix al. 2001) and STOI Taal et al. (01T)) and a MUSHRA for acoustic quality.
TYPe
DIMSS/ RVQ
FRAME RATE (l2) BITRATE (CBF1)))
MOSNEY (t)
ARX
PESQ
STOI (t)
ACOUSTIC QUAL.(T)
VQ. VAE (MIN)
8 RVQ
12.5
1.bxaes
3.11
9.4%
2.13
0,87
$7.7 + 1.3
VAE
32 2044
12.5
3
8.19
2.42
0.90
66.8 = 1.4
pretrained 2B parameters multilingual text LM as backbone, we train on French and English speech data following Défossez et al. (2024b) to learn continuation. To enhance the stability and coherence of speech continuation, concept of inner monologue (Défossez et al., 2024b) latent textual representation of the model's own speech, aligned such that each word is positioned the timestep corresponding to its spoken occurrence. This implies that, at cach timestep the backbone transformer takes both text tokens and speech latents as input, and that its output zong is passed through a linear layer which produces text logits alongside conditioning the consistency head. This internal text stream acts semantic scaffold, as it represents the next word to be pronounced, guiding the generation of audio tokens by grounding them in linguistic form. Crucially, like in Défosssz et al. (2024b), we introduce temporal delay of 2 time steps (160ms) between the inner monologue and the corresponding audio tokens. This delay allows the model access textual content prior to generating acoustic latents, decoupling high-level planning from low-level synthesis. For speech generation, we didn't 't notice any gains from introducing a short context transformer and noising the latents before feeding them to the backbone, resulting in simpler model architecture.
Model and dataset:
Starting from Helium-1 (Kyutai 2025
adopt the
Table 2: Comparison of speech continuation models: 8-RVQ RO-transformer 1-step Consistency model head, with 2 temperature options.
Model Type
Sampling temperature
Overall Sperdup (?)
Sumpler Speetup (*)
Time Sampler (1)
(1)
VERT
Asouttic Quality(1)
Moaninguulmes Rank (I)
(?)
Reference
20.2
25.2
4.012 s
2180
RQ-trnsffmmerr RVQ RÕ tansfomerrss RVQ
1.0
x1.0
x1.0
26.7%
52.5
36.5
2.42 1 0.12
1841 25
0.8
x1.0
x1.0
26.7%
21.88
33. 1
2.75 a 0.14
1870 301
3
CALM Cunsistency
x1.3
x12.3
2.9%
31.3
2.82 0.10
1947
CALM Consistency-
x1.3
2.%%
23.8
8.45=0.14
2023
Results: Tab. shows that our 32-dimensional VAE matches an 8-RVQ Mimi codec on MOSNet
(Lo et al. 2019), which measures audio quality, and exceeds it on the ABX metric (Schatz et al., 2013). ABX evaluates phonetic discriminability by testing whether word like "bat" represented closer to another "bat" utterance than to similar-soundingg word like "bit", , based on latent distances. Tab. 2 shows that the 1-step Consistency model outperforms the RQ-Transformer with 8 RVQ on all our automatic and human based metrics as well as speed. For automatic metrics, we compute PPX and VERT as introduced by Lakhotia et al. (2021). The PPX metric measures the semantic meaningfulness of the generated speech. To do so generate 1000 excerpts of speech of 30 second, we use Whisper (Radford et al. 2022) to compute textual transcriptions and finally compute the negative log- likelihood of the text tokens with Mistral 7B LLM Jiang et al. (2023) and convert to Perplexity. Because a model that generates poorly diverse but good quality sentences would perform well on the PPX metric, the authors of (Lakhotia al. 2021) introduce the VERT metric
8

---

Published as a conference paper at ICLR 2026
(for diVERsiTy) which is geometric mean of self- and auto-BLEU metries. We use the official implementation from the fairseq (otnse et al. (2019) repository.
To assess perceptual quality, conduct two human evaluation studies involving 50 participants and 50 randomly selected examples from the English test set. Each participant rates 10 examples across the following evaluation protocols: For Acoustic Quality, participants are presented with all model continuations for the same prompt, including the ground truth reference, and rate the acoustic quality of each continuation on to 5 scale. For Meaningfulness, participants are shown two continuations of the same prompt and select the one that is the most meaningful. These pairwise preferences are used compute an Elo score (see Sec. HD.
Notably, we note a clear quality and meaningfulness improvement with our temperature method. Given that there is text stream to guide the audio generation, we expected the CALM to match the baseline on meaningfulness rather than outperforming it. This phenomenon could be due to less model capacity the backbone being allocated to audio manipulation, allowing more for text prediction. On the inference speed side, the consistency head is x x12.3 faster than the RQ-Transformer. The overall gain to perform full inference 30 seconds is x1.3.
Temperature Sampling and speaker similarity: In Sec. we show that our gaussian temperature sampling heuristic has similar effects on speaker similarity than the temperature sampling of the discrete model.
5.2 TEXT-TO-SPEECH (TTS)
VAE, model and dataset: We use the same VAE for speech continuation. Our TTS CALM builds on 300M backbone transformer and uses the same architecture for the 10M parameters consistency sampling head. The text fed to the backbone as prefix with SentencePiece model (Kudo & Richardson 2018) with vocabulary size of 4k. The training data is a mix of public datasets totalizing to 88k hours of speech that is detailed in Sec.
Results: We evaluate on the Librispeech test-clean set using the same protocol as F5-TTS (Chen et al. 2025). We compare against four baselines: F5-TTS (Chen et al. 2025), DSM (Zeghidour et al. 2025), DiTAR (Jia et al., 2025),, and SALAD (Turetzky et al. 2024). For DiTAR and SALAD,
report the paper results since the models are closed-souree We report Word Error Rate (WER) and Character Error Rate (CER) using Whisper-large-
(Radford et al. 2022), Speaker Similarity using M- (Chen et al. 2021b) as well as the results of a MUSHRA test for acoustic quality and a pairwise audio test for speaker similarity. By doing distillation the backbone, we obtain Pocket TTS, a 100M parameters model that can run faster than real-time on a CPU (see Sec
Table 3: Text-to-Speech models. Our CALM model with 1-step LSD outperforms baselines on WER, CER and Acoustic Quality. Results for Pocket TTS are in Sec. G
MODEL
NUM, PARAMETERS
WER
CER
SIM (P)
ACOUSTIC QUALITY (f)
SPEAKER SIM HUMAN ELO (f)
(5)
(1)
RESERENCE REPERENCE (WITH VAE)
2:23
0.69
61.8 =
1953 +
0.57
TIS (NPE-32)
336M
2.42
0.66
54.7
+
DSM (16 RVQ CPGS3
750M
1.95
0.67
60.2 4: 2.4
2112 +: 20
W.R.T TEXT AND AUDID PROMPT) ZECUIDOUR IT AL.
2025
DITAR (NFE-10) (TA AL 2021
600M
2.39
0.67
SALAD (NFE=20)
2024
350M
0.74
0.54
CALM w/ LSD (NFE=1, CFG+1.5 WUR.T TEXT)
313M
1.81
0.57
0.32
61.1 #: 2.3
1966 +: 23
Our CALM model with 1-step LSD (Boff et al., 2025) outperforms baselines on WER, CER and Acoustic Quality but obtains a low speaker similarity score. This can be partially explained by the fact that when computing the speaker similarity between the reference prompt and the reference utterance that goes through the VAE (the second line of the tab) we obtain a similarity of 0.57. Yet. we demonstrate in Tab. that we decide to measure speaker similarity with a human study. We observe that all measured methods beat the ground truth, which means that they preserve well the voice of the audio prompt.
VAE faithfully reconstructs speech. Due this surprising behavior,

---

Published as a conference paper at ICLR 2026
5.3 MUSIC CONTINUATION
In this section, we detail our Music Continuation model. Since our dataset does not have any textual information, we use CLAP (Elizalde al. 2023) to train a text-to-music model (see Sec.
Dataset: We use randomly selected subset of 400K songs (approximately 20K hours with 32kHz mono format) from the LAION-Dison 12M dataset, ensuring broad coverage across musical genres.
VAE: Our variational autoencoder (VAE) and codec architecture is adapted from the Mimi codec (Défossez et al. 2024b), originally designed for 24kHz speech at 12.5Hz. We trained it to compress 32kHz mono music with 25Hz frame rate. Details and metrics described in Sec.
Table 4: Comparison of music continuation for 30 seconds generation. Consistency-based models provide up to 2.2x overall speedup and 19.3x sampler head speedup compared to the RQ-
Transformer 32 RVQ baseline, with improved FAD scores and equivalent human ratings. TrigFlow
achieves the best qualitative results but has significantly higher inference time. Since MusicGen only
uses linear layer to sample its token consider its inference cost negligible.
MODEL
OVERALL SPREDUP (T)
SAMPLER SPEEDUP (1)
& TIME IN
FAD (
ACOUSTIC QUALITY (*)
ENJOYMENT
SAMPLER (1)
ELO
RANK (L)
REFERENCE
3,84 1 0.08
2166 1
RO-
RVQ (BASELINE)
1.06
RQ-TRANSPORMER RVQ
X 2.2.2
38 38.0%
1.43 1: 0.07
2.76 0.07
CAL CONSISTINCY- STE
x 2.2
19.3
6.6%
0.83 0.04
2.90 0:07
1857 + -8
CALM CONSISTENCY 44 STEPS
X 1.9
X.4
20.1%
0.71 1 0.05
3.07 1 0.07
18:17 1 21
CALM TRIGFLOW- 100 STEPS
x 0.33
0.2
86.6%
0.64 0.04
3.12 1. 0.07
1921 1
MUSICGIEN MEDIUM
Model: Our music CALM builds on the MusicGen Medium backbone, 1.355 parameter
Transformer (see Sec. [ for all the hyperparameters). We compare against: (1) 32- and 16-RVQ discrete models utilizing the same 1.35B backbone and an RQ-transformer for parallel prediction: (2) MusicGen Medium variant using EnCodec and delay-pattern interleaving. All baselines were trained on our dataset.
Results: In Tab. 4 we report both objective metrics and the results of human evaluation study for the task of music continuation, conditioned on a 3-second prompt. We compute the speed-up compared to the RO-Transformer 32 RVQ, the VGG Fréchet Audio Distance (FAD) on 4,000 model-generated continuations from the test set. The Acoustic Quality is a MOS score between and 5. The Enjoyment metric Elo score (see Sec. H. computed by making human raters choose their favorite music out of generated pairs with the same 38 prompt. We observe that CALM with consistency outperforms the 32 RVQ RQ-Transformer baseline computed and human metrics while being x x1.9 to x2.2 times faster for the overall speedup. While the RQ- Transfoooreer takes 57.7% of the inference time for the baseline, the consistency head only takes 6.6% to 20.1%. As well, we train a CALM model with TrigFlow head instead of consistency and it outperforms all the models but the price of slow inference.
Ablation Study: In Sec. E. we show the importance of each architectural component.
Necessity of Consistency for fast inference: Sec. E.4 shows that consistency models largely outperforms TrigFlow under 10 inference steps (the regime where RTF < 1).
Scalability: We show in Sec, E.S that CALM does improve with bigger 3B parameters backbone. However, we leave complete scalability study for future work.
CONCLUSION
We present Continuous Audio Language Models (CALM), a novel framework for autoregressive audio generation that operates directly in the continuous latent space of VAE, bypassing the limitations of discrete quantization. Replacing RVQ or diffusion heads with consistency models significantly reduces inference cost while improving sample quality as shown by our experiments. Our architecture combines noise-injected long-te term context and clean short-term context, implemented via tral-trrasformmr design. We introduce practical innovations to further improve sampling quality and training efficiency. We demonstrate the effectiveness of our approach across both speech and
10

---

Published as a conference paper at ICLR 2026
music generation tasks. Our results suggest that continuous modeling offers compelling alternative to discrete tokenization for high- qualitu efficient, and scalable autoregressive audio generation.
REFERENCES
Andrea Agostinelli, Timo I. Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, Matt Sharifi, Neil Zeghidour. and Christian Frank. Musiclm: Generating music from text, 2023. URL https: /taps:://rxiv
htgg/abs/an abc/ 11325
Robin Algayres, Yossi Adi, Tu Anh Nguyen, Jade Copet, Gabriel Synnaeve, Benoit Sagot, and Emmanuel Dupoux. Generative spoken language model based on continuous word-sized audio tokens. In E
Placeholder Author and Various Contributors. The emilia dataset: High-quality expressive speech corpus for neural tts. Dataset Release Documentation, 2023. Replace with accurate reference you have one.
Emily Beck, Bowen Zhang, Morgane Riviere, Daniel PW Ellis, Arun Babu, Tatiana Likhomanenko, et al. Libriheavy: A 50,000-hour english speech corpus for fully supervised ast. arXiv preprint 1rXiv:2405.08985, 2024.
Nicholas M. Boffi, Michael S. Albergo, and Eric Vanden- Eijnden., How to build a consistency model: Learning flow maps via self-distillation, 2025. URL https::/arxivv.or abs/
2505. 18825
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghi- dour. Audiolm: language modeling approach to audio generation, 2023. URL https
/ /arxivv.. wwtp/ogg///b..oooooooo..oo...eet 2209. 03143
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: The method of paired comparisons. Biometrika, 39(3-4):3243455, 12 1952. ISSN 0006-3444. doi: 10.1093/ biomet/39.3-44 -..3234999.4.44....
324
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. CoRR, abs/1809.11096, 2018. URL httpp://arxiv.org//aaa.
1809. 11096]
Jean Carletta. Unleashing the killer corpus: Experiences creating the multi-everything ami meeting corpus. Language Resources and Evaluation, 41: 41:181-190, 2007.
Francois Caron and Arnaud Doucet. Efficient bayesian inference for generalized bradley-terry models, 2010. UtL/attpss//axii...z////////1111
Guoguo Chen, Songxiang Chai, Gengsheng Wang, Jun Du, Wenchao Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Jie Zhang, et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv:2106.0668
2021a.
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, and Furu Wei. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. CoRR, abs/21 10. 13900, 2021b. URL htttp:://arxii......m/ http///rrwwwooo//s/////// 2110.13900
Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, and Xie Chen. F5-tts: fairytaler that fakes fluent and faithful speech with flow matching, 2025. URL
https://arxiv org/ /tw//////////////////////////l// 2410. 06885
Michael Chinen, Felicia S. C. Lim, Jan Skoglund, Nikita Gureev, Feargus O' O'Gomman, and Andrew Hines. Visqol v3: An open source production ready objective speech and audio metric, 2020. URL https://arxis org/ abs 2004 09584

---

Published as a conference paper at ICLR 2026
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. In Neurips, 2023.
Miguel Del Rio, Peter Ha, Quinten McNamara, Christopher Miller, and Saurabh Chandra. Earnings-
22: A practical benchmark for accents in the wild. arXiv preprint arXiv:2203. 155911 2022.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and llya Sutskever.
Jukebox: A generative model for music, 2020. URL
00341
Alexandre Défosse7, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. In ICLR, 2024a.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue, 2024b. URL TtR.https://
orgg abs 241 4140 0. 00037
Benjamin Elizalde, Soham Deshmukh, and Huaming Wang. Natural language supervision for general- purpose audio representations, 2023. URL httpt::tt//aaa.. wtw.vv.vvvv. orq abs /2009.23099... C5766
Zach Evans, CJ Carr, Josiah Taylor, Scott H. Hawley, and Jordi Pons. Fast diming-condition latent audio diffusion. ISMIR, 2024.
Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. In ICLR, 2025.
Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. In ICLR, 2025.
Jiatao Gu, Yuyang Wang. Yizhe Zhang. Qihang Zhang, Dinghuai Zhang, Navdeep Jaitly, Josh Susskind, and Shuangfei Zhai Dart: Denoising autoregressive transformer for scalable text-to- image generation, 2025.
Yiwei Guo, Zhihan Li, Hankun Wang, Bohan Li, Chongtian Shao, Hanglei Zhang, Chenpeng Du, Xie Chen, Shujie Liu, and Kai Yu. Recent advances in discrete speech tokens: A review, 2025.
URL /tttps://arxiv.
abs/2502 06490
Tiankai Hang, Jianmin Bao, Fangyun Wei, and Dong Chen. Fast autoregrressive models for continuous abs
latent generation, 2025. WRRL:ht:s///axiiiti wttp://.rggrbbb//////b//
François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Estève. Ted-lium 3: Twice as much data and corpus repartition for experiments on speaker adaptation. In Speech and Computer (SPECOM 2018). pp. 198 208. Springer, 2018.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. URL hotttt/ss////ttt
org/ 2207.22222 12598
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffuuson probabilistic models. In NeurIPS. 2020.
Kuan-Po Huang, Shu wen Yang, Huy Phan, Bo-Ru Lu, Byeonggeun Kim, Sashank Macha, Qingming Tang, Shalini Ghosh, Hung yi Lee, Chieh-Chi Kao, and Chao Wang. Impact: Iterative mask-based parallel decoding for text-to-audio generation with diffusion modeling, 2025. URL https /arxiv.erg/abs//2.
00738
Qingqing Huang, Daniel S. Park. Tao Wang. Timo Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang. Zhishuai Zhang, Jiahui Yu, Christian Frank, Jesse Engel, Quoe V. Le, William Chan, Zhifeng Chen, and Wei Han. Noise2musie: Text-conditioned music generation with diffusion models, 2023. URL
Dongya Jia, Zhuo Chen, Jiawei Chen, Chenpeng Du Jian Wu, Jian Cong, Xiaobin Zhuang, Chumin Li, Zhen Wei, Yuping Wang, and Yuxuan Wang. Ditar: Diffusion transformer autoregressive modeling for speech generation, 2025. URL https/////.rr...orr///////222.. 03930
12

---

Published as a conference paper at ICLR 2026
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud. Marie-Ame Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William RI Sayed. Mistral 7b, 2023. URL https: //axxiv
org/abs/23Dn /oobb//2310.. 06825
Eugene Kharitonov, Damien Vincent, Zalán Borsos, Raphaël Marinier, Sertan Girgin, Olivier Pietquin, Matt Sharifi, Marco Tagliasacchi, and Neil Zeghidour. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision. Transactions of the Association for Computational Linguistics, 11:1703-1718 2023.
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In NAACL-HLT, 2019.
Qiuqiang Kong, Yin Cao, Turab lqbal, Yuxuan Wang. Wenwu Wang, and Mark D. Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. In Transactions on Audio, Speech, and Language Processing, 2019.
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation, 2023. URL
attps::/aariis http///awvvv.oraas//////////////
abs/ 2209.1535
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing, 2018. URL
/bs/
1308806228
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvqgan. In NeurIPS, 2023.
Kyutai. Helium 1: modular and multilingual 1lm. httpss//syttal.orr//20/////// helium. html] april 2025, Blog post.
Tom Labiausse, Laurent Mazaré, Edouard Grave, Patrick Pérez, Alexandre Défossez, and Neil Zeghidour. High-fidelity simultaneous speech-to-speech translation. In ICML, 2025.
Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Adelrahman Mohamed, and Emmanuel Dupoux. Generative spoken language modeling from raw audio. In TACL, 2021.
Doyup Lee, Chihcon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han, Autoregressive image gen- eration using residual quantization, 2022. URLttt:::///xxi org abs /203 01941
Jcan-Maric Lemercier, Simon Rouard, Jade Copet, Yossi Adi, and Alexandre Défossez. An loss for music generation with language models. In ICML, 2024.
independence-promenn
Peike Li, Boyu Chen, Yao Yao. Yikai Wang, Allen Wang, and Alex Wang. Jen-l: Text-guided universal music generation with omnidirectional diffusion models, 2025. WRRLhttps://arx.
erg/abs/230s 2308a 04729
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. In Neurips, 2024.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. In ICLR, 2023.
Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D. Plumbley. Audioldm 2: Learning holistic audio generation with superssde pretraining. 2024. URL https:://// arxia. wrg. abs////bs///a 2ta.b///303. 2308 05734
Chen-Chou Lo. Szu- Wei Fu, Wen-Chin Huang, Xin Wang, Junichi Yamagishi, Yu Tsao, and Hsin-Min Wang. Mosnet: Deep learning- based objective assessment for voice conversion. In Interspeech 2019, interspeech 2019. ISCA, September 2019. doi: 10.21445/iierrppecch.0/// 201. 2003. URL 2019-2003
http://dxx doi www.ddo....oooooooo 2ttp///1..1137//11
13

---

Published as a conference paper at ICLR 2026
Cheng Lu and Yang Song. Simplifying, stabilizing and scaling continuous-time consistency models. In ICLR, 2025.
Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, and Furu Wei. Autoregressive speech synthesis without vector quantization, 2024. http::tts://ecm /ttp://axvvvvv/o.o www.aawwwwww....//aaaaaaaaaaaa wttp/a///gaaaoee 08551
Patrick K O'Neill, Vitaly Lavrukhin, Somshubra Majumdar, Vahid Noroozi, Yan Zhang, Oleksii Kuchaiev, Jagadish Balam, Yury Dovzhenko, Karl Freyberg, Michael D Shulman, et al. Spgispeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognition. arXiv preprint ar 104.02014, 2021.
OpenAl. Gpt-4 technical report, 2024. URL
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli, fairseq: A fast, extensible toolkit for sequence modeling. Proceedings of NAACI-HIIT 20/9: Demonstrations, 2019.
Marco Pasini, Stefan Lattner, and George Fazekas. Music2latent: Consistency autoencoders for latent audio compression. In ISMIR, 2024a.
Marco Pasini, Javier Nistal, Stefan Lattner, and George Fazekas. Continuous autoregressive models with noise augmentation avoid error accumulation. In NeurIPS Audio Imagination Workshop. 2024b.
Marco Pasini, Stefan Lattner, and György Fazekas. Music2latent2: Audio compression with summary embeddings and autoregressive decoding. In ICASSP 2025 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2025.
K R Prajwal, Bowen Shi, Matthew Lee, Apoorv Vyas, Andros Tjandra, Mahi Luthra, Baishan Guo, Huiyu Wang, Triantafyllos Afouras, David Kant, and Wei-Ning Hsu. Musicflow: Cascaded flow matching for text guided music generation. In /CML, 2024.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. URL httph:://aaatt
org/abs/ 04356
Antony W Rix, John G Beerends, Matthew P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq) a new method for speech quality assessment of telephone networks and codecs. IEEE Transactions on Audio, Speech, and Language Processing, 96):749-760, 2001.
Koichi Saito, Julian Tanke, Christian Simon, Masato Ishii, Kazuki Shimada, Zachary Novack, Zhi Zhong, Akio Hayakawa, Takashi Shibuya, and Yuki Mitsufuji. Soundreactor: Frame-level online video-to-audio generation, 2025. URL https:tts/// htt////avvvv/// wttg:bb///bb..rroooe 02110
Thomas Schatz, Vijayaditya Peddinti, Francis Bach, Aren Jansen, Hynek Hermansky, and Emmanuel Dupoux. Evaluating speech features with the minimal-pair abx task: analysis of the classical mfc/plp pipeline. In Interspeech 2013, pp. 1781-1785, 2013. doi: http:///hnerspeccc2200-4-444
Yang Song, Jascha Sohl-Dickstein, Diederík Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. ICML, 2023.
Cees H. Taal, Richard C. Hendriks, Richard Heusdens, and Jesper Jensen. An algorithm for intelligi- bility prediction time frequency weighted noisy speech. IEEE Transactions on Audio, Speech, and Language Processing, 19(7):21252211 2011.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URI ptattp::/arrea
ARI.httpss//arxiiv org/ atprr///ss//3.....//// 13971
14

---

Published as a conference paper at ICLR 2026
Michael Tschannen, Cian Eastwood, and Fabian Mentzer. Givt: Generative infiitt-vocabulayy transformers, 2024. https:
org/ abs/2312.02111
Arnon Turetzky, Nimrod Shabtay, Slava Shechtman, Hagai Aronowitz, David Haws, Ron Hoory, and Avihu Dekel. Continuous speech synthesis using per-token latent diffusion, 2024. URL
https: //axxiv http:///axxvvv....aaaee borg/bbs/413 16048
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio, 2016.
Aaron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning, 2018. U
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, and Lukasz Kaiser and. Attention all you need. In Advances in Neural Information Processing Systems (NeurlPS), pp. 5998-6008, 2017.
Changhan Wang, Morgane Riviere, Anne Lee, Arun Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint arXiv:2101.00390 2021.
Chengyi Wang. Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. Neural codec language models are zero-shot text to speech synthesizers, 2023. URL
/
23C1. 02111
Shu wen Yang, Byeonggeun Kim, Kuan-Po Huang, Qingming Tang. Huy Phan, Bo-Ru Lu, Harsha Sundar, Shalini Ghosh, Hung yi Lee, Chieh-Chi Kao, and Chao Wang. Generative audio language modeling with continuous-valuee tokens and masked next-token prediction. In ICML, 2025.
Canxiang Yan, Chunxiang Jin, Dawei Huang, Haibing Yu, Han Peng, Hui Zhan, Jie Gao, Jing Peng. Jingdong Chen, Jun Zhou, Kaimeng Ren, Ming Yang, Mingxue Yang. Qiang Xu, Qin Zhao, Ruijie Xiong, Shaoxiong Lin, Xuezhi Wang, Yi Yuan, Yifei Wu, Yongjie Lyu, Zhengyu He, Zhihao Qiu, Zhiqiang Fang, and Ziyuan Huang. Ming-uniaudio: Speech I1m for joint understanding, generation and editing with unified representation, 2025. URL http::ttpt:aaatttt http://axxviirro././ abs/
2511. 05516
Donghaa Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00704, 2023.
Ze Yuan, Yanqing Liu, Shujie Liu, and Sheng Zhao. Continuous speech tokens makes Imms robust multi-modality learners, 2024. URL http:/tttt//aaiiiitttta www.wwwwwwoooo/ooooooooero abs/ 22122 04917
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. CoRR, abs/2107.03312, 2021. URL https
wtwp:li..rg/bb///e //aradw.errr/ab//210....333
Neil Zeghidour, Eugene Kharitonov, Manu Orsini, Václav Volhejn, Gabriel de Marmiesse, Edouard Grave, Patrick Pérez, Laurent Mazaré, and Alexandre Défossez. Streaming sequence-to-seequeee learning with delayed streams modeling, 2025. URL https: http//aaxvvv//. or wwwpx//www.g////////2aoa0
08753
15

---

Published as a conference paper at ICLR 2026
A LAGRANGIAN SELF-DISTILLATION
Lagrangian Seff-Distillation (LSD) (Both et al. 2025 extends the consistency- modell framework by introducing an additional time parameter s, enabling the model learn mappings between arbitrary points along the probability flow trajectory rather than only from noisy inputs to clean data. An LSD model a neural network Lo (x, L, 8) that predicts the state of the PF-ODE solution at time a given its state X, at time t: (s,,t) e [0, 1]2, (x,, s, Xg. In particular, with 1, and by sampling (0, 7) we can sample from the data distribution with fx((1, 1 = 1,8 0).
In Boff al. 2025). the authors define Fx 1, 8) as
fo = (s t),,
To train a LSD model from scratch we need to combine flow matching loss as well LSD loss: C = LFM + LLSD. With an adaptive weighting loss w. s) and our notations they derive as:
(0 )b)
Excrppan c-110)))
[e (e-w((ttt)
|| (x,, t,t) (a/xx + ofe)1 + ww (t,
(5)
and
LLSD(Ø, 1/) = cxi, (e ( [|a - F s), w (p((t
(6)
where Xo Panta NN,, I), 1), t).
In practice, we compute the flow matching loss on 75% of batch and the LSD loss on the 25% left.
B OUR MUSIC VAE
Table 5: Music compression models. At least 96 VAE latent dimensions required to outperform the 32-RVQ codec on reconstruction metrics. EnCodec has been retrained on our dataset.
MODRL
RVQ
(Hz)
RITRATE (KUPS)
VIsQOL
SISNR (1)
ENCodsc
VQ-VAE (ISPIRED FROM XIX))
32 RVQ
8.8
3.63
9.61
YAL
32 DIMS
2.23
5.51
YAF YAE
96 DIMS
3.65
9.76
125 DIMS
4.01
10.3
Our variational autoencoder (VAE) and codec architecture is adapted from the Mimi codec (Dérossez) al. 2024b), originally designed for 24kHz speech 12.5Hz. We trained it to compress 32kHz mono music with a 25Hz frame rate. We experiment with bottleneck sizes of 96 and 128 dimensions. For comparison, MusicGen's ''s EnCodec model (Copet et al. 2023) also operates at 32kHz but uses 4-veeel RVQ at 50Hz. In Tab. [ we report reconstruction metrics (audio ViSQOL Chinen et al. (2020) and SISNR), showing that 32-dim VAE matches MusicGien's 's codec, and that at least 96 dimensions are needed for our VAE to match the quality of the 32-level RVQ configuration.
EISQOL,Chinen
C EFFECT OF GAUSSIAN TEMPERATURE SAMPLING
We evaluate how our proposed gaussian temperature sampling method affects acoustic diversity for consistency models, in comparison with temperature sampling on its discrete counterpart. More precisely, we compute WavL.M speaker embeddings (Chen et al. 2021b over 100 unprompted speech generations with different values of temperature, both for an RQ-transformer and CALM. In both cases, the inner monologue text stream is generated with 0.8 temperature to ensure text diversity, We then average the pairwise cosine similarities of these embeddings, as a measure of diversity: the higher the average similarity, the lower speaker diversity in generated audio. The reference speaker similarity number is computed over 100 examples from the ground truth dataset. Fig. 2 shows, as expected, that speaker similarity tends to decrease with temperature, which means that temperature increases diversity with similar trajectory to the discrete models.
16

---

Published as a conference paper at ICLR 2026
Figure 2: Average pair- wise speaker similarity over 100 unprompted 10s generations, 100 10s ex- amples from the ground truth dataset as reference. As expected, for both methods models generate more diverse speakers (i.c. less pairwise speaker sim- ilarity) as temperature in- creases.
Average Speaker Similarity Temperature
Average Speaker Similarity
0.35
Trnntornformrr
0.30
Reference
0.25
0.20
0.15
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Temperature
D DATA USED FOR THE TEXT-TO-SPEECH MODEL
The dataset used to train our TTS models is composed of AMI Carletta (2007) > EARNINGS22
Del Rio et al. (2022), GIGASpeech Chen et al. (2021a), SPGISpeech O Neill et al. (002I). TED-
Hernaneda et al. (2018), VoxPopuli Wang al. (202T), LibriHeavy Beck et al. (2024) and
Emilia Author & Contributors 2003). results into 88k hours of audio.
E SUPPLEMENTARY EXPERIMENTS
E. ABLATION STUDY
Table 6: Ablation study on music CALM Consistency 4-steps model components, af- ter 250K training steps. Removing noise augmentation the short-context transformer leads to significant performance drops. Fi-
MODEL VARIANT
(L)
BASE (CALM CONSISTENCY - STEPS)
0.93
0.06
w/O HEAD BATCH MULTIPLIER
1,32 = 0.09
w/O NOISE AUGMENTATION
1.63 = 0.11
w/o SHORT CONTEXT TRANSFORMER
4.03 = 0.16
w/o ANY OF THE ABOVE
8.38 0.17
nal row approximates the MAR configuration from Li et al. (2024).
An ablation study (Tab. 6 on music CALM Consistency 4 steps shows the importance of each component. The experiments are run for 250K steps, which explains that the base model's FAD is worse than the one reported in Tab. 4 The final row that is the closest to the MAR framework (consistency replacing diffusion) fails to produce high-quality music. In Fig. 3 we show that the FAD decreases much faster over time when we train consistency CALM models with bigger head batch multiplier. All evaluations are done with steps of consistency. We keep the value of 8 for all of our experiments as higher value would lead to out of memory issues at training time.
E.2 HEAD BATCH MULTIPLIER VALUE
3.0
CALM head batch mult.
GAUM head batch mull,
Figure Efffec the head batch multiplier
value. Training model (music consistency CALM) with a higher batch size multiplier fas- tens the convergence for the FAD metric. All the evaluations are done with 4 steps consistency at inference time.
1.5
1.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
Training (days)
17

---

Published as a conference paper at ICLR 2026
E.3 SHORT-CONTEXT TRANSFORMER WINDOW
To study the influence of the context window K of the short-context transformer, we performed an hyperparameter search with K 5, 10, 20, 40 and trained our models for 500K training steps (see Tab. The model is the music continuation CALM and all evaluations are done with 4-steps of consistency. We observe that finding an optimal value of K not critical even though it seems that is too small enough.
Table 7: FAD after 500K training steps for different short-context Transformer contexts K.
in short-context Transformer
FAD
5
0,85 + 0.05
10
0.76 0.04
20
0.73 + 0.04
40
0.78 0.04
E.4 COMPARISON OF TRIGFLOW AND CONSISTENCY ON INFERENCE SPEED AND QUALITY CRITERIA
In Tab. 8 we compare the inference speed for both TrigFlow CALM and Consistency CALM on the task of music continuation on our test set. We see that below the 10 steps regime, only the Consistency model performs well. Moreover, around 10 steps is the limit for streaming applications where the real time factor (RTF) is smaller than 1. This justifies the need of consistency modeling (instead of diffusion or flow matching) for good quality streaming generation.
Table 8: Generation efficiency of TrigFlow CALM and Consistency CALM models. We compute the inference time to generate 30 seconds of audio, the corresponding Real Time Factor (RTF) as well as the FAD metric for different numbers of inference steps. For streaming (RTF<1), only consistency generates good quality audio.
# Steps
Time (s)
RTF
FAD (TrigFlow)
FAD (Consistency)
1
16.7
0.56
0.83 + 0.04
4
20.4
0.68
28.83 + 0.20
0.71 = 0.05
10
27.7
0.92
4.62 + 0.07
0.73 # 0.05
25
46.1
1.54
0.79 + 0.04
0.96 = 0.06
50
76.8
2.56
0.74 + 0.05
1.46 0.06
100
136.4
4.55
0.64 + 0.04
2.05 + 0.07
E.5 SCALABILITY OF CALM
In order to see if all the hyperparameters of our CALM method transfer well to more model parameters. we trained a consistency music CALM model with a larger 3B backhone (with a model dimension of 2048, 32 heads and 48 layers) well discrete (Q-Trsssfrrmer with 32-RVQ model with the same backbone size. For the CALM model, we use 4 consistency steps at inference. Tab, P shows that the FAD are improving with bigger backbone (3B 1.3B) in similar proportions for both discrete-basee
and continuous- based (CALM) models.
Table 9: Scalability of the backbone for CALM and Rraansoffrr methods.
Model
FAD
CALM with 3B backbone
0.62 + 0.05
32 RVQ RQ-Transformer with 3B backbone
0.98 + 0.06
CALM with 1.3B backbone
0.71 + 0.05
32 RVQ Transfforrrr with 1.3B backbone
1.06 + 0.06
18

---

Published as a conference paper at ICLR 2026
We can in Tab. that at this smaller scale (~300M), Lagrangian Seff-Distillation models provide better audio quality than Consistency models. We also conclude that the latent CFG has significant impact over the WER. As well, applying the latent CFG the audio prefix improves the speaker similarity but at the cost of audio quality. All models were run with sampling from a Gaussian with 0.7 temperature (i.e., multiplying the standard Gaussian noise by v0.7).
E.6 TEXT-TO-SPEECH ABLATION STUDY
At the 300M parameter scale, Lagrangian Self-Distillation (LSD) achieves significantly higher audio quality than standard Consistency models. Furthermore, latent CFG has a major positive impact on Word Error Rate (WER), though applying to the audio prefix trades overall audio quality for increased speaker similarity.
Table 10: Ablation study on the Text-to-Speech CALM model (~300M parameters). We evaluate the impact of Lagrangian Self-Distillation (LSD) versus Consistency modeling, as well as the effects of latent CFG applied to text and audio prefixes.
MODea
NUM PARAMETERS
WER
CER
SIM
MUSHRA
XPEAKER SIM
(1)
(()
(1)
[HOMAAN EVAR REO (?)
&D
(NFE=). CFG)
2.39
0.50
0.52
1991
CALM (NFE=1, CPG-l
313M
1,36
0.59
0.54
1995
(NHBel, CPG+1.
1.93
0.62
0.40
2.5
1900
F TEXT-TO-MUSIC GENERATION
We compare CALM on the task of text-to-mun mu-ii generation. Since our dataset do have any text labeling, we use CLAP (Elizalde et al.) 2023) as prefix conditioning and train MusicGen, our RQ-Transtie RQ-Transformer 32 RVQ baseline as well as CALM. During training, we drop the conditioning 20% of the time in order to perform Classifier Free Guidance (Ho & Salimans 2022) at inference. We use CLAP audio mode when training. We use the same training data, VAE/codecs and hyperparameters the models reported in Tab. 4 In Tab. [1] we show the results of our CLAP conditioned model evaluated on our internal test set with the CLAP conditioning being used in audio mode. As well, report the results of our model and several baselines on the text-to-music benchmark the MusicCaps dataset Agostinelli et al. (2023) in Sec
Table 11: Text-to-music generation models on our test set where CLAP is used in audio mode.
MODer.
OVERALL
SAMPLR (1)
ACoUSTIC QUALITY (?)
REJOYMENT
SPEEDUP
SAMPLER
ELO
RISTERINCE:
3.75 0.15
2104
RQ- TRANSFORMER 32 RYQ (CFG=3) CALM CONSISTINCY Stups (CFG-2)
x
x 1.0
57.7%
0.93
0.07
3.16 0.15
1998 24
x 8 5.4
20.1%
0.91
0.08
3.11
0.14
1998 24
MUSICGEN MEDIUM
x 1,3
0.05
1.93
0.14
2.54 = 0.14
1946 : 25
We report the results on the MusicCaps dataset Agostinelli et al. (2023) and report as well the results of MusicLM (Agostinelli al. 2023), the original cext-to-must MusieGen Medium (Copet et al.
2023). AudioLDM2 (Liu et al., 2024), Muise2Music (Huang et al. 2023), Jen-1 (Li et al. (025) as well as MusicFlow (Prajwal al.J 2024) on the FAD, the KL Divergence (KLD) metric based on the pre-trained audio model PANN (Kong et al. 2019) as well the CLAP cosine similarity that measures the matching between the generated music and text description. The results of Tab. 12 show that even though our goal is not specifically build the best text-to-music model, simply applying CALM on our dataset with a CLAP conditioning leads to competitive results. The metrics from the models that we did not train are reported from the MusicFlow (Prajwal al. 2024 paper.
G POCKET TTS
Given 313M parameters text-to-speedt speechh CALM model that has 24 layers backbone transformer (the teacher), we do latent distillation to 6 layers transformer backbone (the student) and CFG coefficient of 1.5 applied to the teacher. We keep the same MI.P sampling head. The final model,
19

---

Published as a conference paper at ICLR 2026
Table 12: Text-to-music results the MusicCaps dataset.
MODEL
FAD (
KLD (1)
CLAP
REFERENCE
0.30
MUSICLM (AGOSTINELLIET
2023
4.00
MUSICGEN MEDIUM (COPET
2023
3.40
1.23
0.37
AUDIOLDM2 LIU AL. 2024
3.13
1.20
0.43
NOISE2MUSIC HHANG ET Al..
2023
2.10
JEN-1 (LI ET AL., 2025
2.00
1.29
MUSICFLOW (UNDIRECTIONAL LM + FM) (PRAIWAL AL.
2024
2.69
1.23
0.52
MUSICGEN MEDIUM (RETRAINED WITH CLAP)
2.70
1.37
0.39
RO-TRANSFORMER 32 RVQ
2.56
1.35
0.43
CALM CONSISTENCY 4 STEPS
2.14
1.30
0.44
named Pocket TTS has a size of 90M parameters while the VAE has 20M parameters. Tab 13 we put 100M parameters to include the VAE decoder in the parameter count.
We evaluate Pocket TTS on the Librispeech test-clean set following the same protocol F5-TTS (Chen et al. 2025), with the difference that we cleaned the audio input using Adobe Enhance Speec to obtain 24kHz high-quality audio. We evaluate baselines with the enhanced sample
We compare against three baselines: F5-TTS (Chen et al. 2025) J DSM (Zeghidour et al. 2025 Chatterbox Turbo and Kokoro TT$ We report Word Error Rate (WER) using Whisper-large v3(Radford et al., 2022). well the results of pairwise human evaluation for audio quality and speaker similarity. For audio quality, we ask raters "Which of the two audio clips has the best audio quality?", , and for speaker similarity, we ask "Which of the two audio clips sounds more similar the reference audio clip in terms voice characteristics?" and provide the voice prompt as a reference.
Table 13: Comparison of TTS models Librispeech test-clean.
MODEL
PARAM SIZE
WER
AUDIO QUALITY
SPEAKER SIM
FASTER THAN REAL-TIME CPU
(GEN. ONLY)
(J)
(ELO) (1)
(ELO) (1)
FS-TTS (CHEN ET AL. 2025
336M
2.21
1949 + 27
1946 = 26
DSM
750M
1.84
1959 + 25
2037 + 21
CHATTERBON TURBO
350M
3.24
2055 + 23
2012 + 22
KOKORo
82M
1.93
No VOICE CLONING
POCKET TTS (OURS)
100M
1.84
2016 + 25
1898 + 26
As seen in Tab. 13 Pocket TTS has the lowest Word Error Rate, a better Audio Quality than the ground truth F5-TTS and DSM, well on-par Speaker Similarity with the ground truth while being significantly smaller model than competitors, and being the only one that can run faster than real-time on CPU (we tested on Apple M3 and Intel core ultra 7 165H). We invite the reader check the blog post where the one-line installation of the model is provided:
H HUMAN EVALUATION METHODS
Audio clips are always 30s second total, with prompt coming from ground truth audio. Each experiments has 50 samples for each method. There are 50 raters. Each of them sees 10 samples. Raters were payed E hour for their contribution.
20

---

Published as a conference paper at ICLR 2026
H.1 SPEECH CONTINUATION
Acoustic quality assessment: How would you rate the overall quality of this audio clip? Consider aspects such as clarity, balance, richness, and naturalness. Listen to least seconds of audio before deciding. 1 clip is presented, possibilities are bad, poor, fair, good, excellent.
Meaningfulness: Which these two audio clips feels more like meaningful and natural speech? The first 3 seconds are identical. Listen to at least 10 seconds of each clip. 2 clips are presented, ties are possible. Elo scores are Bayesian estimates of the posterior mean in a Bradley- Terry model.
H.2 MUSIC CONTINUATION AND CLAP-TO-MUSHA
Acoustic quality assessment: How would you rate the overall quality of this music? Consider aspects such as clarity, balance, richness, and naturalness. Listen to at least 10 seconds of audio before deciding. 1 clip presented, possibilities bad, poor, fair, good, excellent.
Which music do you enjoy listening to more? The first seconds are identical. Listen to least 10 seconds of each clip. 2 clips are presented, ties are possible. Elo scores Bayesian estimates of the posterior mean in a Bradley-Terry model.
Music enjoyment:
H.3 BAYESIAN ELO SCORE
The Meaningfulness metric for speech and the Enjoyment metric for music are both Bayesian Elo Scores. Elo score is used to rank models based on some pairwise comparisons of audio samples. Given two models A and B, the probability that A is preferred over is:
P(A > B)
1 10(En-BA)/aml B4)/400
where E and F are the Elo scores of each model. Unlike a traditional Elo score, the Bayesian Elo score uses a Gamma prior, that one can derive confidence intervals over the posterior distribution.
By defining SA such as FA = 400 log (Sa) + with being constant, we obtain:
SA
P(A > B)
(8)
SA
S
which Bradley-Terry (Bradley & Terry
1952) model. There are a few different methods to estimate the parameters of Bradley--Teree Terry model. We use the iterative one from (Caron Doucet 2010) where 5 follows Gamma prior with parameters a,,8". By denoting times where method A won against any other methods and NAB the number of times where and B are compared, S is computed with the following update rule until convergence:
the number of
a ut WA
St+1
(9)
3+ 3 L B
which is the mean of the Gamma distribution with updated parameters ++1
gh+t,
a c = a WAs
and
B+11 30 +
Σ
WA,B
(10)
St
+ S
B
Iterating over allows to reach a fix point, we rum 30 of them, once we have collected all the pairs. We 0.1, B 0.1, 2000 that in absence of any data, S^ 1 and EΛ 2000. Confidence interval are 95% confidence interval according to the posterior (the 2,5th and 97.5th percentiles).
I HYPERPARAMETERS
We present in Tab. 14 the parameters of the music and speech VAE used for CALMs. For CALMs and the Ronarooffrrrrr based discrete LMs, the hyperparameters presented in Tab. 15
21

---

Published as a conference paper at ICLR 2026
Table 14: VAE hyperparameters
Music VAE
Speech continuation VAE
General
Sample rate
32kHz
24kHZ
Frame rate
25Hz
12.5Hz
Latent dimension
128
32
Architecture
Convolutions ratios
8, 8, 5,
6, 5, 4, 4,4 8
Num transformer encoder layers
4
Num transformer decoder layers
4
8
Transformer context
30s
10s
Training parameters
Batch size
64
64
Audio sample length
12s
12s
KL loss weight
0.01
0.01
Reconstruction loss
X
Distillation loss weight
X
25
LR Schedule
cosine
cosine
Learning rate
10 44
8 10
22

---

Published as a conference paper at ICLR 2026
Table 15:
Model and training hyperparameters
Music
Speech continuation
Text-to-Speeeh
Backbone Tronsformer
Model dimension
1536
2560
1024
MLP dimension
6336
10560
4096
Number heads
24
Number of layens
48
24
24
Learning rate Number parameters
1- 10100
10
lo44
1.35B
2.2B
302M
RQ-Transformer head (RVQ model)
Model dimension
1024
1024
MLP dimension
4096
4096
X
Number
Number of layers
X
Number of parameters
701M
701M
Consistency sampler head (ours)
Number layers
MLP dimension
3072
512
512
Gating Number parameters
SILU
SiLU
SILU
601M
10M
10M
Short transformer
Model dimenssin
1536
MLP dimension
6336
Number heads
24
Number of layens
4
Context
Number of parameters
113M
Audio embeedding manipalations
Center and normalize
<
Noise before entering backbone
x
Training parameters
batch multiplier
Optimizer
AdamW & 0.9, 92 0.95
AdamW 81
0.9, 0.95
AdamW
0.9, 0.95
Batch size
48
144
128
Audio sample length
30s
300s
60s
I.R Schedule
cosine
cosine
cosine
Learning rate
1 10
2 10
le44
GPU
H100
H100 150k
H100
Number of training sleps
S00k
400L X
Initial checkpoint
Helium-1 Kyutai 2025
Inner monologue
X
x
Acoustic delay
frames (160 ms)
23


Reviews:
[{"review_id": "tB1s2zXXLm", "reviewer": "Reviewer_mHKG", "summary": "This work introduces Continuous Audio Language Models (CALM), a new framework for generating high-quality audio (speech and music). It addresses the primary limitation of current state-of-the-art models, which rely on converting audio into discrete tokens (using RVQ codecs). \nCALM bypasses this issue by operating directly in the continuous latent space of a Variational Autoencoder (VAE). Its architecture uses a large Transformer to model the long-term context of the audio. Instead of a slow token-by-token predictor (like an RQ-Transformer) or a multi-step diffusion process, it uses a highly efficient Consistency Model as its head. \nExperiments on both speech and music continuation show that CALM achieves higher or comparable quality to strong discrete baselines while being significantly faster and more computationally efficient at inference time.", "questions": "Can you include at least one external strong discrete audio LM (not re-trained under your codebase) to demonstrate improvements are not an artifact of your reimplementation?\nSince the paper currently focuses on continuation, adding a short section on non-continuation tasks would strengthen the claim that “continuous AR + consistency” is a general recipe.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.Using a  consistency model as the head enables fast single-step generation in a continuous space. Relative to DiTAR-style “continuous latent + AR + diffusion head,” the paper’s contribution is to bring in a consistency-model-style head to reach 1–few steps and to pair this with the long/short context mechanism that specifically stabilizes music-length sequences.\n\n2.Good performance on both speech continuation and music continuation task.", "weaknesses": "1.The paper evaluates mainly on continuation. It does not yet demonstrate text-conditioned generation, fine-grained control, or editing, where discrete-token model currently provide a convenient control surface. \n\n2.Asymmetric gains across domains. Music clearly benefits from “noisy long + clean short,” whereas speech shows weaker or no gains from the same trick, suggesting part of the method is tuned to music-style long dependencies rather than being universally optimal. \n\n3.Comparisons are mostly in-house. Many baselines are re-trained under the authors’ own setup (same data, backbone, RVQ depth). This makes the comparison clean but slightly weakens claims about superiority over the latest large-scale discrete audio LMs trained externally.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MvjY8Qvc6O", "reviewer": "Reviewer_6NWz", "summary": "The paper proposes CALM, a continuous autoregressive audio generation framework without discrete quantization. The idea is to first replace lossy RVQ codecs with a standard VAE, and then generate latent vectors frame-by-frame using transformers with consistency heads. Experiments show that on both speech and music continuation tasks, CALM can generate high-quality, long-context and meaningful audio contents within real-time.", "questions": "1. Have the authors considered methods that directly generates mel-spectrogram, like MELLE [1] and Flow-Omni [2]? What is the advantage of CALM compared to these solutions?\n2. How much data is used to train the speech generation model? Have the authors used in-house data? If not, the source of the training data should be provided.\n3. As the reconstruction quality of VAE sets an upper bound for generated audio quality, it would be better if more detailed evaluation could be provided. For example, reporting additional metrics like PESQ and STOI, as well as perceptual metrics like MOS.\n\n**References**\n\n[1] Meng, Lingwei, et al. \"Autoregressive speech synthesis without vector quantization.\" *arXiv preprint arXiv:2407.08551* (2024).\n\n[2] Yuan, Ze, et al. \"Continuous speech tokens makes llms robust multi-modality learners.\" *arXiv preprint arXiv:2412.04917* (2024).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. Different from previous MAR-style work designed for single tasks like TTS and musical audio generation, CALM aims to build **a unified framework** for multiple audio generation tasks. The authors illustrated the effectiveness of CALM on **two challenging tasks**: speech and music continuation, demonstrating CALM’s ability to generate content with high semantic meaningfulness and acoustic quality.\n2. The idea to replace sampler/diffusion/flow matching heads with consistency head is novel. This approach, along with the newly-proposed training strategy of head batch multiplier, significantly reduces inference and training cost.\n3. The paper proposes an effective solution to the error accumulation problem in continuous audio generation with noise augmentation and short context transformer.", "weaknesses": "1. Although the meaningfulness and enjoyment improves upon baseline methods, the generated audio still lacks long-content semantic continuity as illustrated by samples in the demo page.\n2. The proposed method is evaluated only on unconditional generation tasks, making it unclear whether the proposed method can be extended to more applicable conditioned tasks like TTS and text-to-music. This also makes it difficult to compare CALM with mainstream baselines that work on conditional generation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i72IV0FTc0", "reviewer": "Reviewer_vAFX", "summary": "To address the quality and efficiency issues of discrete tokens, the paper proposes CALM. CALM models directly in the continuous latent space and introduces a consistency model to enhance audio reconstruction quality while significantly improving inference speed. By using the Causal Short-Context Transformer module, the quality of generation is enhanced. In addition, the paper proposes practical techniques such as temperature sampling and Head Batch Multiplier to enhance training stability and controllability of generation.", "questions": "1. Could the authors provide a more detailed explanation of the working mechanism of the Short-Context Transformer, or validate through experiments and visual results what specific information this module captures?\n\n2. Recently, some new continuous modeling-based audio language models have emerged, such as DiTAR and Ming-UniAudio. If possible, could the authors compare the technical similarities and differences as well as the actual performance between CALM and their models?\n\n\nDiTAR: https://arxiv.org/abs/2502.03930\n\nMing-UniAudio: https://mdn.alipayobjects.com/cto_asrtts/uri/file/as/TR-Ming-UniAudio.pdf", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper makes several improvements to existing Audio Language Models, introducing new mechanisms and training methods with a good degree of innovation. The paper is well-written, with the methods section presenting a complete mathematical process and the experimental section validating multiple improvements. CALM is of considerable importance to current audio language models.", "weaknesses": "1. Although a detailed comparison is made with the RQ-Transformer, there is insufficient comparison with other continuous audio generation models (such as SALAD, DiTAR, etc.).\n\n2. As shown in the experimental data presented by the authors (Table 5), the Short-Context Transformer is crucial for generation quality, but the authors do not specifically explain the working mechanism of this module, nor do they explain how the absence of this module would degrade the model's generation results.\n\n3. Similarly, the authors need to more clearly explain the specific impact mechanism of noise injection on model robustness and preferably provide some visual results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xJlk3MpRZR", "reviewer": "Reviewer_ZVTy", "summary": "Discrete-token audio LMs (e.g., RVQ stacks + AR decoders) trade fidelity for latency: deeper codebooks mean more tokens and slower sampling. The paper investigates whether we can get rid of discretization and instead do autoregressive modeling in a continuous latent space while keeping quality and reducing inference cost by few-step sampling with consistency models. \nThe paper contributes CALM, an autoregressive framework that operates directly on continuous VAE latents and replaces RVQ/diffusion heads with a small consistency-model MLP to enable few-step next-latent prediction. This avoids discretization while targeting real-time sampling. It introduces a dual-context design, a noise-injected long-term backbone combined with a clean short-context transformer over the last K latent, whose features are summed to serve as the condition of the head. Empirically, the consistency head reduces inference cost while maintaining or improving quality.", "questions": "Why sample $x_t$ with $x_t = cos(t)x + sin(t) \\epsilon$ instead of $x_t = (1-t) \\cdot x+t \\cdot \\epsilon$ ?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "At a high level, the paper’s short-context pathway is analogous to LocDiT in DiTAR: both introduce an additional local/short-term module on top of a long-range autoregressive backbone to recover fine detail that a purely long-context model tends to miss. That said, the mechanisms and constraints differ: the proposed short-context transformer is causal and consumes a clean recent window whose features are summed with a noised long-context stream to condition a few-step consistency head. By contrast, LocDiT performs bidirectional, intra-patch diffusion with a small prefix of historical patches, and typically runs for many denoising steps. The goals align, but the causality, noise-injection, and sampling budgets (few-step consistency vs. multi-step diffusion) are importantly different.", "weaknesses": "- The paper targets continuation on speech and music, but the evaluation scope is too narrow to position the method within the broader speech modality. In practice, TTS is the most common task for audio generation models. Without a TTS evaluation, it’s difficult to compare against widely used or state-of-the-art systems. Including comparisons with strong baselines (e.g., GLM-4-Voice, DiTAR) on overlapping tasks would be required. In short, the method is well-substantiated for continuation, but its position relative to on-the-shelf speech models is unclear. Adding TTS (with direct SOTA comparisons) would make the contribution much easier to compare and significantly strengthen the paper.\n\n- Lack of references: Suggest citing IMPACT [A] in line 53, which is also a MAR-style audio generation model.\n\n[A] Huang, K. P., Yang, S. W., PHAN, H., Lu, B. R., Kim, B., Macha, S., ... & Wang, C. IMPACT: Iterative Mask-based Parallel Decoding for Text-to-Audio Generation with Diffusion Modeling. ICML 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)