Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
Authors: Shu-wen Yang
Byeonggeun Kim
Kuan-Po Huang
Qingming Tang
HUY PHAN
Bo-Ru Lu
Harshavardhan Sundar
Shalini Ghosh
Hung-yi Lee
Chieh-Chi Kao
Chao Wang
Venue: ICML 2025 poster (2025)
Keywords: text-to-audio
audio language modeling
continuous token
diffusion model
next-token prediction
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We replace discrete tokens with continuous latents for audio language modeling, enhanced by a novel masked next-token prediction task.

Abstract: Autoregressive next-token prediction with the Transformer decoder has become a de facto standard in large language models (LLMs), achieving remarkable success in Natural Language Processing (NLP) at scale. Extending this paradigm to audio poses unique challenges due to its inherently continuous nature. We research audio generation with a causal language model (LM) without discrete tokens. We leverage token-wise diffusion to model the continuous distribution of the next continuous-valued token. Our approach delivers significant improvements over previous discrete solution, AudioGen, achieving 20% and 40% relative gains on AudioCaps in Frechet Audio Distance (FAD) and Kullback-Leibler (KL) divergence, respectively. Additionally, we propose a novel masked next-token prediction task that incorporates masked prediction into the causal LM framework. On AudioCaps, the innovation yields 41% and 33% relative FAD improvements over AudioGen Base (285M) and AudioGen Large (1B) models, respectively, and is on par with the state-of-the-art (SOTA) diffusion models. Furthermore, we achieve these results with significantly fewer parameters—193M for our Base and 462M for our Large models.

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
Shu-wen Yang
Byeonggeun Kim 22 Kuan-Po Huang 122
Qingming Tang?
Huy Phan Bo-Ru Lu
Harsha Sundar? Shalini Ghosh Hung-yi Lee
Chieh-Chi Kao
Chao Wang?
Abstract
Next-Token Prediction
Autoregressive next-token prediction with the Transformer decoder has become a de facto stan- dard in large language models (LLMs), achiev- ing remarkable success in Natural Language Pro- cessing (NLP) scale. Extending this paradigm audio poses unique challenges due its in- herently continuous nature. We research audio generation with causal language model (LM) without discrete tokens. We leverage token-wise diffusion to model the continuous distribution of the next continuous-valuee token. Our approach delivers significant improvements over previous discrete solution, AudioGen, achieving 20% and 40% relative gains AudioCaps in Frechet Au- dio Distance (FAD) and Kullback-Leibler (KL) di- vergence, respectively. Additionally, we propose a novel masked next-token prediction task that in- corporates masked prediction into the causal LM framework. AudioCaps, the innovation yields 41% % and 33% relative FAD improvements over AudioGen Base (285M) and AudioGen Large (IB) models, respectively, and is par with the state-of-the-an (SOTA) diffusion models. Further- more, we achieve these results with significantly fewer parameters- 193M for our Base and 462M for our Large models.
O
Discrote-valued audio tokens
Masked Next-Token Prediction Training
Inference
Continuous-valued audio tokens
Figure Causal Language Modeling on Continuous-valued Audio Tokens with Masked Next-Token Prediction. The audio tokens are low-dimensional continuous latent. We use stan- dard Transformer decoder for the audio language modeling. Our masked next-token prediction learns to predict any future token given any subset of the past tokens, which turns out benefit the standard next-token prediction and rival the bidirectional diffusion models and masked generative models. The figure illustrates the prediction order in which all tokens are conditioned on all previ- ously predicted tokens.
1. Introduction
and reasoning abilities (Achiam al., 2023; Team et al., 2023) after scaling up to thousands of billions of parameters, such as the GPT series (Brown et al., 2020; Achiam et al., 2023). The essence behind LLMs is simple, scaling up the model by more parameters and data with a Trunsformer de- coder (Vaswani. 2017) and next-token prediction. Given the success of this paradigm, plenty of efforts have been put into developing better theories and tools for this standardized model, including the architecture improvement (Chowdhery et al., 2023; Du et al., 2022) scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022), and efficient inference and streaming infrastructure (Kwon et al., 2023; Pope et al., 2023).
Large Language Models (LLMM)) has revolutionized arti- ficial intelligence (Wang et al., 2018; Zagar & Robnik- Šikonja, 2022; Hendrycks et al., 2021). They show remark- able emergent capabilities and human-level understanding
'Equal contribution 'Graduate Institute of Communica- tion Engineering, National Taiwan University. Taipei, Taiwan AAmazon AGI, Bellevue, United States. Correspondence to: Shu-wen Yang <1e019941227 comm>, Chich-Cr Chi Kao <chichchi amazon.com>..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
We study text-to-audio (TTA) with the LLM framework. TTA is critical due to its various applications in virtual (aug- mented) reality, media creation, video editing, and game development (Liu et al., 2023a). Specifically, given natu- ral language prompt, the model generates a relevant sound. The causal language modeling approach is important to audio for its fundamental role in multi-modal LLMs (Wu et al., 2024; Yin et al., 2023; Liu et al., 2024b; Zhou al., 2024) and the streaming capability for real-time user interac- tion (Wang et al., 2024a; Zhang et al., 2024). Hence, we ask whether causal language model can achieve high-fidelity audio generation. Existing works applied a Transformer decoder with next-token prediction on discrete audio tokens for TTA (Kreuk et al., 2023; Yang et al., 2023a). Despite scaling to billions of parameters, these models lag behind the SOTA systems, which typically rely on diffusion tech- niques (Majumder et al., 2024; Liu et al., 2024c). As result, to build an LLM that can speak or produce high- fidelity sound, the prevailing approach relies on external vocoders (Huang et al., 2024; Liu et al., 2023b; Wang et al., 2024b). However, the pipeline approach overlooks the pos- sibility of directly generating audio with scalable and streamable LLM, allowing us to enjoy the aforementioned resources like scaling, efficient inference and streaming in- frastructure.
2024a; 2023; Wei et al., 2023), both of which have been shown benefit generation across various scenarios. Con- cretely, we randomly drop tokens as the masking, forming shorter sequence. Next-token token prediction is then performed on this shorter sequence as the masked prediction, a task we term masked next-token prediction (MNTP). Compared to next-token prediction, MNTP predicts a random future token conditioned on a random subset of past tokens (Fig- ure 1), a task that ultimately benefits next-token prediction. On AudioCaps, our AudioMNTP Base significantly out- performs AudioNTP Base, achieving a 41 % relative FAD improvement over AudioGen Base. Scaling up to 462M AudioMNTP Large yields a 33% relative FAD improve- ment over 1B AudioGen Large, matching SOTA diffusion models (Liu et al., 2024c: Majumder et al., 2024) while remaining streamable and compatible with LLMs Our con- tributions are:
We propose the use of continuous- valued tokens in generative audio language modeling. demonstrating their superiority over discrete tokens.
We introduce novel learning task, masked next-token prediction (MNTP). Training with MNTP enhances the model's decoding performance on the regular next- token prediction.
In this work, we enable audio generation for causal lan- guage models. We address the bottlenecks that block the decoder-only solution by integrating strengths from SOTA diffusion models (Majumder et al., 2024; Liu et al., 2024c), including token types, loss functions, and learning tasks. We reinterpret the variational auto-encoder (VAE) latents in latent diffusion models (LDMs) (Rombach et al., 2022) as continuous-valued tokens', replacing discrete acoustic tokens used in audio language modeling. To model the continuous distribution of the next token, we replace the cross-entropy loss by the token-wise diffusion loss (Li et al., 2024b), leaving the backbone Transformer decoder intact, allowing it to fully leverage the benefits and resources of LLMs. Using continuous-valued audio tokens and next-token prediction (NTP), our 193M AudioNTP Base achieves 20% and 40% relative improvements over 285M AudioGen Base (Kreuk et al., 2023) on AudioCaps (Kim et al., 2019) in Frechet Audio Distance (FAD) and Kullback- Leibler (KL) divergence, respectively.
By integrating continuous-valuee tokens and MNTP, we achieve SOTA-level audio generation within the causal LM framework, establishing a pathway for di- rectly generating high-fidelity audio with LLMs.
2. Related Work
Text-guided Audio Generation. The most relevant works include AudioGen (Kreuk et al., 2023) and UniAudio (Yang et al., 2023a). AudioGen learns a causal LM on discrete tokens (Zeghidour et al., 2021; Défossez et al., 2023) with low downsampling rate to preserve audio fidelity, whereas UniAudio employs a multi-scale Transformer (Yu al., 2023) to reduce RVQ token sequence length, thereby saving computation. On the other hand, LDMs (Rombach et al., 2022) have outperformed language modeling approaches in audio generation (Yang et al., 2023b; Liu et al., 2024c; Majumder et al., 2024) by modeling the joint probability distribution of continuous-valued tokens through diffusion processes. Among them, we adopt a causal LM similar to AudioGen but apply it to continuous-valuce tokens, in LDMs.
To further match the performance of SOTA diffusion mod- els, we incorporate masked language modeling (MIM) and propose a novel learning task. MLM learns contex- tualized dependencies (Kenton & Toutanova, 2019; Vyas et al., 2023; Liu et al., 2024a; Li ct al., 2024b; Chang et al., 2022) and develops understanding capabilities (Li et al.,
Language Modeling on Continuous-valued Tokens. Language modeling on continuous data has traditionally required quantizing data into discrete tokens (Chen et al., 2020b; Esser et al., 2021; Borsos et al., 2023; Kreuk et al., 2023). Recent works in image and speech generation (Li
SSpecifically, we use a VAE to encode the Mel-spectrogram into 2-D feature map, and scrialize it into a 1-D sequence of low-dimensional latents.
2

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
al., 2024b; Tschannen al., 2023; Meng al., 2024) demonstrates that continuous-valued tokens prevent infor- mation loss and achieve higher generation quality. These approaches model continuous next-token distributions using diffusion loss (Li et al., 2024b) or Gaussian mixture mod- els (Tschannen et al., 2025; Meng et al., 2024). In our work, we employ diffusion loss for sound.
Diffusion Process
Diffusion
HOT-1)
HUi--AN Vocuder
MLP
MLP
VAE Decnder
Shared
Conditioaint
Context
Masked Prediction in Generative Models. Masked pre- diction has been shown effective in various generative mod- els. In text generation, masked infilling serves pretext task for bidirectional models (Raffel et al., 2020; Lewis et al., 2020), and iterative MLM has been applied to image gener- ation (Li et al., 2023; Chang et al., 2022). Similarly, masked infilling benefits audio ( (Vyas et al., 2023) and speech (Liu et al., 2024a) generation in the fow-matching (Lipman et al., 2023) framework. Recently, several methods (Fried al., 2023: Aghajanyan et al., 2022; Peng al., 2024; Bavarian et al., 2022) reorder masked tokens to the sequence end, enabling decoder-only models to leverage both past and fu- ture context for the editing tasks. In contrast, we integrate masked prediction without future context, strictly preserving causality and improving unidirectional decoding.
Continuous tokens
Transformer Decoder
VAE Encoder
Emnh
bos
Positional Emb.
Figure 2. proposal. The waveform is transformed into the Mel-spectrogram, subsequently encoded into continuous vaaluud tokens. The Trans- former decoder learns the next token prediction on these tokens via the loken-wis- diffusion loss with a small MLP diffusion head. This framework is termed AudioNTP.
The framework of the continuous- vaalind audio token
tokens in audio language modeling. Figure 2 illustrates the framework of AudioNTP.
Relation to MAR. Our work is based on MAR (Li et al., 2024b), which introduces continuous-valuee tokens in the context of masked generative modeling (MGM) for image generation. In MGM, a bidirectional model iteratively per- forms MLM from an all-zero input, predicting all masked positions but retaining only subset, until all positions re- tained. Our approach differs from MAR by using a standard Transformer decoder with the classic next-token prediction, making it compatible with LLMs. We show that, in the audio domain using continuous-vall valued tokens, the unidirec- tional MNTP outperforms naive next-token prediction and comparable to the bidirectional MAR. Furthermore, in the left-to-right causal inference scenario, MNTP outperforms MAR.
Training. Given input text prompt and a length-s audio waveform {a ,...,,") a"}, we tokenize it into sequence of low-dimensional latents?: x = [x......"" z"), where x² E Rh and h E Z represents the latent di- mension. We refer to this length-n, 1-D latent sequence & as the continuous
tokens. We train the lan- guage model with maximum likelihood on r over an au- dio corpus D {az, Wy], The loss function is then E B(a,oy> [-log p(a w)), where
p(a /w) = pp(x "'.""""..""""
xn (w) = p(x" x , x
(1)
3. Method
To learn the next-token distribution p(a" |x] x=- w). our model consists of a Transformer decoder Co and MLP diffusion head Mg. where 0 and denote the parameters. Firstly, Co encodes the input sequence (w, B, 3, ... into a sequence of context vectors =xz² ...., 20), where w is placed at the front as the input prompt and /3 is the BOS small multi-layer
Our approach includes two main proposals: continuous- valued tokens in Section 3.1 and masked next-token predic- tion in Section 3.2.
3.1. AudioNTP: Continuous-valued Tokens
token: z² = Co (w. ...,,.....-1
Common belief holds that next-token prediction is tied to fixed size dictionary and cross-entropy loss. However, the task by definition is predicting the next token given the previous tokens, independent of the exact realization of the token and how to model the next-token distribution. Following the paradigm shift in CV (Tschannen et al., 2025; Li et al., 2024b), we investigate the use of continuous-valued
See Appendix F.1 for the detailed tokenization pipeline. Compared MAR (Li et al., 2024b) which relies on a bidirec- tional Transformer to achieve competitive results, we stick to the causality to align with the LLM setting. Their use of continuous- valued tokens on causal LM performs poorly, while our approach achieves results comparable the bidirectional counterpart with the masked next-token prediction.
3

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
perceptron (MLP) Ma conditions on of and models the next-
token distribution p(x²* =²) = p(æ² :.... "x.,,..,="" _f1 w) with
Dillision Head
the following diffusion objective (Li et al., 2024b; Rombach
Context
et al., 2022).
Transformer
Decoder
|||e Mo(ab 2t, 01|]]
arg min Best 0,0
Text Emb.
Positional Emb.
bos
P
E E IE is a Gaussian noise sampled from NN(0, I). The diffused token æt is obtained by xi = V0,x² + vI a,s with the noise schedule ā₂ (Ho al., 2020: Nichol Dhari- wal, 2021). 1 E (0, .... 7 is time step sampled from the noise schedule. We train C and My jointly. By backpropa- gating the gradient from the small diffusion head Ma to the Transformer decoder Cg. the decoder represents the next- token distribution into the context vector 2 This facilitates accurate diffusion modeling even with a lightweight Ma-
Drop & Merge
Target Positional Emb.
Pt
On-the-fly Pre- pre-processing
Random Masking
Figure 3. The framework of the masked next-token prediction (MNTP). The continuous- valuudd tokens first masked (dropped) to form a shorter sequence. The Transformer decoder then learns the next-token prediction on the dropped sequence with the diffu- sion loss. This framework is termed AudioMNTP.
Inference. At position i, Cs infers the context vector zf by a single pass given the previously sampled tokens 2 Co "u..""..,,,++ ). Conditioning on 2², MM. iteratively de- noises a Gaussian noise into a clean token 5², known as the sampling process. Sampling is done via a reverse diffusion procedure (Ho et al., 2020):
Training. Before feeding the token sequence x {at, ,c.,....") }"} into Co. we first apply masking on it. We denote the mask as v = {u², ..., v"} where of E {0, and v² = 0 means the position is masked. The masked sequence is then T. = x O v. Recent works show that dropping instead of masking yields similar performances while greatly reduce the training cost². We drop the tokens to form shorter sequence 2, { T. 5 & 0}. We learn the next-token prediction on 2. as the masked predic- tion Upon a closer look, each position i learns to predict various future positions, depending on the random masking patterns. We can then view MNTP as a multi-task learning with imput dropout (Hou et al., 2022; Zaremba et al., 2014; Sennrich et al., 2016), where skip-token prediction benefits next-token prediction during inference?. Since the model is unaware of the current masking pattern, it is confusing to naively ask the model predict various future positions indis- criminately. We introduce the target positional embedding Pt to Cp. as shown by Figure 3. Originally, each token has its own learnable content positional embedding Pc- We learn an additional target positional embeddings representing the
az MalTi, z 1) +ozō. +000 (3)
V 04
at
5 is sampled from NO, I) and o, is the noise level time step f. The decoding de-noising procedure starts from the timestamp T to timestamp O. That is, Zh ~ 10, and ii _ ~ p(x' 2¹) = p(r' I=',..,, x=- w). The sampled tokens can then be de-tokenized into the sampled w
3.2. ANdio Masked Next-Token Prediction
AudioNTP yields significant improvement over AudioGen. However, as shown in our results, the generation quality and diversity still lag behind LDMs (Liu et al., 2024c; Ma- jumder al., 2024). Instead of resorting the bidirectional solution?, stick to the causality maximally follow the LLM framework. Given the success of MLM in generative models, we seek to migrate the core idea of MLM into the causal next-token prediction. We hypothesize that the key mechanism behind MLM is predicting the unseen token given the sparse context', essentially independent of the model directionality. We devise novel learning task tai- lored to the causal architectures termed masked next-token prediction (MNTP). Figure 3 illustrates the framework.
'Due to the drastically reduced sequence length (Huang et al., 2022; He et al., 2022; Baevski et al., 2023).
"Appendix D.1 reviews the similarity and differences between MLM and MNTP.
'Predicting skip positions have been proven effective for audio representation learning (Oord et al., 2018; Chung & Glass, 2020). We verify its effectiveness in autoregressive generative modeling. Compared to text, continuous data exhibits high similarity between consecutive tokens, allowing the model to easily exploit local smoothness (Oord et al., 2018) for next-token prediction. We avoid this trivial prediction and facilitate sparse context required by ML. via skip-token prediction.
In MAE (He al., 2022) and AudioMAE (Huang al., 2022). over 70% tokens are masked.
4

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
position to be predicted and add both embeddings to the au- dio tokens. Consequently, masked next-token prediction can be viewed as generalized form of causal language model- ing. The model can predict any future timestamp given any subset of past information, with predicting the immediate next token and conditioning on all previous tokens as special cases of the standard next-token prediction
Masking Schedule. For each training iteration, we sam- ple masking ratio from distribution over [0, 1] defined by the schedule, and randomly drop the tokens according to the ratio. Specifically, we apply mixture of normal distri- bution and truncated normal distribution, where the former emphasizes the high masking ratio for MLM and the latter preserves the long-tailed distribution on the low masking ratio for next-token prediction. See Appendix B for details.
Inference. The inference procedure of AudioMNTP is identical to that of AudioNTP, except for the inclusion of an additional target positional embedding, which is derived by incrementing the content positional embedding by one.
5. Experiments
Training. We train our model on AudioCaps (AC) (Kim et al., 2019) and WavCaps (WC) (Mei et al., 2024). See Appendix A for details. We use AdamW (Loshchilov & Hutter, 2017) optimizer with a fixed learning rate 100 We train the Base model with 40 NVIDIA VI00 GPUs, and the Large model requires 104. Our effective batch size is 2048 10-second clips. We train the Base and the Large model for 1000 epochs, about 2 days and 5 days, respectively.
4. Implementation
Continuous-valuee Audio Tokenizer. We leverage the to- kenization pipeline of AudioLDM (Liu et al., 2023a), includ- ing a pre-trained VAE and a pre-trained Hifi-GAN (Kong et al., 2020) vocoder. Given an audio, the pipeline extracts the Mel-spectrogram, and the VAE encoder processes into sequence of continuous-valun
tokens = (z.',.....) x"), which is used for training. During inference, we sample J (i:".....") } according to equation 3. We use the VAE decoder to decode ă back to a Mel-spectrogram. Then, the vocoder is used to synthesize the waveform. Refer to Appendix F.1 for details.
Evaluation. We evaluate our model on the AC evalua- tion set, following the protocol in AudioLDM and using its evaluation toolkit For each generated audio and its ground-truth counterpart in AC, we compute several met- ries: Fréchet Audio Distance (FAD) (Kilgour et al., 2019), Fréchet Distance (FD), Kullback-Leible divergence (KL), Inception Score (IS) (Liu et al., 2023a), and Contrastive Language-Audio Pretraining (CLAP) score (Huang et al., 2023b). See Appendix G.1 for details on these metrics. For subjective evaluation, we rate audio samples on text relevance (REL) and overall quality (OVL) using a 1-5 scale (Ghosal et al., 2023; Liu et al., 2023a); additional de- tails are in Appendix G.2. Because speech is challenging in TTA- and even the most advanced systems often produce unintelligible speech-we we split our evaluation into speech and non-speech categories.
Conditional Audio Language Modeling. We mostly fol- low the implementation in MAR (Li et al., 2024b), includ- ing the training/inference details of the MLP diffusion head, and the architecture design of MLP and the Transformer de- coder. We train an audio version of the bidirectional MAR as the topline (AudioMAR)) a causal LM as the baseline (AudioNTP), and our main proposal (AudioMNTP), which is also causal. For most of the studies, we train AudioMAR, AudioNTP and AudioMNTP with the 193M Base model. We also train a 462M AudioMNTP Large to study the scal- ing behavior. For the text prompt, we use CLAP (Wu et al., 2023) and FLAN-T5 (Chung et al., 2024) to extract the text embeddings and concatenate them as the input prompt. Refer to Appendix F.2 for details.
5.1. Main results
Continuous-valued tokens with diffusion loss are com- petitive for audio language modeling. Table shows that our 193M bascline AudioNTP Base outperforms the 285M AudioGen Base and IB UniAudio by a large margin on the FAD and KL metries. Specifically, we achieve 20%, 40% relative improvements over AudioGen Base on FD and KL scores, respectively. Both AudioGen and UniAudio are based on the discrete tokens and use much more data, pa- rameters and compute compared to our method. The results gauge the effectiveness of the continuous-valued tokens and the diffusion loss for audio language modeling"!
Initialization. By default, AudioMAR, AudioNTP, and AudioMNTP are obtained by fine-tuning the pre-trained im- age MAR" on audio data. That is, they all start by learning the MAR task on images and diverge by using MAR, NTP, and MNTP task on audio data, respectively. As shown in the subsequent ablation study, this image initialization only slightly boosts performance and is not critical.
"Appendix D.2 illustrates the idea more formally.
10
eval
Bither the Base or Large model
in
We do not train our model with discrete tokens due to com- putational constraints imposed by their long sequence length. For
httss//gituh..om/..TE http:hhuhh.oomm..H144a//1...
5

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
Table Main results. FD, FAD, KL, IS, and CLAP metrics on the AudioCaps evaluation set. The AS, AC, and WC stand for AudioSet, AudioCaps, and WayCaps, respectively. Bold: the best performance. Underline: the second best performance. "Re-infeeence with the public checkpoint. The detailed datasets used by each model are listed in Appendix
FD4
FAD L
KI.
IS
CI.AP 1
BI-DIRECTIONAL
MAGNET-SMALL (Ziv ET AL., 2024)
AS AC + OTHERS
300M
23.02
3.22
1.42
9.72
0.287
DISCRETE
MAGNET LARGE (ZIV ET Al.,, 2024)
AS + AC + OTHERS
1.5B
26.19
2.36
1.64
9.10
0.253
TANGO (GHOSAL ET AL., 2023)
7.70
TANGO-FULL-FT (GHOSAL
2023)
AS AC WC 5 OTHERS
866M
18.93
2.19
1.12
8.80
0.340
TANGO-AF&ACA (KONG BT AL., AL., 2024n)
AC OTHER
866M
21.84
2.35
1.32
9.59
0.343
TANGO 2 (MAJUMDER ET Al.., 2024)
OTHERS
9.09
CONTINUOUS
MAKE-AN-AUDA 2 (HUANG ET AL., 2023A)
AS + AC WC+ 11 OTHERS
937M
16.23
2.03
1.29
9.95
0.345
AUDIOLDM2-AC (LIU ET AL., 2024c)
AC
346M
1.67
1.10
AcniOL DM2-AC-TARRO
AL., 2024c)
AUDIOLDM2-FULL (LIU AL., 2024C)
AS AC WC 2 OTHERS
346M
1.78
1.60
RE-INPERENCE"
346M
32.14
2.17
1.62
6.92
0.273
ACDioLDDM--Fui DD2-- (LIU AL., 2024c)
AC OTHERS
1.64
INFERENCE"
712M
33.18
2.12
1.54
8.29
0.281
UNI-DIRECTIONAL
AUDIOGEN BASU (KREUX 2023)
AC
285M
2.84
2.14
DISCRETE
AUDIGGEN LARGE (KRRUK ET AT., 2023)
AS AC OTHERS
IB
1.82
1.69
UNIAUDIO (YANG ET AL., 2023A)
AS AC + WC 10 OTHERS
3.12
2.60
CONTINUOUS
AUDIONTP BASE
AC WC
193M
18.52
2.28
1.29
9.42
0.308
AUDIOMNTP BASE
AC + WC
193M
14.81
1.68
1.16
9.67
0.336
AUDIOMNTP LARGE
AC + WC
462M
14.30
1.22
1.17
9.81
0.341
the similar KL, IS and CLAP scores compared the lead- ing diffusion models We believe performance can be improved by further scaling up the model and data, and we leave this for future work due to computational constraints.
Table Human Evaluation Results. We use AudioMNTP Large to compare to other models.
NON-SPRECH
SPEECH
METHOD
REL
OVL.
REL
OVL.
REFERENCE
4.06 L 1.09
3.87 1.1
4.47 0.83
4.61 1 0.95
5.2. Subjective evaluation
RI-DIRECTIONAL
2
3.10
3.32
3.17 2 1.14
3.01 0.91
We compare with the best existing decoder-only solution, AudioGen, and two leading diffusion models, AudioLDM 2 and Tango 2. Table 2 shows that AudioMNTP is signifi- cantly better than AudioGen in both speech and non-speech categories, and approaching the performance of Tango In both categories, our REL scores lag behind those of Tango 2; however, Tango 2 is bi-directional model and utilizes the preference optimization dataset to enhance text-audio alignment. Interesting, we find that our method is especially good at generating authentic speech, indicated by the high- est OVL score the speech category, potentially due to the incorporation of MLM, the de facto standard speech pre-training (Liu al., 2024a; Baevski et al.. 2020)13
TANGO
3.95
3.80 0.99
4.20 1 0.79
3.15 1 0.94
UNIDDIRECTIOAL
AUDIOGEN
3.02
0.97
AUDIOMNTP 3.79 1 1.02
3.46
1.10
3.91
0.94
3.69 1. 0.88
MNTP significantly outperforms next-token prediction. Table I shows that AudioMNTP Base outperforms our base- line AudioNTP Base by large margin, with 26%, 10%, and 9% relative improvements on FAD, KL and CLAP scores respectively, demonstrating the effectiveness of MNTP.
Scaling up MNTP reaches SOTA performances on FD and FAD. We scale up the model to produce the 462M Au-
dioMNTP Large model. The scaling boosts the performance across most of the metrics, including the 27% and 1.5% rel- ative improvements on FAD and CLAP scores compared to AudioMNTP Base. Our model is intrinsically less expres- sive compared to SOTA diffusion models. (1) our model is uni-directional and (2) our model is significantly smaller, i.e. 866M Tango 2. However, AudioMNTP Large demonstrates the best FD and FAD scores across all the models, and reach
5.3. Ablation studies
We ablate the components AudioMNTP in Table 3 and visualize them in the Appendix E. Note that different mask-
We place 5th, 2nd, and 4th based on the KL, IS, and CLAP scores, respectively. Most of the scores are close. Tango 2's espe- cially high CLAP score is attributed to the additional preference dataset Audio-Alpace (Majumder al., 2024).
13 "MLM is effective in both discriminative (Baevski et al., 2020) and generative (Liu et al., 2024a) speech pre-training.
each 10-second clip, method only 256 takens, whereas AudioGin uses 5,000 tokens.
6

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
Table Ablate the components of MNTP with the Base configuration. INIT. mcans initialization; POS. EMB. means positional embedding. By default, the masking schedule is the mixture of the normal and truncated normal distribution, as described in Section 4. 'indicates the masking schedule used in MAR (Li al., 2024b), a truncated normal distribution over [0.7, 1j. Tindicates the fixed masking ratio at 0.7. Findicates masking schedule of uniform distribution over [0, 1].
ID CAUSAL
MAR INIT.
ZERO
GAUSSIAN MASK
PREDICT
PREDICT
TARGET
PREDICT MASKED
FD
FAD
KL
1S
CLAP t
MASK
TOKEN
NEXT
SKIP
POS. EMB.
BASELING AUDIONTP
(A)
X
X
X
X
X
X
18.52
2.28
1.29
9.42
0.308
(B)
X
17.15
2.13
1.25
9.45
0.321
(C)
1
X
X
X
16.78
1.97
1.28
9.33
0.333
(D)
X
X
X
15.15
1.82
1.18
9.22
0.324
(E)
X
X
X
X
X
16.62
1.77
1.32
9.25
0.315
(F)
/
X
X
X
24.45
4.49
1.68
5.87
0.229
AUDIOMNTP
(G)
/
/
x
X
/
V
14.81
1.68
1.16
9.67
0,336
(1)
/
X
X
15.55
1.89
1.16
9.56
0.327
())
/
X
X
20.82
3.12
1.55
8.13
0.301
())
/
X
X
14.85
1.70
1.17
9.62
0.335
TOPLINE AUDIOMAR
(K)
X
x
x
X
X
14.86
1.35
1.17
9.75
0.346
ing strategies, such as masking or dropping, favor different masking schedules. We explored various schedules and report the best result.
due to its computational efficiency. However, our proposal is independent of the exact realization of the masking. Note that other masking strategies, such zero masking, do work well with our final masking schedule, as indicated in Table 3 (F), In contrast, dropping tokens remains effective with different masking schedules in subsequent studies.
Combining MLM and NTP. Table 3 (B) naively com- bines MLM with NTP by using the bidirectional AudioMAR as the initialization and fine-tuning with next-token predic- tion on audio data. That is, Table 3 (A) uses the image MAR as the initialization, but Table 3 (B) uses AudioMAR as the initialization, which additionally learns the MAR task audio data. Table 3 (B) shows that MLM on audio data improves performance, but only slightly.
Migrate MLM into NTP: masked prediction. We study the effectiveness of skip-token prediction, our form of masked prediction, which we hypothesize to benefit next- token prediction. Table 3 (G), AudioMNTP, justifies our hypothesis and outperforms all the rows on all metrics. We ablate mixture of distributions masking schedule to ify the robustness of MNTP. We replace the schedule by the one used in MAR Table 3 (H) shows that our MNTP task, even with the masking schedule of MAR, remains competitive compared to Table 3 (A) and Table 3 (B), only slightly underperforming Table 3 (G). This demonstrates MNTP's robustness the masking schedule. Next, we ablate the target positional embedding used for skip-token prediction. Table 3 (I) is significantly worse than Table 3 (G) on all metrics, suggesting that different future positions conflict each other. Table 3 (I) is even worse than Table 3 (E), which simply predicts the immediate next token. The results demonstrate the importance of the target positional embedding.
masking. The bidirectional MLM is essentially not designed for the causal decoding. We investigate incorporating input masking into next-token prediction. Table (C) follows MAR to apply zero masking, but uses the causal decoder for next-token prediction. Ta- ble 3 (D) tries the Gaussian noises as the masking, since the numerical value space more aligned to that of the clean tokens, which sampled from the same Gaussian by the diffusion head. Appendix E Figure 8 (C) illustrates the idea. Table 3 (E) tries dropping the tokens as a type of masking, which significantly reduces the compute cost, as shown by Appendix Figure& 8 (E). We predict the next tokens directly on top of the visible tokens instead of the masked tokens, and the masked tokens are completely removed. Given that our masking ratio is typically above 70% (Li al., 2024b: Huang et al., 2022), dropping only requires less than half of the original computation during training. According to Table 3 (C), (D), and (E), incorporating input masking to next-token prediction improves the performance on most metrics compared to Table 3 (A), including FD. FAD and CLAP, except for IS. We proceed with the dropping scenario
Migrate MIL into NTP:
Initialization. We replace the image MAR initialization by random initialization for AudioMNTP. Table 3 (J) under- performs Table 3 (G) on all metrics, but slightly, suggesting that the image MAR initialization is helpful but not neces-
"We find that MAR's 's truncated normal distribution on |0.7, 1.0] works poorly, but simply shifting it to the left to [0.55, 0.85) works reasonably well.

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
Table Ablating different MLP diffusion head sizes with the AudioMNTP Base configuration. The dimension of each MLP layer is 1024. denotes the layers of the MLP. The relative size of the heads in the whole model is 5.4%, 10.7%, and 17.4% for = 1,3,0 6, respectively. 5 denotes the inference speed, measured by seconds per 10-second audio with batch size 40 an NVIDIA V100 GPU.
Table 6. Comparing MAR and MNTP on different decoding scenarios. The landom-order decoding is the default decoding method used in MAR. The steps means the number of decoding steps. 64 is the default decoding step of MAR. 256 is the total sequence length the 10-second audio. AudioMAR supports parallel decoding it can roduce the decoding steps, denoted by 1 The bold denotes the best performance with the causal decoding. The underline denotes the globally best performance.
MLP
FD L.
FAD J.
KL 4
IS
CLAP f
S
SIZE
ID
MODEL
STEPS
FD J
FAD
KL
15
CLAP
9.84M
16.97
1.68
1.20
8.87
0.321
3.22
RANDOM DECODING
20.34M
14.81
1.68
1.16
9.67
0.336
3.93
(A)
64
14.86
1.35
1.17
9.75
0.35
36.09M
14.74
1.21
1.15
9.78
0.337
4.84
(B)
AUDIOMAR
256
15.02
1.57
1.13
9.47
0.347
CAUSAL DECODING
Table 5. Comparing the text embedding with the Base configura- tion. EMB. denotes embedding.
(C)
64
21.43
2.17
1.17
7.31
0.287
AUDIOMAR
(D)
256
16.32
1.84
1.14
9.09
0.317
AUDIONTP
256
18.52
2.28
1.29
9.42
0.308
TEXT EMB.
FD 1
FAD 4
KL
1S
CLAP f
AUDIOMNTP
256
14.82
1.68
1.16
9.67
0.336
CLAP FLAN-T5
14.82
1.68
1.16
9.67
0.336
CLAP
15.94
1.75
1.29
8.36
0.297
FLAN-TS
16.43
1.75
1.37
8.22
0.285
dioMNTP under causal decoding. AudioNTP and AudioM- NTP are intrinsically developed for causal decoding, while AudioMAR can perform causal decoding by unmasking from left to right. Causal decoding degrades the perfor- mance of AudioMAR when comparing Table 6 (A) and (C). Interesting, by increasing the decoding steps as shown by Table (D), AudioMAR works much better, and even sur- passes the naive AudioNTP in Table (E). However, Table 6 (F) indicates that AudioMNTP is the best in most metrics in the causal decoding scenario
sary for MNTP to work.
The size of the MLP diffuuson head. Table 4 shows that the size of the diffusion head matters. Increasing the head from 1 layer to layers significantly improve the perfor- mances on most metrics. Further increasing to 6 layers significantly decrease the FAD score. However, with the larger MLP size, the inference time increases. Our results highlight that the size of the diffusion head is an important hyperparameter for balancing performance and efficiency during trade-off's
6. Conclusion
The ensemble text embedding. Table shows that con- catenating CLAP and FLAN-T5 yields the best performance, as both embeddings provide complementary information to the model, and neither individual embedding can match the performance of the joint embeddings.
We propose AudioNTP and AudioMNTP. a high- performing framework for audio language modeling. Our results demonstrate that continuous-valued tokens are com- petitive for audio language modeling, yielding 20% improve- ments over the discrete token- based audio language models. Our masked next-token prediction (MNTP) further vali- dates the benefits of applying masked language modeling concept to causal language modeling on continuous data, achieving another 20% improvements over next-token pre- diction while preserving the causality during inference. By combining continuous-valued tokens with MNTP, our ap- proach achieves SOTA audio generation quality, rivaling the leading latent diffusion models and reviving the language modeling approach for sound generation. Moreover, our models are significantly smaller than most existing solutions, showcasing the efficiency and effectiveness of the proposed learning techniques. These findings point toward promis- ing direction for directly generating high-fideling audio with the streamable and scalable large language models.
5.4. MAR vs. MNTP vs. Next-Token Prediction
MNTP is approaching MAR. First, Table 6A indicates that AudioMAR yields the best results on most metrics ex- cept for FD. This is expected, as it is a bidirectional model with the full context. However, compared to AudioNTP, AudioMNTP significantly bridges the gap and achieves per- formance comparable that of AudioMAR on IS and CLAP score. AudioMNTP even surpasses AudioMAR on FD and KI., suggesting that MNTP is a competitive task for learning causal language modeling on continuous data.
The effectiveness of MNTP in causal decoding. MAR is competitive, but its use of future context does not sup- port streaming in the LLM framework well. We verify this hypothesis by comparing AudioMAR, AudioNTP, and Au-
We verified the idea of simply applying MLM to Transformer decoder Table 3 (C) and (D), which are worse than AudioMNTP.
8

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
Impact Statement
Baevski, A., Zhou, Y.,, Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information process- ing systems, 15:11244412460 2020.
Scaling with Large Language Models (LLMs). Our work demonstrates that causal language modeling can pro- duce high-fidelity audio even with small models and limited data. A natural next step is to scale up billions of pa- rameters like LLMs and incorporate larger datasets, such as A With this scaling, our approach has the potential to set new SOTA in streamable sound generation.
Baevski, A., Babu, A., Hsu, W.-N., and Auli, M. Efficient self-supervised learning with contextualized target repre- sentations for vision, speech and language. In Interna- tional Conference on Machine Learning, pp. 1416-1429, PMLR, 2023.
Merge with other Modalities in LLMs. Since our method only requires changing the prediction head of the Transformer decoder, it can be easily integrated into existing multi-modality LLM training to jointly learn with other modalities such as text and image (Sun et al., 2024). Compared to post-synthesizing with the external vocoders (Huang et al., 2024; Liu et al., 2023b; Wang et al., 2024b), the end-to-end approach avoids error propagation and allows audio data to interact directly and losslessly with other modalities in single model. Audio generation also benefits from the LLM scaling with other modalities.
Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., and Chen, M. Efficient training of language models to fill in the middle. arXiv preprint vrXiv:2207.14255, 2022.
Bertin-Mahieux, T., Ellis, D. P., Whitman, B., and Lamere, P. The million song dataset. In Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR 2011), 2011.
Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Roblek, D., Teboul, O., Grang- ier, D., Tagliasacchi, M., et al. Audiolm: language modeling approach to audio generation. IEEE/ACM trans- actions on audio, speech, and language processing, 31: 2523-25333 2023.
Efficient Inference. Since we primarily adhere to the stan-
dard Transformer decoder, our model naturally benefits from resources developed for this standardized architecture, in- cluding the K V-cache (Pope al., 2023) and PagedAtten- tion (Kwon et al., 2023). Furthermore, our model learns to predict any future timestamp given any subset of past infor- mation. Theoretically. one can devise a parallel decoding algorithm by manipulating the target positional embedding. With these techniques, combined with our inherently stream- ing capability, our method can potentially establish a new SOTA in efficient, high-fidelity sound generation.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Chang. H. Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11315-11325, 2022.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Chen, H. Xie, W., Vedaldi, A., and Zisserman, A. Vg- gsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020a.
Aghajanyan, A., Huang, P.-Y. B., Ross, C., Karpukhin, V., Xu, H., Goyal, N., Okhonko, D., Joshi, M., Ghosh, G., Lewis, M., and Zettlemoyer, L. Cm3: A causal masked multimodal model of the internet. ArXīv, abs/2201.07500, 2022. URL https: /
Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pixels. In International conference on machine learning, pp. 1691 1703. PMLR, 2020b.
aap.apii....... wwt.:esemaniicchhcceeeeee Txg/Corpus : 246035820.
Agostinelli, A., Denk, I., Borsos, Z., Engel, J., Verzetti, M.. Caillon, A. Huang, Q., Jansen, A., Roberts, A., Tagliasacchi, M., et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325. 2023.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
l'We currently rely only on AudioCaps and WayCaps, totaling about 1,000 hours of audio.
9

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. language models. Journal
Fried, D., Aghajanyan, A., Lin, J., Wang. S., Wallace, E., Shi, F,, Zhong, R., Yih, S., Zettlemoyer, I.., and Lewis, M. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2023.
Scaling instruction-finetune of Machine Learning Research, 25(70):1 2024.
Chung, Y.-A. and Glass, J. Improved speech representations with multi target autoregressive predictive coding. In Pro- ceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2353-2358, 2020.
Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A.. Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., and Défossez, A. Simple and controllable music generation. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Ghosal, D., Majumder, N., Mehrish, A., and Poria, S. Text- to-audio generation using instruction tuned LLM and latent diffusion model. CoRR, abs/2304. 137311, 2023, doi: 10448550/ARXIV22344..333
Défosse7, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression. Transactions on Ma- chine Learning Research, 2023.
URL https:// doi doi.org/10 /158/550//iiii 2304. 13731.
Deshmukh, S., Elizalde, B., and Wang. H. Audio retrieval with wavtext5k and clap training. In INTERSPEECH 2023, pp. 2948-2952, 2023. doi: 10.21437/nntesspecch. 1136-1136.
Gong, Y., Chung, Y.-A., and Glass, J. Ast: Audio spec- trogram transformer. In Interspeech 2021, pp. 571-575, 2021. doi: 10.2 1022447//eersseechh222---5
Guo, Z., Leng, Y., Wu, Y., Zhao, S., and Tan, X. Prompttts: Controllable text-to- speech with text descriptions. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2023.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Drossos, K., Lipping, S., and Virtanen, T. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 736-740. IEEE, 2020.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al. Glam: Efficient scaling of language models with mixture-of experts. In International Conference on Machine Learn- ing. pp. 5547-5569 PMLR, 2022.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https: openreview net forumoormm/mmm wit.didddddBBtmmm.m
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 12873-12 12883, 2021.
Hershey, S., Ellis, D. P., Fonseca, E., Jansen, A., Liu, C., Moore, R. C., and Plakal, M. The benefit of temporally- ICASSP
strong labels in audio event classification. 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 366-370. IEFE, 2021.
Fonseca, E., Pons Puig. J., Favory,
X., Font Corbera, F., Bogdanov, D., Ferraro, A., Oramas, S., Porter, A., and Serra, X. Freesound datasets: a platform for the creation of open audio datasets. In Hu X, Canningham SJ, Turn- bull D, Duan Z, editors. Proceedings of the 18th ISMIR Conference; 2017 oct 23-27; Suzhou, China. [Canada]: International Society for Music Information Retrieval; 2017. 486-93. International Society for Music Informa- tion Retrieval (ISMIR), 2017.
Ho, J.. Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. In Advances in Neural Information Processing Systems, 2020.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute- optimal large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, pp. 30016-30030, 2022.
Fonseca, E., Favory, X., Pons, J.. Font, F., and Serra, X. Fsd50k: an open dataset of labelll sound events. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 30:829 8522, 2021.
10

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
Hou, L.,, Pang, R. Y., Zhou, T., Wu, Y., Song, X., Song, X., and Zhou, D. Token dropping for efficient bert pre- training. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3774-3784, 2022.
2019, pp. 2350-2354, 2019. doi: 10.214477/nnerspeech. 2019-2219.
Kim. C. D., Kim, B., Lee, H., and Kim, G. Audiocaps: Gen- erating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long and Short Papers), pp. 192, 2019.
Huang. J., Ren, Y., Huang, R., Yang, D., Ye. Z., Zhangg, C., Liu, J., Yin, X., Ma, Z., and Zhao, Z. Make-an- audio 2: Temporal-enhaacee text-to-audio generation. CoRR, abs/2305. 185.184775 18474, 2023a. doi: 10.48550/ARXIV. 2305.18474, URL https: http:://ooo....g... 00..44/55 arXiv. 2305. 18474.
Kingma, D. P. and Welling, M. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banjf. AB, Canada, April 14-16, 2014, Conference Track Proceedings. 2014.
Huang, P.-Y., Xu, H., Li, J., Baevski, A., Auli, M., Galuba, W., Metze, F., and Feichtenhofer, C. Masked autoen- coders that listen. Advances in Neural Information Pro- cessing Systems, 33-28708-287200 2022.
Ko, T., Peddinti, V., Povey. D., Seltzer, M. L.,, and Khu- danpur, S. study on data augmentation of reverberant speech for robust speech recognition. In 2017 IEEE in- ternational conference acoustics, speech and signal IEEE, 2017.
Huang, R., Chen, F., Ren, Y., Liu, J., Cui, C., and Zhao, Z. Multi-singer: Fast multi-singer singing voice vocoder with large-scale corpus. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 3945-3954, 2021.
processing (ICASSP), pp.
Kong, J., Kim, J., and Bae, J. Hifi-gan: Generative ad- versarial networks for efficient and high fidelity speech synthesis. Advances in neural information processing systems, 33:1 17022-17033, 2020.
Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., Ye, Z., Liu, J.. Yin, X., and Zhao, Z. Make-an- an-audio: Text-to- audio generation with prompt-enhanced diffusion models. In International Conference Machine Learning, pp. 13916-13932 PMLR, 2023b.
Kong, Z., Goel, A., Badlani, R., Ping, W., Valle, R., and Catanzaro, B. Audio flamingo: A novel audio lan- guage model with seo-shot learning and dialogue abilities. arXiv preprint arXiv:2402.01833 2024a.
Huang, R., Li, M., Yang, D., Shi, J., Chang, X., Ye, Z.,, Wu, Y., Hong. Z.. Huang, J., Liu, J., et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 23802-23804, 2024.
Kong, Z.,, Lee, S.-g., Ghosal, D., Majumder, N., Mehrish, A., Valle, R., Poria, S., and Catanzaro, B. Improving rext-to-audio models with synthetic captions. In Proc. SynData4GenAl 2024, pp. 1-5, 2024b.
Kreuk, F., Synnaeve, G., Polyak, A., Singer, U., Défossez, A., Copet, J., Parikh, D., Taigman, Y., and Adi, Y. Audio- gen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenRe- view.net, 2023. URL htt.ttts:/// /ttpaps::///ppeeeiieeeeee www.eviiwwnn... net/ forum? wwt.middccccccccc..
Kahn, J., Riviere, M., Zheng, W., Kharitonov, E., Xu, Q., Mazaré, P.-E., Karadayi, J., Liptchinsky, V., Collobert, R., Fuegen, C., et al. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673. IEEE, 2020.
Kwon, W., Li, 7., Zhuang, S., Sheng, Y., Zheng, I.., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Sym- posium on Operating Systems Principles, pp. 611-626, 2023.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Kenton, J. D. M.-W. C. and Toutanova, L. K. Bert: Pre- training of deep bidirectional transformers for language understanding. In Proceedings of naacl-HLLT, volume 1. Minneapolis, Minnesota, 2019.
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mo- hamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, pre-training for natural language generation, translation, and com- prehension. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. (eds.), Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp-
L. BART: Denoising sequence-to-sequenee
Kilgour, K., Zuluaga, M., Roblek, D., and Sharifi, M. Fréchet audio distance: A reference- free metric for eval- uating music enhancement algorithms. In Interspeech

---

Generative Audio Language Modeling with vontinuous-valued Tokens and Masked Next-Token Prediction
7871-7880, Online, July 2020. Association for Compu- tational Linguistics. doi: 10.186655/11/2022acllmmmm 703. URL httppss//aalatthoooggoo 1ol-main.700
Majumder, N., Hung, C., Ghosal, D., Hsu, W., Mihalcea, R., and Poria, S. Tango 2: Aligning diffusion-based text-to- audio generations through direct preference optimization. CoRR, 0as/2404.099566 2024. doi: 10.48550/ARXIV. 2404.09956. URL https::///// wwoii.org/ae 049//.45500 48550/ arXiv, 2404. 09956.
Li, T., Chang, H., Mishra, S., Zhang. H., Katabi, D., and Krishnan, Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattemn Recognition, pp. 2142-2152, 2023.
Martín-Morató, I. and Mesaros, A. What is the ground truth? reliability of multi-annotator data for audio tagging. In 2021 29th European Signal Processing Conference (EUSIPCO), pp. 76-80. IEEE, 2021.
Li, T., Katabi, D., and He, K. Return unconditional generation: self-superrised representation generation method. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a.
Mei, X.,, Meng, C., Liu, H., Kong, Q. Ko, T., Zhao, C., Plumbley, M. D., Zou, Y., and Wang, W. Wavcaps: A chatgpt- weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Process- ing. 2024.
Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregres- sive image generation without vector quantization. arXiv preprint arXiv:2406. 1838, 2024b.
Lipman, Y., Chen, R. T., nem-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. In The Eleventh International Conference on Learning Repre- sentations, 2023.
Meng. L., Zhou, L., Liu, S., Chen, S., Han, B., Hu, S., Liu, Y., Li, Zhao, S., Wu, X., et al. Autoregressive speech synthesis without vector quantization. arXiv preprint arXiv:2407.0855/, 2024.
Liu, A. H., Le, M., Vyas, A., Shi, B., Tjandra, A., and Hsu, W.-N. Generative pre-training for speech with flow matching. In The Twelfth International Conference on Learning Representations, 2024a.
Mesaros, A., Heittola, T., and Virtanen, T. Tut database for acoustic scene classification and sound event detection. 2016 24th European Signal Processing Conference (EUSIPCO), pp. 1128-1132. IEEE, 2016.
Liu, H., Chen, Z. Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D. Audioldm: text-to-audio generation with latent diffusion models. In Proceedings of the 40th International Conference on Machine Learning, pp. 21450-2141 21474, 2023a.
Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In Proceedings of the International Conference on Machine Learning, 2021.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint arXiv: (0kYv.8807.03337 0077003748 2018.
Liu, H., Li, C., Wu, Q.. and Lee, J. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024b.
Peng, P., Huang, P.-Y., Li, S.-W., Mohamed, A., and Har- wath, D. VoiceCraft: Zero-shot speech editing and text-to- speech in the wild. In Ku, L.-W., Martins, A., and Sriku- mar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. -24422 Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: (0.18653//1/2222.acac / //aclanthology wrg/2024//2222 acl long- 673.
Liu, H., Yuan, Y., Liu, X., Mei, X., Kong, Q. Tian, Q., Wang, Y., Wang, W., Wang, Y., and Plumbley, M. D. Audioldm 2: Learning holistic audio generation with self- supervised pretraining. IEEE ACM Trans. Audio Speech Lang. Process., 32:2871-2888 2024c. doi: 10.1 TASL.P URL https: wtw.loooog/1..... 1109/TASIE 220...........2... 3399607.
URL https:
Liu, X., Zhu, Z. Liu, H., Yuan, Y., Cui, M., Huang. Q., Liang, J., Cao, Y., Kong, Q., Plumbley, M. D., et al. Wavjourney: Compositional audio creation with large lan- guage models. arXiv preprint arXiv:2307.14335 2023b.
Piczak, K. J. Ese: Dataset for environmental sound classi- fication. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 1015-1018, 2015.
Loshchilov. I. and Hutter, F. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2017. URL httpps//appii semant www.cccc.hhooccoccoccocco.ceeeee semaatttccccool.rrrrrroo/oooopss
Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5:606-624, 2023.
12

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., and Collobert, R. Mls: A large-scale multilingual dataset for speech research. In Interspeech 2020, pp. 2757-2761, 2020, doi: 10.21457/nterspeecc.200 20w.eeccchh002222260
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Linzen, T., Chrupala, G. and Alishahi, A. (eds.), Proceedings of the 2018 EMNLP Workshop BlackboxNI.P. Analyz- ing and Interpreting Neural Networks for NLP. pp. 353- 355, Brussels, Belgium, November 2018. Association for Computationall Computational Linguistics. Linguitttio doi: 10.18653 org -14--446
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
URL www.h:ttps///c httttss://cllaatolllllaoo
Wang. X., Li, Y., Fu, C., Xie, L., Li, K., Sun, X., and Ma, L. Freeze-omni: A smart and low latency speech-to- speech dialogue model with frozen Ilm. arXiv preprint arXiv:2411.00774, 2024a.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image symthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Wang, Y., Huang, R., Song, S., Huang, Z., and Huang, G. Not all images are worth 16x16 words: Dynamic transformers for efficient image recognition. Advances in neural information processing systems, 34:1 34:1:44:01101 1960-11973, 2021.
Salamon, J., Jacoby, C., and Bello, J. P. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 1041-1044, 2014.
Wang, Y., Wang, X., Zhu, P., Wu, J., Li, H., Xue, H., Zhang, Y., Xie, L., and Bi, M. Opencpop: A high-quality open source chinese popular song corpus for singing voice synthesis. In Interspeech 2022, pp. 4242-4246, 2022. doi: 10.2 10.21457/mterspeccch22222 2022-448.
Sennrich, R., Haddow, B., and Birch, A. Edinburgh neural machine translation systems for wmt 16. In Proceedings of the First Conference on Machine Translation: Volume 2. Shared Task Papers, pp. 371-376, 2016.
Shi, Y., Bu, H., Xu, X., Zhang, S., and Li, M. Aishell- 3: A multi-speaker mandarin its corpus. In Interspeech 2021, pp. 2756-2760, 2021. doi: 10.214377Anterspeechh. 2021-755.
Wang, Z., Tai, Y.-W,, and Tang, C.-K. Audio-agent: Lever- aging 1ims for audio generation, editing and composition. arXiv preprint UrXiv:2410.03333 2024b.
Wei, C., Mangalam, K., Huang, P.-Y., Li, Y., Fan, H., Xu, H., Wang, H., Xie, C., Yuille, A., and Feichtenhofer, C. Dif- fusion models masked autoencoders. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 16284-16294, 2023.
Sun, Y., Bao, H., Wang, W., Peng, Z., Dong, L.,, Huang. S., Wang, J., and Wei, F. Multimodal latent language modeling with next-token diffusion. arXiv preprinl 21Xiv:241..08635, 2024.
Team, G., Anil, R., Borgeaud, S., Alayrac, J., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K. et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Wu, S., Fei, H., Qu, L.., Ji, W., and Chua, T.-S. Next-gpt: Any-to-any multimodal 11m. In Forty-first International Conference on Machine Learning, 2024.
Wu, Y., Chen, K., Zhang, T., Hui, Y., Berg- King-Kirkpatrick, T., and Dubnov, S. Large-scale contrastive language-audio pretraining with feature fusion and seyword-to-caplee augmentation, ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEBE, 2023.
Tschannen, M., Eastwood, C., and Mentzer, F. Givt: Gen- erative nafnite-vocabulee transformers. In European Conference on Computer Vision, pp. 292-309. Springer, 2025.
Tzanetakis, G. and Cook, P. Musical genre classification of audio signals. IEEE Transactions on speech and audio processing, 10(5):293-302 2002.
Yang, D., Tian, J., Tan, X., Huang, R., Liu, S., Chang, X., Shi, J.,, Zhao, S., Bian, J., Wu, X., Zhao, Z., Watanabe, S., and Meng, H. Uniaudio: An audio foundation model to- ward universal audio generation. CoRR,
Vaswani, Attention is all you need. Advances in Neural Information Processing Systems, 2017.
2023a. doi: 10, 10.4855/AARXI1233100007 / //doi. wag...or/// /0.
URL https: 2310 00704.
Vyas, A., Shi, B., Le, M., Tjandra, A., Wu, Y.-C., Guo, B., Zhang, J., Zhang, X., Adkins, R., Ngan, W., et al. Au- diobox: Unified audio generation with natural language prompts. arXiv preprint arXiv:2312. 15821, 2023.
Yang, D., Yu, J., Wang, H., Wang, W., Weng. C., Zou, Y., and Yu, D. Diffsound: Discrete diffusion model for
13

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
text-to-sound generation. IEEE/ACM Transactions Audio, Speech, and Language Processing, 31:1720-1733, 2023b.
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. A survey multimodal large language models. arXiv 2023.
preprint arXiv:2306.
Yu, L., Simig, D., Flaherty, C., Aghajanyan, A., Zettlemoyer, L., and Lewis, M. Megabyte: Predicting million-byte se- quences with multiscale transformers. Advances in Neu- ral Information Processing Systems, 36:78808-78822 2023.
agar, A. and -iikojja,,, M. Slovene SuperGLUEE benchmark: Translation and evaluation. In Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi. S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Odijk, J., and Piperidis, S. (eds.), Proceedings of the Thirleenth Language Resources and Evaluation Confer- ence, pp. 2058-200800 Marseille, France, June 2022. Eu- ropean Language Resources Association. URL ht htsps: //aclanthology. org/ 2022 1.22-1.221.
Zaremba, W. Sutskever, I., and Vinyals, O. Recurrent neural network regularization. ArXiv, abs/1409.2329) 2014. URL https::/app........ http://///.....saa.ss.... sew.nt. cesmaniccscoole orr//oorpuSDD:171.97..
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
Zen, H., Dang, V., Clark, R., Zhang, Y., Weiss, R. Jia, Y., Chen, Z., and Wu, Y. Libritts: A corpus derived from librispeech for text-to-speech. In Interspeech 2019, pp. 1530, 2019, doi: 10.2 2021443/nnerppecch229992222
Zhang, X., Lyu, X., Du, Z., Chen, Q. Zhang, D., Hu, H., Tan, C., Zhao, T.. Wang, Y., Zhang, B., et al. Intrin- sicvoice: Empowering llms with intrinsic real-time voice interaction abilities. arXiv preprint arXiv:2410.08035, 2024.
Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and dif- fuse images with one multi-modal model. arXiv preprint a4Xiv:2408. 11099 2024.
Ziv, A., Gat, Lan, G. L., Remez, T., Kreuk, F., Copet, J.. Défossez, A., Synnaeve, G., and Adi, Y. Masked audio generation using single non-autoregressive trans- former. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //openreview net. fere//forum22 wiw.rrrmmmmiiinniiiiini.
14

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
A. Data
We compare the datasets used by our method to the existing systems. We only use AudioCaps (Kim et al., 2019) and WavCaps (Mei al., 2024), highlighting the data efficiency of our methods. Audios longer than 10 seconds are randomly cropped into 10 second. That is, the number of the text-audio pairs are the same after the pre-processing. To speedup the training, we pre-extract the text embedding and the continuous-valle audio tokens.
Table The datasets used by the TTA systems. The AS, AC, WC stand for AudioSet (Gemmeke et al., 2017). AudioCaps (Kim et al., 2019), WavCaps (Mei et al., 2024), respectively. WavCaps composed of FreeSound (Fonseca et al., 2017), BBC Sound Effects". " SoundBible"" " and the AudioSet Strongly-Labeles Subset (Hershey et al., 2021), augmented with the ChatGPT-gennerted text prompt.
Some systems the individual datasets in WavCaps without the natural language prompt, i.e. AudioGen.
METHOD
DATASETS
AUDIOGEN BASE (KRRUKK FT Al.,, 2023)
A$ + AC + RBC SOUND EFFECTS CLaTHO v2 (DROsSos ET Al., 2020) VGG-SOUND (CHEN AL., 2020A) FSD50K (FONSECA ET AL., 2021)
AUDIOGEN (KREUK AL.,, 2023)
MAGNET-SMALL (AUDIO"") (ZIV ET AL., 2024)
FREE To USE SOUNDS SONNISS GAME ErrECTS + WISOONDÉWEECS""
MAGNET-LARGE (AUDIO) (ZIv ET AL., 2024)
ODEON SOUND EFFECTS
TANGO (GHOSAL FT Al.., 2023)
AC
AS + AC + WC+ MUSICCAPS (AGOSTINELLI ET AL., 2023) ESC (PICZAK, 2015) URBANSOUND (SALAMON Al., 2014)
TANGO-FULL-FT (GHOSAL ET AL., 2023)
GTZAN (TZANETAKIS & COOk, 2002) MUSICAL INSTRUMENTS"
TANGO-AF&AC-PF (KONG ET AL.. 2024B)
AC AF-AUDIOSEr
TANGO 2 (MAJUMDER ET AL.., 2024)
TANGO- FULL FT DATA + AUDIO ALPACA (AA)
+ WC WAVTEXT5K (DESHMUKH ET AL.. 2023) + ADOBE AUDITION SOUND EFFECTS
MACS (MARTÍN MORATÓ MESAROS, 2021) CLOTHV2 (DRoSSOS ET AL.., 2020)
MAKE-AN-AUDIO 2 (HUANG ET AL., 2023A)
AUDIOSTOCK" EPIDEMIC SOUND" FSD50K (FONSECA ET AL.,, 202 ODFOX SOUND ESFECTS
URBANSOUND (SALAMON ET AL.., 2014) + ESC (PICZAK, 2015) TUT (MESAROS ET AL.., 2016)
AUDIOLDM22AA (LIC ET AL., 2024C) AUDIOLDM2-AC-LACC
AC
2024c)
AUDIOLDM2-HE FULLLDMM--ULLLLLEE (LIU AL.. 2024C)
+ AC+ + WC+ VGG-SOUND (CHEN BT AL., 2020A) MSD (BERTIN-MAHIEU ET AL., 201
AUDIOLDM2-FULL- FULL-LARGE (LIU GT AL.. 2024c)
AS + AC WC LIBRILIGHT (KAHN ET AL.., 2020) + LIBRITTS (ZEN ET AL.,, 2019)
(PRATAP AL., 2020) + AISHELL3 (SHI AL.., 2021) + OPENCPOP (WANG AL., 2022)
UNIAUDIO (YANG RT AI.,, 2023A)
OPENSINGER (HUANG AL., 2021) MSD (BERTIN- MAHIEUX AL., 2011)
PROMPTSPEDCH (GUO BT AL., 2023) OPENSLR OPEN SLR28F (Ko ET AL., 2017)
AUDIO-NTP BASE
AUDIO-MNTP AUDIO-MNTP LARGE
WC
B. Masking Schedules
B.1. Defining masking schedules.
We explored various masking schedules, including the truncated normal distribution used in MAR, the fixed masking ratio, uniform distribution, and our mixture of normal and truncated normal distribution. Different masking strategies, including
18 "https///oound https://sound-effectss... efffects.berrewinde www.bbbeeddcc......
"bttps://suundbible.. comm
T'hhere are music and audio versions of Magnet. The music version follows the datasets used in (Copet et al., 2024) and the audio version follows those in AudioGen.
https:/wwwwffretoosssoouuu httpr/toouusssuuussooollllllliae commall-in-one--undde
> http://deeeeettttoooooooooodeeed
saatuumennt--oundddaaaaes AudAioo is a synthetic dataset released by (Kong al., 2024b), where Audio Flamingo (Kong et al., 2024a) provides the caption for the AudioSet audio.
25 https//www.kaggle.omm/
"The Audio-Alpaca (AA) preference dataset is released by Tango 2. where a text prompt corresponds to winner audio and loser audio. is used after the standard TTA supervised training for the preference alignment.
Phtpps//audiostock.ee//
"htps//www.eepeemmisoonn.ccmmm
15

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
masking, Gaussian masking, and dropping, work best under different masking schedulex, as suggested by Table 3. By default, masking schedule represents distribution over [0, 1], where we sample masking ratio [0, for each training iteration. Then, given length-n sequence of continuous-valued tokens, we mask (drop) n x tokens. For the dropping masking strategy, the remaining sequence length is (1 r). In Figure 4, we visualize a few representative schedules we find useful in this study.
(A) Normal
0.06
(B) Truncated normal
(C) MNTP schedule: mixture of (A) normal and (B) truncated mormal
(D) MAR schedule: truncuted normal
(E) Uniform
Probability
0.00
Masking ratio
Figure 4. Visualizing the masking schedules. We first sample masking ratio from the schedule, a probability distribution over [0, 1], and then sample the masking positions in the token sequence based the ratio. In (C), we average (A) and (B) with equal weights.
B.2. Ablating the MNTP schedule.
We examine the components of the MNTP masking schedule, specifically the normal distribution and the truncated normal distribution. Table shows that the normal distribution (A) plays a more critical role than the truncated normal distribution (B), underscoring the importance of a high masking ratio during MLM. However, the normal distribution alone results in poor IS score. We hypothesize that this issue arises from a training/testing mismatch. During next-token prediction, all previous tokens are presented-a scenario that less frequently encountered during training if the focus is solely on a high masking ratio. address this, we incorporate a long-tailed distribution (B) into normal distribution by averaging them, resulting (C), our final masking schedule. As shown in Table 8, (C) outperforms (D), the MAR default masking schedule, across all metrics, demonstrating the effectiveness of our approach.
Table 8. Ablating the masking schedule of MNTP. The schedule IDs match the IDs in Figure 4.
SCHEdULE ID
FD
FAD
KI.
CLAP
(A)
15.22
1.70
1.17
8.83
0.326
(B)
15.96
2.17
1.25
8.18
0.293
14.81
1.68
1.16
9.67
0.336
(D)
15.55
1.16
9.56
0.327
C. Classifier-free guidance (CFG) and sampling temperature
Classifier-free guidance. Classifier-free guidance is method for trading off diversity in favor of fidelity. We reuse the notations in Section 3.1. During training, the input text prompt is replaced with fake latent wy with a probability of 10%. The fake latent has the same length the text prompt and is learned jointly with the whole model. During testing, we run two inferences for each text prompt: conditional generation and unconditional generation. Specifically, at each decoding position i, we get two conditioning vectors: ø = Co(w, B, "e.,..., xf-1) and : = Co(w,, 3, ...., x²-1). At each diffusion sampling step f. we then induce two noise predictions with the diffusion head: &, Mo 2 t) and
Eu = Mo (Gt, z,, t). The classifier-fre guidance realized by linear interpolating two predicted noises.
16

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
Ec + Wi (Ec Ea)
(4)
where wi E the guidance scale when decoding the position i. Intuitively, the guidance scale represents the degree of moving away from the unconditional generation. With higher wa, the generation becomes aligned with the given text prompt, albeit with reduced diversity. We do not fix guidance scale for all positions. We adopt annealing schedule in (Ziv et al., 2024). That is, given the current decoding position i E {l,..., n and an initial CFG scale wo 2 1, we gradually decrease the guidance scale by:
(1-11-111
(1
1 1---
W/ = 1 + (wo - 1) x
(5)
Intuitively, this means that the initial decoded tokens are more tailored the given condition and gradually allow for greater diversity and uncertainty. We find that this annealing schedule works better than the constant schedule and the linear schedule used in MAR (Li et al., 2024b).
Sampling temperature. Conventionally, the higher sampling temperature corresponds more diverse samples. To facilitate this behavior, we can multiply the noise scale ō in equation by temperature factor (Dhariwal & Nichol, 2021).
Results. Figure 5 demonstrates the effects of the CFG guidance scale wi and the sampling temperature т. Figure (a) shows that wp is critical for the competitive performances, and the best performance is achieved around 7. As result, we set 7 as default thorough the article. Next, we explore different sampling temperatures with wi 7 in Figure 5 (b). Figure 5 (b) shows that sampling temperatures have varying impacts on different metrics. As a result, given specific metric to optimize, it is helpful to tune the temperature. However, no single т achieves the best performance across all metrics. Therefore, we set as the default.
19,663
2,172
7,815
0.298
8.98
14.735
1.749
1.174
9.635
0.331
1.344
0.5
1199
0.5
15.
1.854
1.168
9.174
0.339
CFG scale
0.6
Temperature
84.825
1.685
1.165
9.635
1.335
0.6
14,879
1.429
1.151
9.447
0.336
14.814
1.679
1.165
9.671
0.336
14,814
1.679
1.165
9.671
0.336
0.4
0.4
14.851
1.646
1.147
4.540
0.336
15.111
1.71
1.219
9.346
0,330
1.842
9.424
0.311
1,02
1.661
9.427
Objective metries
Objective metrics
(a) Classifier-Free Guidance
Temperature
Figure 5. Ablating the (a) classifier-free free guidance scale (CFG) and the (b) temperature during the MLP diffusion sampling. The darker color represents the better performance. We use the AudioMNTP Base configuration for the ablation. In (a), the temperature is fixed 1.0; In (b) the CFG is fixed to 7.0, as suggested by (a).
D. MNTP v.s. MLM v.s. Next-Token Prediction
We compare MNTP to the closely related MLM and next-token prediction for their similarities and differences.
D.I. Comparing MNTP to MIMM
MNTP draws inspiration from MLM to improve next-token prediction. They share the same spirit learning the contextualized dependencies, where the model generates the representation from a sparse context which is predictive to the unseen data, Figure 6 illustrate the idea. In MLMM, the model predicts the unseen 2t conditioning on a masked sparse context
17

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
fee , ",,,,,,r x.). In MNTP, the model predicts the unseen x² also conditioning dropped sparse context {a a0 x² The differences include the following:
1. Directionality: MLM relies on a bidirectional model, while MNTP is designed for the causal model. As a result, the sparse context of MLM includes both the past {a", xx x²) and future {x"} context, while that of MNTP includes only the past context {x x= ,x*)
2. Mask token& & Target positional embedding: We remove the mask tokens which do not appear in the NTP decoding stage and cost computation, and predict the unseen tokens directly the position of the tokens with the additional target positional information.
3. Prediction target: We consider the right-masked span Figure as an example. ML.,, all tokens in the masked span are predicted using both left and right context, e.g., p(x x4 ....) and p(z x², 2²....). In contrast, MNTP predicts only the rightmost token using left context alone, e.g.. p(æ² .=...... To quantify task difficulty, we only list the closest observed token to the unseen token as the condition. As result, MNTP is uni-directional and relies on more
distant tokens on average compared to MLM, making it intuitively more challenging.
P(x
P(x | 129,2 xt)
Output
2
5
Pos. Emb.
2
5
6
Input æ
Target Pos. Emb.
(a) MLM
(b) MNTP
Figure 6. Comparing the differences between MLM and MNTP.
D.2. Comparing MNTP to Next-Token Prediction.
We can view MNTP as a Generalized form of Causal Language Modeling (GCLM), where the model predicts any future timestamp given any subset of past information. We illustrate the idea with an simple example in Figure 7. Given the length-5 sequence and the current token æ², all the possible prediction patterns in GCLM are enumerated Figure (b), including possible subsets of the past conditions and the possible future prediction targets. All the corresponding masking patterns are listed in Figure (a), and they are all possible masking patterns our masking schedule since we sample the masking ratio from the entire [0, 1). To be formal, we follow the notation in Section 3.1 and Section 3.2. Given unmasked sequence x {a+ ...., x²) ), without loss of generality, we pick current input token x' where i < n. We then have the past token set P = {z ...., x² " } and future token set F = {x==1_ ..., x²} For any prediction pattern in GCLM, there is past token subset P C P and target future token x E F,, we can derive at least one masking pattern = {u) ...., " which satisfies this prediction pattern by:
1 if (i, < and x P), 0 otherwise.
==1
(6)
Note that this not the only masking pattern satisfying this prediction pattern, our goal to show that the prediction pattern would be sampled and learned. This masking pattern can be sampled when setting the masking ratio = 1 --En of Since our masking schedule is a distribution over the entire [0, 1] (See Appendix B. the pattern would be sampled and learned. As a result, our MNTP at position models pa(25 P) over all æf E F and P C P. By setting P = P and x! +1, we restores the original next-token prediction with the full context.
18

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
2
Pivot 2
2a
6
7
8
10
(a) Masking pattern
(b) Condition Target
Figure 7. Visualizing the generalized causal language model. (a) shows the masking patterns. We enumerate the masking patterns with binary numbers where masked position is denoted by I (white) and unmasked position is denoted by 0 (gray). (b) shows the corresponding dropped sequence and the prediction pattern given the current gray token x. The conditions are in white and the prediction larget black.
E. Ablating the components of MNTP
We visualize the process of MNTP ablation in Figure 8. Refer Section 5.3 for the discussion.
Output
Pos. Emih.
NTP
(C)D) Drup
Target
(G) MNTP
Drop predict
Figure 8. Visualizing the differences between the ablation variants. The subfigure IDs match the IDs in Table 3.
F. Implementation
F.1. Continuous-valued audio tokenizer
Tokenize. We leverage the pre-processing pipeline of AudioLDM (Liu al., 2023a), including pre-trained variational
autoencoder (VAE) (Kingma & Welling, 2014) V (Ve, VD) and pre-trained Hifi-GAN (Kong al., 2020) vocoder G. VE and Vp are the encoder and the decoder, respectively. We first extract the 64-band Mel-spectrogram with the hop size of 10 ms from audio: m R where T 1000/10 and F = 64. Given that all our training samples are 16 kHz 10-second audios,
160000. The VAE encoder VE then encodes m into 2-D map of continuous latents E R xx where 4 is the compression level and 8 is the latent dimension. To reduce the sequence length and the training cost, we further patchify into z E 7 X x(e-p") by stacking the latents a p X p neighborhood. We set p = 4. Finally, we flatten z into a 1-D sequence x = ".!,,..,"") in a row-major order, where n x and x² E R That is, different frequency components are placed together in the flatten sequence. With the tokenization, we represent a 10-second clip by 256 continuous-vall valued tokens with the latent size 128.
19

---

Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
De-tokenize. Given the sampled, flattened tokens ā = y....y _r, J. undergo the reverse process of flatten and patchity to get the 2-D latent map ö E _ xc, The VAE decoder VD then decodes e into the sampled Mel-spectrogram m. Finally, the vocoder G synthesizes waveforms â from m.
F.2. Conditional audio language modeling
Diffusion head. We follow (Li et al., 2024b) for its diffusion process, including the cosine noise schedule, training/sampling (1000/100) diffusion steps, and the prediction target (noise). We reuse the MLP architecture introduced in (Li et al., 2024b). The MLP of the Base model has layer, and that of the Large model has 6 layer. Both versions use 1024 dimension for all the MI.P layers.
Transformer. We use the Transformer (Vaswani, 2017) implementation in VIT (Wang et al., 2021). same as MAR (Li et al., 2024b). The Base model adopts 24 layers with the hidden dimension of 768; the Large model adopts 32 layers with the hidden dimension of 1024. The model sizes are about 170M and 420M, respectively. Since we use the same architecture, we can leverage the image pre-trained weights in MAR. The image pre-trained weights help the spectrogram-basee audio models, as suggested by AST (Gong et al., 2021). We reuse only the weights of the backbone Transformer, excluding those of the diffusion head, since the latter entangled to the modality-speche token dimension.
Text prompt. We use CLAP (Wu et al., 2023) and FLAN-T5 (Chung et al., 2024) to extract the text embeddings and concatenate them as the input prompt. Our ablation study shows that using both is slightly better than using either individually, In practice, FLAN-T5 produces variable-length embeddings, which we truncate to the first 77 embeddings. Combined with single CLAP embedding and the linear projections, our text conditioning is equal to 78 tokens, which we use as the initial input prompt in the causal LM for audio generation. The positions of the text prompt do not contribute to the loss.
G. Evaluation
G.1. Objective evaluation
We evaluate our model on the AC evaluation set. Each audio is labeled by captions, and we follow AudioLDM randomly select one text description as the input prompt. We leverage the AudioLDM evaluation toolkit"" Given the generated audio and the ground-truth audio in AC, calculate several metries: Fréchet Audio Distance (FAD) (Kilgour et al., 2019), Fréchet Distance (FD), Kullback-Leibler divergence (KL), Inception Score (IS) (Liu et al., 2023a), and Contrastive Language-Audio Pretraining (CLAP) score (Huang et al., 2023b). Intuitively, lower FD and FAD indicate higher similarity to the paired ground-trutt The lower KL indicates that the set of the generated audios share the similar distribution as the ground-truth audios. The higher IS indicates that given an audio classifier, each audio is well-reoognized by the " and the classifier classifies all the generated audios into diverse classes Intuitively, these two conditions jointly imply the generated audios are diverse and authentic. Finally, the higher CLAP score indicates the better alignment between the generated audio and the input text prompt.
G.2. Subjective evaluation
We follow a similar approach to Tango (Ghosal et al., 2023), we assessed 20 generated audio samples based on text relevance (REL) and overall quality (OVL) but used 1-to-5 rating scale instead of a 1-to-100 scale. Each sample rated by at least 10 participants. We separate the evaluation into speech and non-speech categories to understand the differences in model behaviors. The speech prompts are sampled by selecting the text prompts containing man, woman, and person and people. The non-speech prompts are sampled from the remaining. Note that in the AudioCaps evaluation set, most prompts consist of both speech and non non-speech descriptions. We compare our method to the existing decoder-only solution AudioGen and two SOTA diffusion models AudioLDM 2 and Tango 2.
"/naps//gittuu..com/// com/hhahhlli//uudiodnn eval
s2 Indicated by the low entropy of the output distribution.
33 Indicated by the high entropy of the average output distribution over the generated audios.
20


Reviews:
[{"review_id": "CifaOLgLU0", "reviewer": "Reviewer_8Ca7", "summary": "This paper investigates generative audio language modeling with continuous-valued tokens. It begins with a next-token prediction approach in which each latent embedding (i.e., token) from an autoencoder is iteratively produced via a token-wise diffusion process. Building on this, the authors propose Masked Next-Token Prediction (MNTP), a strategy that predicts future tokens rather than strictly the next one. Experimental results demonstrate that MNTP not only improves both next-token and masked-token prediction but also matches or surpasses strong audio generation baselines. Furthermore, an extensive ablation study covering architecture design, training objectives, and token masking techniques, highlights the impact of each component on the model’s overall effectiveness.\n\n\n## update after rebuttal\nI appreciate the authors' response to my concerns. Although the rebuttal did not present extensive new experimental evidence on those concerns, the work’s potential impact on its field remains noteworthy, albeit not groundbreaking. Consequently, I maintain my initial overall assessment leaning towards acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful for your kind words and thoughful comments. Regarding our weaknesses, we address them in the following:\n\n# 1. Larger scale in both model size and dataset\n\nWe fully agree that exploring the scaling behavior of our proposed methods would provide valuable insights. In this current submission, our primary focus is on introducing continuous-valued tokens and the MNTP method. With smaller-scale models and datasets, we achieve SOTA performance, justifying the effectiveness of our proposal. Nonetheless, we acknowledge the importance of examining whether these methods maintain their effectiveness at larger scales.\n\nWe have preliminary evidence of scaling effectiveness, where increasing model parameters from Base to Large by approximately 140% led to notable improvements in overall performance metrics. This positive trend is noteworthy, especially given that such improvements are not universally observed, as exemplified by the comparative performance of AudioLDM-full vs. AudioLDM-full Large and Magnet small vs. Magnet large.\n\nWe agree that scaling analyses are broadly relevant in the context of large language models. We would be eager to further investigate scalability; however, the substantial increase in convergence time and resource requirements associated with training larger models and datasets means we cannot provide these results during the rebuttal period. Should our submission be accepted, we commit to including detailed scalability analyses in the final manuscript.\n\n> **Note.** As a practical consideration, training the Large model on AudioSet, similar to the approach adopted by AudioGen, requires approximately one month of computational resources utilizing around 100 V100 GPUs. Even scaling down to the Base model demands around two weeks of training. These estimates do not account for the additional resources necessary to explore even larger model configurations.\n\n\n# 2. Generalizability of MNTP to discrete tokens\n\nThanks again for the helpful comments! In this work, our primary claim emphasizes that both continuous-valued tokens and MNTP are crucial for achieving SOTA audio generation using language models, supporting MNTP's efficacy specifically for continuous data. However, extending MNTP methodologies to discrete tokens is indeed a compelling new direction and will be part of our future research efforts. We appreciate your suggestion!\n\n# 3. Typo\n\nWe will fix the typo. Thanks for the comment!\n\n\n# 4. Demo\n\nWe omitted the demo from our initial submission. Here is our [demo page](https://audiomntp.github.io/). Please take a look!\n\n---\n\nWe appreciate your valuable comments! If you feel we have adequately addressed your concerns, please consider increasing the score. If you have any further suggestions for improving our paper, please do not hesitate to let us know. Thank you!", "claims_and_evidence": "The proposed methods and evaluation criteria are relevant.", "methods_and_evaluation_criteria": "Proposed methods and/or evaluation criteria are reasonable and aligned with the problem/application.", "theoretical_claims": "I reviewed proposed methods and found them sound.", "experimental_designs_or_analyses": "I reviewed the experimental design and analyses and found them sound.", "relation_to_broader_scientific_literature": "This work contributes to improving language modeling in the audio domain and introduces innovations to further enhance generative performance, which could also be applicable or relevant to other generative modeling domains, such as image generation or text-to-speech.", "essential_references_not_discussed": "Essential references are well discussed.", "other_strengths_and_weaknesses": "Strengths:\n* The proposed Masked Next-Token Prediction (MNTP) approach is both conceptually simple and effective. It delivers stronger results than standard next-token or masked-token prediction techniques.\n*  The authors' explanations throughout the manuscript, including insightful footnotes, effectively highlight the contributions of this work and clearly distinguish it from other methods. Their detailed discussion enriches the understanding of both the proposed method and experimental analysis.\n*  The authors conducted extensive and rigorous experiments, providing comprehensive results. The detailed ablation study, covering architecture variations, training objectives, and token masking strategies, clearly illustrates how each individual component contributes to the overall performance and effectiveness of the method.\n\nWeaknesses:\n* Only two model size variants (base and large) are considered, with an ablation study limited to the size of the MLP-based diffusion module. Including additional experiments focused on scalability, both in terms of model size and dataset size, would offer deeper insights into the proposed approach’s potential at larger scales.\n* The method is currently evaluated exclusively on language modeling involving continuous-valued tokens, leaving uncertainty regarding its applicability to discrete-valued tokens. Exploring the effectiveness of MNTP when applied to token sequences generated by vector-quantized VAEs, for example, could broaden its scope and demonstrate greater versatility.", "other_comments_or_suggestions": "Page 6, line 326: Typo – “(2) our model is significantly smaller, i.e., 866M Tango 2.”", "questions_for_authors": "I don't have any other questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nfhn2H5woj", "reviewer": "Reviewer_v7YS", "summary": "This paper presents a novel approach for generative audio language modeling using continuous-valued tokens instead of discrete tokens. The key contributions include:\n\n1. Following previous works, such as masked autoregressive (MAR), which introducing continuous-valued audio tokens to replace discrete ones, improving generative modeling by preserving more information.\n\n2. Proposing a novel Masked Next-Token Prediction (MNTP) learning task that enhances next-token prediction by incorporating masked token prediction.\n\n3. Demonstrating that their approach achieves significant improvements over AudioGen and UniAudio.\n\n4. Achieving results comparable to state-of-the-art diffusion models while maintaining a more efficient and streamable transformer-based causal language modeling framework.\n\n5. Validating the approach with extensive quantitative and qualitative evaluations, including human evaluation on speech and non-speech audio generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# 1. Inference Latency Comparison with Diffusion Models\n\nWe appreciate the reviewer’s comment and will include a latency discussion in the final version. We present the latency comparison in Table A.\n\n**Table A. Latency comparison of TTA models**. Measured with **batch size = 1** on a single NVIDIA A100 using 10-second clips. RTF = inference time / 10 sec. **Bold**: best; *Italic*: second best. AudioGen Base has no reported latency.\n\n| Model               | Type           | Param | Latency (s) ↓ | RTF ↓   | FD ↓    | FAD ↓  | KL ↓   | IS ↑    | CLAP ↑   |\n|--------------------|----------------|--------|----------------|---------|---------|--------|--------|---------|----------|\n| AudioLDM 2         | Diffusion      | 346M   | 32.03          | 3.203   | 32.14   | 2.17   | 1.62   | 6.92    | 0.273    |\n| AudioLDM 2 Large   | Diffusion      | 712M   | 65.23          | 6.523   | 33.18   | 2.12   | 1.54   | 8.29    | 0.281    |\n| Tango 2            | Diffusion      | 866M   | 60.77          | 6.077   | 20.66   | 2.69   | **1.12** | 9.09    | **0.375** |\n| AudioGen Base      | Discrete LM    | 285M   | -              | -       | -       | 2.84   | 2.14   | -       | -        |\n| AudioGen Large     | Discrete LM    | 1B     | **12.4**       | **1.24**| -       | 1.82   | 1.69   | -       | -        |\n| **AudioMNTP Base** | Continuous LM  | 193M   | *13.43*        | *1.343* | *14.81* | *1.68* | *1.16* | *9.67*  | 0.336    |\n| **AudioMNTP Large**| Continuous LM  | 462M   | 15.77          | 1.577   | **14.3**| **1.22** | 1.17 | **9.81**| *0.341*  |\n\n#### **AudioMNTP v.s. Diffusion Models**\n\nAudioMNTP Base and Large achieve much lower latency and better quality than diffusion models. This is due to using a small MLP for diffusion steps, instead of repeatedly applying the full model. Over 90% of the parameters run only once per token.\n\n> The token sequence is 256 long—much shorter than the typical 1000+ denoising steps used in diffusion models.\n\n#### **AudioMNTP v.s. AudioGen**\n\nCompared to AudioGen Large, AudioMNTP gives much better quality with only a small latency increase. Both use a single-pass Transformer, but AudioMNTP replaces the single-pass categorical head with a multi-pass diffusion MLP. The added diffusion steps greatly boost generation quality with a slightly higher latency.\n\n#### **Real-Time Applications**\n\nOnly AudioGen and AudioMNTP reach near real-time generation (RTF ≈ 1). On an H100 GPU, real-time is easily achievable. Diffusion models remain too slow for such use cases.\n\n# 2. Discussion on MNTP Schedule\n\n#### **Masking v.s. Dropping**\n\nTable 3 (rows C–E) shows similar results for zero masking, Gaussian masking, and dropping:\n\n| Masking Scheme     | FD ↓  | FAD ↓ | KL ↓  | IS ↑  | CLAP ↑ |\n|--------------------|-------|-------|-------|-------|--------|\n| Zero Masking       | 16.78 | 1.97  | 1.28  | 9.33  | 0.333  |\n| Gaussian Masking   | 15.15 | 1.82  | 1.18  | 9.22  | 0.324  |\n| Dropping           | 16.62 | 1.77  | 1.32  | 9.25  | 0.315  |\n\nWe chose dropping due to its lower compute cost, which improves efficiency and scalability.\n\n#### **Max Mask Ratio**\n\nWe sample mask ratios from \\[0, 1\\]. When the ratio hits 1, we retain one random token and use the BOS token and positional embedding to predict it. We’ll clarify this—thank you for the observation.\n\n#### **Positional Encoding**\n\nWe use **absolute positional embeddings**, as both training and inference operate on 10s clips. Extending this to longer sequences using RoPE is a promising direction for future work.\n\n#### **Can you give more explanation about the design of target positional embedding?**\n\nCertainly! Traditional prediction always targets the immediate next token. **MNTP**, in contrast, predicts tokens at variable distances. For instance, if the sequence is **0, 3, 6, 9**, token 3 predicts 6, not 4. Since the offset varies, the model can't guess the target from context alone.\n\nTo help, we add a **target positional embedding** that tells the model which position to predict.\n\nIn **Figure 3**, each token predicts the next unmasked token at a variable offset. The embedding $p_t$ encodes this offset (e.g., $x^0 + p_t^3$ → $x^3$, $x^3 + p_t^5$ → $x^5$). We use absolute embeddings for $p_t$.\n\nThanks again—we agree the paper’s explanation was brief and will expand it in the revision.\n\n# 3. Demo\n\nWe didn’t include the demo in the original submission. It’s now live at [https://audiomntp.github.io/](https://audiomntp.github.io/). Please have a look!\n\n---\n\nWe truly appreciate your feedback. It helped improve the paper in concrete ways. If you feel your concerns are resolved, we’d be grateful if you could consider a higher score. Let us know if there's anything else we can improve. Thank you again!", "claims_and_evidence": "The claims made in the submission is supported by clear and convincing evidence. \n\n1. The claim that continuous-valued tokens lead to better generation quality is validated by improvements in FAD and KL divergence compared to discrete-token-based models like AudioGen. \n\n2. The effectiveness of MNTP is demonstrated through significant improvements over their baseline AudioNTP model, suggesting that masked token prediction enhances next-token prediction.", "methods_and_evaluation_criteria": "The proposed methods, including the use of continuous-valued tokens and MNTP, are well-motivated and clearly explained.\n\nThe choice of evaluation metrics (FAD, KL divergence, IS, CLAP score, and subjective human ratings) is appropriate and widely used in audio generation tasks.", "theoretical_claims": "This paper is experiment-driven research. Without any theoretical claims.", "experimental_designs_or_analyses": "The experimental designs effectively validate the proposed method in the appropriate benchmarks and comparisons.", "relation_to_broader_scientific_literature": "This study follows the continous LM training strategies in image/audio generation. Although the idea is not novel, this paper validates the effectiveness of such training strategy in audio generation domain. Futhermore, they also introduce masked next-token\nprediction (MNTP) strategy.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Weaknesses:\n\n1. The paper does not include a detailed comparison of inference latency between their model and diffusion-based methods, which would be critical for real-time applications.\n\n2. The proposed MNTP schedule needs more discussion:\n\n   (1) The authors point out that \"Recent works show that dropping instead of masking yields similar performances while greatly reduce the training cost\". Whether authors conduct experiments to support this claim?  \n   \n   (2) What is the max mask ratio for MNTP?\n\n   (3) which types of positional encoding?", "other_comments_or_suggestions": "Refer to Weaknesses parts.", "questions_for_authors": "Can you give more explanation about the design of target positional embedding? From Figure 3, I can only find the it starts from BOS token. I cannot fully understand it's value.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "idkymvTTpn", "reviewer": "Reviewer_eX1D", "summary": "The paper proposes a text-to-audio model that leverages diffusion-based designs and causal language models, named AudioMNTP. In detail, the model applies a transformer-based decoder for the next-token prediction of the feature in latent space before being forwarded into the diffusion-based structure, then follows a VAE decoder and Gan-based vocoder to reconstruct the waveform. In addition, a token masking strategy is proposed to improve the model performance. Experiments illustrate that the proposed model achieves state-of-the-art performance with a significantly lower model size.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback! Below, we address all comments and outline planned improvements.\n\n---\n\n# 1. Presentation Improvement\n\nWe agree the current version can be clearer and will revise structure, improve diagrams, and expand explanation in the final version.\n\n> If you have specific instructions on the organization, please let us know!\n\n#### **Relation between AudioNTP and AudioMNTP. Why show both AudioNTP and AudioMNTP? Are they both necessary? What is the final pipeline?**\n\nAudioNTP introduces **continuous-valued tokens**, and AudioMNTP extends it with **MNTP** training. Both are our novel contributions, so introducing both is necessary. Table 1 shows both are key to achieving SOTA audio generation with LMs. AudioMNTP—integrating both—is our final proposal. For clarity, we illustrate the full AudioMNTP pipeline below, covering training and inference.\n\n#### **Training**\n\n> Please see [Figure A](https://imgur.com/XIsMoZB.png) for the training pipeline.\n\nWaveforms are encoded into latents, masking is applied, and remaining tokens form a new sequence. A Transformer decoder performs next-token prediction on this masked sequence, guided by target positional embeddings. The LM outputs ($z$) go into a small MLP diffusion head, trained with diffusion loss to predict the next unmasked token.\n\n> See our response to **Reviewer v7YS** for more on target positional embeddings.\n\n#### **Inference**\n\n> See [Figure B](https://imgur.com/RmptxLD.png) and [Figure C](https://imgur.com/aX2ivm4.png) for step-by-step inference.\n\nGiven the BOS token and text embedding, the model generates latent token $z^0$ using one Transformer pass. Conditioning on $z^0$, the audio token $x^0$ is sampled from noise via diffusion. Then $x^0$ is used to generate $x^1$ autoregressively. After generating a chunk of tokens, they are decoded into waveform using a VAE decoder and HiFi-GAN vocoder.\n\n#### **Are all the tokens general noise in the beginning?**\n\nYes — each token is initialized with Gaussian noise and refined via the MLP diffusion head.\n\n#### **Why use MLP for diffusion?**\n\nA simple MLP keeps latency low and enables real-time generation, which current TTA diffusion models don’t support. If the diffusion head is too slow, denoising becomes a bottleneck. See our response to **Reviewer v7YS** for latency discussion.\n\n#### **Does the model mainly replace LDMs with a Transformer?**\n\nNo. Our Transformer runs once per token to produce a conditioning input for the small MLP, which does iterative diffusion. LDMs run the full model at each diffusion step, causing high latency.\n\n# 2. Does not compare to Diffusion-Based Audio LM Baselines\n\nTo our knowledge, this is the first use of continuous-valued tokens in LMs for TTA. If we missed related work, we’d appreciate pointers. The most [relevant work](https://openreview.net/pdf?id=y4LniyJIWi) cascade a discrete audio LM and a diffusion model, not a single diffusion-based audio LM. Also, it's not open-sourced.\n\nWe propose **AudioNTP** (the first diffusion-based Audio LM) and improve it with **AudioMNTP**. Table 1 compares our methods to LDMs and discrete LMs, supporting two key claims:\n\n1. **Continuous audio LMs** outperform discrete ones.  \n2. **MNTP** improves training over next-token prediction.\n\n# 3. Demos\n\nThanks for the suggestion! Please see our [demo page](https://audiomntp.github.io/). Our method outperforms AudioGen Large and AudioLDM2-Full-Large and matches Tango 2. Due to internal download limits, we can’t include more samples for the ablation. But Tables 3 & 4 support our claims.\n\n# 4. Why not scale up the datasets?\n\nOur method already achieves SOTA on standard datasets. We are scaling up and will include results in the final version if accepted (see reply to **Reviewer 8Ca7**). Thanks for the suggestion!\n\n# 5. Why split into speech vs. non-speech?\n\nSpeech is harder to generate and not a common case in TTA. Most TTA systems — including ours — can’t generate intelligible speech. Since ~50% of AudioCaps involves speech, it can obscure performance on sound. So, we split evaluation into **pure sound** and **sound + speech** (see Appendix G). We agree current naming can be confusing and will revise them. The rationale will also be added to the main text.\n\n# 6. Missing experimental details?\n\nMost details are in Appendix F and G due to page limits. Let us know if anything should be moved to the main text or is missing.\n\n# 7. AudioBox and Re-AudioLDM\n\nThese models are not public, so direct comparison is difficult. We will include discussion in the final version.\n\n# 8. Inference time\n\nLatency details will be added in the final version. Please see our response to **Reviewer v7YS** for results and discussion.\n\n---\n\nWe appreciate your thoughtful feedback! Your comments helped us improve clarity and depth. If you feel your concerns are addressed, we’d be grateful if you could consider raising your score. Let us know if anything can be further improved. Thank you!", "claims_and_evidence": "The authors claim that previous diffusion-based audio language models present bottlenecks and limitations. However, evidence supporting the effectiveness and superiority of the proposed AudioMNTP models over simpler baselines is not adequately provided. A suggestion is to provide some demos to support such a claim and also present the enhancement by the proposed methods Additionally, the introduction of both the transformer-based decoder and the overall structure is not very understandable.", "methods_and_evaluation_criteria": "The proposed system mainly applies AudioCaps and WavCaps for training. However, the availability of upscaling the dataset with numerous other audio-language datasets remains unclear. Although the author claims that the upscale of the model is limited by the computational constraints, the model is trained using 104 V100 GPUs for 5 days, such resource is already enough for any large model. The author provides various evaluation matrices to illustrate and compare the performance, however, the result including speech and non-speech categories is hard to understand, Does this mean that the proposed model can correctly generate speech contents?", "theoretical_claims": "All the theoretical claims are discussed in the paper, however, the written format of the paper could be improved, currently,y the structure is hard to follow and understand. Why does the paper actually provide the details of two models, AudioNTP and AudioMNTP? The introduction of the inference section for both models is also ambiguous. It is hard to understand how the decoder works during inference, is all the tokens are general noise in the beginning?", "experimental_designs_or_analyses": "The paper lacks most of the essential details of the experiments, such as specific training/inference parameters of each model, e.g., the inference time. After reading the whole paper, it is still hard to understand the pipeline of the system and how the model actually operates.", "relation_to_broader_scientific_literature": "The paper discusses a new idea of applying the advantage of causal language models for generative models. In addition, MAE-based training strategies are applied to improve performance.", "essential_references_not_discussed": "There are actually more text-to-audio models which have achieved better performance, such as AudioBox and Re-AudioLDM. Although it might be hard to run these models, the paper could be improved by discussing them.", "other_strengths_and_weaknesses": "The overall structure, especially the two graphs, is excessively complex and hard to understand or follow. \nThe format of the paper should also be improved, some sections are overlapped and the order of some parts should be improved. \nThe paper introduced many techniques but lacked the details of the proposed model. Currently, it is even unclear the overall pipeline of the model and how the transformer-based decoder with diffusion-head works. \nThe model did not provide any demos on the model.", "other_comments_or_suggestions": "None", "questions_for_authors": "Overall, the model provides an enhanced performance with the smaller size of the system, however, the writing of the paper needs to be improved as the current file is hard to follow. I would be happy to increase the score if the author could answer the following questions and also explain the overall pipeline of the model. \n\nQuestion 1:\nWhat is the main proposed model, why introduce both AudioNTP and AudioMNTP? Are both necessary? \n\nQuestion 2:\nWhy use MLP architecture for the diffusion-based section? Any consideration for other architectures? \n\nQuestion 3\nWhat is the overall structure of the model, is the proposed model mainly replacing the previous LDM/transformer-based feature generation model with the transformer-based decoder? \n\nQuestion 4\nCan we provide some demos of the result?  \n\nQuestion 5\nWhat is the inference time of the proposed model?\nQuestion 6\nWhy the evaluation in Table 2 being categorized into non-speech and speech, Does that mean the model can successfully generate real-speech content?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper explores generative audio language modeling using continuous-valued tokens. It starts with a next-token prediction framework, where each latent embedding from an autoencoder is generated through a token-wise diffusion process. Building on this, the authors introduce Masked Next-Token Prediction (MNTP)—a strategy aimed at predicting upcoming tokens beyond the immediate next one. 

While reviewers initially noted missing results and comparisons, the authors addressed most of these concerns during the discussion phase. Despite the authors’ responsiveness and willingness to provide additional results and clarifications, the novelty and results of the proposed approach are not convincing enough for an ICML publication. I encourage the authors to better justify their architectural and design choices, and include results from additional speech / audio domains. This will greatly improve the quality of the submission.
Decision: Accept (poster)