UALM: Unified Audio Language Model for Understanding, Generation and Reasoning
Authors: Jinchuan Tian
Sang-gil Lee
Zhifeng Kong
Sreyan Ghosh
Arushi Goel
Chao-Han Huck Yang
Wenliang Dai
Zihan Liu
Hanrong Ye
Shinji Watanabe
Mohammad Shoeybi
Bryan Catanzaro
Rafael Valle
Wei Ping
Venue: ICLR 2026 Oral (2026)
Keywords: Audio Language Model
Audio Understanding
Audio Generation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: This paper introduces UALM, an audio language model designed to unify audio understanding, generation, and reasoning

Abstract: Recent advances in the audio language modeling (ALM) domain tackle audio understanding and text-to-audio generation as separate tasks. Very few studies attempt to unify these tasks -- an essential step toward advanced multimodal reasoning. This paper introduces Unified Audio Language Model (UALM), which aims to unify audio understanding, text-to-audio generation, and multimodal reasoning in a single model. To achieve this goal, we first present UALM-Gen, a text-to-audio language model that directly predicts audio tokens and is comparable to state-of-the-art diffusion-based models. We then demonstrate, using proper data blending, training recipes, and inference techniques, that our single UALM model matches the quality of state-of-the-art specialized models in audio understanding, text-to-audio generation, and text reasoning.  Furthermore, we present UALM-Reason, a multimodal reasoning model that utilizes both text and audio in the intermediate thinking steps to facilitate complex generation tasks. To our knowledge, this is the first demonstration in audio research of cross-modal generative reasoning, with its effectiveness confirmed by subjective evaluations.

Published as a conference paper at ICLR 2026
UALM UNIFIED AUDIO LANGUAGE MODEL FOR UN- DERSTANDING, GENERATION AND REASONING
Jinchuan Tian Sang-gil Lee Zhifeng BaaeeennkKnnn,Seeeeerree Sreyan Ghosh 231 Arushi Goel?, Chao-Han Huck Yang", Wenliang Dai Zihan Liu", Hanrong Ye?, Shinji Watanabe Mohammad Shoeybi', Bryan Catanzaro', Rafael Valle", Wei Ping"
j jinchuat @andrew.. emmuedd,, (sanggill 11, zkong, wping ) @nvidia.com
ABSTRACT
Recent advances the audio language modeling (ALM) domain tackle audio understanding and text-to-audio generation as separate tasks. Very few studies attempt to unify these tasks an essential step toward advanced multimodal rea- soning. This paper introduces Unified Audio Language Model (UALM), which aims to unify audio understanding, text-to-audio generation, and multimodal rea- soning in single model. To achieve this goal, We first present UALM-Gen, rext-to-audio language model that directly predicts audio tokens and is compara- ble to state-of-the-an diffusion-based models. We then demonstrate, using proper data blending, training recipes, and inference techniques, that our single UALM model matches the quality of state-of-the-ae specialized models in audio under- standing, text-to-audio generation, and text reasoning. Furthermore, we present UALM-Reason, multimodal reasoning model that utilizes both text and audio in the intermediate thinking steps to facilitate complex generation tasks, To our knowledge, this is the first demonstration in audio research of cross-modal gener- ative reasoning, with its effectiveness confirmed by subjective evaluations.
INTRODUCTION
Human auditory intelligence is characterized by two fundamental capabilities: perception (understand- ing) and production (generation). This duality not merely conceptual; neuro-scientific evidence re- veals a profound connections between these func- tions, where impairment in one often corresponds to a deficit in the other (Liberman et al., 1967; Hickok & Poeppel, 2007: Rizzolatti & Craighero, 2004). Furthermore, resolving complex acoustic challenges requires sophisticated reasoning process that is in- herently multimodal (McGurk MacDonald, 1976; Leman, 2007; Denes & Pinson, 1993; Liberman Mattingly, 1985). This cognitive loop, which of- transcends purely textual representation, is ex- emplified by music composer who iteratively cre- ates piece (generation), critically listens (un- derstanding), and refines it (self-reflection) (Hallam al., 2016). This human paradigm of tightly inte- grating understanding, generation, and reasoning motivates work and suggests that unifying these three pillars is a crucial step toward advanced and general audio intelligence.
AUDIO UNDERSTANDING
AUDIO GENERATION
REASONING
Figure Humans need understanding, gen- eration, and reasoning to handle complex tasks, like composing music.
However, realizing the vision of unified audio intelligence faces significant challenges rooted in the prevailing research landscape. First, prior works predominantly tackled audio understanding
Equal Contribution. CMU NVIDIA?. UMD t: Work done during an internship at NVIDIA. rch. nvi dia. com/ a /dd r T/UAM. Code: https:
Demo samples: intelll--nnnll11 wtw.gggnnnneeeeeeeeeeeeeeeeeeeee
gi http://////aaooiiiieneeeeeeeeeee

---

Published as a conference paper at ICLR 2026
and text-to-audio generation as separate tasks. This gap is further entrenched by a divergence in modeling paradigms: understanding tasks are mainly addressed with auto-regressive large language models (Goel et al., 2025; Xu et al., 2025), while state-of-ther art generation models are mostly based on diffusion models (Lee et al., 2024; Fei et al., 2024; Valle et al., 2025). Second, reasoning within the audio domain remains highly under-explored. Current reasoning studies are confined to the text- only reasoning trajectory in service of audio understanding tasks (Xie et al., 2025; Diao et al., 2025; Goel al., 2025; Rouditchenko et al., 2025). However, is as important to have the ability to reason for guided generation or problem solving that requires a multimodal thinking process. In this work, we aim to close this gap and present the Unified Audio Language Model (UALM) for supporting audio understanding, text-to-audio generation, and multimodal reasoning concurrently in a unified manner: UALM handles all tasks via a single language model and is capable of reasoning through an interleaved and flexible understanding-generaal chain-of-thouge creativity.
mechanism central to human
The first challenge is to achieve high-quality text-to-audio via a language model. Prior diffusion- based text-to-audio models (Evans et al., 2024; Lee et al., 2024; Fei et al., 2024) outperform autore- gressive models (Yang al., 2023; Copet et al., 2024) in quality, potentially due to better inductive bias (Vastola, 2025) and data efficiency (Prabhudesai et al.. 2025). We discover several important findings for successful text-to-audio generation via a decoder-only language model: (i) in terms of data scaling, we need an order of magnitude more audio data than diffusion-based models; (ii) we show that training and sampling with classifier-free guidance (Ho & Salimans, 2022) - technique widely used in diffusion models but rarely in multimodal language models is critical to high- quality generation in our model; (iit) we find a versatile codec (Ye et al., 2025) suitable for audio token generation, and it is crucial to train and infer audio tokens with the delay pattern (Copet et al., 2024) for efficiency; (iv) we find that applying a self- adaptatio stage followed by direct preference optimization (DPO) (Rafailov et al., 2023) can further improve our audio generation quality and aesthetics. With these techniques, we introduce UALM-Gen for text-to-audio generation, and ex- periments confirm that UALM Men achieves quality comparable to frontier cifffusion-basee models.
The second challenge to unify all three tasks in a single language model. While there are several unified generation and understanding models in the vision or speech domains (Wu et al., 2024; Team, 2024; Tian et al., 2025a), in our preliminary experiments we find their recipes do not directly apply to the broader audio domain, and it is hard to balance between different tasks. To tackle these challenges, we (t) carefully design the data blending ratios and up- up-weight generation data due its slower convergence, and (ii) apply modality alignment stage warm up the MLP adapter layers and all token embeddings before unfreezing the full language model backbone. With these, we present single UALM that is comparable to ofth--of-the-art specialized models in each of the three domains: text problem solving, audio understanding, and text-to-audio generation.
The third challenge is to achieve generative multimodal reasoning beyond the text domain. The formal definition, training data, and training recipes for reasoning in audio generation are not well-defined yet. This work takes the first attempt toward this challenging task by investigating three specific steps. (i) We first introduce rich cuptions structured and comprehensive descriptions of audio- as an intermediate blueprint for generation. (fi) We then enable the model to chat with the user to consolidate all details for generation. (iii) We further guide the model to understand and critique its self-generated content, and produce an improved follow-up generation. To achieve these abilitics, we present principled data curation and training recipe, and introduce UALM- M-Rasonn. Experiments show reasoning in generation improves controllability towards nuanced prompts. To our knowledge, UALM-Reason is one of the earliest works to achieve audio reasoning with the multimodal thinking trajectory beyond the text-only domain.
In summary, this work presents the following contributions toward general audio intelligence. (i) We present UALM-Gen. an L.LM that predicts audio tokens and achieves state-of-the-art text-to-audio generation quality ($2.2). (ii) We introduce UALM, a single LLM that unifies audio understanding, text-to-audio generation, and text-only tasks. UALM achieves competitive results across all three domains ($2.3) (iii) We demonstrate UALM-Reason, a reasoning model focused on multimodal reasoning beyond the text domain. UALM-Reason unifies reasoning across understanding and gen- eration tasks and demonstrates better controllability ($2.4). (iv) We present practically effective data strategies, training recipes, and inference techniques to enable unified audio language modeling with ablations.
2

---

Published as a conference paper at ICLR 2026
+ 16KHZ
Enhancement VAE
+0- 48KHz
Always trainable
Text Tokens
Audio Takens (delay patter))
Pre-trained and frozen
Frozen in modality allgnmmet only
Output Embeddings
Decoder-only Transformer (Initialized LLM)
Simples
Trkkens
Adapter
Embeddings
Text Tokens
(30.4M)
24
(34.48)
(24.30)
27.7%
Audio understanding
Toxtonly
Audio generation (2x weigheed
37.95 (59.0M)
53.3%
4p 16kH2z Mono
[29.08]
Data blending ratios
Figure 2: UALM architecture overview and the multimodal pre-training data blending ratios.
2
UNIFIED AUDIO LANGUAGE MODEL (UALM)
2.1
ARCHITECTURE
The architecture UALM presented Fig.2, which extends pre-trained decoder-only text LLM with audio inputs and outputs.
Audio Input and Output: We adopt the well-established Encoder-Adapter-LLe LLM architecture (Liu et al., 2023b; Goel et al., 2025) for audio input, which connects the modules with continuous rep- resentations and avoids the input information loss caused by discrete tokenization. We adopt the acoustic encoder from Goel et al. (2025) for audio input, which operates at 25Hz frame rate and sliding window chunk size of 30 seconds. The adapter is a single-layer MLP.
Audio output is achieved by predicting discrete audio codec tokens, common practice adopted by previous works (Tian et al., 2025a; Copet et al., 2024; Yuan et al., 2025). We use X-codee (Ye et al., 2025) which operates at 50Hz frame rate. Each frame is discretized via residual vector quantization (RVQ) (Zeghidour et al., 2021) producing tokens per frame. We employ the delay pattern (Copet al., 2024) for intra-frame auto-regression of RVQ, technique proven effective in prior audio generation research (Yang et al., 2023). See Appendix B. for details of RVQ and the delay pattern.
Both the acoustic encoder's input and the audio codec decoder's output operate at monophonic 16kHz waveform. We additionally introduce an enhancement VAE module to improve the output waveform 48kHz stereo with better perceptual quality (Appendix B.2).
Initialization: UALM initialized from Qwen2.5-7B (Yang et al., 2024), a text LLM with reason- ing capabilities. We extend its vocabulary to accommodate additional audio tokens from the audio codec. The additional audio embeddings are randomly initialized together with the MLP adapter.
Implementation Details: In all training stages, we only compute the loss over the model out- put tokens, which could be either in text or audio domains. We consider that one audio frame is cquivalently important to one token', Sequence packing (Krell al., 2021) is crucial during pre-training to accommodate interleaved samples of varying lengths and target modalities, thereby avoiding skewed sample and/or length distributions within a mini- match and stabilizing training.
2.2 UALM-GEN: LANGUAGE MODEL-BASEE AUDIO GENERATION
Using an ruto-regressive LM for audio understanding (Goel et al., 2025) and text reasoning (Liu et al., 2024d; Bercovich et al., 2025b) has been well explored in prior works. However, LM-based text-to-audio generation, although proven feasible (Yang et al., 2023; Copet et al., 2024), is empir- ically found to be inferior to the diffusion-based counterparts (Evans et al., 2024; Lee et al., 2024). This section builds UALM-Gen, which demonstrates that the LM paradigm simple, scalable, and achieves frontier results for audio generation. We build UALM-Gen with 1.5B parameters, with its weights initialized from Qwen2.5-1.5B.
Since each audio frame contains tokens, we scale the loss of each audio token by 1/8.
3

---

Published as a conference paper at ICLR 2026
Keywords:
UALM
SFT-1
DPO-1
SFT-2
Brons bund iusis Plrsarsion
Layout:
hand conss follwed percunnoo.
Ando Rich Caption (2106)
(500k)
DPO Puis (MI))
self: Refleciio (Wik)
DPO Pais (29k)
Dislogue (250k)
Description:
Brono band musi: Nowey brco
band plantng tog apleat riusionic
jesviiant tranquats
melod: The music will festure
Phorpt
Sampling
Sempling Cánque Generatian
Sempling Selection (latonal Julpt
Prrsuerion:
pronoding vindly drising Bost
Figure 3: Rich caption example (left) and post-training workflow (right).
Removal of External Text Encoder for Caption Embedding: For both LM and diffusion ap- proaches, prevalent practice in audio generation (Kreuk et al., 2022; Copet et al., 2024; Lee et al., 2024; Hung al., 2024) is to cross-attend the caption embedding from an external text encoder (e.g., T5 (Xue et al., 2022)), which is not compatible with our architecture in Section 2.1. For the first time, we show that L.Mbased audio generation can process text prompts standard BPE tokens by initializing from a pre-trained text LLM.
Data Scaling: Our investigation reveals that LM-based audio generation needs significantly more data than diffusion-bel bassd methods. While previous diffusion models have achieved strong results on relatively small data volumes (often <2M samples or <4k hours (Lee et al., 2024)), we found that approach cannot reach competitive quality at similar data scale. We therefore scaled our training data volume up to 30M samples (approx. 80k hours and 17B tokens), crucial step that enabled UALM-Gen to match the result of frontier diffusion-based counterparts.
the
Classifier-Free Guidance (CFG): CFG (Ho & Salimans, 2022) a widely used inference-time technique in generative models that enhances instruction following, which generates the sequence 91.r based on an interpolation between the conditional and unconditional distribution:
CFG #8
(y) 11:1-11, x) = To(uulii. + +1 -1-
X) -1-))
and 0 means null condition. Consistent with findings from previous LM (Kreuk et al., 2022; Copet et al., 2024; Hussain et 2025) and diffusion models (Lee et al., 2024; Hung et al., 2024), we show that CFG is also an important component for improving the quality of LM-based audio generation.
where À 2 1 the CFG
Direct Preference Optimization (DPO): After the base model is trained with cross-entropy loss, we further conduct DPO training. DPO (Rafailov et al., 2023) an offline reinforcement learning algorithm, first applied in text models (Rafailov et al., 2023) and later to audio generation in diffusion (Hung al., 2024) and LMs (Hussain et al., 2025; Tian al., 2025b). It optimizes a model using a preference pair of winning and losing samples (yw,B)) for given prompt
Te (a (x) Teer (z)
To (
[TPO(TT) -B(...... log Blog
log
(2)
Tree
where xp is the trained model; is the reference model initialized from no and frozen. is sigmoid function and 3 hyper-parameter. To obtain such preference pairs, we generate 10 synthetic samples for each prompt and select the winning and losing samples with a judge model. We found necessary to first adapt the model the self- genereeee winning samples using cross-entropy loss. Since the base model was pre- pre-trained exclusively synthetic pairs can induce out-of-domain issue and subsequently performance degradation issues that, in our experiments, could not be mitigated solely by tuning the hyperparameter 3. It also found that the cross-entropy regularizer helps to reduce the divergence (i.e., ne (Fw (x) - Trer(y (x)))))))) (x)) from the base model. Experimental results are in $3.2.
natural audio, directly applying DPO to
2.3 UALM: UNIFIED AUDIO UNDERSTANDING AND GENERATION PRE-TRAINING
Having established strong foundation for single-tar task audio generation, proceed to the continued multimodal pre-training UALM from the text LLM. This phase is to simultaneously cultivate ca- pabilities in audio understanding, audio generation, and text-based reasoning within single model.

---

Published as a conference paper at ICLR 2026
Enrichment
b. Dialogue
C. Self-Reflection
Generate a bress band music percussion
Generate music that contains Bress band music followed by percussion. Ok! What kind brass band music
Generate music that containe Brass band music followed percussion. wantt trunpets and trombones in brass band music
(Rich Caption)
<Generated Audiic>
should fively, containing you want? prominett trumpets and trombones.
piece nusk for parade scene. should be celebratory, capturing the eecitement a street performencs.
(the plan) [Rich Caption]
<Generated Audio 1> [the result) (Rich Caption] (crtique] The brass band music and percussion heard concurtentty. correct, need to generate the brass band music first and then percussion. <Generated Audio 2>
Cool! Any requirement for the percussion? steady and
Okeyy to express that scene, I'll include brass band music with trumpets
sounds like a diivng best. Now start the generation.
percuasion. [Rich Caption]
Caption] <Generated Audlio-
<Generated Audioa
Figure 4: Demos: audio generation reasoning and joint understanding generation -ernettttnnnnngggeeataitooo reasoning.
Pre-training Data Mixture: To achieve this unification, we create a comprehensive data mixture that fuses datasets from all three target domains. Audio understanding and generation data combined with text-only reasoning data, enabling the model to develop shared representational space. The blending ratios of our data mixture are in Fig.2.
Modality Alignment Stage: While our single-task generation model could be trained effectively without a specific curriculum, we found that a dedicated modality alignment stage is critical for the success of our unified pre-training, an observation consistent with prior work (Wu et al., 2025b). In this initial phase, we freeze the Transformer body and acoustic encoder, updating only the MLP adapter and audio embedding tables using small number of steps but large batch size. After this stage, we unfreeze all parameters for all follow-up training stages, except the acoustic encoder.
2.4
FOR MULTIMODAL REASONING
To unlock more advanced cognitive abilities, we advance the pre-trained UALM to its reasoning- enhanced version, UALM-Reason, through a dedicated post-training procedure. This stage intro- duces a multimodal Chain-of-Thoughh (CoT) paradigm where the model generates intermediate multimodal reasoning steps, in audio and/or text, to deconstruct a user's request, formulate a de- tailed generation plan, and even critique its own output before producing the final audio.
We note that the reasoning capabilities for audio understanding have already been established during pre-training'. This text-based reasoning has already been addressed in prior works (Wu et al., 2025c) and is not a contribution of this work. Thus, our primary focus in post-training shifts to pioneer reasoning for audio generation. This is achieved through a two-stage interleaved SFT-DPO recipe, which instills three novel reasoning patterns: enrichment, dialogue, and self-reflection.
2.4.1 MULTIMODAL REASONING FOR AUDIO GENERATION
Central to our approach is the concept of rich caption a structured and highly detailed textual description that serves as an intermediate blueprint for audio generation. Unlike conventional short prompts, a rich caption provides a comprehensive plan by specifying: Keywords: A list of the core acoustic events. Layout: The temporal arrangement of these events. Description: A detailed characterization of each event's acoustic properties. An example of the rich caption in Fig.3.
This detailed intermediate representation provides nuanced guidance that is critical for high-fidelitty and controllable audio synthesis. The following reasoning capabilities are designed to bridge the gap from simple and diverse user queries to this rich, machine-usable format.
User prompts are often abstract, short, and underspecified. The enrichment capability allows UALM-Reason to autonomously translate user prompt into a detailed rich caption. The model faithfully incorporates all user-provided details while inferring and adding necessary specifics (e.g., environmental context, instrument textures) to create a complete acoustic scene. We further support the abstract user prompt that describes a scenario, a feeling, or a genre, rather than the exact audio events. For these imaginary prompts, the model would enrich with all suitable audio events and the corresponding details automatically. Examples are in F.g.4.a.
Enrichment:
'The AF3 (Goel et al., 2025) data mixture for audio understanding contains massive reasoning samples for audio understanding. To preserve this capability during post-training, we uniformly sample the AF3 mixture to account for 20% volume during each SFT stage.
5

---

Published as a conference paper at ICLR 2026
(a)
(b)
(c)
(d)
DPO Loss
(ng
Divergence
0.514
CL
3/8
Initaa Less adieptation adaptation
1/2
w/ CE loss
1/4
1/32
1.00
2,00
22.50
2.75
3.50
0.15
100K
2000
-100
2000
Training Steps
DPO Steps
DPO Steps
Figure 5: Statistics of UALM-Gen model. (a) The CLAP scores (CL) with various CFG X; (b)
The CLAP scores (CL) with various training data volume down-weighting; (c) the DPO loss w/o
adaptation synthetic data before DPO training; (d) the divergence we (Fwl(z) (ee (x) from the reference model w/o CE loss added in DPO training.
an interactive alternative to enrichment, the model can engage multi-tun dialogue to collaboratively construct the rich caption. It actively queries the user for specific details, guiding them to provide the information needed for a successful generation, thereby resolving ambiguity before synthesis begins. Examples are in Fig.4.b.
Dialogue:
This represents the most advanced form of reasoning, creating reasoning traces that leverage all audio understanding, generation, and text capabilities. The process unfolds follows: Generate: Following enrichment or dialogue paradigm, the model first generates a rich caption and subsequently audio clip based output and generates another new rich caption describing what it actually produced. Critique & Refine: The model compares the two rich captions (the plan the result), identifies discrepancies or flaws in textual critique, and uses this feedback to generate a second, improved audio clip. This generate-unde www--uunddeddddneee aataannnrriiuu-reeieeeee
Self-Reflection:
user prompt. Understand: It then "listens" to its
cycle, also known self-reflection (Liu al., 2024a), mimics human creative iteration and marks significant step towards higher-level intelligence in multimodal models. Examples are in Fig.4.c.
2.4.2 TWO-STAGE SFT-DPO TRAINING RECIPE
UALM-Reason's reasoning capabilities are instilled via two sequential rounds of interleaved SFT- DPO curriculum.
Round 1: Building Foundational Generation Reasoning. This round focuses teaching the enrichment and dialogue capabilities. We begin with 250k internal rich suption-audi pairs. For SFT, a text LLM used synthetically generate diverse user prompts and conversational dialogues that correspond to these rich captions, resulting in 750k-sample training set for the SFT-1 model. We then apply DPO to 250k subset of these samples to create the DPO-1 model: we only use the keywords from the rich caption to compute CLAP scores for preference ranking due to context length limitations of existing CLAP models. We obtain around 60k DPO pairs after threshold-bane filtering, in $3.2.
Round 2: Enabling Relf-Reflection. The second round introduces the self-reflection capability. We create dataset from 60k samples from the first SFT round. For each sample, model to generate the initial audio, then curate a rich caption for it. text L.LM then generates a textual critique comparing the planned and actual rich captions, highlighting the most salient flaw and suggesting a fix. This self-reflection data is combined with the first-round SFT data to train the SFT-2 model. Finally, we perform targeted DPO step on 20k samples (with and without self-reflection). For preference selection, we choose the sample that better adheres to the detailed instructions in the original rich caption. This final optimization step yields the UALM-Reashe model.
the DPO-1
3 EXPERIMENTS
3.1 EXPERIMENTAL. SETUP
Data: Starting from a text LLM, our continued pre-training data is a mixture designed to support audio understanding, generation, and text-only reasoning. For all audio understanding tasks,

---

Published as a conference paper at ICLR 2026
Table 1: Audio Generation results of UALM-Gen ($2.2) and UALM ($2.3) compared to LM-based and diffusion- -baeed baselines. 5-scale subjective scores (OVL, REL) 95% CI = 0.10. Bold indicates best, underline second-best.
Model
SongDescriber
AudioCaps
FDJ
KL4
ISn
CLt
AEST
OVL?
REL
FDJ
KLJ
ISt
CL1
AES
OVL
REL1
Ground Truch
0
1.88
0.48
7:20
4.10
4.03
0
13.49
0.62
4.50
3.91
3.96
Gen-stee--e (Copet al,, 2024)
228.94
0.84
1,69
6.64
3.91
3:97
AndioGen-M (Krenkk at al., 2022)
149.70
1.35
12.26
0.57
4.21
3.83
3.80
MAGNeT-M (Ziv es al,, 2024)
191.49
0.70
1.47
0.41
6.65
3.84
3.86
149.05
08
1.86
7.13
0.46
4.08
3.08
3.72
AM2 (Liu al, 202
351.73
0.68
3,80
0.52
1.31
3.85
3.75
TangoPlux (Huug el al., 2024)
235.61
0.71
1.70
0.41
6.46
3.80
3.89
103. 01
1.02
15.13
0.65
1.42
3.72
3.93
Stable Audio Open (Evans es al, 2024)
138.58
1.01
2.25
0.12
6.37
3.92
3.97
100. 93
2 2.22
11.80
0.35
1.47
3.81
3.80
ETTA (lece ul. 20124)
95.66
0.80
2:15
3.92
3.93
80.13
14.36
0.54
4.51
3.73
3.94
UALM-Gen (Ous)
74.45
0.63
1.87
0.54
7.36
4.07
3.96
75.
112
14.52
0.65
5.08
3.79
3.92
UALM (Ours)
83.69
0.50
2.00
7.28
3.9T
3.09
65.87
0.62
1.92
3.89
3.86
dataset is identical to that used by AF3 (Goel et al., 2025), which already contains reasoning con- tent. We curate a large-scale audio generation dataset of 30M text-audio pairs with length of 10s each. Notably, the majority of the text captions are pseudo-labels generated by open-source audio captioning models (Xu et al., 2025; Ghosh et al., 2025; Goel et al., 2025). The convergence of audio generation is slow ($2.3) and its data is comparatively small in volume, so we empirically up-sample it by 2x. To preserve and enhance the model's native text reasoning capabilities, we in- tegrated 21 million samples of math and code reasoning data from Liu et al. (2024d) and Bercovich al. (2025b). An additional million in- house text samples were included bolster commonsense knowledge. Other data usage in post- bost-training is described in $3.2 and $2.4.
Optimization and Inference: The model was pre-trained on a compute cluster of 16 nodes, each equipped with 8 NVIDIA A100 80GB GPUs We utilized a per-GPU batch size of 5,000 tokens and trained the model for a total of 660,000 steps. We always use greedy search for text generation. For audio generation, we use top-k sampling. Our detailed configurations are in Appendix C.I.
Objective Evaluation: For audio generation, we evaluate our model on the AudioCaps (Kim et al., 2019) and SongDescribber (Manco et al., 2023) test sets. We follow standard evaluation protocols (Evans et al., 2024; Lee et al., 2024): : (1) Frechet distance (FD) using OpenI (Cramer et al., 2019) at 4k.1kHz; (2) Kullback-Leibler divergence (KL) using PaSST (Koutini al., 2022) at 32kHz; (3) Inception Score (IS) using PANNs (Kong et al., 2020) at 16kHz; (4) CLAP scores (CL) using LAION-CLAP (Wu et al., 2023) at 48kHz; (5) AudioBox-Aeshhetic score (AES) (Tjandra et al., 2025) using an average of (CE, CU, PC, PQ) at 16kHz. For audio understanding, we perform evaluations on MMAU(Sakshi al., 2024) and MMAR (Ma et al., 2025). For text-only evaluations, we always use the "reasoning mode" " and test its zero-shot accuracy on MMLU (Hendrycks et al., 2020), GSMSK (Cobbe et al., 2021), and HumanEval (Chen et al., 2021), which corresponds to common sense, math, and code capability, respectively.
Subjective Evaluation: For audio generation, we additionally conduct subjective human evalua- tions of 5-scale mean opinion scores using mechanical turk, following established practices (Kreuk al., 2022; Copet et al., 2024; Liu al., 2023a; Lee et al., 2024): (1) OVL: an overall quality of sample without seeing captions; (2) REL: relevance of the sample to the provided caption.
3.2 LANGUAGE MODEL-BASED TEXT-TO-AUDIC GENERATION RESULTS
With the base UALM- Gen model trained with cross-entropy loss, we first show that CFG is neces- sary for model inference. As suggested in 5.a, audio generation without CFG encounters severe degradation. As defined in $2.2, we find the weight constantly use k = 20 without temperature rescaling.
3.0 for CFG is optimal. For top-k sampling,
Secondly, in Fig.5.b, we show the impact of data scaling where we reduce the data volume down to 1/32 of its full size (30M). The result indicates that data scaling is necessary for the success of the LM-based approach. Note that overfitting is clearly observed with 1/32 data somn-sampling, where the data volume is comparable to prior of diffusion model ETTA (Lee et al., 2024) (1.0M vs. 1.3M). Our finding aligns with Prabhudesai et al. (2025), comparing scaling laws for auto-regressive models trained with cross-entropy and liffusion-based models.
7

---

Published as a conference paper at ICLR 2026
Table 2: Audio understanding results of UALM ($2.3) versus open- understanding models.
Model
Base Model
MMAU-v06.155
MMAR
Sound
Musict
Speech
Meant
Meant
GAMA-IT (Gloseh et al,, 2024)
LLaMA-2 (713
32.T
22.4
11.6
22.2
17.4
SALMONN (Tang et sl., 2021)
Vicuna (13B)
12.1
37.8
28.3
36.2
33.2
Sree2-AAdio-nnsttiee Instrucs (Chu et al,, 2024)
Quen2-Audio (7B)
61.2
56.7
65.4
57.4
30.0
DeSTA2.5-Audo (Ln al. 2025)
Jlama3.1-SB-Ime
66.8
57.1
71.9
63.2
Audio Reasoner (Xie et sl., 2026)
Quen2-Audio Instruct (7B)
G7.3
61.5
62.5
63.8
36.8
Stop-Andios (Wi nl., 2025a)
Unknown (<110RR)
68.2
72.8
73.9
Qwen2.5-Omni (Xu es al. 2026)
Qwen2.5 (7B)
76.8
67.3
68.9
71.0
M6.7
Audio Flamingo 3 (Goel al., 2025)
Q.en2.5 (7B)
78.7
73.3
64.9
72.3
58.5
UALM (Ours)
Qwen2.5 (7B)
77.9
77.6
06.7
74.1
55.2
Text capability of prior unified multimodal language models (in the vision domain) and our UALM. Our model is initialized from Qwen2.5-7B.
Table 3:
Model
MMLUt
GSM8Kt
HumanEvalt
Meant
OpusLM (Tian al., 2025a)
52.5
Liquid-7B (Wu al, 2024)
56.0
Chameleon-7B (Team, 2024)
52.1
Qwen2.5 B-BB-nsstttt (Yong al., 2024)
74.5
91.6
84.8
83.6
UALM (Ours)
71.6
92.1
81.1
81.6
Thirdly, we conduct DPO on the UALM-Gen base model. We sample 250k prompts uniformly from the pre-training data and generate 10 audio clips for each prompt. We then select the preference pairs using CL and (CE, CU, PC, PQ) metrics', which ultimately yields 50k pairs. As the base UALM-Gen trained real audio, fine-tuning on the winning examples (typically Ik steps). Without adaptation, the DPO loss would spike in the early training phase before convergence as Fig.5.c. DPO training causes divergence from the base model, which could be alleviated by enforcing the cross-entroee cessopy loss over the winning samples, together with the DPO loss, as Fig.5.d.
found necessary
first adapt it synthetic audio by
Ultimately, Tab. shows that UALM-Gen outperforms previous LM-based approaches (Kreuk et al., 2022; Copet et al., 2024) and achieves competitive results leading diffusion models, including TangoFlux (Hung et al., 2024), Stable Audio Open (Evans et al., 2024), and ETTA (Lee et al., 2024). Techniques introduced in $2.2 are further ablated in Tab.8.
3.3 MUL.TIMODAL PRE-TRAIE TRA-NIGINN RESULTS
We demonstrate that the pre-trained base model UALM matches the quality of frontier special- ists in text-to-audio generation, audio under- standing, and text-based reasoning. The Au- dio generation capability is reported in 1.
(b)
0.5
Acc.
MMAU
0.4
MMAR
Like UALM-Gen, the UALM outperforms or matches the quality of prior diffusion models.
The audio understanding capability is reported in Tab.2. The results show that the accuracy of our model can match prior state-of-the-an open-source models like Audio Flamingo 3 and Qwen2.5-Omni. Finally, as presented in Tab 3, our UALM only encounters marginal degrada- tion on MMLU, GSM8K, and HumanEval, compared with Qwen2.5-7B-Insstee model maintains strong common sense and text reasoning ability. Compared with the prior unified model in vision (Team, 2024; Wu et al., 2024) and pure speech (Tian et al., 2025a), our text-only metries show clear advantage. noticeable observation during this training is that audio under- standing converges much faster than audio generation, which is evident in Fig.6.
120k
600k
200k
Training Steps
Training Steps
Figure 6: Audio understanding (a) and audio gen-
eration (b) capabilities along training steps
which proves our
For CL., enforce the winning-losing gap of 0.15; for (CE, CU, PC, PQ), enforce that gap to be all positive. We only select one pair for each prompt.
8

---

Published as a conference paper at ICLR 2026
Table 4: 5-scale subjective score of UALM-Reason on reasoning-oriented generation with 95% CI.
Model
Enrichment
Dislogue
Self-reflection
UALM
3.77 0.11
3.92 0.11
3.82 + 0.11
FALM-Reesson
4.01 + 0.10
4.02
0.10
4.04 +: 0.09
3.4 MULTIMODAL DOST-TRAINING ANALYSIS
As the generative reasoning for audio models is still nascent, our evaluation mostly relies on quali- tative analysis and subjective evaluation. We show that, with reasoning enabled, the UALM-Reason superior than the base UALM.
Experimentally, our qualitative analysis shows that FALM-Reason excels in three key areas, compared with the base UALM. First, demonstrates superior detail control- lability, faithfully rendering nuanced acoustic details from text prompts that previous models find challenging. The model can discern concepts like number (a dog barks vs. dogs bark), spatial properties (in the far distance), temporal sequencing (event A follows event B), and audio texture (distorted audio)-a capability we attribute to our use of rich captions. Second, the model exhibits sophisticated understanding of human intention. It successfully supports human-centric interac- tions like imaginary content enrichment and dialogue and subsequently generates appropriate inter- mediate rich captions. This is attributed to the base UALM's strong text capability, which connects its broad textual knowledge to the audio generation domain. Third, UALM-Reason creates a joint modeling scheme between understanding and generation. The model can analyze and criticize its own generated audio and leverage that critique for iterative refinement, an outcome of fine-tuning the base UALM for strong audio understanding. These observations are already beyond the existing evaluation protocols for audio generation. We thus demonstrate these experimental observations in our demo webpage.
Qualitative Analysis:
Subjective Evaluation: Table 4 shows a 5-scale subjective evaluation for scenarios discussed in Section 2.4 (detail in Appendix C.3). The results demonstrate that generation-oriente reasoning from audio understanding and text capabilities is highly effective in advancing audio production.
4 RELATED WORK
Language Model-Based Audio Generation: Modern text-to-audio generation models can be broadly classified as diffusion models (Ziv et al., 2024; Liu et al., 2024b; Hung et al., 2024; Lee et al., 2024), auto-regressive language model (LM) approaches (Yang et al., 2023; Copet et al., 2024), and their hybrids (Lam et al., 2024). Recently, diffusion-based models are shown to have better generation quality (Hung et al., 2024; Lee et al., 2024). However, we demonstrate that with proper data scaling and carefully designed training recipe (see $2.2), LLM- based architecture can outperform diffusion models on this task. Furthermore, while conventional text-to-audio mod- els (including both diffusion and LM approaches) often take contextual contrastive embeddings (Chung et al., 2024; Wu et al., 2023) of the caption as cross-attention inputs, we show that built-in BPE text tokenizer in decoder-only LLM can achieve similar or better results.
While DPO (Rafailov et al., 2023) has been applied to diffusion-based text-to-audio generation (Hung et al., 2024; Majumder et al., 2024), this work is one of the first to integrate DPO into an LM-based eox-to--adiio framework and demonstrate its effectiveness. Both Hung et al. (2024); Ma- jumder al. (2024) and this work leverage the standard CLAP model for preference pair selection, though we acknowledge the potential benefits of adopting human preference-aware CLAP (Takano et al., 2025) model to better align with human perception.
Unified Audio Understanding and Generation: There are many audio foundation models spe- cialized in understanding (Kong et al., 2024; Goel et al., 2025; Chu et al., 2023; Xu et al., 2025) or generation (Yang et al., 2023; Liu et al., 2024b; Lee et al., 2024; Hung et al., 2024). To our knowledge, UALM is the first audio language model that successfully unifies these distinct capa- bilities within a single framework. Works with similar motivation have been explored in the vision and pure speech fields, but they often suffer from severe degradation on text-only tasks (Wu et al., 2024; Team, 2024; Tian et al., 2025a; Hori et al., 2019; Tjandra al., 2017). In contrast, UALM

---

Published as a conference paper at ICLR 2026
maintains high text reasoning capabilities, showing minimal degradation text-only benchmarks during multimodal pre-trainn pre-training (Dai et al., 2024).
reasoning in existing audio language models is mostly for understanding and therefore confined to text-centric analysis of audio inputs, where the model connects acoustic cues to its pre-existing world knowledge (Xie et al., 2025; Diao et al., 2025; Goel et al., 2025; Rouditchenko et al., 2025; Deshmukh et al., 2025; Wu et al., 2025c). Reasoning in the generative domain sig- nificantly under-explored. MusiCoT (Lam et al., 2025) is the most relevant recent work, which im- plements a chain-of-thougld (CoT) process for music generation by predicting intermediate CLAP latents. By contrast, UALM-Reason carves out a new frontier by universally applying reasoning to understanding and generation either separately or jointly. We demonstrate that reinforcement learning explored prior understanding works (Diao et al., 2025; Rouditchenko et al., 2025) are also effective for reasoning beyond the text domain.
Reasoning in Audio:
Relationship with Speech Language Models (SLM); The pre-trained UALM is designed for uni- fied understanding and generation, capability also explored in concurrent Speech Language Models (SLMs) (Défossez et al., 2024; Zeng et al., 2024). While these SLMs primarily target low-latency speech-to-speech dialogue and conversational smoothness, UALM targets at unified modeling of audio and text tasks in order to achieve higher-level audio intelligence. Furthermore, some SLMs employ joint modeling of text and audio within a single timestamp, relying on the strong assump- tion that speech and text are monotonically aligned. Since this assumption does not hold for general audio modeling, UALM treats text and audio distinct, separate modalities.
CONCLUSION
We introduce the Unified Audio Language Model (UALM), single model that unifies audio derstanding, text-to-audio generation, and text problem solving. We further present UALM-Reason, which leverages the wwwrsstannnngggnrraaii bilities such as iterative refinement of its own outputs well-known behavior of high-level intelligence. This work marks a significant step towards more controllable, intelligent, and holistic audio AI. There are a number of important directions we plan to address in our future work.
together through novel multimodal reasoning capa-
key future direction of this work is to build a unified audio representation. As in $2.1, current UALM adopts the continuous audio encoder for audio input and the discrete audio codec tokens for audio output. A unified audio representation could further facilitate scalable training of joint understanding, generation, and reasoning.
Unifying audio representation:
Quality Assessment of Synthetic Audio Captions: Our SFT and DPO data curation is based on synthetic captions. However, even with our best efforts in data curation, there exists certain amount of hallucination and misalignment between the audio and captions through manual inspections. is an important future direction to design quantitative methods at scale to assess the quality of the synthetic audio captions (especially the rich caption) to build a robust data curation pipeline.
Quality Assessment of Audio Quality: While there are many existing audio quality evaluation metrics (Lerch et al., 2025), there is still gap from human perception. The lack of accurate and layered evaluation of acoustic quality, generation diversity, musical aesthetics and correctness, and faithfulness limits better RI. multimodal reasoning. In our future works, we plan to develop better audio quality evaluation metrics that are suitable for complex audio generation, including those measuring the multimodal reasoning chains.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To facilitate reproducibility, we thoroughly document our data and training procedures from the pre- training data mixture and recipes to the post-training curriculum for UALM-Reason. All training and inference hyper-parameters are provided in Appendix C.I. We also release the training code.
ETHICS STATEMENT
This research adheres to the ICLR Code of Ethics. Our model was trained on large-scale datasets, including standard benchmarks and audio with synthetically generated captions. We acknowledge that generative audio models carry a risk of misuse for creating misleading content. While this work intended for creative and assistive applications, we encourage responsible use. The training data may reflect societal biases, which the model could perpetuate. We transparently note the use of LLMs for data curation and writing assistance in Appendix A.
REFERENCES
Jisheng Bai, Haohe Liu, Mou Wang, Dongyuan Shi, Wenwu Wang, Mark D Plumbley, Woon-Seng Gan, and Jianfeng Chen. Audiosetcaps: An enriched audio-caption dataset using automated gen- eration pipeline with large audio and language models. IEEE Transactions on Audio, Speech and Language Processing, 2025.
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran EI- Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zil- berstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Ger- ald Shen, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Zijia Chen, Zhilin Wang, David Mosallanezhad. Adi Renduchintala. Haifeng Qian, Dima Rekesh, Fei Jia, Somshubra Majumdar, Vahid Noroozi, Wasi Uddin Ahmad, Sean Narenthiran, Alek- sander Ficek, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Igor Gitman, Ivan Moshkov, Wei Du, Shubham Toshniwal, George Armstrong. Branislav Kisacanin, Matvei Novikov, Daria Gitman, Evelina Bakhturina, Jane Polak Scowcroft, John Kamalu, Dan Su, Kezhi Kong, Markus Kliegl, Rabech Karimi, Ying Lin, Sanjeev Satheesh, Jupinder Parmar, Pritam Gundecha, Bran- don Norick, Joseph Jennings, Shrimai Prabhumoye, Syeda Nahida Akter, Mostofa Patwary, Abhinav Khattar, Deepak Narayanan, Roger Waleffe, Jimmy Zhang, Bor- Boi-Yiing Su, Guyue Huang, Terry Kong, Parth Chadha, Sahil Jain, Christine Harvey, Elad Segal, Jining Huang, Sergey Kashirsky, Robert McQueen, Izzy Putterman, George Lam, Arun Venkatesan, Sherry Wu, Vinh Nguyen, Manoj Kilaru, Andrew Wang, Anna Warno, Abhilash Somasamudramath, Sandip Bhaskar, Maka Dong, Nave Assaf, Shahar Mor, Omer Ullman Argov, Scot Junkin, Olek- sandr Romanenko, Pedro Larroy, Monika Katariya, Marco Rovinelli, Viji Balas, Nicholas Edel- man, Anahita Bhiwandiwalla, Muthu Subramaniam, Smita Ithape, Karthik Ramamoorthy, Yut- ing Wu, Suguna Varshini Velury, Omri Almog, Joyjit Daw, Denys Fridman, Erick Galinkin, Michael Evans, Katherine Luna, Leon Derczynski, Nikki Pope, Bileen Long, Seth Schneider, Guillermo Siman, Tomasz Grzegorzek, Pablo Ribalta, Monika Katariya, Joey Conway, Trisha Saar, Ann Guan, Krzysztof Pawelec, Shyamala Prayaga, Oleksii Kuchaiev, Boris Ginsburg, Oluwatobi Olabiyi, Kari Briski, Jonathan Cohen, Bryan Catanzaro. Jonah Alben, Yonatan Geif- man, Eric Chung, and Chris Alexiuk. Allma--nmmotee Efficient reasoning models, 2025a. URL https:://axxtt g alos/ 2255555552. 00949.
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El- Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, et al. Llama-nemotron Efficient reasoning models. arXiv preprint arXiv:2505.0094 2025h.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2/07.03372 2021.
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07990 2023.

---

Published as a conference paper at ICLR 2026
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2- -audio technical report. arXiv preprint arXiv:2407.10759, 2024.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling -iruuttto--iietuueed lan- guage models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint a 10. 14 168, 2021.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défosse7. Simple and controllable music generation. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Aurora Linh Cramer, Ho-Hsiang Wu, Justin Salamon, and Juan Pablo Bello. Look, listen, and learn more:
Design choices for deep audio embeddings. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3852-3856. IEEE, 2019.
Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang. Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NVLM: Open frontier-class multimodal llms. arXiv preprint arXiv:2409. /402, 2024.
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: speech-text foundation model for real-time dia- logue. arXiv preprint arXiv22410.00037. 2024.
Peter B. Denes and Elliot N. Pinson. The Speech Chain. W. H. Freeman, 1993.
Soham Deshmukh, Satvik Dixit, Rita Singh, and Bhiksha Raj. Mellow: small audio language model for reasoning. arXiv preprint arXiv:2503.088 08540, 2025.
Xingjian Diao, Chunhui Zhang, Keyi Kong, Weiyi Wu, Chiyu Ma, Zhongyu Ouyang, Peijun Qing, Soroush Vosoughi, and Jiang Gui. Soundmind: Rl-incentivized logic reasoning for audio- language models. arXiv preprint arXiv:2506. /2935, 2025.
Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. arXiv preprint arXiv:2407.14355 2024.
Zhengcong Fei, Mingyuan Fan, Changqian Yu, and Junshi Huang. Flux that plays music. arXiv preprint 00t.22499005887 2024.
Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. In EMNLP, 2024.
Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jachyeon Kim, Wei Ping, Rafael Valle, Di- nesh Manocha, and Bryan Catanzaro. Audio flamingo 2: An audio-language model with long- audio understanding and expert reasoning abilities. arXiv preprint arXiv:2503.03983 2025.
Arushi Goel, Sreyan Ghosh, Jachyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao- Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, et al. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. arXiv preprint arXiv:2507.08122. 2025.
Susan Hallam, Ian Cross, and Michael Thaut (eds.). The Oxford Handbook of Music Psychology. Oxford University Press, 2 edition, 2016. ISBN 9780191034459.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.088: 033000 2020.
12

---

Published as a conference paper at ICLR 2026
Gregory Hickok and David Poeppel. The cortical organization of speech processing. Nature Reviews Neuroscience, 8(5):393-44) 402, 2007. doi: 10. 10.1038/nmm213.
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
arXiv preprint
arXiv:2207.12500 12598, 2022.
Takaaki Hori, Ramon Astudillo, Tomoki Hayashi, Yu Zhang, Shinji Watanabe, and Jonathan
Le Roux. Cycle-consistency training for end-to-end speech recognition. In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6271- 6275, 2019.
Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pp. 13916-13932. PMLR, 2023.
Chia-Yo Yu Hung, Navonil Majumder, Zhifeng Kong, Ambuj Mehrish, Amir Ali Bagherzadeh, Chuan Li, Rafael Valle, Bryan Catanzaro, and Soujanya Poria. Tangoflux: Super fast and faithful text to audio generation with flow matching and clap-ranked preference optimization. arXiv preprint arXiv:2412.21037, 2024.
Shehzeen Hussain, Paarth Neekhara, Xuesong Yang, Edresson Casanova, Subhankar Ghosh, Mikyas T Desta, Roy Fejgin, Rafael Valle, and Jason Li. Koel-tts: Enhancing Ilm based speech generation with preference alignment and classifier free guidance. arXiv preprint arXiv:2502.05236, 2025.
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume (Long and Short Pupers), pp. 119-132, 2019.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Qiuqiang Kong. Yin Cao, Turab Iqbal. Yuxuan Wang. Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transac- fions on Audio, Speech, and Language Processing, 28:2880-2888 2894, 2020.
Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. Interna- tional Conference on Machine Learning, 2024.
Khaled Koutini, Jan Schlüter, Hamid Eghbal- and Gerhard Widmer. Efficient training of audio transformers with patchout. In Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pp. 2753-2757. ISCA, 2022. doi: 11444//nntespppcchh2222222 Interspeech.c 2022-222222 227.
URL https: /col:////oo....// org 10. 0. 21437/
Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. Efficient sequence packing without rross-contamination: Accelerating large language models without impacting per- formance. arXiv preprint vxXiv:2107.02021 2021.
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209.
2022.
Rithesh Kumar, Prem Seetharaman, Alejandro Luebs, Ishaan Kumar, and Kundan Kumar. High- fidelity audio compression with improved rvggan. Advances in Neural Information Processing Systems, 36:27980-27993, 2023.
Max WY Lam, Qiao Tian, Tang Li, Zongyu Yin, Siyuan Feng, Ming Tu, Yuliang Ji, Rui Xia, Mingbo Ma, Xuchen Song, et al. Efficient neural music generation. Advances Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2026
Max WY Lam, Yijin Xing. Weiya You, Jingcheng Wu, Zongyu Yin, Fuqiang Jiang, Hangyu Liu, Feng Liu, Xingda Li, Wei-Tsung Lu, et al. Analyzable ofain--f--usicllll thouuhtt prompting for high-fidelity music generation. arXiv preprint arXiv:2503.19611 2025.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using a learned similarity metric. In International conference on machine learning, pp. 3558-1566 PMLR, 2016.
Sang-gil Lcc, Wei Ping. Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training. In The Eleventh International Conference on Learning Representations, 2023. URL https: : /ttp/eeeeeeeee.ecec//e net/ forum? wwo.rummidi1ttCCC0
Sang-gil Lee, Zhifeng Kong, Arushi Goel, Sungwon Kim, Rafael Valle, and Bryan Catanzaro. Etta: Elucidating the design space of text-to-audio models. arXiv preprint arXiv:2412.195511 2024.
Mare Leman. Embodied Music Cognition and Mediation Technology. MIT Press, Cambridge, MA, 2007.
Alexander Lerch, Claire Arthur, Nick Bryan-Kinns, Corey Ford, Qianyi Sun, and Ashvala Vinay. Survey on the evaluation of generative models in music. arXiv preprint urXiv:2506.05104, 2025.
Alvin M. Liberman and Ignatius G. Mattingly. The motor theory of speech perception revised. Cognition, 21(1):1-36, 1985.
Alvin M Liberman, Franklin S Cooper, Donald P Shankweiler, and Michael Studdert-Kemndd... Per- ception of the speech code. Psychological Review, 74(6):431-40) 1967. doi: 10. 10.1037/h0020279.
Fengyuan Liu, Nouar AlDahoul, Gregory Eady, Yasir Zaki. and Talal Rahwan. Self-reflection makes large language models safer, less biased, and ideologically neutral. arXiv preprint arXiv:2406.104m 2024a.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei. Xubo Liu, Danilo Mandic. Wenwu Wang, and Mark Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. Interna- fional Conference on Machine Learning, pp.
PMLR, 2023a.
Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. Audioldm Learning holistic audio generation with self superviee pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-3491 34916, 2023b.
Huadai Liu, Rongjie Huang, Yang Liu, Hengyuan Cao, Jialei Wang. Xize Cheng, Siqi Zheng, and Zhou Zhao. Audiolcm: Text-to-audio generation with latent consistency models. arXiv preprint
14Xiv:2406600356 2024c.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-11986. 2022.
Zihan Liu, Yang Chen, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. vancing frontier math reasoning with post-training and reward modeling. 0rXiv:2412.15084 2024d.
Acemath: Ad- arXiv preprint
Zihan Liu, Yang Chen, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. AceMath: Ad- vancing frontier math reasoning with post-training and reward modeling. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the As- sociation for Computational Linguistics: ACL 2025, pp. 3993-4011 Vienna, Austria, July 2025. doi: 00.18653/11/2023 Undings-acl.22 200. URL httpt:tpp///cc http://tcha/aolaaaa/oo howphhooooyyyy...o.o oog.gooy.oooogoo fi findings-ach 2067.
Association for Computational Linguistics. ISBN 979-8-89176-25655
14

---

Published as a conference paper at ICLR 2026
Ke-Han Lu, Zhehuai Chen, Szu-Wet Fu, Chao-Han Huck Yang. Sung-Feng Huang, Chih-Kai Yang, Chee-En Yu, Chun-Wer Chen, Wei-Chih Chen, Chien-yu Huang, et al. Desta2. 5-audio: Toward general-purpose large audio language model with self-generated cross-modal alignment. arXiv preprint S5iv:2507.00766 2025.
Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang. Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, et al. Mmar: A challenging benchmark for deep rea- soning in speech, audio, music, and their mix. arXiv preprint arXiv:2505. 2025.
Navonil Majumder, Chia-Ye Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, and Sou- janya Poria. Tango 2: Aligning diffusion- based text-to-audio generations through direct prefer- ence optimization. arXiv preprint arXiv:2404.0995. 2024.
lariia Manco, Benno Weck, Seungheon Doh, Minz Won, Yixiao Zhang, Dmitry Bogdanov, Yusong Wu, Ke Chen, Philip Tovstogan, Emmanouil Benetos, Elio Quinton, György Fazekas, and Juhan Nam. The song describer dataset: a corpus of audio captions for music-and-language evaluation. In Machine Learming for Audio Workshop NeurIPS 2023, 2023.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international confer- ence on computer vision, pp. 7794-2802, 2017.
Harry McGurk and John MacDonald. Hearing lips and seeing voices. Nature, 264:746-748, 1976. doi: 10.1038/264746a0.
Mihir Prabhudesai, Mengning Wu, Amir Zadeh, Katerina Fragkiadaki, and Deepak Pathak. Diffu- sion beats autoregressive in data-constrained settings. arXiv preprint arXiv:2507. 15857, 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 33728-53741, Curran Associates, Inc., 2023.
Giacomo Rizzolatti and Laila Craighero. The mirror- system. Annual Review of Neuro- science, 27: 2004. doi: 10.1 1444aammeev..euuo27777000......0
Andrew Rouditchenko, Saurabhehand Bhati, Edson Araujo, Samuel Thomas, Hilde Kuehne, Roge- rio Feris, and James Glass. Omni-rl: Do you really need audio to fine-tune your audio Ilm? arXiv preprint arXiv:2505.00 09439, 2025.
S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ra- mani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint 100 2024.
Hubert Siuzdak. Vocos: Closing the gap between time-domain and fourier-based neural vocoders for high-quality audio synthesis. arXiv preprint arXiv:2306.008 2023.
Christian J Steinmetz and Joshua D Reiss. auraloss: Audio focused loss functions in pytorch. In Digital music research network one-day workshop (DMRN+ 15), 2020.
Christian J Steinmetz, Jordi Pons, Santiago Pascual, and Joan Serra. Automatic multitrack mixing with a differentiable mixing console of neural audio effects. In ICASSP 2021-2021 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 71-75. IEEE, 2021.
Taisei Takano et al. Human-clap: Human--erceetioon-aasee contrastive language-audio pretraining. APSIPA, 2025.
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Chameleon Team. Chameleon: Mixed-modaa early-fusion foundation models. arXiv preprint
arXiv:2405.09818. 2024.
15

---

Published as a conference paper at ICLR 2026
Jinchuan Tian, William Chen, Yifan Peng, Jiatong Shi, Siddhant Arora, Shikhar Bharadwaj, Takashi Mackaku, Yusuke Shinohara, Keita Goto, Xiang Yue, Huck Yang, and Shinji Watanabe. OpusLM: A Family of Open Unified Speech Language Models. In Interspeech 2025, pp. 3259-3299666 2025a. doi: 10.21 12221433/ntesspecchhh.2 2025- 184.
Jinchuan Tian, Chunlei Zhang. Jiatong Shi. Hao Zhang, Jianwei Yu, Shinji Watanabe, and Dong Yu. Preference alignment improves language model-based tts. In ICASSP 2025 2025 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp- 1-5, 2025b.
doi: 10. 10.1109/CAASSP446600 20255000500022
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Listening while speaking: Speech chain by deep learning. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 301-308, 2017.
Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Lc, Nick Zacharov, et al. Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound. arXiv preprint arXiv:2502.051399 2025.
Rafael Valle, Rohan Badlani, Zhifeng Kong, Sang gil Lee, Arushi Goel, Sungwon Kim, Joao Fe- lipe Santos, Shuqi Dai, Siddharth Gururami, Aya Aljafari, Alexander H. Liu, Kevin J. Shih, Ryan Prenger, Wei Ping, Chao-Han Huck Yang, and Bryan Catanzaro. Fugatto I: Foundational gener- ative audio transformer opus 1. In The Thirteenth International Conference on Learning Repre- sentations, 2025.
John J Vastola, Generalization through variance: how noise shapes inductive biases diffusion models. arXiv preprint urXiv:2504./2532 2025.
Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, et al. Step-audio 2 technical report. arXiv preprint 1663222077..663, 2025a.
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingehao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 12966-12977. 2025b.
Junfeng WU, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable and unified multi-modal generators. arXiv preprint arXiv:2412.043322 2024.
Shu Wu, Chenxing Li, Wenfu Wang, Hao Zhang, Hualei Wang, Meng Yu, and Dong Yu. Audio- thinker: Guiding audio language model when and how to think via reinforcement learning. arXiv preprint arXiv:2508. 20X0022508.08039, 2025c.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Kerg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and seyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao. Audio- reasoner: Improving reasoning capability in large audio language models. arXiv preprint 0011032503.02318. 2025.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, al. Qwen2.5-omni technical report. arXiv preprint N.iv:2503.202/5 2025.
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards token-free future with pre-trained byte-to-byte mod- els. Transactions of the Association for Computational Linguistics, 10:291-306, 2022.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu. Fei Huang. Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint 1515 2024.
16

---

Published as a conference paper at ICLR 2026
Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang. Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. Uniaudio: An audio foundation model toward universal audio generation. arXiv preprint arXiv:2310.00704, 2023.
Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, et al. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 25697-25705, 2025.
Ruibin Yuan, Hanfeng Lin, Shuyue Guo, Ge Zhang, Jiahao Pan, Yongyi Zang, Haohe Liu, Yiming Liang, Wenye Ma, Xingjian Du, et al. Yue: Scaling open foundation models for long-form music generation. arXiv preprint arXiv:2503.0866 08638, 2025.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Sound- stream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495-507, 2021.
Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glice-voice: Towards intelligent and human-like end-to-end spoken chatbot. arXiv preprint arXiv:2412.02611, 2024.
Alon Ziv, Itai Gat, Gael Le Lan, Tal Remez, Felix Kreuk, Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. Masked audio generation using single non-autoregressive transformer. arXiv preprint arXiv:2401.04571 2024.
17

---

Published as a conference paper at ICLR 2026
A THE USE OF LLMS
We use LLMs assist the writing of the paper in terms of: (1) grammar check, and (2) occasionally choosing the best word in writing. We use LLMs to verify that we have properly included all relevant papers in the field. We use LLMs as part of automated text data curation in our research, in a similar way as many other LLM-related research papers.
B TECHNICAL DETAILS
B. RESIDUAL VECTOR QUANTIZATION AND DELAY PATTERN
For an audio frame e RD, RVQ discretizes z into multiple discrete tokens by selecting one codebook per stage to successively quantize the residual. Concretely:
Cn,in in
arg min ||r Ga,ills To x, In Tn-1 Cn,in"
Here My is the number of RVQ stages, Ca.k RDD is the k-th codebook stage in is the selected index, In the post-stage residual, and & the reconstructed frame. For each Cn,ks we assign a unique discrete index An,
to represent it.
Flattening all tokens per frame into single sequence yields prohibitively long token streams (T' x ny steps for T frames). To reduce effective audio sequence length, MusicGen (Copet al., 2024) introduced the delay pattern: single autoregressive Transformer predicts all ng codebooks in parallel each step, with one LM head per codebook. As shown in Figure the n-th codebook at frame t An,: is predicted at sequence step offset. Equivalently, at sequence step s, the LM predicts ng tokens {A1,.s A2, An in parallel. The delay pattern allows the LM to capture dependencies across RVQ tokens while maintaining the required autoregressive sampling steps (T + n, 1) close to the number of audio frames (T).
t + (n 1), i.e. with fixed temporal
In detail, the X-Codee (Ye et al., 2025) uses ny = and 50Hz frame rate (T = 50 x seconds). Therefore, for a 10-second audio, there 4000 tokens to be predicted in 507 sequence steps.
Delay pattern
A1
A
A
A k-3
A k-2
RVQ layers
A2
A2,2
A
A2,4
A k-2
A k-1
A
A1
A
A14
A1,s
A k-1
A
-
5eg steps
1
2
3
4
5
k-1
k
Figure 7: Illustraiion of delay pattern of RVQ audio tokens A. An, the k-th audio token at n-th RVQ layer, < n ng. k < T. At step s, the LM predicts {Ai,s, A2,5-1," Angis -nn in parallel.
B.2 ENHANCEMENT VAE
UALM operates in 16kHz monophonic audio waveform, following previous works in continuous encoder (Goel et al., 2025) and discrete codec (Ye al., 2025). To improve perceptual quality of generated audio, we attach enhancement module based on VAE (Kingma & Welling, 2014). It receives 16kHz mono waveform generated by the audio codec decoder and upsamples into 48kHz stereo waveform. Note that the input wavefonm to the enhancement VAE is a lossy version of the target with codec distortions; the enhancement VAE implicitly learns the joint process of blind audio restoration, super-sampling, and spatialization.
18

---

Published as a conference paper at ICLR 2026
The encoder consists of an STFT layer (hop length=8) that converts the waveform to spectro- gram, followed by 3 ConvNeXt (Liu et al., 2022)- downsampling blocks (stride= [4, 4, parameters (starting with 512 channels, multiplied by 2x per block) similar Vocos 5]) with 153M
(Siuzdak, 2023). Thus, the encoder downsamples the input by 640x producing latents at 25Hz frame rate. The latent dimension is 512. The decoder consists of 5 convolutional upsampling blocks (stride- [8, 6, 5, 4, 2]) with 489M parameters (starting with 2560 channels) based on Stable Audio Open (Evans al., 2024), upsampling the latent by 1920x to generate 48kHz stereo waveform. All layers use SnakeBeta activation functions (Lee et al., 2023).
The enhancement VAE is trained with a combination of reconstruction, adversarial, and regulariza- tion losses. Let Tin denote the distorted waveform, and Zgtt the corresponding clean ground-truth waveform. The encoder E maps Zin to a latent distribution (=(=)zmn) and the decoder /) recon- structs - (22) from a latent sample qE- The enhancement VAE is trained with the following objective:
LVAE = LaterroMRSTTFT + Loogmll + Ladv + Lreat + Ç LILE,
(3)
where each loss term defined as follows:
Stereo sum and difference MR-STFT loss (Steinmetz & Reiss, 2020; Steinmetz et al., 2021)
m Σ
()TFF;;()) (f)
1 +
log STFT. II).
(x) =
(4)
(T)F,,(:)]I=
LstereoMRSTET (Tgt, 2) LMRSTFT (Tsums Fsum) + LMRSTET (Faiff, Fam))
(5)
with Tsam Tgl. + Tg.R and Taa = Tgl.l Tgt..
Multi-scale log-mel L1 loss (Kumar al., 2023)
Llogmel (Igr. .t)
Σ
llogio (xg1) log (mcl,
(6)
where mely denotes a mel spectrogram at resolution j.
Least-squares adversarial loss (Mao et al., 2017) with stereo BigVGAN-v2 discriminator (Lee et al., 2023)
Ladv (Zg. - (D&
where Da is the k-tth discriminator head.
Feature matching L1 loss (Larsen et al., 2016)
1
( (+)l
Lieat (Fig, 5t)
(8)
KL
where D)((.) is the I-th feature map of discriminator Dg-
KL. divergence regularization
LKL. (L(qe(>(<m) (0)
19

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL DETAILS
C.1 TRAINING AND INFERENCE HYPER-PARAMETERR
Table 5: Pre-Training configurations
Modality Aligument
Pre-Training
Batch Size (Token per GPT)
25k
5k
Peak Learning Rate
5e-4
le-4
Learning Schedule
Constant
Cosine Decay
Warmup Steps
25k
#Steps #GPUs
1.8k
660k
128
128
Table 6: Post-Training configurations
SFT1
DPO1
SF12
DPO2
Batch Size (Taken per GPU)
Peak Learning Rate
2e-6
2e-7
2e-7
2e-7
Learning Schedule
Constant
Constaut
Constant
Constant
Warmup Steps
#Steps
15k
15k
#GPUs
Saamples
750k
60k
850k
20k
DPO B
0.1
0.1
DPO Cross Entropy Weight
1.0
1.0
Table 7: Inference Configuration
Modality
Method
Candidate k
CFG À
Temperature
Text
Greedy Search
Audio
Top-k Sampling
20
3.0
1.0
20

---

Published as a conference paper at ICLR 2026
C.2 AUDIO GENERATION: ABLATION STUDY
Table Ablation study of UALM-Gen and UALM showing the effect of CFG, DPO, and Enhance-
ment VAE. Objective metrics include: (1) Frechet distance (FD) using OpenL (Cramer et al., 2019) at 44. 14.1kHz; (2) Kullback-
divergence (KL) using PaSST (Koutini et al., 2022) at 32kHz; (3) Inception Score (IS) using PANNs (Kong al., 2020) 16kHz; (4) CLAP scores (CL) using LAION-CLAP (Wu et al., 2023) at 48kHz; (5) AudioBox-Aesthete score (AES) (Tjandra et al.,
2025) using an average of (CE, CU, PC, PQ) 16kHz.
Song Deseriber
AudioCaps
(kHba)
KL
Grond Trrth
0
1.88
0.48
7:200
0
13.49
0.62
4.50
Ground Troth (X-eodee roconstructed 16kila)
16
183.90
0.16
1.68
0.45
7.27
172.74
0.58
8.29
0.45
4.41
MustCen-sterro-l (Copet
2024)
32
228.94
0.84
0.36
6.64
AndioGes-M (Krouls
2022)
MAGNeTTM (2te et al., 2024)
32
191.29
0.70
1.47
0.41
6.65
140.68
1.96
7:73
0.46
4.08
AM2-L (Lin al,, 20245)
16
331.73
0.68
1.96
0.45
6.33
121.63
1.72
8.30
0.52
4.31
Make-Ana Andioo (Ilangg
20233
16
1.85
2.68
0.25
5.44
189.25
1.51
10.31
0.54
4.30
A
2024e)
TaugoFlux (Hung el al., 2024)
11.1
233,00 01
0.71
1.70
0.51
6.46
103.04
1.02
15.13
0.65
4.42
Stable Aucio Open (Evous et al. 2024)
14.1
1:58 58
1.01
2.25
0.42
6.37
100.95
2.22
11.80
0.35
4.47
ETTA (Lee 2024)
41.1
95.08
0
2.15
0.41
6.71
80.13
1.22
14.36
0.54
4.51
UALM
(w/o CFG)
UALM-Gea Gear-Bree Base
16
217.90
0.84
2.13
0.45
6.70
186.01
1.23
10.86
0.51
4.47
+ DPO
16
224.72
0.668
1.85
0.51
7.36
214.89
1.16
13:43
0.57
4.99
+ Fabanomment VAB (UALM-Gmm)
48
74.48
0.66
1.87
0.54
7.36
75.14
1.10
14.52
0.65
5.08
DPO
16
207 82
0.67
0.52
7.28
196.56
1.25
14.38
0.53
4.87
+ Enlanneeeet VAE (UALM)
48
83.60
0.59
2,00
0.54
7:28
65.87
1.35
15.62
0.62
4.92
Table shows an ablation study of UALM-Gen and UALM. We note the ablation model without DPO and the enhancement VAE module using '-Base' suffix. First, activating CFG significantly improves prompt adherence measured by CI., along with improving all other objective metrics. Ap- plying DPO provides further improvements, especially CL and AES. Finally, applying enhancement VAE gives significant improvements FD along with overall improvements of other metrics.
21

---

Published as a conference paper at ICLR 2026
C.3 DETAILS SUBJECTIVE EVALUATION OF REASONING- BASED GENERATION
We conducted a 5-scale mean opinion score analysis using Mechanical Turk for three reasoning-oriented generation scenarios described in Section 2.4. We uniformly curate 20 test prompts/dialogue for each of the below categories, with instructions for evaluators as follows:
Enrichment the model's ability to interpret creative user requests, where evaluators rated how well generated audio fulfilled imaginative descriptions (e.g.. "I need an audio track for club scene something energetic and modern").
Request: scundscape
cleaning. trying get something realyy dirty, sparkling
10111110
well
creative request?
Excellent pertectl
with minor ober described
4: Good Audio moddy lufflls the
S: Fai Audio somewhat matches user's request out looks important cloments
21 Poor- Audio banely monee what user requeeted
1: Bad Audie doos not mutch the usar's sequent at
Dialogue evaluates the model's capacity handle multi-turn conversations, where users iteratively refine their audio requirements with an assistant. Evaluators judge whether the final audio matched the user's accumulated specifications based on the dialogue.
Distogue Conversation
usor noed an audio track with an acousti sttin instrument and some poroussoo.
user. The strin relrument ahould play melocko phraso accomponied iigh peroussion Then simpl meledy jaroo metody detal??
asa stntt Can you descrice iatumeents
repettivv mekedy, how atart the audio senerntioon..
REL Pldecaanck How relevnt is the autio sample the usot's requirements from the dialogue?
Ausia eviial aaches roga calique Audio does starts scalnners dalguu
Sell-Reflection checks event correctness, focusing on semantic accuracy, requiring evaluators to whether all specified audio events were present and correctly ordered in the generated output.
Description: Chaotic ringing large metallc bells.
Remember: whether
(GNORE qualkylattracts
CORRECTNESS (Event Correctness: Are all the audio events mentioned the text present and correct? (GNOORE audio quelity/artiffcts)
E: Perfect ALL events from the present and comrec iinore audio quality
4. Good MCST events present and correct minor events might missing
3: Fair SOME important ovents present, but soveral missing wrong
2: Poor FEW events are coneet, mest are missing wrong
1: Bod Events are completely wrong missing
22

---

Published as a conference paper at ICLR 2026
D DATA CURATION DETAILS
This appendix details the data curation strategy employed for constructing our text-to-audio genera- tion training pairs.
Data Sources: We collect audio data and annotations from existing works including: Stable-Audio- Open (Evans al., 2024), ETTA (Lee al., 2024), Audio Flamingo 3 (Goel al., 2025), and AudioSetCaps (Bai et al., 2025), following the data preparation instructions of each of these works. For text-only data, we use corpus from Liu al. (2025) and Bercovich al. (2025a).
We use the full audio understanding data and text data from these sources. We only conduct synthetic data labeling and data filtering for eex--to-audio data as follows.
Labeling: For audio without provided caption. we segment them into non-overlapping 10-second segments. We use open- sourced captioning models including Qwen-2.5 5-Omni (Xu et al., 2025) and Audio Flamingo 3 (Goel et al., 2025) to generate multiple captions for each clip. These models were prompted with the instruction: "Generate audio caption for the input audio." "
Data Filtering: To ensure high data and caption quality, we improve the filtering pipeline of ETTA (Lee et al., 2024) follows:
1. De-duplication: We compute AF-CLAP (Goel al., 2025) audio embeddings and remove repetitive audio using FAISS-GPU.
2. Quality Filtering: We identify repetitive, low-quality keywords, and low- semantic-content keywords to remove low quality audio and captions. The full keywords are: ["noise", "noisy", "unclear", "muffled",
" "indistinct" "inaudible", "distorted", "garbled", "unintelligibbe"" "static", "interference" "echo", "background noise", "low volume", "choppy", "feedback", "crackling", "hissing", "fuzzy", "murmur", "buzzing", "scrambled", "faint", "broken up", "skipped", "irrelevant", "overlapping speech", "reverberation" "clipping", "sibilance", "popping", "unspecific", "gibberish", "unknown sounds", "vague", "ambiguous", "incoherent", "misheard", "uncertain", "distant", "irregular", "glitch", "skipping", "dropout", "artifact", "undermodulated" "overmoculltted" "off-mic", "misinterpretationn" "unreliable", "fluctuating", "low-quality", "low quality" "compromised", "substandard", "inferior", "deficient", "poor", "suboptimai"" "flawed", "unsatisfactory", "inadequate", "faulty", "second-rate", "mediocre", "insufficient". "lacking", "imprecise").
3. Audio-Text Alignment:
We calculated the AF-LAAP (Goel et al., 2025) audio-text sim- ilarities for each sample pair and set threshold of 0.25. We note that AF-CLAP is only used for filtering and is different from the CLAP model used for text-to-audit evaluation.
23


Reviews:
[{"review_id": "hgzqC4a1HV", "reviewer": "Reviewer_BqjX", "summary": "The paper proposes a audio language model that unifies audio understanding, text-to-audio generation, and text-only reasoning within a single framework. The model jointly learns to perceive, generate, and reason across audio and text modalities by extending a text LLM with modal alignment and post-training. The paper also introduce a caption-rich multimodal dataset that links audio with diverse textual descriptions, which enables downstream tasks, such as controllable audio generation. Experiments demonstrate competitive results on benchmarks for audio understanding, text-to-audio generation, and text-based reasoning. The paper is well written.", "questions": "The authors are encouraged to address or answer the question raise in Weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "*Unified multimodal design and empirical performance*. The paper presented a unified framework for audio understanding, text-to-audio generation, and reasoning. The proposed model achieves competitive or comparable results across diverse tasks, showing that the multimodal post training is effective.\n\n*Data contribution*. The introduction of a caption-rich dataset with detailed text descriptions for audio clips is an important resource. It supports both controllable generation and fine-grained understanding, addressing a bottleneck in current audio-language research.", "weaknesses": "*Description of data curation*. As described in the paper, the desirable performance appears to rely heavily on the richness of the captions; however, the process of dataset curation remains unclear. A more detailed explanation beyond Sections 2.4.2 and 3.1 would strengthen the paper, particularly regarding data filtering, caption generation, and quality control.\n\n*Preference design in stage 1*. I am somewhat concerned about the use of the CLAP model for DPO training, as it may not accurately capture human preferences in text–audio alignment. The authors could consider or at least discuss preference-aware alternatives such as Human-CLAP [1].\n\n*Lack of ablation*. It is also unclear how much the model benefits from each post-training stage (e.g., instruction tuning, alignment tuning, DPO). Providing ablation studies or quantitative comparisons would help clarify their relative contributions.\n\nOther minor issue:\n*Comparison with speech language model* It is recommended that the authors discuss speech language models (SLMs) and clarify how their proposed framework differs conceptually and technically from SLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pdcg7fjyLh", "reviewer": "Reviewer_Gqug", "summary": "ALM), an ambitious framework designed to unify audio understanding, text-to-audio generation, and multimodal reasoning within a single model. The work presents a three-stage approach: (1) It first develops UALM-Gen, a language model-based text-to-audio generator that, through extensive data scaling and techniques like Classifier-Free Guidance (CFG) and Direct Preference Optimization (DPO), achieves quality comparable to state-of-the-art diffusion models. (2) It then presents UALM, a single model trained via a carefully designed data blending and curriculum learning strategy to concurrently handle audio understanding, generation, and text-only reasoning tasks without significant performance degradation. (3) Finally, it introduces UALM-R1, a reasoning-enhanced model post-trained to perform novel multimodal generative reasoning tasks, such as generating audio from abstract prompts (Enrichment), interactive refinement (Dialogue), and iterative self-correction (Self-Reflection). The authors claim that UALM achieves state-of-the-art or competitive performance across all three domains in a unified manner.", "questions": "1.  Given the significant performance boost from the Enhancement VAE, would it be more transparent to present the primary results of `UALM-Base` (without enhancement) and compare those directly to other models? How does the core unified model's raw output quality stand against SOTA baselines before any post-processing?\n2.  The paper argues for the \"synergy\" of unification. Can you provide any analysis or evidence showing that the unified pre-training of `UALM` leads to improvements in one capability (e.g., generation) as a direct result of being trained on another (e.g., understanding), compared to training them separately on the same data?\n3.  The \"Self-Reflection\" mechanism is a highlight of the paper. However, it operates on a macro, \"post-hoc\" level. Have you considered or explored more fine-grained, internal self-correction mechanisms that could operate during the token-by-token generation process itself?\n4.  The success of the reasoning abilities seems tied to the structure of the \"Rich Caption.\" How robust is this approach to tasks that do not fit neatly into this predefined structure? Does the model exhibit more general, free-form planning capabilities beyond this format?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  **Ambitious and Visionary Goal:** The paper sets out with a highly ambitious and forward-looking vision: to create a single, unified model for audio intelligence that integrates understanding, generation, and reasoning. This conceptual goal is well-motivated by parallels with human cognition and represents a significant step towards more general and capable AI systems.\n\n2.  **Impressive System-Building and Engineering:** The sheer scale and complexity of the engineering effort are commendable. The authors successfully tackle numerous technical challenges, from scaling up a language model-based audio generator to rival diffusion models, to carefully balancing a multi-task, multi-modal training regime. The methodical approach, breaking down the grand vision into manageable technical problems and providing effective solutions for each, is a major strength.\n\n3.  **Pioneering Exploration of Generative Reasoning:** The introduction of multimodal reasoning for audio *generation* is the most novel and exciting contribution of this work. The concept of \"Self-Reflection\"—a cycle of generating, understanding, critiquing, and refining—is a powerful paradigm that mimics human creativity and problem-solving. This pushes the boundary beyond simple input-output generation towards more intelligent, iterative systems.", "weaknesses": "Despite the impressive scope and results, the paper suffers from several significant weaknesses that temper its contributions and raise questions about the validity of some of its core claims.\n\n1.  **Misleading Performance Claims Due to External Enhancement Module:** The paper's headline claim of achieving state-of-the-art generation quality is confounded by its reliance on a powerful, external \"Enhancement VAE.\" This module, which upsamples the model's 16kHz mono output to 48kHz stereo, is a separate, large post-processing network. The ablation study in Table 8 reveals that this VAE is responsible for a massive improvement in objective scores (e.g., FD). This raises two critical issues:\n    *   **Lack of Fairness:** The performance of the core `UALM` is conflated with the performance of this post-processor. Comparisons against other SOTA models are unfair unless those models are also afforded the same enhancement module. The true generative capability of the unified model itself remains unclear.\n    *   **Contradiction of \"Unified\" Claim:** The necessity of a separate, large enhancement module contradicts the central premise of achieving all capabilities \"in a single model.\" A truly unified model should be able to generate high-fidelity audio end-to-end.\n\n2.  **Lack of Evident Synergy in the \"Unified\" Model:** The paper’s foundational argument is that unifying understanding, generation, and reasoning will create a synergistic system where abilities mutually reinforce each other. However, the paper provides little to no evidence that this synergy occurs during the unified pre-training of the main `UALM` model. The training appears to be a form of multi-task learning, demonstrating that a single model *can* perform these tasks concurrently, but not that it performs them *better* because of the unification. The most compelling synergy (Self-Reflection) only emerges after extensive, specialized post-training in `UALM-R1`, which calls into question the actual benefits of the unified pre-training stage itself.\n\n3.  **Incomplete Literature Review:** The paper's literature survey for audio generation is not up-to-date. It omits several recent and highly influential text-to-audio generation models, such as **Make-An-Audio** and **AudioLCM**. These models represent the cutting edge of the field. To robustly claim that a language model-based approach can be \"comparable to state-of-the-art diffusion-based models,\" it is essential to benchmark against these latest and strongest competitors. The current comparison set, while respectable, is incomplete.\n\n4.  **Heavy Reliance on a Non-Transparent Intermediate Representation:** The advanced reasoning capabilities of `UALM-R1` are heavily dependent on a manually designed intermediate representation, the \"Rich Caption.\" This structured format simplifies the reasoning task into what can be seen as a sophisticated form of \"template filling.\" This raises questions about the generality of the learned reasoning. It is unclear how the model would perform on complex tasks that cannot be easily decomposed into the predefined \"Keywords, Layout, Description\" structure. This suggests the model may be learning a specific workflow rather than a general, flexible reasoning ability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1MJ5fK9pmO", "reviewer": "Reviewer_jHYo", "summary": "The paper proposes UALM, a single decoder only LLM (initialized from Qwen2.5 7B) that is extended with an audio encoder + MLP adapter for inputs and an audio codec head for outputs. It aims to unify three capabilities in one model: (i) audio understanding, (ii) text to audio generation, and (iii) multimodal reasoning that interleaves text and audio in intermediate steps. Empirically, UALM matches or outperforms strong baselines on AudioCaps and SongDescriber.", "questions": "Please refer to the weaknesses above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. \nThe design choices (25 Hz input frames, 50 Hz codec frames with 8 RVQ codes, loss scaling so one audio frame ≈ one text token) are well motivated and concrete.  \n\nS2. \nUALM Gen and UALM deliver competitive objective scores vs. leading diffusion models on both SongDescriber and AudioCaps.\n\nS3. \nThe rich caption representation (keywords, temporal layout, and detailed descriptions) plus enrichment, dialogue, and self reflection workflows are well illustrated.", "weaknesses": "W1.\nThe generation corpus is largely pseudo labeled, and DPO pair selection uses CLAP + aesthetics metrics. This raises a risk that the model (and the enhancement VAE) overfit to the judge/feature space that also underlies the reported objective scores. \n\nW2. \nUALM R1’s core claim is generation oriented reasoning, yet the subjective study uses only 20 prompts per scenario with AMT raters, and there is no standardized benchmark for audio CoT controllability (e.g., temporal ordering, count, spatial positioning) beyond demos. \n\nW3. \nThe alignment warm up is asserted to be critical, but there’s no ablation isolating its contribution vs. simply training longer/fewer steps or different adapter capacity.\n \nW4. \nWeights/code are promised “upon publication\", but several key assets (the 30M corpus, rich caption pairs, selection heuristics for DPO, and the enhancement VAE checkpoints) are not available. Given the centrality of the enhancement VAE and the data scale, releasing exact configs and checkpoints (even if data cannot be released) is important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qDZitZ0vGV", "reviewer": "Reviewer_jfVq", "summary": "This paper presents UALM, a unified audio-language model designed for understanding, generation, and reasoning. The authors further introduce UALM-R1, which demonstrates enhanced reasoning capabilities. Overall, the paper is clearly written and easy to follow.", "questions": "N.A.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "- The paper is clearly written and well organized. The accompanying demo page provides convincing qualitative results.\n\n- The three main challenges are well defined and clearly articulated.\n\n- The proposed methodology—pre-extracting audio tokens and continuing the training of a pretrained LLM—appears effective and contributes to the model’s success.\n\n- The paper provides several insightful observations, such as the comparable data requirements between LLMs and diffusion models, and the use of classifier-free guidance. These are valuable takeaways for the research community.", "weaknesses": "- The main concern lies in the dataset preparation and processing, which currently lack sufficient detail. I would be like to accept this paper if the dataset construction process is clearly described.\n\n- The paper should provide detailed information on how the 30M text–audio pairs were created, including data sources, filtering, and preprocessing steps.\n\n- In Equation (1), the meaning of the symbol \\pi is unclear — does it denote a probability distribution?\n\n- The reproducibility of the research is uncertain, as important implementation and dataset details are not provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9XUqcS0tEj", "reviewer": "Reviewer_9tTC", "summary": "This paper introduces UALM, a single AI model that unifies three audio tasks: understanding, generation, and reasoning. Instead of using separate models, UALM uses one large language model to handle everything. The special version, UALM-R1, pioneers multimodal reasoning by using intermediate \"rich captions\" (detailed audio plans) and a \"self-reflection\" capability, where it generates an audio clip, \"listens\" to its own output, critiques flaws, and then generates an improved version.", "questions": "1.In Table 2 (Audio Understanding), the paper did not provide the parameter counts for competitor models (like Audio Flamingo 3 or Qwen2.5-Omni). Can the paper provide these numbers?\n\n2.The reasoning (SFT-1/SFT-2) and generation (30M) data rely heavily on synthetic creation or pseudo-labels. Could the author detail the exact data generation pipeline for reproducibility? For example, what models and prompts were used to generate the SFT data, and how were the initial 250k \"rich captions\" created?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.This paper uses one model to do three jobs: audio understanding, audio generation, and reasoning. This is simpler and more efficient than making a separate model for each task.\n\n2.The paper introduces a \"self-reflection\" feature. The model can \"listen\" to the audio it generated, find what's wrong, and then make a better version. This helps improve the final audio quality and accuracy.", "weaknesses": "1.The comparison in many places such as Table 2 (Audio Understanding) lacks rigor. The paper does not list the specific parameter counts for competing models like SALMONN or Step-audio-2. These models often have multiple versions of different sizes (e.g., 1B, 7B, 72B). Without this, we cannot know if the comparison against UALM is fair, making the results inconclusive.\n\n2.The paper's most novel feature, \"self-reflection\" (UALM-R1), lacks strong evidence. Its evaluation (Table 4 & Appendix C.3) relies only on a tiny subjective test (20 prompts per scenario), which is not enough to prove it works reliably. Furthermore, in the demos, the \"improvement\" sounds like it just adds new audio on top of the original, rather than truly \"correcting\" or \"replacing\" the mistake.\n\n3.Limited innovation and poor reproducibility. The paper's approach relies heavily on combining existing techniques (like the Encoder-Adapter-LLM architecture and DPO) rather than introducing new methods. Its strong performance seems to come from brute-forcing the training with a massive 30 million sample (80k hours) dataset. The paper does not detail how this dataset was collected or created, and the dataset is private. This makes the work difficult to reproduce, and it's unclear if the success is due to the model's design or just the enormous amount of private data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hT0z8gkVHT", "reviewer": "Reviewer_tZVa", "summary": "This paper introduces UALM : a joint LLM model for audio understanding, text-to-audio generation, and text reasoning.\nThey train an audio-reasoning model UALM-R1 that has the ability to correct its own generation. They show that LLMs can beat diffusion models at text-to-audio generation, but have different scaling laws and require much more data. \n\nMain contributions:\n- They introduce a very carefully crafted training recipe, with modality alignment, multi-task pre-training, SFT and DPO. Showing that LLM based audio gen models need 10x data compared to diffusion audio gen models.\n- Unified understanding + generation model with state of the art results on text to audio generation- the authors plan to open-source model weights upon publication\n- First attempt at reasoning for audio generation (UALM-R1) with decent increase in performance compared to UALM", "questions": "- For DPO, you mention the cross entropy loss is necessary to adapt to generation. Do you adapt to generations (generated with CFG) from the CFG=1 model? How does this regularization compare to using different DPO beta to stick closer to base policy?\n- For the RVQ and delay pattern appendix - you describe quite clearly how this works at the output layer, how does this work at the input layer?\n- For the enhancement VAE - why use a VAE instead of e.g a GAN or diffusion model to upsample and correct the audio? This looks like a key step in getting SOTA FD metric", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- State of the art results on text-to-audio generation with a LLM, whereas all previous work pointed towards diffusion models being the strongest models for text-to-audio. The authors show that with ~10X more data, LLMs can outperform diffusion models. This is a very promising direction for future research around audio-gen LLMs and scaling laws\n- A very well engineered pipeline for model training and post-training - with details clearly laid out in the paper.\n- Ablation studies of newly introduced parameters : CFG for LLM sampling, self-adaptation before DPO,  cross entropy loss regularization for DPO\n- Comparison to previous state of the art audio-generation models on established baselines\n- Supplemental material shows audio generation of excellent quality + comparison to state of the art models for the same prompts, very well documented", "weaknesses": "- The reasoning model UALM-R1 is only evaluated on \"instruction following\" tasks but its audio quality isn't compared directly with audio metrics to the base model. Reasoning is done mostly through this rich caption text abstraction but there's no \"soft\" signal extracted from the generation to improve it.\n- The authors do not show synergies between audio-understanding and audio-generation joint training - the synergy is exploited for the reasoning model but they don't discuss if audio-understanding improves audio-generation performance or the other way around.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)