AudioX: A Unified Framework for Anything-to-Audio Generation
Authors: Zeyue Tian
Zhaoyang Liu
Yizhu Jin
Ruibin Yuan
Liumeng Xue
Xu Tan
Qifeng Chen
Wei Xue
Yike Guo
Venue: ICLR 2026 Poster (2026)
Keywords: Audio and music generation
DiT
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Audio and music generation based on flexible multimodal control signals is a widely applicable topic, with the following key challenges: 1) a unified multimodal modeling framework, and 2) large-scale, high-quality training data. As such, we propose AudioX, a unified framework for anything-to-audio generation that integrates varied multimodal conditions (i.e., text, video, image, and audio signals) in this work. The core design in this framework is a Multimodal Adaptive Fusion module, which enables the effective fusion of diverse multimodal inputs, enhancing cross-modal alignment and improving overall generation quality. To train this unified model, we construct a large-scale, high-quality dataset, IF-caps, comprising over 7 million samples curated through a structured data annotation pipeline. This dataset provides comprehensive supervision for multimodal-conditioned audio generation. We benchmark AudioX against state-of-the-art methods across a wide range of tasks, finding that our model achieves superior performance, especially in text-to-audio and text-to-music generation. These results demonstrate our method is capable of audio generation under multimodal control signals, showing powerful instruction-following potential. We will release the code, model, and dataset.

Published as a conference paper at ICLR 2026
AUDIOX: A UNIFIED FRAMEWORK FOR ANYTHING- TO-AUDIO GENERATION
Zeyue Tian', Zhaoyang Liu', Yizhu Jin', Ruibin Yuan', Liumeng Xue', Xu Tan?, Qifeng Chen', Wei Xuet Yike Guotl
Hong Kong University of Science and Technology Independent Researcher
T2A-bench
HHA Audio
VGGBBaad Iionsto-daade
Muci
32h6-bench Tay Mesie
Dya Music
AndioCaps
VEM-bondh
444
Polonus Meric
AVVP
VOKISound
Calax
Ord--ae
Dppinting
Emageds-daal
Audio Time
VCGSsuml
Jusbo
WVV2
XVVP FMdr-s--Aaaa
Seas Sal--allesoddall
VOGScuud
Ordering
FrequRAY
AuskoGann
AubloLWw2
Malio An Audiit
IIAALCO
Tangs 2
Stabls Asaio Opan
Ouri
(a) Comprchensive performance comparison
(b) cnstruction-oolloinnn text-to-audio comparison
Figure 1: Performance comparison of AudioX against baselines. (a) Comprehensive comparison across multiple benchmarks via Inception Score. (b) Results on fostloction-followiing benchmarks.
ABSTRACT
Audio and music generation based on flexible multimodal control signals is a widely applicable topic, with the following key challenges: 1) a unified multi- modal modeling framework, and 2) large-scale, high-quality training data. As such, we propose AudioX, a unified framework for soything-to-audin generation that integrates varied multimodal conditions (i.e., text, video, and audio signals) in this work. The core design in this framework is a Multimodal Adaptive Fusion module, which enables the effective fusion of diverse multimodal inputs, enhane- ing mooss- alignment and improving overall generation quality. To train this unified model, construct large-scale, high-quality dataset, IF-caps, compris- ing over million samples curated through structured data annotation pipeline. This dataset provides comprehensive supervision for multimodal-conditooeel au- dio generation. We benchmark AudioX against state-of-the the-thh-rrt methods across a wide range of tasks, finding that our model achieves superior performance, espe- cially in text-to-audio and vext-to-must generation. These results demonstrate our method capable audio generation under multimodal control signals, showing powerful instraction-followiii potential. The code and datasets will be available at /eyuet..eeeetee/ee github. io/ Lo/AudioX/
"Corresoondigg Authors

---

Published as a conference paper at ICLR 2026
1 INTRODUCTION
In recent years, audio generation, especially for sound effects and music, has emerged as a crucial element in multimedia creation, showing practical values in enhancing user experiences across a wide range of applications. For example, in social media, film production, and video games, sound effects and music significantly intensify emotional resonance and engagement with the audience. The ability to create high-quality audio not only enriches multimedia content but also opens up new avenues for creative expression.
However, the manual production of audio is time-consuming and requires specialized skills, pre- senting a compelling research opportunity to automate audio generation. Despite notable advance- ments (Liu et al. 2023 Copet al., 20244 Wang et al. 2024), the field has predominantly focused on specialized models with constrained inputs and outputs. These models often operate with a single conditioning modality, such as text-to-audio or video-to-audio, and are typically restricted to single output domain, like generating either sound effects (Cheng et al. 2025) or music (Tian
al. 2025) exclusively. While recent trend towards unification is emerging, with some pioneering works accommodating multiple inputs (Polyak et al., 2024] Zhang et al. 2024), they often lack the flexibility to support diverse modal combinations and exhibit weak destruction-olloviin
abilities. As result, the potential of unified models still remains underexplored. We find that a major factor behind these limitations is the scareity of high-quality, multimodal data suitable for training unified systems. Existing datasets are often task-specific, typically providing supervision for only one con- ditioning modality, such as text-to-audio (Kim et al. 2019), video-to-audio (Chen et al. 2020), or video-to-music (Tian et al. 2025). This lack of datasets with diverse and combinable control signals has significantly hindered the development and training of unified models.
To this end, propose unified framework termed AudioX for anything-to--an to-auioi generation. We observe that Transformer- basedd works (Wu et al.] 2023a Liu et al.) [224b]] Lin et al. (202) have ef- fectively tackled multi-modal alignment, and build on this success by incorporating Transformer- based methods into our framework for multi-modal condition handling. Furthermore, diffusion mod- els have increasingly become leading-edge techniques in the field of high-quality audio and music generation (Evans et al. 22024a b), outperforming next-token prediction in terms of audio fidelity (Evans et al.) 2024a] Majumder et al. 2024). Therefore, we mainly build on Diffusion Transformer (DiT) to unity multimodal conditions and generate high-fidelity audio. To further enhance mul- timodal representation learning and alignment, we introduce lightweight Multimodal Adaptive Fusion module that adaptively weights and aligns conditioning modalities before fusion, enabling stronger cross- modal control and yielding significant improvements in generation quality.
To support the training of unified model, we designed pipeline using structured annotation and data augmentation to build IF-caps (Instruction- Folllwion-..looiing). a large-scale, high-quality multimodal dataset. The dataset serves as robust foundation for our approach, containing over 1.3 million general audio samples and 5.7 million music samples. Training on this large-scale, fine- fine-graine grained dataset allows our model to handle flexible multimodal conditions and generate diverse audio genres, including music and sound effects. Consequently, AudioX enables a range of tasks, including text- to-audio generation, video-to-audio generation, audio inpainting, and text-guided music completion.
With this unified design and trained on our large-scale dataset, our model demonstrates exceptional performance and strong mnstruction-folloiing capabilities. To validate our model's capabilities, we benchmark it against state-of-the-ar methods across a comprehensive suite of tasks and established benchmarks. In addition, rigorously evaluate its inslrwction-following ability on T2A tasks, we construct a new benchmark, T2A-bench. As demonstrated in Sec. 5.3 AudioX achieves of the-art or comparable results across multiple benchmarks and various tasks, while substantially out- performing prior methods in instraction-followe training approach is that we observe a cross-modal regularization effect under unified training: improving the quality and granularity of textual supervision reduces alignment noise and leads to better modality alignment, which jointly improves performance across conditioning modalities (see Sec. 5.4) This observation provides empirical insight for future multimodal audio generation.
capabilities. A notable finding from our unified
In summary, the main contributions of this work are as follows: 1) We propose AudioX, a unified framework for anything-to -oything-to-audio generation that overcomes the limitations of constrained inputs and outputs. The proposed framework supports audio and music generation from varied multi-modal conditions, contributing to a new insight into studying generalist models for audio generation.
2

---

Published as a conference paper at ICLR 2026
2) To overcome data scarcity for unified training. design data curation pipeline and construct a large-scale, high-quality dataset, IF-caps, containing over million samples with fine- -reigra annotations.
3) We conduct comprchensive experiments on a wide array of tasks, systematically benchmarking state-of-the-art methods categorized by their input modalities and output domains. Our extensive periments demonstrate our model's 's strong multi-task capabilities and superior cntrructioo-follwiing ability.
2 RELATED WORK
Audio and music generation. Recent advances in deep generative models have greatly broadened the scope of audio and music synthesis. However, most existing methods remain confined to a single
modality or support only limited types of conditioning. For instance, text- coct-to-audic approaches (Liu
itt al. 2023; Majumder et al.] 2024: [Evans et al., [2024||b Diang et al. [025] Huang et al.] 2023]
He et al. 2024) focus on generating diverse soundscapes from textual prompts, while text-lo-music
systems Copet et al. 2024 Ghosal et al. 2023: Deng et al.] 2024 Yuan et al. 2024 2025 Ma
al. 2024) specialize in composing coherent musical pieces. Separate lines of work tackle tasks 2023 2024a). primarily with text conditioning, Mcanwhile, video-
like audio inpainting (Liu et al.
to-audio methods (Zhang et al. 2024 Luo et al. 2024; Wang et al.
2024 Polyak et al.) 2024] Chen
et al. 2024) typically generate foley or environmental sounds synchronized to visual cues. Some of
these also incorporate text for additional context, thereby bridging visual and textual modalities. Be-
yond sound effects, video-to-music approaches (Kang et al. 2024 Liu et al. [2024c] Di et al.) [202T]
Tian et al. 2025 Li et al., 2024b) [in et al.] 2024] et al. 2024a) align musical compositions with the visual content to enhance narrative depth in multimedia applications. Despite these advances, current frameworks often specialize in only one modality or rely on a limited set of input conditions, hindering multi-task adaptation and restricting their ability to scale transfer knowledge across related tasks. In contrast, our unified approach supports both audio and music generation for a broad range of input
text, video, and audio all within single framework.
Audio Datasets. While substantial research efforts have led to the creation of valuable datasets for spe- cific tasks like text-to-audio (Kim et al., 2019: Xue al. 2025 Drossos et al.
1- bn I- do
2020
Wu et al. 2023b).
mexi-to-music (Copet et al.
2024
Liu et al. 2024c
Ramires al. 2020), video-to-audio (Chen et al.
Gemini
2020 Hershey et al. 2021 Tian et al., 2020), and
Video-to-music (Tian al.
2025| Zhou al.] 2025
Chi et al. 2024), their utility for training generalist unified model remains limited. These resources are typically constrained to single conditioning modal- ity and a narrow output domain (c.g., only sound effects only music). This fragmentation of data has significantly hindered progress towards devel- oping more versatile and robust systems. To over- come this critical data scarcity, we introduce a large- scale, multimodal dataset constructed via novel notation and augmentation pipeline, specifically de- signed to provide the comprehensive supervision re- quired for unified audio and music generation.
Raw caption
Audio
Music
Audio Caption
Mustic Caption
Category & Count
Genre & Mood
Ordering Relation
Instrument
Sound Event Detection
Tempo
Qwen2-Audke
Diffusion models. Denoising diffusion models (Ho) et al. 2020 [Song et al.] (020) have become a cor- nerstone of modern generative modeling, achiev- ing state-of-the-art results in image (Rombach et al.
Augmentation
2022
Ramesh et al.) 2022; Brooks et al.) 2003).
Figure 2: Dataset process pipeline.
video
(Chen et al. 2023] Ho et al.] 2022 Guo et al.
2023
He et al. 2024). and audio synthesis (Popov
al.
2021 Jeong al. 2021 Liu et al. 2024a 2023
2022
Evans et al.
2024a). However, their
3

---

Published as a conference paper at ICLR 2026
application in the audio domain has predominantly been limited single tasks (e.g.. text- to-audio), falling short of the more generalized "anything-te toaiiniiot-uioo" scenarios where inputs can be multimodal. To bridge this gap, unified framework leverages the power of diffusion models for multi-condition generation, offering a more flexible and universal solution.
DATASET PROCESS
Existing audio datasets often lack the high-quality, multimodal conditioning signals necessary to train versatile, unified models. To address this gap, we designed an effective annotation pipeline that processes existing video datasets (Chen et al. Hershey et al. 202T] Tian et al. 2025). allowing us to construct IF-caps, large-scale dataset with diverse, mooli conditions. Our pipeline, as shown in Fig. operates as follows: First, we employ powerful multimodal LLM (Gemini 2.5 Pro) to generate comprehensive set of initial annotations by processing the audio track of each 10- second video-audio clip. These annotations consist a holistic global caption and set of structured fields. For general audio, these fields include sound event classification and count; for music, they specify attributes like genre and instrumentation. Then, since using the resource-intensive Gemini model for the entire dataset is costly, we leverage the open-source Qwen2-Audio (Chu et al. 2024) model to augment these structured fields at a large scale. Conditioned on both the initial annotations and the raw audio, the model generates varied captions, enhancing data diversity while manag- ing costs. Finally, this process yields comprehensive, fine-grained captions for approximately 1.3 million rideo-audio clips and 5.7 million video-musie clips. The diversity of our curated dataset More details and samples of our annotated data are is highlighted by the word clouds in Fig. 3 provided in the Appendix 1..
Clapping Shout
Seo) Mechan S
cuee currmmssi electron C
Speech Tit vehicle
inst a
Singing
electri
R RICK
Wind
Tock
000
Cr Crowd
d
amb
Lenn
guitar UMS
bpn
Breathine
Lemp
Triing
Symbat
Girnny
BUZZ
Beep
audio
music
Figure 3: Word clouds for our curated dataset, illustrating the diversity of terms for the general audio (left) and music (right) domains.
METHOD
4.1
MODEL DESIGN
Our framework, AudioX, as shown in Fig. 4 is built upon a DiT backbone designed for high- fidelity audio synthesis. Given video X., text X., and audio X., each modality is passed through corresponding specialized encoders. To capture the temporal dynamics, the resulting video and audio features are then processed by temporal transformer. Finally, the features from all three modalities are mapped through projection head to produce the domain-specine embeddings (H.,, H. H.) These embeddings are then fused into a unified condition embedding, which ultimately passed to the Diffusion Transformer to guide the generation process.
key challenge in training a unified model is that signals from different modalities can interfere with each other, making effective fusion and well-aligned conditioning critical. To address this, we introduce the lightweight Multimodal Adaptive Fusion (MAF) module. As shown in Fig. (right), the MAF module operates as follows: First, the initial feature embeddings from each modality are fed into gates, which filter and reweight them to suppress noise and retain the most informative cues. concatenated and attended by learnable queries via cross-attention. These queries are organized into three modality -speciffc sets, acting as experts to assess and ag- gregate evidence across the different data streams. Finally, self-attention layer consolidates this
Next, the gated embeddings
4

---

Published as a conference paper at ICLR 2026
Trinable
G7 Jones
Audio Eneader
--+
Fresx
Diffusion Transformer
Setf-Att
Multimodal Adaptive Fusion Module
Prujection
Projection
Projection
Cuncut
Visual Encoder
Text Encoder
Audio Encoder
volscie eviffity the distance.
Fideo
Tert
Ando
Figure 4: The AudioX Framework. Specialized encoders process diverse modalities, and a MAF module unifies these signals into conditioning embedding He. The DiT backbone processes the noisy latent input 2, conditioning on He via cross-attention to generate high-quality audio and music. (ze and H. notations omitted for visual clarity).
aggregated context, and the refined information is dispatched back to the modality paths via residual updates. This process yields calibrated, modality-specific outputs which are then concatenated to form the final multimodal condition embedding, Ho:
H. H. H. = MAF(H,, H. H.), H. Concat H., H. H
(1)
This final embedding, along with a diffusion timestep I, is what conditions the DiT backbone for the final audio synthesis. As we demonstrate in our ablation studies (Sec. (.4)), the MAF module is essential for reducing cross-modal interference while improving both the overall generation quality on multimodal tasks and the model's instruction-followiin capabilities.
4.2 TRAINING
The objective of the training process is to effectively integrate multimodal inputs and optimize the DiT model for generating high-quality audio music through robust diffusion and denoising framework. The details of the training data are provided training. for each pair (X,. X. X. A). where A is the ground truth we aim generate, the pair lacks video or audio modality input, we use zero-pus zero-padding padding to fill the missing modality. If it lacks text modality input, we substitute with natural language descriptions, such as "Generate music for the video." for the video-to-musé generation task. For the tasks of audio inpainting and music completion, the audio modality input is required. In audio inpainting, X, is a masked version of the ground truth audio A, and the model's objective is to fill in the masked sections. For music completion, X, is the preceding music segment of A, and the model aims to generate the subsequent music segment of X..
Table A. in the Appendix. During
Diffusion process. The DiT model processes the multimodal embedding H. in the latent space through a denoising diffusion process. Initially, the ground truth A is encoded using encoder E, which projects A into the latent space, yielding the latent representation z = {(A). The data then undergoes a forward diffusion process, producing noisy latent states at each timestep t.
The forward diffusion defined as a Markov process over timesteps, where the latent state at timestep / is produced based the latent state
q(2, 24-1) N(z;; V1 8,4,11,811).
where 3, represents the predefined variance at timestep t, and N denotes a Gaussian distribution. The forward diffusion process gradually adds noise the latent state.
5

---

Published as a conference paper at ICLR 2026
The reverse denoising process involves training transformer network eo gradually remove noise at each timestep and reconstruct the clean data. The reverse process is modeled as follows:
pa (Z--1 = N (Z--1; pe (Zr, L, Ho), Ee (Ze. t, H.)),
(3)
where M4 and Eg are the predicted mean and covariance of the reverse diffusion, conditioned 2,, t, and H.. These parameters define the Gaussian distribution from which 2,-1 sampled.
The denoiser network takes input the noisy latent state Zz, timestep i, and the multimodal condition embedding H.. The goal is to minimize the noise estimation error at each timestep. which is formulated as:
min E,, ||c co (Ze, He)l
where is the simulated noise at timestep and eo (ze, L, Ho) is the predicted noise from the model. The training objective is to minimize the mean squared error between the simulated and predicted noise across all timesteps.
By training the DiT model in this manner, we effectively unify multimodal inputs into a latent space, enabling the generation of high-quality audio or music that is coherent and aligned with the input conditions.
EXPERIMENTS
In this section, we provide the implementation details our experiments and conduct extensive evaluations. These assessments comprehensively measure the effectiveness of our proposed method from both subjective and objective viewpoints. The evaluations aim to offer valuable insights into the generation of audio and music from various inputs.
5.1 IMPLEMENTATION DETAILS
For encoding the visual features, we use CLIP-ViT-B/32 (Radford et al. 2021) to extract video frame features at rate of fps, and Synchformer (lashin et al. (024] to extract synchronization features at 25 fps. The CLIP and Synchformer features are fused via addition. The text inputs encoded using T5-base (Raffel et al.] 2020), while the audio is encoded and decoded using audio Autoencoder (Evans et al. 2024b). The model has a total of 2.4B parameters (1.1B trainable). Our proposed MAF module constitutes only 60M of these parameters, highlighting its lightweight nature. The DiT model, consisting of 24 layers, uses a pretrained model from (Evans et al. 2024b).
The training process the AdamW optimizer with base learning rate of 1e-5, weight decay of 0.001, and a learning rate scheduler incorporating exponential ramp-up and decay phases. To im- prove inference stability, maintain exponential moving average the model weights. Training is conducted on three clusters of NVIDIA H800 GPUs, each with 80GB of memory, requiring ap- proximately GPU hours in total. The batch size steps using classifier-free guidance with scale of 7.0. Please refer to Appendix A.I.1 for further details on our training and evaluation datasets.
48. During inference, we perform 250
5.2 EVALUATION METRICS
To provide a comprehensive assessment of our model, we employ a suite of objective and subjective metrics. Further details for each metric are provided in the Appendix A.2
Objective Evaluation. For overall audio quality and semantic alignment, we use several estab- lished metrics. These include: Kullback-Leibler Divergence (KL); Inception Score (IS); Fréchet Distance (FD) with PANNs embeddings (Kong et al.) 2020); Fréchet Audio Distance (FAD) with VGGish embeddings (Hershey et al., 2017); Production Complexity (PC) and Production Quality (PQ) TTannra et al., 2025). As a prompt-free metric for both quality and diversity, we chose IS for the unified comparison in Fig. T For alignment,
the CLAP score (Wu et al. 2023b) for text inputs and the Imagebind AV score (Girdhar et al. 2023) for video inputs. To assess the model's instruction-follwviii capabilities in 12A, report metrics
benchmarks. On proposed

---

Published as a conference paper at ICLR 2026
Table
Performance evaluation across various tasks and datasets. Task abbreviations are: '2A (Text-to-Audio), V2A Aideo-to-Audio) TV2A (Text-and- Video-to-Audics T2M (Text-to-Music), V2M (Video-to-Musiis and TV2M (Text-and- Video-to-Musiin For alignment (Align.), we use the CLAP score for text and the Imagebind AV score for video inputs.
Dataset
Method
Task
KL 4
IS 1
FD
FAD
PC
PQ
Align.
AudioCien(Kreul al. 20128]
T2A
1.39
10.22
13.29
1.72
3.26
5.25
0.27
Audiol ADM-t (eurecii Alpas
T2A
2.00
6.51
37.27
8.37
2.82
5.67
0.20
AMdioLDDM22d (DM0-222222 Rerpeee pe
1.49
1.97
2.86
0.22
AudioCaps
Tango 2Majumder Caii 20223
Stuble Audiio Openn Exatt Eeats
2024b
T2A
2.01
10.37
3.15
2.77
6.16
0.21
MAGGNET-Jazzd ao Zit dive r
T2A
1.62
7,46
24.88
2.99
3.25
5.15
0.15
MMAudio Chenty or al. 2023
T2A
1.35
12.03
12.63
4.71
3.06
5.64
0.30
AudhoX
T2A
1.27
12.48
11.51
1.59
3.32
5.80
0.30
AudioGen Geeuueuuk
2022 civeseut
2.16
3.30
AudioLDMi
7.05
Audiol .D.22-1 I PFlree 20244
2.10
2,05
Tango MMajumdr Cali M17G)
T2A
2.31
10.00
22.96
3.47
3.93
0.29
Stable Alndo Exa In 2024b
T2A
2.36
14.45
26.00
2.00
2.64
6.53
0.33
MACNET-Jarse larg 21 Ztreeu Cour A044
T2A
2.03
8,53
22.17
2.74
3.65
5.25
0.26
MMAudin ( C Cheny y
2025)
T2A
2.17
17.83
11.52
2.50
XOZ
6.12
D.32
VGGSound
AudioX
T2A
1.74
19.58
9.01
1.33
3.34
6.31
0.33
& Hearing Xine
2024
Foley CreyCrrftet//// Chanc 8024
2.39
2.23
3.31
0.27
Doll-Foley to oeal 201
V2A
3,01
$6.54
5,89
2.57
5.85
0.20
FRIEREN
Warr ar
VZA
2.58
6.91
50.88
3.13
2.98
6.06
0.20
MMAudio Audio Chenry c1 2023
V2A
1.97
14,95
6.18
2.04
3.38
5.91
0.35
AudioX
V2A
2.21
12.60
7.84
1.28
3.49
6.21
0,26
Foley
Zhans
3.38
0.26
MMAudio C bongg of 2025
TV2A
1.51
17.79
6.60
2.20
3.31
5.99
0.33
AndioX
TV2A
T.48
17.91
6.97
1.06
3.46
6.29
0.26
& Xing 2024
V2A
2.30
4.02
40.38
8.66
3.64
5.16
0.35
FolevCraften// C
V2A
2.13
28.68 68
3,77
3.25
3.87
0.28
Difl Diff-Foley
3.14
10.95
2.55
FRIEREN Nanyy
AVVP
MMAudio C Chong crall
V2A
1.22
13,51
3.55
5.89
0.34
AudioX
V2A
1.89
8.60
17.2
2.24
3.65
6.09
0.28
FoleyCrafted Zhane
2024
TV2A
1S1
6.22
26.76
2.85
3.62
5.60
0.27
MMAudio Chang
IV2A
1.74
9.52
2.74
3.64
0.34
3,65
MusicGen
Copet Copet et a 2024
T2M
1.43
2.24
25.40
4.35
5.19
1.16
0.18
Audiol DDM- Poth preeati ai R028
T2M
1.45
2.49
34.44
6.34
4.72
6.10
0.22
2-1 arger DM-2
an 20240
T2M
1.26
2.84
15.61
2.80
5.22
6.70
0.23
MusicCaps
TangoMusic Ghosal beati
T2M
1.13
2.86
15.00
1.88
5.57
7,06
0.23
Stable Audio Pxent
2024b
T2M
T.5T
2.94
36.33
3,23
3.91
0.23
MAGNET- largellrgelll (Zil AudioX
T2M
0.96
3.55
9.76
5.21
6.70
0.24
MusieGen Copet all 12024
T2M
0.76
1.31
40.59
3.25
5.57
7.43
0.14
ADdioLDME=t Po-i Cil 60243
T2M
0.72
1.37
36.63
2.97
5.08
7.01
0.16
AMdioLDM- 2-----1-- ard uore
susai
T2M
0.62
1.46
25.80
1.63
5.57
6.90
0.14
TangoMusic
(aalussiiCGoaaae Chosa bratin 210253
T2M
0.72
T,46
19.
2.43
5.78
7.46
0.14
Stable Open Fivans i
MAGNET-large Zit diteerat 2024
T2M
0.60
3.15
7,04
V2M bench
AudioX
T2M
0.47
1.50
19.62
5.91
7.12
0.14
Video2Musia Kanp et al 2044
V2M
1.78
1.01
144.88
18.72
3.34
8.14
0.14
MuMu LaM fi 20246
V2M
1.00
1.25
$2.25
5.10
5.60
T:97
0.18
CMT st 20211
V2M
1.22
1.24
85.70
8.64
4.98
8.20
0.12
VidMosar
2025
V2M
7.04
AudioX
TV2M
0.45
1.52
18.64
1.44
5.42
7.24
0.22
T2A-bench (detailed in Appendix A.3 racy (Cat-ace, Cnt-acc, Ord-acc, TS-acc). On AudioTTime (Xie et al.) 2025), we use its established metrics for Ordering, Duration, Frequency, and Timestamp.
we measure category, count, ordering, and timestamp aceu-
Subjective Evaluation. We conducted formal user study with professional audio experts to evaluate the subjective quality of generated samples against baselines. The study followed the established methodologics of prior work (Kreuk et al. 2022 Liu et al.) 2023), where experts rated anonymized samples from I to 100 on Overall Quality (OVL) and Relevance (REL) to the prompt.
5.3
MAIN RESULTS
This work introduces a unified model capable of generating audio and music from flexible combina- tions of video, text, and audio inputs. Through extensive experimentation, benchmark model against SOTA specialist models across all supported tasks. Results demonstrate that our single model consistently achieves SOTA highly competitive performance on the majority of metrics.
7

---

Published as a conference paper at ICLR 2026
Audio generation. Results of our audio generation are in Table I, which includes the outcomes of generating audio or music from any combination of video and text modalities. The upper part of the table presents the audio generation tasks, while the lower part displays the music generation tasks.
For text-to-audio generation, we evaluate on the AudioCaps (Kim et al, 2019) and VGGSound model achieves SOTA performance, while on
(Chen et al., 2020) datasets. On AudioCaps, VGGSound, the advantage is even more pronounced. This demonstrates that our model is powerful text-to-audio generator. Furthermore, both our model and baseline results on VGGSound confirm the effectiveness of our curated caption data. For video-to-audio generation, we experiment on VGGSound and AVVP (Tian al., 2020), AVVP is an out-of-domain test dataset for all methods. Our model achieves results comparable to SOTA on both VGGSound and AVVP, proving that it not only strong video-to-audin generator but also exhibits excellent generalization on out-of- domain datasets. For audio generation conditioned on both text and video, we benchmark against the strong baselines FoleyCrafter (Zhang et al. 2024) and MMAudio (Cheng al. 2025), achieving results that are comparable to them. We find that when both text and video inputs are provided, the model effectively integrates the information from both modalities to generate better results.
The bottom part of Table [ shows the results of music generation tasks. On the V2M dataset (Tian) The text-
jet al., 2025), we evaluate -usii,, aideo-t-musse and sileo--nd-teet-to-meee to-music task is additionally evaluated on the MusicCaps (Copet et al.) 2024) dataset. Our model achieves SOTA performance across these tasks, demonstrating its effectiveness in generating high- quality music conditioned diverse inputs.
Table 2: Evaluation of instructioo T2A ability on the T2A-bench and AudioTime.
T2A-bench
TideoTime
Method
Cat-acc 1
Cut-ace 1
Ord-acc
TS-ace
Ordering
Duration 1
Frequency
Timestamp"
AudioGen
5.40
0.91
3,73
1.58
0.54
ADdiol.DM
18.60
4.00
3.40
11.60
0.97
3.41
1.54
0.41
AD-
20.10
7.40
1,20
13.40
0.96
3,40
1.64
0.54
Tango 2
25.20
4.60
10.20
18.80
0.86
3,70
1.52
0.61
Make-An An-Andin
4.00
0.76
1.42
0.56
Stable Audio Open
31.20
9.80
6.00
21.80
0.9%
3.07
1.46
0.53
MMAudio
26.60
4.80
2.40
21 40
0.98
3.33
1.54
0.50
AudioX
34.20
12.40
23.60
28.20
0.34
1.30
0.74
0.81
cnstruction-folloviin text-to-audio generation. As shown in Figure and Table 2 AudioX sub- stantially outperforms all baselines in tasks requiring fine-grat fine-grained control. On our T2A-bench, Au- dioX demonstrates a commanding lead across all dimensions, from category generation to count and temporal control. For instance, it surpasses the temporally-enhanncl Make-An-Audi22 base- line Ord-acc. This advantage is reaffirmed on the AudioTime benchmark. We also note that the performance trends between AudioTime's 's Ordering metric and our Ord-acc are consistent across all models, which helps validate the design of our benchmark for evaluating temporal adherence. Fur- thermore, an insight from these comparisons is that high audio fidelity does not necessarily correlate with enstruciion-ollowiii
prowess. For instance, Tango2, despite its high-quality synthesis, delivers only moderate performance these control-focused metrics. Collectively, these results underscore our model's 's superior fine-grained control, setting a new standard for controllable T2A generation.
User study. We conducted user study to evaluate the quality of the generated audio and music. We randomly selected 25 samples for each audio generation task, including T2A, T2M, V2A, and V2M. 10 audio experts are asked to rate the quality of the generated audio and music. The results are shown in Fig. A.2 in the Appendix. The evaluation shows that our model achieves subjective SOTA performance terms of OVL and REL scores in most tasks, indicating high user satisfaction.
To further demonstrate the versatility of our model, we present results for additional tasks, including audio inpainting, music completion, and image-to-audio generation. in Appendix(AA44 The results further underscore our model's 's strong performance and broad applicability across variety of audio generation tasks.
5.4 ABLATION STUDY
In this section, we conduct a series of ablation studies to investigate the contribution of our key design choices. We systematically validate the efficacy of our data curation strategy and the
8

---

Published as a conference paper at ICLR 2026
chitectural integrity of the proposed MAF module. An additional ablation study the impact of different conditioning modalities detailed in Appendix A.4.2
Efficacy of data curation strategy.
To verify the impact our data curation strategy, we evaluate models trained on different textual
supervision sources (Table 3): 1) Labels: using raw class labels from the source datasets; 2)
MadioSSeCaps: using captions from a recent concurrent dataset (Bai et al. 2025); ; 3) QwenCap: using captions generated directly by Qwen2-Audio;
GeminiCap: using only the initial annota- tions generated by Gemini 2.5 Pro; and GeminiCap-auue our full pipeline. The results show that Gemi n Cap-aug outperforms all baselines, including the external AudioSetCaps dataset and the single-stage generation methods. It not only achieves the best scores on general-purposs tasks (T2A, V2A, TV2A) but also enhances the model's instruction-followinn capabilities. Collectively, these results validate the superior quality of our constructed dataset and the effectiveness of the proposed two-stage curation pipeline. Notably, we observe that the benefits of high-quality textual supervision are not limited lext-to-audio generation. The marked improvement in the V2A task provides strong empirical evidence of cross-modal regularization effect. This insight leads to crucial conclusion for future work: high-quality textual data should be viewed not only as input, but also as an effective strategy for building more capable and robust multimodal models.
Table Ablation study on data curation strategies. We compare our model's performance when trained with captions from different sources. The results show clear trend of improvement with higher-quality data. Our full pipeline (Gemi Cap-aug) not only achieves the best performance on all general tasks (T2A, V2A, TV2A) but also essential for enabling fine-grained control.
Caption Method
lnstlwntionfoolooiingg T2A
T2A
V2A
TV2A
Cat-ace f Cnt-ace f
Ord-ace f
f
FAD 1
IS f
FAD 1
IS
FAD
|
Labels
17.35
2.80
4.60
7.59
6.02
10.46
1.81
10.62
3.41
AudioSetCaps
27.85
6.40
4.80
10.08
3.19
11.35
1.33
12.39
1.56
OwenCap
24.60
6.40
6.20
9.74
4.40
10.57
1.67
11.79
1.95
GeminiCap
28.05
9.60
7.60
10.81
3.02
11.48
1.31
12.78
1.70
GeminiCap-an
28.91
10.20
7,80
T0.93
2.91
11.69
1.15
12.90
1.48
Table 4: Ablation study of the MAF architecture components. We evaluate the contribution of the Gate and Query mechanisms by removing them individually. The results show that the Full MAF, which includes both components, achieves the best performance across most metrics. This confirms that our complete design is essential for effective multimodal fusion.
Components
Gate
Query
KL
IS
f
FD 4
FAD 4.
Duration Frequency Ordering
w/o MAF
x
x
1.83
10.70
11,60
2.67
3.022
1.359
0.912
w/o Gate
x
1.69
11.66
9.72
2.00
2.945
1.348
0.876
w/o Query
1.71
11.72
9.65
2.08
2,841
1.328
0.912
Full MAF
1.68
11.84
9.64
1.98
2.827
1.302
0.888
Architectural ablation of the MAF module. We conduct an architectural ablation of the MAF module to validate its design (Table The results confirm that each component is integral, with the most severe performance deterioration observed when the MAF module is omitted entirely. Re- moving the Gate mechanism the Query-based attention individually also results in performance decline, confirming their respective contributions. This analysis validates our design choices, un- derscoring that the complete MAF architecture is critical for optimal multimodal fusion, thereby enhancing cross- modal alignment and improving generation quality.
5.5 DISCUSSION
Our extensive experiments provide a multi-faceted validation of AudioX, consistently demonstrating state-of-the-art performance from broad audio generation to commanding lead in fine-grained instruction-follont following..... Our ablation studies confirm that this success directly attributable to two principles: data curation strategy that provides rich semantic foundation via a powerful cross-mos cross-modal regularization effect, and an MAF architecture essential for translating these signals into precisely controlled outputs. The synergy between this data-centric foundation and purpose-

---

Published as a conference paper at ICLR 2026
built architecture culminates in model'ss SOTA performance challenging instruction-folloiinn following benchmarks, validating our approach for unifying generative versatility with fine-grained control.
CONCLUSION
In this work, we present AudioX, unified framework that transcends the modality and domain con- straints prevalent in prior specialist models for audio generation. By leveraging DiT backbone and our designed MAF module, our model effectively unifies diverse inputs like text, video, and audio produce high-quality outputs. The training of our model is supported by IF-caps, our large-scale, fine-grained dataset, which provides a robust foundation for unified training and evaluation. No- tably, our training methodology induces an effective cross-modal regularization effect, enhancing the model's 's internal representations. Extensive experiments demonstrate that our single, unified model only matches or outperforms specialist models but also unlocks superior instruction instruction-folooiing capabilities, showcasing its command of both generative versatility and fine-grained control.
7
ACKNOWLEDGMENT
The research was supported by the Early Career Scheme (ECS-Hong Kong University of Science and Technology 22201322), the Theme-based Research Scheme (T45-205/21-N), as well as the Generative AI Research and Development Center from InnoHK. We thank Fortuna Peng for helpful discussions and Jingyi for assistance with the figures.
ETHICS STATEMENT
The authors have adhered to the ICLR Code of Ethics. Our work is intended to foster positive and creative applications in media, and we have carefully considered the ethical implications of our search. The IF-caps dataset is curated exclusively from publicly available sources, and its release will fully comply with the licenses of the original material. Our human evaluation involved compen- sated professional experts who provided informed consent. No conflicts of interest or sponsorship bias exist in this work, and all authors adhere to research integrity practices, including transparent documentation of data sources, collection procedures, and evaluation protocols. By committing to open-soureing our code, model, and dataset, we aim to ensure transparency and support the commu- nity in the responsible advancement of generative audio technology.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our work and have provided comprehensive details of our methodology and experiments. The complete architecture of our AudioX frame- work, including the Multimodal Adaptive Fusion module, is detailed in Section 4. and visualized in Figure All implementation details, including optimizer settings, learning rates, batch sizes, and computational resources, provided in Section Our data curation pipeline for constructing the IF-caps dataset is described in Section 3 with further details on the annotation schema, data statis- ties, and augmentation samples available in Appendix(A.I.I Our full evaluation protocol, including precise definitions for all objective and subjective metrics and the design of our new benchmark, T2A-bench, is detailed in Section 5.2 Appendix A.2 and Appendix A.3] To further aid replication and future research, we will open-source our code, pretrained model checkpoints, and the complete IF-caps dataset upon publication.
REFERENCES
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
Jisheng Bai, Haohe Liu, Mou Wang, Dongyuan Shi, Wenwu Wang, Mark Plumbley, Woon-Seng Gan, and Jianfeng Chen. Audiosetcaps: An enriched audio-caption dataset using automated gen-
10

---

Published as a conference paper at ICLR 2026
eration pipeline with large audio and language models. IEEE Transactions on Audio, Speech and Language Processing, 2025.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high- quality video generation. arXiv preprini urXiv:2310.195/22 2023.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020.
Xin Chen. Animeganv2. 2022.
Ziyang Chen, Prem Seetharaman, Bryan Russell, Oriol Nieto, David Bourgin, Andrew Owens, and Justin Salamon. Video-guided foley sound generation with multimodal controls. arXiv preprint 141. 2024.
Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mit- sufuji. Mmaudio: Taming multimodal joint training for high-quality video-to-audio synthesis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 28901-28911, 2025.
Xiaowei Chi, Yatian Wang, Aosong Cheng, Pengjun Fang. Zeyue Tian, Yingqing He, Zhaoyang Liu, Xingqun Qi, Jiahao Pan, Rongyu Zhang, al. Mmtrail: multimodal trailer video dataset with language and music descriptions. arXiv preprint arXiv:2407.20962, 2024.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024.
Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexan- dre Défossez. Simple and controllable music generation. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Qixin Deng, Qikai Yang, Ruibin Yuan, Yipeng Huang, Yi Wang, Xubo Liu, Zeyue Tian, Jiahao Pan, Ge Zhang, Hanfeng Lin, al. Composerx: Multi-agent symbolic music composition with Ilms. arXiv preprint arXivv2404. 1..400.8800 18081, 2024.
Shangzhe Di, Zeren Jiang. Si Liu, Zhaokai Wang, Leyan Zhu Zexin He, Hongming Liu, and Shuicheng Yan. Video background music generation with controllable music transformer. In Proceedings of the 29th ACM International Conference Multimedia, pp. 2037-2045, 2021.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. arXiv preprint arXiv:1802.04228, 2018.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 736-740. IEEE, 2020.
Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons. Long-form music generation with latent diffusion. arXiv preprint arXiv:2404. 10301, 2024a.
Zach Evans, Julian D Parker, CJ Car,, Zack Zukowski, Josiah Taylor, and Jordi Pons. Stable audio open. arXiv preprint arXiv:2407. 14358, 2024b.

---

Published as a conference paper at ICLR 2026
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
Deepanway Ghosal, Navonil Majumder, Ambuj Mchrish, and Soujanya Poria. Text-to-audio gen- eration using instruction-tuned Ilm and latent diffusion model. arXiv preprint arXiv:2304.13731 2023.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang. Yaohui Wang, Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. arXiv preprint arXiv:2307.04722 2023.
Yingqing He, Zhaoyang Liu, Jingye Chen, Zeyue Tian, Hongyu Liu, Xiaowei Chi, Runtao Liu, Ruibin Yuan, Yazhou Xing, Wenhai Wang, et al. Lims meet multimodal generation and editing: A survey, arXiv preprint vrXiv:2405.19333 2024.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, speech and signal processing (icassp), pp. 131-135. IEEE, 2017.
Shawn Hershey, Daniel PW Ellis, Eduardo Fonseca, Aren Jansen, Caroline Liu, R Channing Moore, and Manoj Plakal. The benefit of temporally-strong labels in audio event classification. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 366-370. IEEE, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, (166800-6551, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances Neural Information Processing Systems, 35:8633- 8646, 2022.
Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. Make-an-audio Temporal-enhanced text-to-audio gen- eration. arXiv preprint arXiv:2305.1844 18474, 2023.
Vladimir lashin, Weidi Xie, Esa Rahtu, and Andrew Zisserman. Synchformer: Efficient synchro- nization from sparse cues. In ICASSP 2024-2024 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 5325-5329. IEEE, 2024.
Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Diff-tts: A denoising diffusion model for text-to-speech. arXiv preprint 1rX1v22104.01409, 2021.
Yuxuan Jiang, Zehua Chen, Zeqian Ju, Chang Li, Weibei Dou, and Jun Zhu. Freeaudio: Training-free free timing planning for controllable long-form text-to-audio generation. arXiv preprint arXiv:2507.08557, 2025.
Jacyong Kang, Soujanya Poria, and Dorien Herremans. Video2music: Suitable music generation from videos using an affective multimodal transformer model. Expert Systems with Applications, 249: 123640, 2024.
12

---

Published as a conference paper at ICLR 2026
Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Vol- ume I (Long and Short Papers), pp. 119-132, 2019.
Qiuqiang Kong, Yin Cao, Turab lqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. JEEE/ACM Transac- lions on Audio, Speech, and Language Processing, 28:2880-2894, 2020.
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209. 2at.:22229.155,,,, 2022.
Ruiqi Li, Siqi Zheng. Xize Cheng, Ziang Zhang, Shengpeng Ji, and Zhou Zhao. Muvi: Video- to-music generation with semantic alignment and rhythmic synchronization. arXiv preprint 1rX1v:2410.12957, 2024a.
Sizhe Li, Yiming Qin, Minghang Zheng, Xin Jin, and Yang Liu. Diff-bgm: A diffusion model for video background music generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27348-273577 2024b.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:231111012,, 2023.
Yan-Bo Lin, Yu Tian, Linjie Yang, Gedas Bertasius, and Heng Wang. Vmas: Video-to-music gen- eration via semantic alignment web music videos. arXiv preprint arXiv:24090074 0744000007 2024.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.125500 2023.
Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. Audioldm 2: Learning holistic audio generation with self supervipe pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In Proceedings of the AAAI conference artificial intelligence, volume 36, pp. 11020-1 1028, 2022.
Shansong Liu, Atin Sakkeer Hussain, Qilong Wu, Chenshuo Sun, and Ying Shan. Mumu-llama. Multi-modal music understanding and generation via large language models. arXiv preprint arXiv:2412.06660, 2024c.
Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized tudoo-t--audio synthesis with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Yinghao Ma, Anders Øland, Anton Ragni, Bleiz MacSen Del Sette, Charalampos Saitis, Chris Don- ahue, Chenghua Lin, Christos Plachouras, Emmanouil Benetos, Elona Shatri, et al. Foundation models for music: A survey. arXiv preprint arXiv:2408. 143400 2024.
Navonil Majumder, Chia- Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, and Sou- janya Poria. Tango 2: Aligning diffusiin-based text-to-audio generations through direct prefer- ence optimization. In Proceedings of the 32nd ACM International Conference on Multimedia, pp.
564-572, 2024.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yar Ma, Ching Yao Chuang, et al. Movie gen: A cast media founda- tion models. arXiv preprint Xixv:2410..37220 2024.
13

---

Published as a conference paper at ICLR 2026
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad- tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning. pp.
PMLR, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2XXi::2244006155 1(2):3, 2022.
António Ramires, Frederic Font, Dmitry Bogdanov, Jordan BL Smith, Yi-Hsuan Yang, Joann Ching, Bo-Yo Yu Chen, Yuoh-Kao Wu, Hsu Wei-Han, and Xavier Serra. The freesound loop dataset and annotation tool. arXiv preprint arXiv:2008. 11507, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Roy Sheffer and Yossi Adi. hear your true colors: Image guided audio generation. In ICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 134556 13466, 2020.
Yapeng Tian, Dingzeyu Li, and Chenliang Xu. Unified multisensory perception: Weakly-supervised audio-visual video parsing. In Computer -CCV 2020: 16th European Conference, Glas- gow; UK, August 23-28, 2020, Proceedings, Part 16, pp. 436-454. Springer, 2020.
Zeyue Tian, Zhaoyang Liu, Ruibin Yuan, Jiahao Pan, Qifeng Liu, Xu Tan, Qifeng Chen, Wei Xue, and Yike Guo. Vidmuse: A simple video-to-music generation framework with long-short-term modeling. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp-
18782-18793, 2025.
Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Le, Nick Zacharov, et al. Meta audiobox aesthetics: Unified automatic quality assessment for specch, music, and sound. arXiv preprint arXiv:2502.05133 2025.
Yongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang. Zehan Wang, Fuming You, Ruiqi Li, and Zhou Zhao. Frieren: Efficient video-to-audio generation with rectified flow matching. arXiv preprint wrXXv:2400600:22 2024.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multi- modal 11m. arXiv preprint arXiv:2309.055//3 2023a.
Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Keng-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-cappeee augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023b.
Zeyu Xie, Xuenan Xu, Zhizheng Wu, and Mengyue Wu. Audiotime: A cumporally-alignee audio- text benchmark dataset. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1-5. IEEE, 2025.
Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open- domain visual- audio generation with diffusion latent aligners. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7151-7161, 2024.
14

---

Published as a conference paper at ICLR 2026
Liumeng Xue, Ziya Zhou, Jiahao Pan, Zixuan Li, Shuai Fan, Yinghao Ma, Sitong Cheng, Dongchao Yang, Haohan Guo, Yujia Xiao, et al. Audio-flan: preliminary release. arXiv preprint arXiv:2502216684, 2025.
Ruibin Yuan, Hanfeng Lin, Yi Wang, Zeyue Tian, Shangda Wu, Tianhao Shen, Ge Zhang, Yuhang Wu, Cong Liu, Ziya Zhou, et al. Chatmusician: Understanding and generating music intrinsically with Ilm. arXiv preprint arXiv:2402.16153, 2024.
Ruibin Yuan, Hanfeng Lin, Shuyue Guo, Ge Zhang, Jiahao Pan, Yongyi Zang, Haohe Liu, Yiming Liang, Wenye Ma, Xingjian Du, et al. Yue: Scaling open foundation models for long-form music generation. arXiv preprint arXiv:2503.088 08638, 2025.
Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. Foleycrafter: Bring silent videos life with lifelike and synchronized sounds. arXiv preprint urXiv:2407.01494 2024.
Zitang Zhou, Ke Mei, Yu Lu, Tianyi Wang, and Fengyun Rao. Harmonyset: A comprehensive dataset for understanding video-music semantic alignment and temporal synchronization. Pro- ceedings of the Computer Vision and Pattern Recognition Conference, pp. 3152-3162, 2025.
Alon Ziv, Itai Gat, Gaël Le Lan, Tal Remez, Felix Kreuk, Jade Copet, Alexandre Défossez, Gabriel Synnaeve, and Yossi Adi. Masked audio generation using a single non-autoregressive transformer. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Aus- tria, May 7-11, 2024. OpenReview.net, 2024. URL /owpr///eeeeeeeeee.ee.e. net forum? Ny-
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
USE OF LARGE LANGUAGE MODELS
Large Language Models (LLMs) utilized in two parts in this work. As components of our methodology, Gemini 2.5 Pro is employed for high-quality initial data annotation, benchmark gen- eration, and an automated evaluator, while Qwen2-Audio performs large-scale data augmentation. Additionally, am LLM assistant (Google's ' Gemini) is used as a writing tool to improve the clarity, grammar, and vocabulary of the manuscript. All scientific ideation, experimental design, and analy- sis are conceived and performed exclusively by the human authors.
APPENDIX OVERVIEW
This appendix supplements the main paper with expanded details our datasets, evaluation methodologies, and a broader range of experimental results. We begin by detailing our data and evaluation frameworks: Section A.I delves into the specifies of our datasets and the annotation process, Section A.2 introduces evaluation metrics, while Section A.33 introduces T2A-bench, our benchmark for cieructiion-ollowiing along with its automated evaluation pipeline. Subsequently, we present an expanded set of results. Section A.4 provides further quantitative comparisons, and finally, Section A.5 showcases a comprchensive gallery of qualitative examples and analyses.
A.1 DATASETS
A.1.1 TRAINING AND TEST DATASETS
Table A. provides an overview of all datasets used this work. Table outlines the new captions annotated for training and testing unified model. We will open-source these caption datasets to facilitate further research.
A.1.2 FURTHER DETAILS ON THE IF-CAPS DATASET
As described in the main text, the IF-caps dataset is generated via a multi-step pipeline designed to produce rich, structured annotations for existing video-audio clips. This section provides detailed breakdown of our annotation schema and showcases representative samples.
Annotation Schema. Each sample in IF-caps is accompanied by comprehensive set of annotations designed to provide multi-faceted supervision for training. The key fields are follows:
caption:
holistic, high-level natural language description of the audio content, summa- rizing the main events and their context.
category: A structured dictionary that provides sound event classification and, where ap- plicable, the discrete count of each event. For continuous or unquantifiable sounds (e.g.. background noise, speech), the count marked as null.
SED (Sound Event Detection): A list providing fine-grained temporal localization. Each entry the list maps a precise timestamp (e.g., "00:02-00:06") to description of the sound event occurring within that specific time frame.
time Lime-relation A field describing the temporal relationship between distinct sound events. This can specify a sequential order (e.g., "Event A, Event B") or more complex relation- ships like "interleave" for overlapping sounds.
This structured format allows our model learn not just what sounds are present, but also how many, when, and in what order, which is critical for developing advanced cestruction-oollwiing capabilitics.
Annotation Samples. Below are two examples from IF-caps that illustrate the richness and de- tail of our annotation schema. The first example demonstrates a complex scene with overlapping, continuous, and countable events. The second example shows clear sequence of discrete events.
16

---

Published as a conference paper at ICLR 2026
Table A.I: Comprehensive overview of training and test datasets, detailing the number of clips (# Clips), average duration clip (Dur/Clip in seconds), and total duration (Dur. in hours) for each task and split. T2A: Texi-to-Audio, V2A: Vidco-to-Audio, TV2A: Text-and- Bext-and-Video--oe Video-to-Audio, T2M:
Text-to-Music, V2M: Video-to-Music TV2M: Text-and- Video-to-Music.
Split
Task
Dataset
# Clips
Dur//Cipp (s)
Dur. (h)
AudioCaps
45.0k
10
125.1
WavCaps
08.33
10
300.8
T2A
IF-caps
1268k
10
3524.4
AudioTTime
20k
10
355.5
VGGSound
176.9
10
491.4
V2A
AudioSet Strong
67.3k
10
187.14
Greatest Hits
1.0k
10
2.71
Train
IF-caps
1268k
3524.4
TV2A
Greatest Hits
1.0k
2.71
Private
175.2k
240
11679.3
T2M
V2M
5685.7k
15793.58
MUCaps
22.0k
208
1273.6
V2M
V2M
5685.7k
10
15793.58
TV2M
V2M
5685.7k
15793.58 10
Audio Inpainting
All audio data
398.5k
10
1107.15
Music Completion
All music data
5882.9k
17.6
28746.48
AudioCaps VGGSound
4,875
10
13.54
14,931
10
41.475
T2A
T2A-bench
2000
10
5.55
AudioTime
2000
5.55
VGGSound
14,931
10
41.475
V2A
Test
AVVP
1.120
10
3.11
TV2A
VGGSound
14,931
10
41.475
MusicCaps
5,526
15.35
T2M
V2M
3105
9.01
V2M
V2M
300
108
9.01
TV2M
V2M
300
108
9.01
AudioCaps
4,875
13.54
Audio Inpainting
AVVP
1,120
10
3.11
Music Completion
V2M
300
108
9.01
Table A.2:
Overview of our labeled captions, detailing the number of clips, average duration per clip, and total duration for each source dataset.
Source Dataset
Data Type
# Clips
Dur./Clip (s)
Dur. (h)
VGGSound
Audio
191.8K
10
532.81
AudioSet Strong
Audio
67.3K
10
187.14
AVVP Test Split
Audio
1.1K
10
3.11
Greatest Hits
Audio
1.0K
10
2.71
V2M
Music
5.7M
10
15793.58
Example
"caption": "A weman is speaking centinuously, while dog yips twice in the background.",
"category";
17

---

Published as a conference paper at ICLR 2026
"Female speech": null, "Yip": "Background noise"; null b
"SED": ("00:00-00:09": "A
is speaking throughout the audio, accompanied by faint background roise."), ("00:00-00:01"; "A dog lets out a yip in the background, "), ("00:08-00:09"; "A dog yips again in the background.")
Itime_relationn "interleave", "audio_id": "IATdKPmaMUB_90000
Example
"caption":
"The audio features the mechanical sound of firearm being handled, immediately followed by two separate bursts of machine gun fire.",
"category":
"Machine gun": 2, "Generic impact sounds";
"SED": ("00:00-00:01":
"The mechanical sound of a firearm being handled, possibly being cocked or loaded. "I, ("00:01-00:05"; "A sustained burst of automatic gunfire from a machine gun."),
("00:06-00:08"; "A second, shorter burst of machine gun fire
"Generic impact scunds, Machine gun", "audio_id": "c9Onuthhv2y_0"
Data Augmentation Process. As mentioned the main text, a key step in our pipeline is to leverage cost-effective model (Qwen2-Audio to augment the initial, high-quality annotations generated by Gemini 2.5 Pro. The goal is to increase the linguistic and structural diversity of our dataset. By generating multiple, semantically equivalent but stylistically different captions for the same audio clip, we train our model to robust to variations in user prompts and to develop a more generalized understanding of the relationship between language and sound. The augmentation process is guided by the structured fields of the original annotation. The model is prompted to generate new cap- tions from different perspectives: rephrasing the original description, or generating new descriptions based purely on the category and count, the SED timestamps. or the time relation fields. Below, we use the second example from the previous section illustrate this structured augmentation process.
Original Audic Annotation (Generated by Gemins 2.5 Pro)
("caption":
"The audio features the rechanical sound of firearm being handl ed, immediately followed by two separate bursts of machine gun fire.", "category"; { ("Machine gun": 2, "Generic impact sounds": 11, "SED": I...l, "time relation"; "Generic impact sounds, Machine gun", "audio id": "c90nubhhvZy _0"]
This single, rich annotation serves as the seed for generating a variety of new training captions, each emphasizing different aspect of the audio content.
18

---

Published as a conference paper at ICLR 2026
Augmented Audic Captions (Generated by Cwen2-Audio)
Caption Rephrasing (Linguistic Diversity)
"A gun is cocked, followed by bursts machine gun fire." "After the sharp, metallic sound of firearm mechanism, two rapid-fire bursts from a machine gun are heard."
2. Augmentation from Category and Count
"The audio contains two sounds of a machine gun and one generic impact sound."
3. Augmentation from SED
"The sound of firearm being handled is audible for the first second, fol- lowed by a burst of machine gun fire from I to seconds and a second burst from 6 seconds."
4. Augmentation from Time Relation
"In this audio, the sound of a generic impact occurs first, followed by two distinct machine gun sounds."
This structured augmentation strategy ensures model is exposed to wide variety of textual descriptions, learning to associate not only high-level captions but also explicit instructions about count, timing, and order with the corresponding audio features. Similarly, for music data, this process generates varied descriptions of genre, mood, instrumentation, and tempo, teaching the model comprehend both high-level artistic direction and specific musical components.
Original Music Annotation (Generated by Gemini Pro)
["caption"; "A heartwarming acoustic track featuring a blend of softly strumned guitar and simple, melodic plano line, creating gentle and uplifting atmosphere.", "genre": "Acoustic Pop, Instrumental", "nood": "Heartwarming, Gentle, Uplifting", "instrument": ("Acoustic Guitar", "Piano"), "tempo": "Slow to Moderate")
This structured music annotation then used to generate diverse new training captions, each focus-
ing on a different attribute:
Augmented Music Captions (Generated by Qwen2-Audio)
1. Caption Rephrasing
"A gentle instrumental piece with the interwoven sounds of an acoustic guitar and piano." "Soft guitar strumming and a simple piano melody combine to create an uplifting acoustic pop track."
2. Augmentation from Genre
"An instrumental acoustic pop track featuring piano and guitar."
3. Augmentation from Mood
"A heartwarming, gentle, and uplifting piece of music featuring acoustic guitar and piano."
4. Augmentation from Tempo
A slow to moderate tempo instrumental track with piano and acoustic guitar."
19

---

Published as a conference paper at ICLR 2026
A.2 DETAILS OF EVALUATION METRICS
Fréchet Audio Distance (FAD). To evaluate the perceptual quality of the generated audio, we em- ploy FAD, a reference-free metric analogous to the FID (Heusel et al.) 2017) score used in image generation. The metric functions by comparing the statistical distance between embedding distribu- tions of generated audio and real-world audio. A smaller distance suggests the generated audio is of
higher acoustic quality. For our calculations, we utilize the VGGish (Hershey et al. 2017) feature extractor.
Fréchet Distance (FD). While similar in principle to FAD, FD serves a distinct measure of audio similarity by employing a different feature extractor. We use an FD variant based on PANNs (Kong et al. 2020) embeddings. Given that PANNs models pretrained on the extensive AudioSet Gemmeke et al. 2017), this metric is considered to be highly robust for evaluating audio fidelity.
Kullback-Leible Divergence (KL). The KL divergence is used approximate the acoustic simi- larity between generated and reference audio samples. This is achieved by measuring the divergence between the multi-label class prediction distributions produced by PANNs model for both sets of samples.
Inception Score (IS). The IS is a widely used metric evaluate the performance of generative models. Besides assessing the diversity of the generated samples, IS also evaluates their quality, measuring the clarity and recognizability of individual audio events (Donahue et al.] 2018 Ma- jumder et al. [2024] Liu et al. (023). Given its ability to provide a single, holistic score reflecting both of these aspects without needing reference prompt, we selected IS as the unitied metric for the comprehensive performance comparison in our teaser Fig. [ This allows for a fair and consistent visualization of our model's capabilities across the wide array of supported tasks.
ImageBind Score (Girdhar et al., 2023). We assess the semantic alignment between generated audio and conditioning videos using the ImageBind Score. This score is calculated as the cosine similarity between the audio and video embeddings from the respective branches of the ImageBind model.
CLAP Score. The Contrastive Language-Audio Pretraining (CLAP) model (Elizalde et al., 2023) learns a joint embedding space where audio clips and their corresponding text descriptions are aligned. We use the CLAP Score to evaluate the semantic alignment between generated audio and a text prompt, calculated as the cosine similarity between their respective embeddings from the pretrained CLAP encoders (Wu et al. 2023b). A higher score indicates better alignment.
Production Complexity (PC) and Production Quality (PQ). These metrics are derived from the Meta Audiobox Aesthetics framework TTandra al., 2025). PQ focuses the objective, technical aspects of an audio recording, such as its clarity, fidelity, dynamics, and frequency balance. In contrast, PC evaluates the complexity of an audio scene by measuring the number of distinct audio components present, such as multiple instruments or the co-occurrence of speech, music, and sound effects. Both are designed no-reference metrics, allowing for the assessment of individual audio clips without needing ground-trute truth comparison sample.
Ordering, Duration, Frequency, and Timestamp. These metrics are components of the STEAM evaluation framework, proposed in the AudioTime (Xic et al. 2025) to assess the temporal control- lability of audio generation models. Ordering is an error rate that measures whether sound events are generated in the specified sequence. Duration and Frequency are calculated as the LI error between the specified and detected event durations and occurrence counts, respectively. Timestamp evaluates the precise timing of events (onset and offset) using the Fl-score, a common metric in sound event detection.
Category, Count, Ordering, and Timestamp accuracy. See
Overall Quality (OVL) and Relevance (REL). For our subjective evaluation, 10 professional audio experts rated each generated sample
scale of to 100 on two standard criteria. OVL assesses the intrinsic perceptual fidelity of the audio fooufffoonnngggooo on aspects like clarity and freedom from arrifacts-independdee independent of the prompt. In parallel, REL measures the semantic alignment between the audio and its conditioning input, evaluating how accurately the content reflects the instructions from the provided text video. This evaluation protocol follows the established methodologies of
20

---

Published as a conference paper at ICLR 2026
prior work (Kreuk et al. 2022 Liu et al. 2023). Example of the questionnaire interface is shown in Table A.3
Table A.3: Simplified example of the questionnaire for human evaluation, showcasing the four main task types. Experts provided scores for OVL and REL.
File Name
Prompt (Text Video)
OVL (1-100)
REL (1-100)
9964.wav
A loud white noise and then some beeping.
55
65
0928. wav
uplifting folk-pop instrumental track.
70
55
1441.wl wav
[Video a person walking on dry leaves]
70
1701wav
[Video a drone shot over a sunrise mountain]
65
60
A.3 BENCHMARK AND METRICS FOR INSTRUCTTON-FOLLOONNG IN T2A
To rigorously and scalably evaluate the cerruction-follwiinn capabilities of Text-to-Audio genera- tion models, we introduce a new benchmark, T2A-bench, and corresponding automated evaluation pipeline. This framework is designed to dissect a model's ability to adhere to complex compositional instructions.
punch
seagulls
criciets'
a
wht phisper LOR spicting
SCTEME
inn
pour
thundeer
swi Aemimmnn
Tock Type
uitt
hammer
whistle
Categgiy e Tineetiin
Gdlgge -Dunt:
yawn
splash
gasp
ng
exp o
thunde
b blnder Auaigh g
gunshot
C Calegory Counts
Cipato 47
Cileto
250
S sheeze
ii
apgl
Bal
footsteps
slap
(a) Sound category word cloud.
(b) Task & Category distribution.
Figure A.1: The composition of the T2A-bench benchmark. (a) Word cloud of sound event cate-
gories. (b) Distribution task types and category counts.
T2A-bench Composition and Design. T2A-bench is prompt-based benchmark comprising 2k challenging, natural language prompts generated by Gemini 2.5 Pro. structured to systemat- ically probe four key dimensions of controllability. As illustrated in Figure A. our benchmark encompasses a diverse vocabulary sound categories and a balanced task structure to enable a rig-
orous and comprehensive evaluation. The benchmark is divided into four task types, each containing 500 prompts:
Category-only: Evaluates the generation of correct sound events. Prompts contain be- tween one and five distinct sound categories (100 prompts for each count).
Category+Count, Assesses the ability to generate a precise number of sound events. To avoid ambiguity, prompts in this category feature only single sound type, with the required count ranging from one to five (100 prompts for each count).
Measures adherence to temporal sequence. Prompts specify an order for either two three distinct sound categories.
Category+ EategoryyTimestamppe Timestamp: Tests temporal localization. To ensure clarity, prompts specify start and end time for a single sound category.
Below are representative examples for each task type, including the prompt and its corresponding structured metadata.
21

---

Published as a conference paper at ICLR 2026
T2A-bench Examples
"id": "T2A_01565",
"type": "categery-only",
"prompt": "A viclent storm at with loud clap of thunder and huge crashing the deck.", "category": "thunder, wave crash"
"id": "12A_00031",
"type": "categery+count",
"prompt": "A single, loud bark from dog in the distance." ,, "category": "dog bark", "count": ("dog bark": 1)
"id": "T2A_00575",
"type": "catcgery+ordering",
"prompt"; "The sound of person gargling, followed by the splash of water in the sink.",
"category": "gargle, water splash", itime_relation", "gargle, water splash"
"id": "T2A 01105", "type": "category+timestann
"prompt"; "The scund of a crowd cheering is present from 2.0 seconds to 6.0 seconds.",
"category": "crewd cheering", "timestamp": ("crowd cheering"; ("start": 2.0, "end": 6.0}}
Evaluation Metries. Corresponding to the benchmark's structure, we define four strict, accuracy- based metrics: Category Accuracy (Cat-acc), Count Accuracy (Cnt-acc), Ordering Accuracy (Ord- acc), and Timestamp Accuracy (TS-ace). The final score for each metric the percentage "cor- rect" judgments.
Cat-ace: A judgment is "correct" only if all sound categories specified in the prompt are detected in the generated audio. This is evaluated on all 2,000 samples.
Cnt-ace: A judgment "correct" only if the detected count for the specified category exactly matches the prompt's instruction.
Ord-ace: A judgment is "correct" only if the detected temporal order of sound events exactly matches the specified sequence.
TS-acc: judgment is "correct" only the detected event's start and end times fall within a 1-second tolerance window of the target times specified in the prompt.
Automated Evaluation Pipeline. To ensure objective and scalable evaluation while preventing information leakage, we designed novel two-step pipeline that leverages the statc-of-the-an audio understanding of powerful Multimodal Large Model (MLLM), Gemini 2.5 Pro, automated judge.
Step 1: Blind Audio Annotation. In the first step, the MILM judge receives only the audio sample generated by the model under evaluation. It performs blind, detailed analysis to produce a structured annotation the audio's 's content. This annotation includes detected sound categories, their counts, temporal relationships, and precise sound event detection (SED) timestamps. For sounds where counting ambiguous (e.g., continuous water flow) ordering is not distinct, the corresponding fields are populated with null.
22

---

Published as a conference paper at ICLR 2026
Example of Step Output (Structured Annctation)
"caption"; "The audio contains two distinct loud sounds. First, there is a deep, rolling thunderclap. After bricf pause, powerful and sudden explosion is heard.", "category"; ("Thunder"; 1, "Explosion": 1), "SED":
("00:01.734-00:1 7300000333:000114 "A deep, relling thunderclap is heard.")
000:00.241-0::99.00 s11"": Loud and sudden explosien with distinct poom."Hl
"time relation"; "Thunder, Explosion",
Step 2: LLM-based Judgment. the second step, the MLLM judge is provided with the original prompt from T2A-bench and the structured annotation generated in Step 1. Acting like an examiner with an answer key, the MI.LM compares the annotated audio content against the prompt's instructions. It then outputs binary score (1 for correct, for incorrect) for the relevant metric, along with a detailed textual analysis explaining its decision.
Example of Step 2 Output (Final Judgment)
"prompt": "A medieval battlefield, with the sound of catapult launching a stone and the subsequent explos osion. "prediction"; ""cat_ace"; 0,
mull, "ord acc": null
"ts ace": "ts_acc": null,
"analysis"; "The audio contains a clear and prominent sound of thunder, which is audible from the beginning and culminates in loud clap around 00:03. However, the required categery 'wave crash! is missing. While there is sound of water starting around 00:05, it
acoustically identifiable as heavy rain rather than distinct, powerful wave crashing.";
In summary, our framework, combining T2A-bench, fine-grained metrics, and robust two-step evaluation pipeline, provides a comprehensive and replicable methodology for quantifying the instruction-followiin capabilities of T2A models. We will open-souree proposed benchmark and evaluation pipeline to facilitate future research this area.
A.4
MORE RESULTS
A.4.1 COMPARISON RESULTS
Audio inpainting. As shown in Table A.4 where our model outperformed the baselines (Liu et al. 2023 2024a) on the AudioCaps (Kim et al., 2019) and AVVP (Tian al., 2020) test datasets. Additionally, to explore audio inpainting with various input modalities, we performed experiments as video-guided and text-and-video sawaaddvvio--uudeeee -uided--geeed audio inpainting tasks (on AVVP). The results indicate that both text and video can effectively guide the audio inpainting task, with text providing better guidance than video. When both text and video are conditioned, the model can integrate the two modalities to achieve superior results.
conducted experiments on audio inpainting tasks,
unconditioned audio inpainting, as well
Music Completion. Music completion is task where the model generates music based on a given music clip. We evaluate model on the V2M-bench (Tian et al. 2025) dataset. The results shown in Table A.3 We find that our model can generate music that extends the input music clip. As the number of input modalities increases, the model's performance improves, demonstrating its
23

---

Published as a conference paper at ICLR 2026
Table A.4: Inpainting Performance Comparison. This table shows the performance comparison for audio inpainting on the AudioCaps and AVVP datasets. The values before and after the slash represent the IS and FAD metrics, respectively. A, V, and represent Audio, Video, and Text conditions. The baseline methods are all under audio and text conditions.
Dataset
Method
Input
AudioCaps
AVVP
Unprocessed
6.51/11.34
4,94/6.70
AudioL.DM-L-Fi
Liu al.
2024a
A+T
8.06/2.64
5.11/3.30
AMdiol-D-22Fuull Large Luu et al. 2024a
A+T
4.24/10.17
3
AudioX
A 4.63/5.35
3.94/5,44 44
AudioX
A+T
9.84/2.25
6.12/2.05
AudioX
A+V
N/A
5,63/2.16
AudioX
A+T
N/A
6,25/1.99
strong inter-modal learning capability and ability to leverage multi-modal information to generate better music.
Table A.5: Performance for our method under different conditions in the music completion task. M, T, and V represent Music, Text, and Video, respectively.
Input
KI.4
IS
f
FD
+
FAD 4
M
0.96
1.21
52.77
5.76
T+M
0.51
1.49
21.42
2.14
V+M
0.70
1.37
24.28
2.29
T+V+M
0.46
1.52
18.69
1.67
Image-to-audio generation. To evaluate the model's capability in handling static visual inputs, we conduct zero-shot image-to-audio generation experiment. Adopting the experimental protocol of Seeing&Hearing (Xing et al.) 2024), we perform evaluations on 3k clips from the VGGSound test set, where keyframes were processed using AnimeGANv2 (Chen) (022) transfer them into "Pa- prika style" prior to generation. For comparison, we benchmark AudioX against Beeing&&Harringg (Xing et al., 2024), Im2Wav (Shetfer & Adi 2023). and also constructed baseline by combining an image caption model (Bai et al. 2023) with text-to-audio model (Majumder et al. 2024). The results are shown in Table A.6 the Appendix. We find that our model demonstrates excellent performance in the image-to-audio generation task even without any specific training with image data.
Table A.6: Comparison of Methods for the Image2Audio Task.
Method
KL.
IS 1
FD
FAD +
Align. t
2uption2Audio IhhWfrrShefferr & Adi 2023
2.76
7.48
32.97
5.54
0.21
2.61
7.06
19.63
7.58
0.41
& (caringXing et al. 2024
2.69
6.15
20.96
6.87
0.29
AudioX
2.90
13.48
16.42
2.71
0.23
User study, We conducted user study to evaluate the quality of the generated audio and music. We randomly selected 25 samples for each audio generation task, including text-to-audio (T2A), text- to-music (T2M), video-to-audio (V2A), and video-to-music (V2M). 10 audio experts are asked to rate the quality of the generated audio and music. The results are shown in Fig. A.2 The evaluation shows that our model achieves subjective SOTA performance in terms OVL and REL scores in most tasks, indicating high user satisfaction.
A.4.2 ABLATION RESULTS
Unified model performance.
24

---

Published as a conference paper at ICLR 2026
T2A
V2A
T2M V2M
DVL
REL
Auttiniinn
Macuer Sarge Audiaaiio Full
Lar Mathun
Open
MAGIRET caree
RudhulDi C-iagg AudioL Aull
aund propuattie Aucurtio Ownn stabil
Auulor
Vitec/iMuncc
Hotto Lasta Visidaas
Agtior
Audio
Figure A.2: User study results of generated audio and music. The values represent the average OVL and REL scores across Text-to-Audio (on AudioCaps), Text-to-Music (on MusicCaps), Video-to- Audio (on VGGSound), Mileo-to-Music (on V2M-bench).
We investigate our unified model's intra- and inter-modal performance in Fig. A.3 modal study, we compare our single unified model against specialist models trained on individual tasks (T2A, V2A, and audio inpainting). The results show our unified model consistently outper- forms these specialist models, demonstrating strong intra-modal capabilities. For the inter-modal study on music generation, we find that performance progressively improves as more conditioning modalities are added (e.g., from video-only to video+text). This confirms the model's 's robust ability to effectively integrate multiple modalities to enhance generation quality.
For the intra-
32
1.48
10
L.3
2
12
Intro Modal Perocmazze Computionn
Integration Compartson
Figure A.3: Ablation study comparing intra-modal and inter-modal performance of the unified model. The left compares single-modality models on text-to-audio, video-to-audio, and audio in- painting tasks. The right shows the effect of adding modalities on music generation, with perfor- mance improvements noted for each added modality. Results are based on the Inception Score (IS)
metric.
A.5 QUALITATIVE RESULTS
Figures
A.4 and
A.5 present comprehensive qualitative results.
25

---

Published as a conference paper at ICLR 2026
Input Stexp: A small dog barks repeatedly at multippl intervals.
clock liking
Music performel sicclpnn instrunes.
Input svidevo:: reference for <andio- to be ispainted
Ground Teub
Input soudio>; Unprocessed
ArdwiGen
Ground Truth
L.I
AMdioLDM-L L-Full: audio
Audio DV- Lange
Ausien Lad-2-Large
AudioLDM-2-Lange texttrudio>
TarpoMuniz
Ours: Saudios
Sule Audio Open
Suke Auaioo Open
Ours: -audi-
MAGNET-LaRg
MAGNET Large
ClUs
Ours: Seddeoteet-undion
(a) T2A and T2M Results.
(b) Inpainting Results.
Seeing& Hearing
FoleyCrafter
DifFolly
FRIEREN
Ours (c) V2A Results.
Ground Truth
Figure A.4: Qualitative comparison across various tasks: (a) In Text-to-Audio (T2A) and Text-to- Music (T2M) tasks, our model uniquely excels by consistently generating the "ticking" sound of clock and accurately following the prompt "Music performed on a steelpan instrument," outperform- ing baselines in both rhythmic precision and genre fidelity.
(b) Audio inpainting results demonstrate our model's strong context-aware capabilities and its ability to effectively integrate different input modalities. (c) Video-to-Audio (V2A) results show our model's proficiency in capturing dynamic motion sounds, such as the immersive "drifting" of a car, providing a richer auditory experience compared to baselines.
26

---

Published as a conference paper at ICLR 2026
Test-to-didlto
Test-to-MMezic
Pronge mecwing torke
Prorge Orcheaml, spic, dram, sring, and
Pooppt dog is contenncssl bukkigg wlinig.
Procpe: Orckeszr movi misk with srigg, trass and percessice, croticaal and spi.
Pronge The sourd of oypiz computer keyboard.
Proceee Electonic darce musi wit systhesizers, bass, druns, and slow build
Prompt medody is being played
Proppe guar hamoayy grony
(a) Text-to-Audio and Text-to-Music
Vislooto-AAniio
Videv-to-Musiic
(b) Video-to- Audio and Video-to-Music
Audio Impainting (with text)
Music Completion (with text)
Proege hhs engin hom iiewn inllaad firz Neviin
Prompt Ordisiaa dasatic intoe, penaffll repenieaa trilles chaste.
Unproosed
Roadi
spech Miloodd
Uxpoossed
Rasit
Prrms Roving ratrecck qualing
Propgt Actiraactee echene mekk vit You and prrcsionn
Ueproosed
Uppocesed
Reslt
Andi
(c) Audio Inpainting and Music Completion
Figure A.5: Comprehensive qualitative analysis of our model's performance across various tasks: (a) Text-to-Audio and Text-to-Music synthesis, (b) Video-to-Audio and Video-to-Music generation, and (c) Audio Inpainting and Music Completion.
27


Reviews:
[{"review_id": "aEUTckRjfE", "reviewer": "Reviewer_iAvY", "summary": "This paper presents AudioX, a unified framework for \"anything-to-audio\" generation, designed to produce high-quality audio (both sound effects and music) from a flexible combination of multimodal inputs, including text, video, and audio signals. The work makes four primary contributions:\n- AudioX Framework: A unified model built on a Diffusion Transformer (DiT) backbone.\n- MAF Module: A novel \"Multimodal Adaptive Fusion\" (MAF) module that adaptively fuses these diverse input modalities to reduce interference and improve conditioning.\n- IF-caps Dataset: A new, large-scale (7 million sample) dataset for multimodal audio generation. This dataset is not manually labeled but created via a structured data annotation pipeline that uses a powerful MLLM (Gemini 2.5 Pro) for high-quality initial annotations, which are then augmented at scale by another model (Qwen2-Audio).\n- T2A-bench: A new benchmark designed specifically to evaluate fine-grained, instruction-following capabilities in text-to-audio models, such as event counting and temporal ordering.", "questions": "- The data generation (IF-caps) and the new benchmark (T2A-bench) both rely heavily on Gemini 2.5 Pro. Can the authors comment on the risk of this self-referential loop, where a model trained on Gemini-annotated data is then shown to be SOTA on a benchmark judged by the same Gemini model?\n- The T2A-bench's automated evaluation pipeline relies on Gemini 2.5 Pro's ability to perform blind annotation (SED, counting, etc.) . How was the judge (Gemini) itself validated for accuracy on these tasks? If the judge is inaccurate, the benchmark scores in Table 2 are not reliable.\n- Why is \"image-to-audio\" listed as a core input modality in the abstract when it is not part of the main architecture diagram and only appears as a zero-shot appendix task? Was the model ever trained on image-audio pairs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Unified Generalist Model: The paper successfully presents a single generalist model, AudioX, that performs competitively across a very wide array of \"anything-to-audio\" tasks (T2A, T2M, V2A, V2M, inpainting, etc.).\n- Superior Controllability: The model's key strength is its SOTA performance on instruction-following benchmarks (T2A-bench and AudioTime), demonstrating a new level of fine-grained control over audio content, such as event ordering and counting .\n- Novel Data Pipeline: The IF-caps dataset and its generation pipeline are a significant contribution, providing a practical blueprint for overcoming data scarcity by using LLMs for high-quality annotation and augmentation.\n- Strong Ablation Studies: The paper includes excellent ablation studies that validate both the architectural contribution (the MAF module, Table 4) and the data contribution (the IF-caps pipeline, Table 3).", "weaknesses": "- Dependency on Proprietary Models: The entire 7-million-sample IF-caps dataset is bootstrapped from annotations generated by Gemini 2.5 Pro. This makes the data curation process heavily reliant on a closed-source, proprietary, and expensive black-box model, raising significant concerns about reproducibility.\n- Potential Evaluation Bias: The new T2A-bench is also evaluated using Gemini 2.5 Pro as the \"automated judge\". While the paper describes a thoughtful two-step \"blind annotation\" pipeline to mitigate bias , this is still a form of LLM-as-a-judge. Using the same model family to both generate the training data (via IF-caps) and judge the evaluation (T2A-bench) creates a potential for self-referential bias that is not fully addressed.\n- Overclaim on \"Image\" Modality: The abstract and introduction repeatedly claim the framework is for \"text, video, image, and audio\". However, \"image\" is absent from the main model architecture (Fig. 4), training process (Sec 4.2), and main results tables (Table 1, 2). It only appears as a zero-shot task in the appendix (Table A.6). This is an overstatement of the model's core, trained capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RRH1wZj1d0", "reviewer": "Reviewer_azZ7", "summary": "This paper proposes AudioX, a unified \"anything-to-audio\" generation framework to resolve the lack of unified frameworks for handling diverse input modalities generation.  It also provides the  IF-caps dataset to resolve the scarcity of large-scale, high-quality multimodal training data. Experiments show that AudioX enables high-fidelity audio/music generation from flexible inputs while enhancing instruction-following capabilities.", "questions": "1. Whether the images include \"rare modalities\" (e.g., underwater scenes, ancient artifacts, abstract art) that are less frequent in training data.\n2. Whether the image-to-audio generation relies on human-written prompts, and whether rare objects/scenes were tested.", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Audiox present a unified framework to support audio/music generation from text, video, image, and audio inputs in a single model, breaking modality/domain constraints of specialist models.\n2. The authors build a large-scale Dataset, IF-caps, provides structured, fine-grained multimodal supervision, solving data scarcity for unified training.\n3. The authors plans to release code, model, and IF-caps to support reproducibility", "weaknesses": "1. The paper’s experiments and dataset design are focused on short audio clips (mostly 10 seconds), with no discussion or evaluation of long-form audio generation (e.g., clips longer than 30 seconds. For long-form generation, there are many aspects, including temporal coherence, memory and computational efficiency, error acccumulation should be evaluated. Without them, the paper cannot fully demonstrate AudioX’s practical value for real-world applications that require extended audio.\n2. The experiment only compares 4 methods (Caption2Audio, Im2Wav, Seeing&Hearing, AudioX) on a single set of metrics (KL, IS, FD, FAD, Align.), using \"the same settings as in (Xing et al., 2024)\" () but without specifying:\n3. The paper does not provide any strategies to reduce computational cost to make it reproducible for small labs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8GurwyclOn", "reviewer": "Reviewer_uvVZ", "summary": "The paper proposes a unified any-to-audio framework based on diffusion transformers. The main contributions of the paper are: \n1) large-scale dataset. 2) a unified framework for any-to-audio and 3) a proposed benchmark for instruction-followed in audio generation. The dataset is constructed by annotating the audio from the video dataset, while the proposed benchmark is constructed to test the instruction following of the model under controlled settings, and the model is proposed to use a Multimodal adaptive fusion module (MAF), which processes the input conditions before feeding it to the DiT.", "questions": "- Is the video and audio passed to Gemini or just the audio? Figure 2 suggest only audio while L169 suggests both.\n- Figure 4 is confusion. z_t is not mentioned in the figure suggesting that the combination of the conditions is fed as input for DiT rather than conditioning.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to follow \n- The model weights and proposed dataset and benchmark are promised to be released, which will make a good contribution to the field \n- The direction of having a unified framework that combines multiple conditions is important.\n- Extensive experiments are provided on multiple tasks. \n- The proposed benchmark is solid.", "weaknesses": "Dataset contribution:\n- The construction pipeline of the dataset is not convincing: how are the videos selected? The authors just mentioned using \"existing video datasets\". Were the audio-video alignment considered in the selected samples? Also, the authors mentioned that they used Gemini 2.5 to provide general captions, then Qwen2-Audio for augmentation in order to save compute. I am not convinced why Gemini 2.5 cannot provide the complete augmented instruction. The two-stage process is not well-justiied. \n- The dataset quality is not well-tested. I could not find any analysis on the text-audio or audio-video alignment or the dataset in general in the paper or the supplement. Without this, it is hard to understand the dataset contributions or their comparison to existing datasets beyond scale (e.g AVSync15, AutoRecap, or even VGGSounds)\n\nModel contribution:\n- While the motivation of having an any-to-audio generation framework is interesting, it is not well justified in the paper experiments. Main experiments support text, audio, and/or video conditioning, which is already supported by existing frameworks (e.g MMAudio), except for the audio in-painting, making the framework not appear as truly any-to-audio framework and just another baseline for T2A, V2A. \n- The architectural contribution is limited to the MAF module to process multiple conditions. However, it is not discussed well. For example, how is the positional encoding handled in this module? How is the temporal alignment between different conditions handled?  How does it compare to existing conditioning mechanism? (e.g MMAudio style). \n- Other components of MAF are also not well justified. The gating is not intuitive and does not seem to bring significant improvement in Table 4\n\nIn general, while the paper presents significant experiments at a large scale, most of the author's choices are not well justified, and the relation to existing work (e.g, existing architecture and datasets) is not well discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O7r2Mc1hN4", "reviewer": "Reviewer_CYKM", "summary": "The paper proposes a unified DiT model for anything-to-audio generation conditioned on text / video / image / audio. The model adds a lightweight Multimodal Adaptive Fusion (MAF), per-modality gates and learnable query sets with cross/self-attention, to calibrate condition embeddings before a DiT generator. To train the model, the authors proposed a large-scale dataset, IF-caps, sourced from existing joint audio-video dataset, by using frontier audio-video LLMs for providing detailed captions for audio / music.", "questions": "See weaknesses section.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Unified model architecture: A single model handling text/video/audio conditions for both sound effects and music is non-trivial, and it seems like the results are promising.\n\n2. Detailed evaluations and benchmarking: Evaluations cover many tasks (T2A/V2A/TV2A; T2M/V2M/TV2M; inpainting; completion; image->audio zero-shot), and the results look solid.\n\n3. Novel evaluation prototype: beyond fidelity and diversity (FD / KL metrics), they propose to evaluate more fine-grained aspects,  temporal/count/order control (incl. AudioTime) and adds a new benchmark (T2A-bench).\n\n4. Detailed user studies: the supplementary material provides detailed user studies and qualitative results over different aspects of the proposed method, covering a wide variety of tasks.", "weaknesses": "1. Lack of reference / comparisons against some important prior works in V2A / V2M, including but not limited to [1] - [5]. Except [3], all the rest are either open-sourced or providing samples for evaluation comparison. And also, [4] provides a new benchmark, which I think should be considered to evaluate.\n\n2. Lack of important metrics regarding temporal alignment evaluation. The main table (Table 1) should also incorporate metrics that address temporal alignment, such as ones \"Align Acc\" proposed in [6] and \"AlignSync\" proposed in [7].\n\n\n\n\n[1]. Tell what you hear from what you see-video to audio generation through text, NeurIPS 2024, Xiulong Liu, et.al.\n\n[2]. From vision to audio and beyond: A unified model for audio-visual representation and generation. Kun Su, et.al.\n\n[3]. V2meow: Meowing to the visual beat via video-to-music generation, AAAI 2024, Kun Su, et.al.\n\n[4]. Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation.\n\n[5]. ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing, NeurIPS 2025, Huadai Liu, et.al.\n\n[6]. Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models, NeurIPS 2023, Simian Luo, et.al.\n\n[7]. Audio-Synchronized Visual Animation, ECCV 2024, Shentong Mo, et.al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)