ThinkSound: Chain-of-Thought Reasoning in Multimodal LLMs for Audio Generation and Editing
Authors: Huadai Liu
Kaicheng Luo
Jialei Wang
Wen Wang
Qian Chen
Zhou Zhao
Wei Xue
Venue: NeurIPS 2025 poster ()
Keywords: Any-to-Audio Generation
Chain-of-thought
Multimodal Large Language Model
Primary area: applications
TL;DR: we introduce ThinkSound, a framework that utilizes Chain-of-Thought reasoning to systematically break down audio generation for videos into a step-by-step interactive process.

Abstract: While end-to-end video-to-audio generation has greatly improved, producing high-fidelity audio that authentically captures the nuances of visual content remains challenging. Like professionals in the creative industries, this generation requires sophisticated reasoning about items such as visual dynamics, acoustic environments, and temporal relationships. We present **ThinkSound**, a novel framework that leverages Chain-of-Thought (CoT) reasoning to enable stepwise, interactive audio generation and editing for videos. Our approach decomposes the process into three complementary stages: foundational foley generation that creates semantically coherent soundscapes, interactive object-centric refinement through precise user interactions, and targeted editing guided by natural language instructions. At each stage, a multimodal large language model generates contextually aligned CoT reasoning that guides a unified audio foundation model. Furthermore, we introduce **AudioCoT**, a comprehensive dataset with structured reasoning annotations that establishes connections between visual content, textual descriptions, and sound synthesis.  Experiments demonstrate that ThinkSound achieves state-of-the-art performance in video-to-audio generation across both audio metrics and CoT metrics, and excels in the out-of-distribution Movie Gen Audio benchmark. The project page is available at https://ThinkSound-Project.github.io.

ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing
Huadai Liu Kaicheng Luo", Jialei Wang', Wen Wang"
Qian Chen', Zhou Zhao", Wei Xue-
Hong Kong University of Science and Technology (HKUST) Tongyi Fun Team, Alibaba Group *Zhejiang University
Abstract
While end-to-end tndoo-o--adiio generation has greatly improved, producing high- fidelity audio that authentically captures the nuances of visual content remains challenging. Like professionals in the creative industries, this generation requires sophisticated reasoning about items such as visual dynamics, acoustic environments, and temporal relationships. We present ThinkSound, novel framework that leverages Chain-os Thought (CoT) reasoning to enable stepwise. interactive audio generation and editing for videos. Our approach decomposes the process into three complementary stages: foundational foley generation that creates semantically coherent soundscapes, interactive object-centric refinement through precise user interactions, and targeted editing guided by natural language instructions. At each stage, multimodal large language model generates contextually aligned CoT reasoning that guides a unified audio foundation model. Furthermore, we introduce AudioCoT, a comprehensive dataset with structured reasoning annotations that establishes connections between visual content, textual descriptions, and sound synthesis. Experiments demonstrate that ThinkSound achieves state-of-the-art performance in video-to-audio generation across both audio metrics and CoT metrics, and excels in the out-o--distribution Movie Gen Audio benchmark. The project page is available at wtttttt///// hattps://Thinkooms -artSSounn-Projjcctt seajjeet.gittule github..th..... io
Introduction
brandh
Generating realistic sound for video demands more than recognizing objects; it requires reasoning about complex visual dynamics and context determining when an owl is chirping versus flapping its wings, identifying the subtle sway of tree branches, and syn- chronizing multiple sound events within scene, illustrated in Figure [ Current end-to-end video-to- audio (V2A) generation systems (Luo et al., 22044] Xing et al. 20224 Zhang et al. 2024), while having largely improved, often struggle with this composi- tional complexity and contextual nuance. They may produce generic sounds or fail synchronize pre- cisely with subtle visual cues, thus limiting fidelity and user control.
con ready
Troes 12
rosuroff
wit wief flapiig
Terget
notcling dlmmrctio and aarke drrale atto with hrocod insmuction
Figure 1: ThinkSound with CoT: (1) CoT- driven foley synthesis captures semantic and temporal details (2) interactive object-centric refinement for control (3) targeted editing.
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent breakthroughs in Multimodal Large Language Models (MLLMs) (Lu et al. 2024 Liu et al. 2023b Wang et al., 2024b) offer powerful capabilities in understanding and reasoning across multiple modalities. Concurrently,
Thuughh (CoT) prompting (Wei et al. 2022 Zhang et al. 2023 Wang et al. (025) has proven effective at eliciting structured, step-by-step reasoning from large models. These advancements create great potential to fundamentally rethink V2A by decomposing complex sound design into explicit reasoning steps and actionable synthesis instructions. The necessity for CoT in V2A becomes evident when examining the process of sound designers. They employ a multi-stage approach: analyzing visual content, then reasoning about acoustic properties, and finally synthesizing and refining sounds. End-to-end approaches compress the process into single black-box transformation, losing the nuanced reasoning required for audio generation.
Some approaches have attempted to adopt MLLMs for multi-stage V2A. Sonic (Xie et al.) 2024) converts video to textual captions using MI.LMs and employs a separate model for text-to- audio (T2A) generation, which inevitably loses critical visual details and motion dynamics essential for realistic sound synthesis. More recently, DeepSound-V V1 (Liang al. (025). while introducing MLLM-generated CoT for V2A, fragments the process into three separate tasks (audio generation, vocal removal, and silence detection) using specialized models rather than a unified framework. This fragmentation fails to fully leverage the deep understanding and reasoning capabilities that MLLMs could bring to comprehensive audio design.
To overcome these limitations and unlock the full reasoning potential of MLLMs for V2A, we present ThinkSound that harnesses CoT reasoning enable stepwise, interactive generation and editing for V2A. ThinkSound decomposes audio generation into three intuitive and user-centric stages: (1) foundational foley generation to synthesize semantic and temporal matching soundscapes, (2) interactive object-centric refinement via user clicks, and (3) targeted audio editing guided by high- level natural language instructions. At each stage, an MLLM produces semantically and temporally aligned CoT instructions that guide unified foundation model for audio generation, ensuring that the generated audio remains coherent, contextually grounded, and high quality throughout the workflow. Moreover, to support the training of ThinkSound and advance research in this area, we introduce AudioCoT, a comprehensive large-scale dataset with structured CoT reasoning annotations.
Technically, three key innovations are proposed: a) We fine-tune MLLMs on AudioCoT to generate structured, audio-specific reasoning chains that explicitly capture temporal dependencies, acoustic properties, and the decomposition of complex audio events. b) We design unified audio foundation model based on flow matching that supports all three stages with the capabilities of synthesizing high- fidelity audio from arbitrary combinations of input modalities. This foundation model directly benefits from the detailed CoT reasoning provided by MLLMs, which effectively decomposes complex audio scenes into manageable components, enabling focused sound event generation while maintaining global coherence. c) We introduce novel click-based interface that enables users to target specific visual objects for audio refinement, with CoT reasoning translating visual attention into contextually appropriate sound synthesis. The main contributions are summarized as follows:
A novel three-stage interactive framework for V2A that progressively builds soundscapes through initial generation, object-centric refinement, and targeted editing, all unified by CoT reasoning from MLLMs.
A unified multimodal foundation model capable of high-quality audio synthesis from an arbitrary combination of video, text, and audio inputs, leveraging CoT instructions to decompose complex scenes into manageable sound components.
AudioCoT, large-scale multimodal dataset with audio-specific CoT reasoning annotations that bridges visual content, textual descriptions, and sound synthesis.
Experimental results demonstrate that ThinkSound achieves the state-of-the-ar perfor- mance across objective metrics and subjective metries, highlighting the effectiveness of our soaniing-uiieed guided generation.
reasoning-
2 Related Work
2.1
Video-to-Audio Generation
V2A (Cheng et al. 2024a| Wang al. 2024c audio that aligns scamlessly with the visual content of video clip. Earlier work (Luo al. 2024
eue 2024 Liu et (225)) focuses on synthesizing

---

Xing et al. 2024; Zhang et al. 2024 Viertola et al. 2024) try to generate audio samples based on silent video only using latent diffusion models (Rombach et al. 2022) or language models (Floridi) & Chiriatti 2020). For example, Diff-Foley (Luo et al. 2024) employs an audio-visual contrastive feature and latent diffusion to predict the spectrogram latent, while FoleyGen uses autoregressive techniques. However, recent research (Chen et al., 2024] Mo et al. 2024 You et al.) (2255 pay more attention to generating audios using multimodal control, including videos, audios, and texts. For example, MMAudio (Polyak et al. 2024) uses flow matching conditioned multi-modal inputs, including videos and texts. Multifoley (Wu et al., 2024) further uses audio context as an additional imput. Despite these advancements, existing work struggles reason beyond simple object recognition and fails to conduct a step-by hep-by-step interactive process with users for audio generation. In this work, we propose ThinkSound, V2A model with CoT reasoning in MLLMs, which supports step-by-step and interactive audio generation and editing.
2.2 Large Language Models and Reasoning
LLMs (Liu et al. 2024a Guo et al. 2025 Hurst et al.
2024 have demonstrated remarkable reasoning capabilities through Col prompting, enabling complex problem decomposition via intermediate rea-
soning steps. This paradigm, pioncered by (Wei et al. 20022. has been extended to MLLMs (Alayrac ct al. 2022; Yang et al. 2023 Chu et al. (024) that integrate visual, audio, and textual understanding
through cross-modal aligmment. Recent works (Rubenstein et al. 2023 Li et al., 2023 Wu et al. 2024) have explored MILLMs potential in multimodal reasoning, particularly in visual-audio-textuu grounding and cross-modal causal reasoning. Despite these, MILMs remain under-explored for audio generation. While models like Sonic Vision (Xie et al., (024) and DeepSound-Vi (Liang et al. 2025) incorporate V2A, they lack mechanisms decompose user intent into semantic and temporal reasoning steps. Our ThinkSound proposes novel framework for audio generation and editing that decomposes the complex task into foundation Foley generation, interactive object-centric
refinement, and targeted audio editing (Wang et al. 2023
Liu et al. 2024d).
2.3 Flow Matching
Flow Matching has emerged powerful alternative to diffusion models for high-quality audio synthesis (Evans et al. 2024; Liu et al.] 2024bb (023a), directly learning continuous normalizing flows between distributions by training time-dependent vector field. Recent works Lipman et al.
(2022); Liu et al. (2022) establish its theoretical foundations, while Liu et al. (2024c) demonstrated its advantages for audio generation with improved quality and faster sampling. Polyak et al. (2024) further advances this approach by applying conditional flow matching to multimodal audio generation, showing its effectiveness in maintaining temporal coherence across complex audio signals. Our work extends these advances through CoT-guided generation and a novel multi- multti-uiidanee strategy that integrates control signals from multiple modalities. Our modality-agnoste training approach with classifier-free guidance dropout allows the model to generate high-quality audio from flexible input combinations, addressing both control precision and data scareity challenges that have limited previous approaches.
3 AudioCoT Dataset for CoT-Guided Generation and Editing
3.1 Multimodal Data Sources
The AudioCoT dataset comprises both video-audio and audio-text pairs. For video-audio data, we utilize VGGSound (Chen et al.) 2020) and a curated, non-speech subset of AudioSet (Gemmeke et al.) 2017) to ensure broad coverage of real-world audiovisual events. For audio-text data, we aggregate pairs from AudioSet-SL. (Hershey et al. (021), Freesound (Fonseca et al., 2017), AudioCaps (Kim) et al. 2019), and BBC Sound Ffffects resulting diverse and representative corpus for training multimodal models. Further details on data processing and statistics are provided in the Appendix A.I A.]
https://sound-effie wwf.eeeffeeeeeeeeeeeeeee www.bbwwwd.dcco.u///
3

---

Stagel : Foundational Faley CoT Generation
Audio Andlysis
creating symphony of loud bongs and crecktes en crenalger prrn//per/wwence foreence sett
"Gundhots
Frewerks explede,
Integnated Analysi
aequence loud, shorp freearks sound frowirks exploding sucsession. Intermined with these, accasione deep booms sinalete the larger explosione backdrap brief, shere rrowerk loud amphosizing the interne, celebratory aftresphieee No rumeon voices er other sounds are present. focusing salety the firewrrss' custory effects
VideoLLaMA
Poose
Temporal Anclysis
firework being
The sound the
Tireorks loud and fils the entiine video.
aganet
Qwen2 Audio
Content Andlysis
firevoriss and Freeneckers, creating festive chrassheree There humen the video, and the focus is solely on the vouu display of fineworks
The bociground
Grounded
with
GPT--41-nano
Stoge2: Interactive Object-Gentric CoT Generation
Infegrated Analysis
Roi Arolysis
fieowork being up the dark
Bar
Target audio: A.RE.i.....
by soms The sound exploding tirewrks leud and hils the entire video.
"Extroct confent avodd tthe car2
sccording the audis and videe'
another
Stage Instruction Based Audio Editing GoT Generation
Intagrated Analysis
Torget audio Andio Aodi
Zestroationn Noe def:
Ando
Cot Seneration
tth content
ther ridea
detals
Figure 2: Overview of our AudioCoT dataset construction pipeline.
3.2 Automated CoT Generation Pipeline
Stage Foundational Foley CoT Generation. construct the Foley generation CoT data, we employ multi-stage processing pipeline. For video-audio pairs, utilize VideoLLaMA2 (Cheng) ctt al. 2024b) to extract both temporal and semantic information from the videos through differ- prompting strategies, while for audio-text pairs (which lack video data), we employ more streamlined approach without VideoLLaMA 2. both cases, we generate audio descriptions using Owen2-Audio (Chu et al., 2024), combining them with existing video captions text annotations. The collected information is then integrated using GPT-4. 1 1-and to synthesize comprehensive CoT reasoning chains that capture the complex relationships between content conditions and the corre- sponding audio elements, ensuring both data types contribute to a comprehensive understanding of audio generation reasoning.
Stage 2: Interactive Object-Centric CoT Generation. To facilitate object-focused audio generation. develop Region of Interest (ROI) extraction framework leveraging Grounded SAM2 (Ren et al.) 2024; Ravi et al. 2024). Using audio captions as prompts, we identify and generate bounding boxes for potential sound-emitting objects. These coordinates tracked temporally across video frames, while VideoLLaMA2 provides detailed semantic descriptions for each ROI segment. For complex audio manipulations (extraction/removal)) we construct hierarchical reasoning strueture where the CoT of the target video is merged with another video's ' CoT to establish global context. This composite representation is then integrated with ROI-specific generation information and processed by GPT-4.1 -nano to formulate coherent manipulation rationales (detailed in Appendix A..2).
Stage 3: Instruction- Based Audio Editing CoT Generation. For instruction-guiie guidedd audio editing, we analyze and integrate CoT information from Stage based on four primary operations: extension, inpainting, addition, and removal. These operations address scenarios from extending sequences to eliminating unwanted segments. GPT-4. -nano processes this integrated information to generate instriction-speccfiic CoT reasoning chains while perform corresponding audio operations, creating (Instruction-CCn CoT, input audio, output audio) triplets for model training and evaluation.
4
ThinkSound
4.1
Overview
ThinkSound introduces novel step-by-step, interactive framework for audio generation and editing guided by CoT reasoning. Our approach decomposes the complex V2A task into three intuitive stages: (1) foundation Foley generation that creates semantic and temporal
As depicted in Figure 3.

---

VidcoFROt
Caption
Muhti-Modal Flow Matching Tronsformer
Pasding
Gross-Entrapy Loss--Chhii Thought Cuption
Video
Projection
Projectir
Conditice
Mulh-Streen Inoeefenne llace
MLLM becoder !
Hitti-Steamm Transforer
74.
Mult--Strazn Tranforner llack
Gatez Add
Siglle SkgleeShnen Traneforpar Beck
Instruction
Cot (3)
Coption Visual Encoder
Ausio Encoder
Siyli Strees Trandfameu Btock
Senle-Stron Transformar Block
Roi Cot: Extroct audio of scott> Ausio audio of
AddlN
Summash on
exbeoding
Video
Flav
Concoherch
Figure 3: Overview the ThinkSound architecture. Left: our Multimodal LLM framework, where fine-tuned VideoLLaMA 2 model generates CoT reasoning for audio generation and editing. Right: our enhanced Multimodal Transformer architecture, which features an MM-DiT backbone with dedicated pathways for processing multimodal inputs and CoT-driven conditioning to enable high- fidelity, contextually grounded audio generation.
matching soundscape, (2) interactive region- region-based refinement through user clicks, and (3) targeted audio editing based on high-level instructions. each stage, an MLLM generates CoT reasoning that guides a unified audio foundation model to produce and refine the soundtrack.
4.2 CoT Reasoning with Multimodal LLM
To enable stepwise, context-aware audio generation, we leverage VideoLLaMA2 (Cheng et al. 2024b) as the core multimodal reasoning engine. VideoLLaMA2 is selected for its state-of-the-ant capability in fusing video, text, and audio modalities, and its advanced spatial-temporal modelling is essential for capturing the nuanced interplay between visual events and their corresponding acoustic manifestations.
We further adapt VideoLLaMA2 to the audio reasoning domain by fine-tuning it on our AudioCoT dataset, which contains rich, annotated reasoning chains tailored for audio-visual tasks. This fine- tuning process is designed to equip the model with three core competencies: (1) audio-centric understanding the ability to infer acoustic properties. model sound propagation, and reason about audio-visual correspondences, including temporal and causal relationships among audio events (c.g., "footsteps occur before the door opens, then conversation ensues"); (2) structured CoT decomposition-the capacity break down complex audio generation or editing tasks into explicit, actionable steps; and (3) multimodal instruction following following-robustly interpreting and executing diverse generation or editing instructions across modalities. As illustrated in Figure 33, the fine- tuning objective is the standard cross-entrope loss for next-token prediction. Through this targeted adaptation, VideoLLaMA2 is transformed into a specialized audio reasoning module, capable of producing contextually precise CoT instructions that drive each stage of the ThinkSound pipeline.
4.3 CoT-Guided Unified Audio Foundation Model
The core of ThinkSound is our unified audio foundation model that seamlessly translates CoT reasoning into high-quality audio, whose detail shown in the right part of Figure 3, We encode audio into latent representations using pre-trained VAE (Pinheiro Cinelli et al.] (02T) and train our model with conditional flow matching (Lipman et al.) (022). where the velocity field prediction is conditioned multimodal context, including visual content, CoT reasoning, text captions, and audio context. To support any combination of input modalities, we adopt the integration classifier- free guidance dropout during training. By randomly dropping different modality combinations with probability Pdrop. we enable the model to handle arbitrary input configurations at inference time essential for our interactive framework. We also incorporate strategic audio context masking to support advanced editing operations such as inpainting and extension.
For text processing, we employ dual-pathway encoding strategy: MetaCLIP Xu et al. 20233
encodes visual captions to provide scene-level context, while T5-v1-xl (Raffel al. 2020) processes structured CoT reasoning capture detailed temporal and causal relationships. These complementary
5

---

representations are effectively combined, with MetaCLIP's features serving as global conditioning signals while T5's outputs enable fine-grasn ine-grained reasoning-driver control.
Our enhanced MM-DiT architecture builds on recent advances in multimodal generative model- ing (Esser et al. 2024 Labs 2024 Cheng et al. 2024a) with three key components: (1) implement
a hybrid transformer backbone that alternates between modality-specific and shared processing. Multi- stream transformer blocks maintain separate parameters for each modality while sharing attention mechanisms, allowing efficient processing of diverse inputs without sacrificing cross-modal learning. (2) We design an adaptive fusion module that upsamples video features and fuses them with audio latents via gated mechanism (Cho et al. (014). This not only highlights salient visual cues and suppresses irrelevant information, but also ensures that video information directly involved in subsequent single-stream transformer blocks. By integrating video into the audio latent space, the model can better capture subtle visual details and their nuanced effects on the soundscape, enabling richer cross-modal reasoning than using audio latents alone. (3) We implement global conditioning by mean-pooling CLIP features from both the caption and video, and, following MMAudio (Cheng et al., 2024a)), incorporating sync features to improve audio-visual temporal alignment. The resulting global condition is added to the timestep embedding and injected via adaptive layer normalization layers (AdaLN) Peebles & Xie 2023) into both multi-stream and single-stream blocks.
4.4 Step-by-Step CoT-Gu Audio Generation and Editing
By enabling flexible combinations of input modalities along with CoT, ThinkSound supports de- composing audio generation into three intuitive stages shown in Figure T This stage pipeline enables progressively refined, highly customized audio generation through an intuitive interactive workflow, with CoT reasoning bridging user intent and audio synthesis each step.
CoT-Guided Foley Generation. In the first stage, our framework analyzes the entire video to identify acoustic elements and their relationships. The fine-tuned MLLM generates detailed CoT reasoning that explicitly identifies primary sound events, ambient elements, acoustic properties, and their temporal dependencies determining when objects make sounds and how these sounds interact. This structured reasoning guides the audio foundation model to synthesize high-fidelity audio that precisely matches both the semantic content and temporal dynamics of the visual scene. By decomposing complex audio scenes into explicit sound components through CoT reasoning, the model generates a diverse and coherent soundscape that captures subtle visual cues and motion dynamics essential for realistic audio synthesis.
Stage 1:
Stage 2: Interactive Object-Focusee Audio Generation. Stage 2 introduces an interactive frame-
work that enables users to refine the initial soundscape by focusing on specific visual elements. Through a simple click-based interface, users can select objects of interest for audio enhancement. Unlike the holistic approach in Stage 1, this object-centric refinement leverages the segmented ROI to guide targeted audio synthesis. The fine-tuned MLLM analyzes the selected ROI and generates specialized CoT reasoning focused on the object's 's acoustic properties within the global context. This structured reasoning conditions the audio foundation model to synthesize object-specific sounds, seamlessly integrating them with the initial soundtrack produced in Stage 1. Notably, in this stage, the foundation model incorporates the existing audio context as an additional conditioning signal.
Stage 3: Instruction-bann based Audio Editing. the final stage, users can provide high-level editing instructions refine audio quality modify specific elements. The MILM translates these natural language instructions into precise audio processing operations through CoT reasoning, considering both visual content and current audio state. The foundation model, conditioned on both this reasoning and the existing audio context, applies targeted modifications while maintaining overall coherence. This natural language-driven approach enables non-technical users to perform sophisticated audio manipulation, including adding sounds, removing sounds, audio inpainting, and audio extension.
5 Experiments
5.1 Experiment Setup
Evaluation Metrics We conduct comprehensive evaluations using both objective and subjective metrics to assess audio quality, text-audio alignment, and video-audio synchronization. For objective

---

Table I: Comparison of our ThinkSound foundation model with existing video-to-audio baselines on the VGGSound test set. + indicates lower is better, 1 indicates higher is better. For MOS, we show the mean and variance of the MOS scores. indicates that the method does not use text for inference.
Method
Objective Metrics
Subjective Metries
Efficiency
FDJ
Kl pust
Klane
DeSync)
ClAPPop
CLAPest
MOS-Q1
MOS-AT
Params
Time(s)
4.37=0.21
4.56:20.19
Seu&Heur
118.95
2.26
2.30
1,20
0.32
0.35
1.75
2.87.10.99
415M
19.42
V-AURA"
46,99
2.23
1.83
0.65
0.23
0.37
3,42+1033
5.20:11.17
695M
14.00
CraeyCrafier
39.15
3.08
2.63--0.88
3.84
Frieren"
74.96
2.55
2.64
1,00
0.37
0.34
3.27+1.11
2,95++1.09
159M
V2A-Mapper
1.31
3.16:1:.04
MMAadio
43,26
1,65
1.40
0.44
0.31
0.40
3.84-0.89
3:97++0.82
1.038
3.11
ThinkSound
34.56
1,52
1.32
0.46
0.33
0.46
0.02
4.18.10.9
1.30B
1.07
w/o CoT Reasoning
39.84
1.59
1.40
0.48
0.29
0.41
3.91
4.04::0.75
1.30B
0.98
evaluation, we compute the Fréchet Distance (FD) (Kilgour al. 2018 in the OpenL3 feature space (Cramer et al. 2019 Evans et al., 2024) to measure distribution-leve similarity, which we extend to evaluate stereo audio. We also use the Kullback-Leibler (KL) Divergence (Copet et al. 2024) based on predictions from PaSST model (KLPassT) and PaNNs (KLPuNNs) to evaluate label- consistency. Temporal alignment between audio and video is assessed using DeSyne predicted by the Synchformer model (Iashin et al. 2024). while semantic alignment with text is measured using the CLAP score (Wu* et 20033 Chen et al., 2022), including both CLAP cap (caption) and CLAPcsr (CoTT) For subjective evaluation, we collect human ratings using the Mean Opinion Score (MOS) to evaluate perceived audio quality (MOS-Q) and alignment with video and CoT (MOS-A). Further
details are provided in the Appendix C
Implementation Details For VAE training. we initialize our VAE using the VAE model weights trained on stereo data at 44. 1kHz sample rate provided by Stability AT We employ mixed precision training with batch size of 144 across 24 A800 GPUs for 500, 500,000 steps. Subsequently, following Evans et al. (2024). we freeze the VAE encoder and train the VAE decoder with a latent mask ratio of 0.1 for additional 500,000 steps. We use AdamW W (Loshchilov & Hutter 2019] as the optimizer, setting the generator learning rate to 3e-5 and the discriminator learning rate to 6e-5. In the foundation model training phase, we utilize exponential moving average and automatic mixed precision for 100,000 steps on 8 A100 GPUs, with an effective batch size of 256. We adopt a efg dropout of 0.2 for each modality with a learning rate of le-4. During the task-specific fine-tuning stage, we similarly apply exponential moving average and automatic mixed precision for 50,000 steps on 8 A 100 GPUs, maintaining an effective batch size of 256. AdamW remains our optimizer of choice, with learning rate set at le-4. We attach the benchmark details of VGGSound, Movie Gen Audio Bench, and AudioCoT test set for stages and into Appendix A.3
5.2 Main Results
Video-to-Audio Generation Results We compare our foundation model against existing video- to-audio baselines including Seeing and Hearing (Xing et al.) (024). V-AURA (Viertola et al. 2024). FoleyCrafter (Zhang et al. 2024), Frieren (Wang et al.] 2024d), V2A-Mapper (Wang et al. 2024a), and MMAudio (Cheng et al. 2024a). From Table TJ we observe three key find- ings: (1) GT audio exhibits low CLAP cup scores (0.28), revealing that original VGGSound captions inadequately capture the semantic content and temporal relationships needed for high- fidelity audio generation. (2) ThinkSound outperforms all baselines across most objective met- rics and all subjective metrics. Compared to the strongest baseline (MMAudio), our model achieves substantial improvements in audio quality (KLPsSST: 1.52 vs. 1.65) and semantic alignment (CLAPcor: 0.46 vs. 0.40). while maintaining comparable temporal synchronization. Subjective evaluations further confirm these im- provements (MOS-Q: Q: 4.02 vs. 3.84, MOS-A: 4.18 vs. 3.97). (3) Removing CoT reasoning notably degrades both audio quality and align- ment metrics, especially for CLAPeoT, decreas- ing from 0.46 to 0.41, confirming that CoT pro- vides crucial information about sound events, their temporal relationships, and acoustic char- acteristics.
Table 2:
Out-of-distribution evaluation on MovieGen Audio Bench. This benchmark does not provide the GT audios, so we cannot compare FD and KL.
Method
CLAP 1 DeSyne)
MOS-QI
MOS-A
MMAudio
0.45
0.77
3.95++0.87
3.62+:1033
MovieGen
0.47
1.00
3.98-10.77
3.70.10.96
ThinkSound
0.51
0.76
4.11:10.74
3.87-10.82
'naps://iituib.co//aabbll
7

---

Furthermore, we conduct an out-of distriibbtii evaluation on the MovieGen Audio Bench (Polyak) al., 2024) assess the generalization capability of our model. The results in T show that our ThinkSound model still achieves the best CLAPPcT score of 0.51 while DeSyne is on par with the best baseline. For subjective evaluation, ThinkSound achieves the best performance both alignment and fidelity metrics. This demonstrates that ThinkSound exhibits strong generalization capability across different scenarios.
Object-Focused Audio Generation and CoT-
Table 3: Object-focused generation performance.
Guided Audio Editing For object-focused generation, we compare two approaches: (1) MMAudio, which does not have the ROI design, and (2) ThinkSound w/o CoT reasoning. Results in Table 3 show that our interactive approach with Co'I" reasoning achieves significantly better results, demonstrating superior object-specific sound quality and integration with the foundation audio.
Method
FDL
KL
CLAP1|
MOS-Qt
MOS-AT
MMAudio
44.46
1.38
0.41
33611 3.64
ThinkSound w/ CoT
43.27
1.32
0.48
3.89:
3.91100,53
45.28
1.34
0.43
3.77=0.64 3.81
For text-guided audio editing, we compare ThinkSound with AudioLDM-2 (Liu et al., 2024e) and Edit Friendly DDPM (Huberman-
Table 4: Audio editing results AudioCoT test set (MOS-A: alignment between audio and text; DDPM: DDPM- JDPM-Friendly).
Spiegelglas et al. 2024), adapting these models our experimental setup. As shown in Table ( ThinkSound outperforms the baselines across all objective and subjective metries. Specifically, ThinkSound achieves the lowest FD (34.78) and
Method
FD4
KLPıSST | CLAPt
MOS-Qt
MOS-Af
ADdieLDM-22
61.28
1.94
0.35
0,28
3,48-0082 3.34.10.2
DDPM
55.56
1.75
0.39
3.6710.56
Shund w/o CoT
34.78
0.51
3.92.10.82
3.85
KLPaSST (1.45), and the highest CLAP score (0.51), indicating better audio fidelity and se- mantic relevance. In human evaluation, ThinkSound also gets the best MOS-Q and MOS-A. Remov- ing CoT reasoning clearly drops all metrics, showing the importance of CoT--guided reasoning for text-based audio editing.
45.78
1.58
0.44
3.53:+0.45
3.52+00.2
5.3 Ablation Studies
To better understand the contribution of each component in ThinkSound and to validate the
effectiveness of our design choices, we conduct comprehensive ablation studies on the VG-
GSound test set. Mainly focus on: 1) text encoding strategies and 2) multi-modal integra- tion mechanisms. For more ablation and exploratory results, we attach them to Appendix D
Table 5: Comparison of text encoder fusion strate- gies. The input of the CLIP text encoder is the caption.
Text Encoding Strategies. We evaluate dif- ferent text encoding strategies with or without CoT reasoning, and the results are compiled in Ta The results show that (I) CoT reason- ing substantially improves audio fidelity, with the FD score improving from 39.84 to 37.65 when comparing CLIP-only to ba-based CoT ap- proaches. (2) The integration of contrastive fea- tures from CLIP with contextual features from T5 further enhances performance, reducing KI. divergence metrics from 1.54 to 1.52 (KLPasST) and from 1.35 to 1.32 (KLPaNNs).
The CLAP score means the value of CLAPcoT.
Method
FDL
KLeassT 4 KL + DeSync| CLAP t
CLIP
39.84
1.59
1.40
0.48
0.41
T5 (CoT)
37.65
1.54
1.35
0.46
0.44
CLIP+T55
34.56
1.52
1.32
0.46
0.46
Multi-Modal Integration Mechanisms. We investigate different multi-modal integration mechanisms for video and audio before feed- ing them into the single-stream transformer, and the results are displayed in Ta We find that (1) the element-wist addition video and au- dio features performs better than audio-only in- put with superviuevii global conditioning, especially for synchronization metric DeSync,
Table 6: Comparison of different multi-moda in-
tegration mechanisms between video and audio features.
Integration
FDJ
KL
KL
DeSyne]
CLAP
audio only
37.13
1.58
1.37
0.50
0.43
linear video
38.96
1.58
1.38
0.46
0.45
gated vidco
34.56
1.52
1.32
0.46
0.46
8

---

Open digar
Cluse
Vacnitze
Seode Hear
MMAudia
ThinkSound
Ground-trulh
Degin with quiet ambieni backdrup faint wind noise
Regin with quiet, serene backdrop- soft breeze or rastling leaves. Subdenly, clear pheasani
remains open view. mechanical thad then marks it elasing. After sevena seconds, it reopens, and few seconds later slams shut Ambiast sounds stay minimal, anchoring renlism without distracting from the doer's tacille motion.
through the calm fades, subtle chirps gentle clucking resumee sustaining a natuma atmoophere. Firally, the aodio gradially quies, balancing dynamism and tranquility.
impact allowed fully
Figure 4: Qualitative Comparison: Left:: Spectrograms for a door movement sequence (closed opened + closed), showing ThinkSound's precise alignment of each door sound versus the baseline's premature opening effect. Right: Spectrograms for grassy-ficld pheasant scene (ambient bird calls - >wing--lapp chirp + ambient calls), illustrating ThinkSound's accurate detection and timing of the transient chirp compared the baseline's s omission delay.
decreasing from 0.50 0.46. (2) The gated fusion mechanism outperforms simple element-wise addition and audio-only input across all metrics.
5.4 Case Study
our qualitative analysis, we compare spectrograms audio generated by ThinkSound and those produced by baselines, illustrated in Figure 4 We make the following observations: (1) As demonstrated case car door sequence- -losee--peeeddd-oooee--------- ThinkSound, whereas the bascline models erroneously introduce extra opening sound at the start. This highlights ThinkSound's s ability, via CoT prompting, to track temporal and causal event order in structured scenes. (2) In case 2, pheasant moving in grassy held-accompantin bird calls, suddenly flapping its wings and emitting a sharp chirp before returning to background noise-is is faithfully reproduced by ThinkSound. The baselines, however, often miss delay this brief chirp. These cases underscore ThinkSound's enhanced temporal reasoning and sensitivity to subtle visual cues, resulting in more precise and context-aware audio synthesis.
accurately reflected by
by ambient
Conclusion
This paper presented ThinkSound, novel CoT reasoning framework for audio generation and editing that decomposed complex audio generation into three interpretable and user-centric stages, Our evaluations showed that ThinkSound outperformed state-of-the-arr methods, producing contextually appropriate and temporally precise soundscapes. The framework's 's interactive nature enabled users refine generated audio through intuitive pointing and natural language instructions, addressing the gap between creative intent and automated generation. While challenges remained in capturing nuanced acoustic properties, our AudioCoT dataset and training strategy established foundation for future research in intelligent audio generation with applications across film, gaming, and social media. In future work, we will explore incorporating physical acoustic modeling and developing more sophisticated reasoning capabilities for complex multi-object sound interactions.

---

Acknowledgements
The research was supported by the NSFC (No. 62206234) of Mainland China.
References
Alayrac, J..B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: visual language model for few-shot learning, Advances in neural information processing systems, 55:23716-23736 2022.
Chen, H., Xie, W., Vedaldi, A., and Zisserman, A. Vggsound: A large-scale audio-visual dataset.
arXiv preprint arXiv:2004. 14368, 2020.
Chen, K., Du, X., Zhu, B., Ma, Z., Barg-Kirkpatrick, T., and Dubnov, S. Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2022.
Chen, Z., Seetharaman, P., Russell, B., Nieto, O., Bourgin, D., Owens, A., and Salamon, J. Video- guided foley sound generation with multimodal controls. arXiv preprint arXiv:2411.17690 2024.
Cheng. H. K., Ishii, M., Hayakawa, A., Shibuya, T., Schwing, A., and Mitsufuji, Y. Taming multi- modal joint training for high-quality iidoo-t--addoo synthesis. arXiv preprint 0rXiv:2412.15322 2024a.
Cheng, Z., Leng, S., Zhang, H., Xin, Y., Li, X., Chen, G., Zhu, Y., Zhang, W., Luo, Z., Zhao, D., et al. Videollama 2: Advancing hempalltempooal modeling and audio understanding in video-lims. arXiv preprint arXiv:2406.00 20X7:2206607476,, 2024b.
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. Learning phrase representations using mnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: /406. 1078, 2014.
Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z.,, Leng, Y., Lv, Y., He, J., Lin, al. Qwen2-audio technical report. arXiv preprint arXiv:2407. 10759, 2024.
Copet, J., Kreuk, F., Gat, I., Remez, T., Kant, D., Synnaeve, G., Adi, Y., and Défossez, A. Simple and controllable music generation. Advances in Neural Information Processing Systems, 36, 2024.
Cramer, A. L., Wu, H.-H., Salamon, J., and Bello, J. Look, listen, and learn more: Design choices for deep audio embeddings. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 3852-3856. IEFE, 2019.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. In forty-first international conference machine learning, 2024.
Evans, Z., Carr, C., Taylor, J., Hawley, S. H., and Pons, J. Fast Liming-condition latent audio diffusion. arXiv preprint vrXiv:2402.04822 2024.
Floridi, L. and Chiriatti, M. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681-694, 2020.
Fonseca, E., Pons, J., Favory, X., Font, F., Bogdanov, D., Ferraro, A., Oramas, S., Porter, A., and Serra, X. Freesound datasets: platform for the creation of open audio datasets. In Cunningham, S. J., Duan, Z., Hu, X., and Turnbull. D. (eds.), Proceedings of the 18th International Society for Music Infommaion Retrieval Conference, ISMIR 2017, Suzhou, China, October 23-27, 2017, pp. 486 493, 2017.
Gemmeke, J. F., Ellis, D. P., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., and Ritter, M. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780, IEFE, 2017.
10

---

Guo, D., Yang, D., Zhang, H., Song. J., Zhang, R., Xu, R.,, Zhu, Q., Ma, S., Wang. P., Bi, X., et al. Deepseek-rl: Incentivizing reasoning capability llms via reinforcement learning. arXiv preprint arXiv:2501. 122448, 2025.
Hershey, S., Ellis, D. P., Fonseca, E., Jansen, A., Liu, C., Moore, R. C.,, and Plakal, M. The benefit of remporally-strone labels in audio event classification. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 366-370. IEEE, 2021.
Huberman-SSieeglllas,, I., Kulikov, V., and Michaeli, T. An edit friendly ddpm noise space: Inversion and manipulations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21277, 2024.
lashin, V., Xie, W., Rahtu, E., and Zisserman, A. Synchformer: Efficient synchronization from sparse cues. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5325-5329. IEEE, 2024.
Kilgour, K., Zuluaga, M., Roblek, D., and Sharifi, M.
audio distance: metric for evaluating music enhancement algorithms. arXiv preprint 0rXiv:/8/2.08466) 2018.
Kim, D., Kim, B... Lee. H.,, and Kim, G. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 139- 2019.
Kong, Q. Cao, Y., Iqbal. T.. Wang, Y., Wang, W., and Plumbley, M. D. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880-2899 2020.
Koutini, K., Schlüter. J., Eghbal-Zadch, H., and Widmer, G. Efficient training of audio transformers with patchout. arXiv preprint arx::.11100000000000 2021.
Labs, B. F. Flux.
http:taabb.ccc/lllcccooooeeeeeee laaese--aeet----aallllaae 2024.
Li, K., He, Y., Wang, Y., Li, Y., Wang. W., Luo, P., Wang. Y., Wang, L., and Qiao, Y. Videochat: Chat-centric video understanding. arXiv preprint (tXXi::3305.06355 2023.
Liang, Y., Chen, Z., Ding, C., and Di, X. Deepsound-yl: Start to think step-by-step in the audio generation from videos, 2025. URL httttttss///riiiii org/ abs abs/ 22208
Lipman, Y., Chen, T., Ben-Hamu, H. Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747 2022.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024a.
Liu, H., Huang, R., Lin, X., Xu, W., Zheng, M., Chen, H., He, J., and Zhao, Z. Vit-tts: visual text-to-speech with scalable diffusion transformer. arXiv preprint vrXiv:2305.12700 2023a.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tuning. Advances neural information processing systems, 01344892-449111 2023b.
Liu, H., Huang, R., Liu, Y., Cao, H., Wang, J., Cheng. X., Zheng, S., and Zhao, Z. Audiolcm: Efficient and high-quality text-to-audio generation with minimal inference steps. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 7008-7017, 2024b.
Liu, H. Wang, J., Huang. R., Liu, Y.. Lu, H., Xue, W., and Zhao, Z. Flashaudio: Rectified flows for fast and high-fidelity text-to-audio generation. arXiv preprint arXiv:2410.12266 2024c.
Liu, H., Wang, J., Huang, R., Liu, Y., Xu, J., and Zhao, Z. MEDIC: zero-shot music editing with disentangled inversion control. CoRR, abs/2407. 13220, 2024d. doi: 10448500AARXI.2200..2222 URL https:: //oi org/10. 48550/arXis 2407 13220

---

Liu, H.. Yuan, Y., Liu, X., Mei, X., Kong, Q. Tian, Q. Wang, Y., Wang. W., Wang, Y., and Plumbley. M. D. Audioldm Learning holistic audio generation with supe-suue pretraining. JEEE/ACM Transactions Audio, Speech, and Language Processing, 2024e.
Liu, H., Luo, T., Jiang, Q. Luo, K., Sun, P., Wan, J., Huang, R., Chen, Q. Wang, W., Li, X., et al. Omniaudio: Generating spatial audio from 360-degree video. arXiv preprint arXiv:2504. 14906, 2025.
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint 22Xhh:2209.03003 2022.
Loshchilov, I. and Hutter, Decoupled weight decay regularization. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. /ttpp:tts///opeereviieee http::/wwww..eeooooodddddddaaaae
Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., Sun, J., Ren, T., Li, Z., Yang, H., et al. Deepseek-vi. vl: towards real-world vision-languase understanding. arXiv preprint :xXv::2403.05525, 2024.
Luo. S., Yan, C., Hu, C., and Zhao, H. Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Mo. S., Shi, J., and Tian, Y. Text-to-audio generation synchronized with videos. arXiv preprint WxXiv:2403.07988 2024.
Peebles, W. and Xie, S. Scalable difffusion models with transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp. 4195-4205, 2023.
Pinheiro Cinelli, L., Araújo Marins, M., Barros da Silva, E. A., and Lima Netto, S. Variational autoencoder. In Variational methods for machine learning with applications to deep networks, pp. 111-149. Springer, 2021.
Polyak, A., Zohar, A., Brown, A., Tjandra, A., Sinha, A., Lee, A., Vyas, A., Shi, B., Ma, C.- Y., Chuang, C.-Y,, al. Movie gen: A cast of media foundation models. arXiv preprini arXiv:2410.13720, 2024.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W. and Liu, P. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
http://aprss/vv1..
074. html
Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali, C.. Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson. L., Mintun, E., Pan, J., Alwala, K. V., Carion, N., Wu, C.-Y., Girshick, R., Dollár, P. and Feichtenhofer, C. Sam 2: Segment anything in images and videos, 2024. URL https 7/arxiv. org/ abs/ 1.408.0077
Ren, T., Jiang, Q., Liu, S., Zeng, 7.,, Liu, W., Gao, H., Huang, H., Ma, 7., Jiang, X., Chen, Y., Xiong, Y., Zhang, H., Li, F., Tang, P., Yu, K., and Zhang, L. Grounding dino 1.5: Advance the "edge" of open-set object detection, 2024.
Rombach, R., Blattmann. A., Lorenz, D.,, Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022.
Rubenstein, P. K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., Quitry, F. d. C., Chen, P., Badawy, D. E., Han, W., Kharitonov, E., al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306. /2925, 2023.
Viertola, I., lashin, V., and Rahtu, E. Temporally aligned audio for video with autoregression. arXiv preprint arXiv:2409. 13699, 2024.
Wang, H., Ma, J., Pascual, S., Cartwright, R., and Cai, W. V2a-mapper: A lightweight solution for vision-to-audio generation by connecting foundation models. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 15492-15501, 2024a.
12

---

Wang, P., Bai, S., Tan, S., Wang, S., Fan. Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision-lange language model's perception of the world any resolution. arXiv preprint arXiv:2409. 12191, 2024b.
Wang. X., Wang, Y., Wu, Y., Song, R., Tan, X., Chen, Z., Xu, H., and Sui, G. Tiva: Time- aligned video-to-audio generation. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 573 582, 2024c.
Wang, Y., Ju, Z., Tan, X., He, L., Wu, Z., Bian, J., al. Audit: Audio editing by following instructions with latent diffusion models. Advances in Neural Information Processing Systems, 36: 71340-71357, 2023.
Wang, Y., Guo, W., Huang, R., Huang, J., Wang. Z., You, F., Li, R., and Zhao, Z. Frieren: Efficient video-to-audio generation network with rectified flow matching. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024d. URL httpss//oppereevie petporummmddppXMM5XXX6
Wang. Y., Wu, S., Zhang, Y., Yan, S., Liu, Z., Luo, J., and Fei, H. Multimodal chain-of-thought reasoning: A comprehensive survey. arXiv preprint 1rX::22533.12605, 2025.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2493 2022.
Wu, S., Fei, H., Qu, L., Ji, W., and Chua, T.-S. Next-gpt: Any-to-any multimodal Ilm. In Forty-first International Conference on Machine Learning, 2024.
Wu*, Y., Chen*, K., Zhang* T., Hui*, Y., Berg- Kirkpatrick, T., and Dubnov, S. Large-scale scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023.
Xie, Z., Yu, S., He, Q., and Li, M. Sonievisionlm: Playing sound with vision language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26866-2008 26875, 2024.
Xing. Y., He. Y., Tian, Z., Wang, X., and Chen, Q. Seeing and hearing: Open-domain viuuol-audio generation with diffusion latent aligners. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 7151-7161. IEEE, 2024. doi: 10. 10110099222223.000000 200./7733...2200066.8 URL httph: btttpa://do... http///o..oo/////// 1109//999511 2068.00683 00683
Xu, H., Xie, S., Tan, X. E., Huang, P.-Y., Howes, R., Sharma, V., Li, S.-W., Ghosh, G., Zettlemoyer, L., and Feichtenhofer, C. Demystifying clip data. arXiv preprint arXiv:2309.16677 2023.
Xu, M., Li, C., Tu, X., Ren, Y., Chen, R., Gu, Y., Liang, W., and Yu, D. Video-to-audio generation with hidden alignment. arXiv preprint 2xXi:24407.07444 2024.
Yang. Z., Li, L., Lin, K., Wang, J., Lin, C.-C., Liu, Z., and Wang, L. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309. /7421, 9(1):1, 2023.
You, Y., Wu, X., and Qu, T. Ta-v2a: Textually assisted video-to-audio generation. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025.
Zhang, Y., Gu, Y., Zeng, Y., Xing, Z., Wang, Y., Wu, Z., and Chen, K. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds. arXiv preprint arXiv:2407.01494 2024.
Zhang, Z., Zhang, A., Li, M., Zhao, H., Karypis, G., and Smola, A. Multimodal chain-of-thought reasoning in language models. arXiv preprint
2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We accurately reflect the paper's contributions and scope in the abstract and section introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed limitations in section E Limitation and Future.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
We provide theory assumptions and proofs in section 4
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: we will open-source our codes(contains model or their access method), which is enough reproduce experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: We provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/ppp wwwpppblccgguiicuceccccceooooope Detgguddsse/CeSbbbiiiinn htt.SSbbbiiiiooooiiiiiooooi for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: specify all the training and test details Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We only use one fixed random seed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information on the computer resources in training details
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed Positive impacts in section 6 Conclusion and negative impacts in section F
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We take actions for safeguards, which is described in section G Safeguards
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Creators or original owners of assets, used in the paper, are properly credited and the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our assets well documented and the document is provided
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We carefully follow the LLM policy and fully describe the usage of LLMs
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur//e /2tp://s//////ooeeeeeee///////// for what should or should not be described.
20

---

A AudioCoT Dataset Details
A.1 Data Collection and Preprocessing
To ensure high data quality and consistency, we employ comprchensive preprocessing pipeline. We begin by removing silent audio-vides clips to retain only those with meaningful content. For the AudioSet subset (Gemmeke et al. (017). we further exclude segments containing human voices based on tag information, as our focus is on non-speech audio. All audio-video clips are then segmented into fixed-lenge intervals of 9.1 seconds, with any shorter clips discarded to maintain uniformity. To achieve balanced dataset, we maintain an approximately 1:1 ratio between music and sound effect samples, ensuring equal representation of both categories.
A.2 Quality Control for Automated Data Pipeline
To enhance the effectiveness of CoT' reasoning in preserving audio characteristics while integrating visual context, we implemented a comprehensive multi-stage quality control pipeline:
Stage I: Audio-Text Alignment Filtering We employ systematic approach to ensure high- quality audio-CoT pairs. First, we calculate the CLAP score between each audio sample and its corresponding CoT description quantify semantic alignment. For pairs exhibiting low CLAP scores (below 0.2), we regenerate the CoT using an enhanced prompt specifically designed to emphasize audio characteristics and features. After regeneration, we recalculate the CLAP score to assess improvement. Audio samples that continue to demonstrate poor alignment (persistent low CLAP scores) are excluded from the dataset maintain quality standards.
Stage 2: Object Tracking Consistency To ensure reliable audio-visual correspondence, we retain only those video sequences containing at least one Region of Interest (ROI) that remains consistently visible throughout the entire duration. Videos with objects that disappear from view or exhibit inconsistent tracking are filtered out. This criterion ensures that our dataset maintains high-quality visual references for audio generation tasks, providing consistent visual anchors for the audio generation process.
Stage 3: Semantic Pairing of Audio Components For tasks requiring paired audio components, we utilize GPT-4.1 -nano analyze tag categories from VGGSound (Chen et al.) (2020) based on two critical criteria. First, we ensure semantic distinctiveness, where tags must be sufficiently distinct to avoid confusion during audio extraction and removal tasks. Second, we verify contextual plausibilityy ensuring that the co-occurrence of paired sounds is contextually reasonable within the same acoustic scene. This balanced approach ensures that our audio pairs are both semantically meaningful and practically useful for audio generation tasks.
Human Verification Protocol To validate our automated filtering processes, we implement rigorous manual review at each pipeline stage. No less than 5% of the total data volume undergoes human inspection to ensure quality. This verification step helps validate the effectiveness of our automated filtering criteria and ensures the overall reliability and quality of our dataset. The human reviewers assess both the technical aspects of alignment and the perceptual quality of the audio-visual correspondence. When samples fail human verification, they are immediately removed from the dataset. Additionally, if the human rejection rate for any filtering criterion exceeds 5%, we recalibrate the corresponding automated filtering parameters and reprocess the entire batch to maintain dataset integrity. This feedback loop between automated filtering and human verification ensures continuous improvement of our quality control pipeline.
A.3 Benchmark Construction
We evaluate the performance of ThinkSound on three different tasks: video-to-aus -udio generation, object-focused audio generation, and audio editing. For the video-to-audio generation task, we use the VGGSound test set as the in-distribution evaluation set while the MovieGen Audio Bench is the out-of-distributtin evaluation set. For the VGGSound test set, we use the same quality filtering protocol as our training data preparation. Given that our primary focus is on video-to-soume video-to-sound/music generation, construct three different difficulty levels based on the complexity of the audio-visual
21

---

Table 7: Overview of datasets used in our work.
Dataset
Modality
Text Format
Hours
VGGSound Chen al.
2020)
Audio-Viden
Caption
453.6
AudioSet Gemmeke al.
2017
Audio-Video
Caption
287.5
AudioSet-SI (Hershey et
202
Audio-Text
Caption
262.6
Freesound Fonseca et
2017
Audio-Ters
Caption
1286.6
AudioCaps Kim al.
Audio-Texx
Caption
112.6
BBC Sound Effects
Audio-Text
Tags
128.9
Total Hours
2531.8
relationships. Specifically, we distinguish the difficulty levels based on a multi-dimensional scoring system that evaluates:
Semantic Consistency: The alignment between the audio and the visual content is evaluated by the ImageBind score (0.3+ for easy, 0.25-0.3 for medium, 0.2-0.25 for hard), and the CLAP score between audio and CoTT (0.4+ for easy, 0.3-0.4 for medium, 0.2-0.3 for hard).
Temporal Synchronization: The degree of synchronization between visual events and corresponding sounds evaluated by DeSync score (0-0.3 for easy, 0.3-0.6 for medium, 0.6+ for hard).
Acoustic Scene Complexity: The audio events' numbers (one dominant sound for easy, 2-3 distinct sounds for medium, multiple overlapping sounds for hard)
According the above evaluation criteria, input the scores and criteria into GPT-4.1-name to generate the difficulty level for each sample. The final difficulty assignment follows a tertile distribution: the lowest-scoring third is classified as "easy," " the middle third "medium," " and the highest-scoring third as "hard." This stratified approach ensures balanced representation across difficulty levels while maintaining meaningful distinctions task complexity. For each difficulty level, we construct a benchmark subset containing around 2000 samples.
For stages 2 and 3, maintain methodological consistency with our training protocols while adapting the evaluation criteria to each task's 's specific requirements. For stage 2, we select samples with clearly identifiable visual objects that produce distinct sounds, while for stage 3, focus on samples with different audio categories suitable for manipulation tasks. Each evaluation subset contains approximately 2,000 samples.
B
Model Configurations and Architecture
B.1 Model Configurations
ThinkSound consists of two primary components: hierarchical variational autoencoder (VAE) for audio compression and reconstruction, and a flow-matching multimodal transformer.
The encoder consists of five convolutional blocks with channel multipli- ers [1, 2, 4, 8, 16] and strides [2, 4, 4, 8, 8), projecting the stereo waveform into a 28-dimensional latent space. The decoder mirrors this architecture with transposed convolutions reconstruct 64-dimensional latent representations back into the time-domain waveform.
Variational Autoencoder
Multimodal Diffusion Transformer ThinkSound employs an enhanced Multi-modal Diffusion Transformer (MM-DiT) with a hidden size dimension of 1024. It comprises 14 multi-stream trans- former layers and single-stream transformer layers, with 16 attention heads. We further attach our different model scale parameters for reference. We use the large model by default.
B.2 Model Architecture
The architecture of multistream transformers is depicted in Figure
22

---

Table 8: Diffusion Transformer Configurations at Different Model Sizes
Model Scale
Hidden Size
Depth
Attention Heads
Multistream Layers
Singlestream Layers
Total Parameters
Large
1024
21
16
14
7
1.3B
Medium
768
21
12
14
7
724M
Small
512
18
8
12
6
533M
Query
RoPE
norm
F
adal.it
Linear
RMS porm
RoPE
aifoL
ConvMLP
RMS SOOTT
Key
Joint
adaLN,
Linear
RMS Viorm
attention
Linear
adul.NNg
MLP
RMS
RoPE
Xz
aduL.y,
RoPE
adaLI,,
morry
Value
CanvMLP
f
Concalie
Add
G
Gutey
Figure 5: Multi-stream blocks: F, is the video features, F; is the text features, xa is the audio latents, and 0g denotes the global condition.
C Evaluation
C.1 Objective Metries
To comprehensively evaluate the generated audio, we adopt set of objective metrics targeting different aspects: perceptual quality, semantic consistency, temporal alignment, and cross-modal correspondence.
Feature Distribution Alignment: We project both generated and reference audio into the OpenL embedding space Cramer et al. (2019): Evans et al. (2024) and compute the Fréchet Distance (FD) Kilgour et al. (2018): Copet al. (2024) to assess the similarity between their distributional statistics. We chose OpenL3 because it accepts signals of up to 48kHz while VGGish operates at 16kHz, which is more suitable for our 44kHz audio. Following the previous work Evans et al. (2024). we extend the FD to evaluate stereo signals by projecting left and right channels separately and then averaging the results. Moreover, to evaluate whether the generated audio matches the reference in terms of its distribution, we compute the Kullback-Leine between class probability distributions predicted by the PaSST mode(Koutini et al. (2021) and PaNNs model (Kong et al. 2020) as classifiers.
(KL) DivergenccCopet et al. (2024)
Temporal Alignment: To evaluate the synchronization between generated audio and its correspond- ing video, we adopt the DeSyne score predicted by the Synchformer model [ashin et al. (2024). following the practice of Chenn al. (2024a). For each sample, we truncate the video match the duration of the generated audio and compute the DeSync score using Synchformer, which operates with 4.8 8-second context window. Specifically,, we extract both the first and last 4. 8-second segments from each video-audio pair, calculate the DeSync score for each segment, and report the average as final temporal alignment metric. Text-Audio Correspondence: To assess the semantic alignment between generated audio and textual prompts, we utilize the CLAP score Wu* et al. (2023): Chen jtt al. (2022), which measures similarity in a shared audio-text embedding space. Specifically, we report CLAP cap for evaluating alignment with the original video captions and CLAPcoT for alignment
23

---

with our constructed CoT descriptions. As discussed in Section 5.2, the original VGGSound captions are often low quality and yield lower CLAP scores, whereas our CoT annotations provide richer semantic detail and achieve higher alignment. Consequently, we primarily use the CoT--audio alignment metric in our evaluations, except Table where caption-based alignment is also reported.
C.2 Subjective Metrics
Our subjective evaluation framework employs the Mean Opinion Score (MOS) methodology along two critical dimensions comprehensively assess the generated audio:
Audio Quality Assessment (MOS-Q) We evaluate the intrinsic perceptual quality of generated
audio through rigorous assessment protocol where participants instructed to focus four specific aspects:
Clarity: The absence of unwanted artifacts, distortions, or noise
Naturalness: How realistic and non-synthetic the audio sounds
Fidelity: The richness and accuracy of acoustic characteristics
Overall impression: The holistic listening experience
Each listener rates these qualities using standard 5-point Likert scale (I: Poor, 2: Fair, 3: Good, Very Good, 5: Excellent). The final MOS-Q score for each audio sample represents the average rating across all evaluators, providing robust measure of perceived audio quality.
Semantic and Temporal Alignment Assessment (MOS-A) To evaluate the cross-moss moosll coher- ence between generated audio and visual content, assess both semantic relevance and temporal synchronization (we also provide CoT text as the auxiliary information for semantic alignment):
Semantic alignment: How well the audio content matches the objects, actions, and environ- ment depieted in the video
Temporal alignment: How accurately sound events correspond to visual events in time
Participants judge the alignment according to three categories on the same 5-point scale:
Fully aligned (4-5 points): Complete semantic correspondence with precise temporal syn- chronization
Mostly aligned (2.5-3.9 points): Good semantic match with occasional minor temporal misalignments
Partially aligned (1-2.4 points): Noticeable discrepancies in either semantic content or temporal synchronization
Evaluation Protocol
To ensure evaluation reliability and consistency, we implemented the follow-
ing protocol:
All assessments were conducted in controlled in-person sessions with standardized audio equipment
15 raters with normal hearing ability were recruited and briefed on the evaluation criteria
Each rater evaluated random subset of 50 video-audio pairs from our test collection
Samples were presented in randomized order to prevent ordering bias
Reference examples of each quality level were provided before the evaluation sessions
Raters were given sufficient time to carefully evaluate each sample

---

D Additional Quantitative Results
D.1 Details on Video-to-Audio Comparison
For the results in Table we reproduce the results Seeing and Hearing (Xing et al. 2024). V-AURA (Viertola al., 2024), FoleyCrafter (Zhang al. (024), and MMAudio (Cheng al. 2024a) using the official code and pre-trained models. For the other baselines, we use the generated samples provided by the authors, i.c., Frieren, V2A-Mapper, and Movie Gen (Polyak al. 2024). Furthermore, the CLAP cup scores of MMAudio, MovieGen, and ThinkSound are 0.43, 0.44, and 0.49, respectively.
D.2 Impact of Model Size
We compare three model size of ThinkSound: Large (1.3B), Medium (724M), and Small (533M). The Large model achieves the best performance across all metrics as shown in Table 2 These results demonstrate that model capacity significantly enhances audio quality and improves alignment with ground truth distribution. As model size decreases, performance degrades substantially, highlighting the necessity of adequate model capacity for effective audio generation.
Table 9: Impact of model size results.
Size
FDJ
KL $
KLPaNNs +
DeSynek
CLAPcor
Small
43.26
1.64
1.39
0.50
0.43
Medium
37.62
1.56
1.34
0.47
0.44
Large
34.56
1.52
1.32
0.46
0.46
D.3 Performance across different difficulty levels
To better validate the performance of our CoT-Guided generation, also report the results in the
video-to-audio generation of different difficulty levels. We illustrate the construction of different difficulty levels in Section A A.3 The results shown in Table TO] and conclude that (1) As expected, the performance all models decreases as the difficulty level increases, and (2) Our CoT--Guided generation outperforms other baselines across all difficulty levels.
Table 10: Performance across different difficulty levels.
Difficulty
FDJ
KLPasST +
KLPaNNs +
DeSyne
CLAPcor ?
Easy
31.32
1.35
1.16
0.42
0.52
Medium
35.45
1.46
1.31
0.46
0.49
Hard
48.78
1.63
1.40
0.57
0.41
D.4 Performance Comparisons between coarse-grained and fine-grained CoT
To further validate the effectiveness of our fine-grained CoT, we compare the performance of our model with the coarse-grained COT. The results are shown in Table II We can conclude that our fine-grained CoT outperforms the coarse-grained CoT across all metrics.
Table 11: Performance comparisons between coarse-grained and fine- lineggrinned COT.
Granularity
FDJ
KLPaSST |
KLPaNNs +
DeSyncl
CLAPcor
Coarse
42.72
1.58
1.41
0.52
0.34
Fine
34.56
1.52
1.32
0.46
0.46
D.5 Efffectiveness of T5 Encoder for CoT Structure
A key assumption of our work is that the T5 encoder can effectively capture the logical structure within the CoT reasoning steps. We provide both a theoretical rationale and empirical validation for this.
25

---

Theoretical Rationale While T5 is pre-trained on general web text (C4), its Transformer archi- tecture is fundamentally designed to capture long-range dependencies and compositional structure. The reasoning steps in our CoT (e.g., "first do A, then B happens, which causes C"") are expressed through natural language syntax and logical connectives. T5's contextual emheddings are well-suited to capture these structural and causal cues. The goal is not for T5 to "understand" reasoning in human sense, but to produce a rich, structured conditioning signal that the downstream diffusion transformer can effectively leverage.
Empirical Validation To prove that the T5 encoder effectively utilizes the structure of the CoT, we conducted an ablation study comparing our full model against two degraded variants: (1) Tags Only, which provides the model with only extracted keywords (e.g., "car," "rain," "dog bark"), removing all reasoning structure; and (2) Randomized Co'T, which randomly shuffles the sentences within the CoT, preserving all keywords but destroying the logical flow. As shown in Table 12| both ablations lead to significant performance drop. This strongly indicates that the performance gain attributable the stepwise logical structure of the CoT encoded by T5 and not merely the presence of more keywords.
Table 12: Ablation study the importance of CoT structure. We compare full model against variants with only keywords (Tags Only) and shuffled reasoning sentences (Randomized CoT). Results show that the logical structure is critical for performance.
Setting
FD]
KL $ KLPaNNs $ DeSynck
CLAPcor
Randomized CoT
40.52
1.56
1.35
0.51
0.43
Tags Only
39.35
1.60
1.38
0.50
0,42
Ours (Ordered CoT)
34.56
1.52
1.32
0.46
0.46
D.6 Impact of CoT Verbosity on Generation Quality
To mitigate the risk of verbose or hallucinated CoT text, we employ a two-pronged strategy: (1) rigorous quality control during the creation of our AudioCoT training data (see Appendix A.2).
and (2) direct constraints on CoT generation during inference (e.g., S 3 sentences, S 77 tokens). To empirically validate this approach, verbosity. We removed our length constraints to generate "Overly Detailed" CoT prompts and compared them against our proposed "Concise COT." The results in Table 13 demonstrate clear performance degradation with verbose reasoning, confirming that our dual strategy is effective and that concise reasoning is critical for achieving high-quality results.
conducted ablation study quantifying the impact of
Table 13: Impact of CoT verbosity. compare our standard concise CoT against an overly detailed version generated without length constraints. Verbose reasoning leads to a clear performance degradation.
Setting
FDJ
KL + KL +
DeSynel
CLAPPor 1
Overly-detailed CoT
43.56
1.61
1.55
0.54
0.35
Concise CoT (Ours)
34.56
1.52
1.32
0.46
0.46
D.7 Dedicated MILM Reasoning Evaluation
The quality of the CoT reasoning the linchpin of our framework. To substantiate our claims, we conducted a comprehensive evaluation of the generated CoT quality using both expert human annotators and automated LLM-based metrics.
randomly sampled 100 VGGSound test cases. Two expert annotators rated each generated CoT on 0-1 scale across five dimensions: (1) multimodal integration, (2) specificity of audio details, (3) feasibility for audio generation, (4) logical consistency, and (5) brevity/format compliance. The final score is the sum (max 5).
Human Evaluation
26

---

LLM-Based Similarity Evaluation For large-scale analysis, we compared the generated CoTs to ground-truth CoTs using GPT-4-nano. The model was prompted score similarity on 0-5 scale, focusing on reasoning structure, causal/temporal relationships, and object-sound associations.
The results, presented in Table 14 provide strong empirical evidence that our fine-tuned MLLM (ThinkSound) greatly outperforms other powerful MLLMs in generating high-quality struc- tured CoT for video-to-audio reasoning.
Results
Table 14: Evaluation of MLLM-gemne MLLM-generated CoT quality. Our model compared against baselines using human evaluation and LLM-based similarity scoring, demonstrating superior performance in generating high-quality COT.
Model
Human Score (0-5)
LLM CoT Similarity (0-5)
Qwen2.5-VL-77
3.78
3.95
Qwen2-Audio-78
3.82
4.09
ThinkSound (VideoLLaMA2)
4.13
4.31
E
Limitation and Future
While the current MLLMs are capable of strong understanding and reasoning semantic informa- tion, they still have limitations in understanding the precise temporal and spatial information of video. For example, in the case of locating the exact timestamp of the sound event, MILMs often fail to provide accurate results or provide wrong results. Moreover, the current open-source video-audio datasets for audio generation are limited diversity and coverage, which may lack rare or culturally specific sound events. In the future, we will continue to explore more diverse and comprehensive datasets to improve the performance of our model. Furthermore, we will explore more effective methods to improve the temporal and spatial alignment of generated audio.
F Potential Negative Societal Impacts
ThinkSound carries potential risks if misused. Malicious actors could exploit the system to generate fake audio for synthetic media, thereby contributing the spread of misinformation. Moreover, if the training data underrepresents certain cultures or environments, the model may unintentionally amplify biases-for instance, by reinforcing stereotypes or misassociating sounds with particular demographic groups.
F.1 Ethical Considerations
The dataset used in this research is strictly for academic and non-commercial purposes. We implemented several measures to ensure compliance with ethical standards, as follows.
Data Transparency and Anonymization. We only provide ASR transcripts after rigorous text anonymization processes, visual features of video clips, annotations, and links to the original videos, to ensure transparency regarding the data sources and their usage while maintaining anonymity.
Authorization. Any personal data should be used only with express authorization, ensuring lawful and fair processing accordance with applicable laws.
G
Safeguards
We used diverse training dataset covering wide range of acoustic scenes to minimize reinforcing stereotypes or incorrect associations between sounds and specific demographic groups. The model. will be released in stages to better assess its impact and improve safeguards. However, once the model is openly released, we cannot control how others use it. Therefore, we provide clear usage guidelines encourage responsible and help mitigate potential misuse.
27


Reviews:
[{"review_id": "HBRTqmBbHG", "reviewer": "Reviewer_Ajam", "summary": "This paper proposes ThinkSound, which is a multimodal large language model based on chain-of-thought reasoning for text,video-guided sound generation.\nThrough a 3-step pipeline, it can generate and edit customized sounds by borrowing knowledge from MLLM.\nMoreover, it proposes AudioCoT benchmark, carefully collecting structured reasoning annotations.\nIt shows remarkable performance compared to the current SOTA (MMAudio) in both in-domain and out-of-distribution experiments.", "questions": "- The paper reports CLAP<sub>CoT</sub> scores on MovieGen Audio Bench (Table 2), suggesting that CoT reasoning captions were used. However, it is unclear whether these CoT captions were newly generated for MovieGen samples. Could the authors clarify whether reasoning-based captions were generated for MovieGen, and if so, what CLAP scores are for original captions?\n\n- While the use of CoT reasoning appears to improve performance across metrics such as CLAP<sub>CoT</sub> and MOS, it remains unclear whether verbose or overly detailed reasoning could lead to redundant or hallucinated audio content. I suggest the authors discuss how they handle potential over-generation and whether any such cases were observed during evaluation.\n\n- The benchmark used in Table 3 for object-focused audio generation is not clearly described. Is it based on a specific split of the AudioCoT dataset? If so, how does it differ from the test set used in Table 4? Please clarify the evaluation setup and explain any factors that may account for the differences in results.\n\n\n- Are the parameter and inference time numbers in Table 1 inclusive of the VideoLLaMA2 reasoning module?\n\n\n- Why does ThinkSound achieve 3× faster inference time than MMAudio, despite having slightly more parameters?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "S1. It shows remarkable performance for TV2A.\n\nS2. The AudioCoT benchmark would be valuable in this community.\n\nS3. The authors provide comprehensive experiments to show the robustness and effectiveness of CoT reasoning in the TV2A task.\n\nW1. Given the structural similarity to MMAudio, it needs to clarify the architectural advantages of the proposed method.\n\nW2. Some parts are unclear in the experiments.\n\nW3. More deeply failure case analysis is needed.", "was_revised": true, "final_justification": "The authors have addressed all my concerns, so I will keep my rating for the acceptance.", "rebuttal": "## Response to Reviewer Ajam\n\nWe sincerely thank the Reviewer for the positive feedback and insightful questions. We are encouraged by your recognition of our work's strong performance and the value of our benchmark, and find your questions very helpful for clarifying our contributions. Below, we hope the point-to-point responses could completely address your concerns. We are fully committed to open-sourcing all code, models, and data upon acceptance.\n\n---\n\n> **1. Architectural Advantages Over MMAudio**\n\nThank you for this important question. While both MMAudio and ThinkSound leverage multimodal integration, our framework introduces several fundamental distinctions:\n\n*   **Explicit, Interpretable, and Controllable Pipeline:** ThinkSound decomposes generation into three intuitive stages: (1) foundational foley synthesis, (2) interactive object-centric refinement, and (3) instruction-guided editing. As detailed in **Sec. 4.4** and illustrated in **Fig. 1**, each stage is driven by structured CoT reasoning, making the process transparent and controllable. This is a stark contrast to MMAudio’s single-stage, black-box generation, offering users more control over the creative process. Furthermore, this design supports a richer set of conditioning, including flexible ROI inputs, audio context, global semantics, and structured CoT context (ablated in Table 5).\n\n*   **Highly Efficient Audio Latent & Faster Inference:** At the core of our efficiency lies a fully convolutional VAE achieving **32× audio compression** and enabling **direct waveform synthesis**. This design completely **obviates the need for an external vocoder**, a significant bottleneck in models like MMAudio (~0.45s overhead). This architectural choice is the primary reason ThinkSound achieves **3× faster inference** despite having a comparable parameter count, as detailed in **our response to your Question 7**.\n\n*   **Adaptive Multimodal Fusion for Enhanced Synchronization:** Our model introduces an adaptive fusion module (**Sec. 4.3**, ablated in **Table 6**) that intelligently integrates upsampled video features with audio latents via a gated mechanism. This allows the model to dynamically emphasize salient visual cues and suppress irrelevant information, leading to more precise audio-visual synchronization compared to the audio-only transformer blocks in MMAudio.\n\nWe will add a dedicated paragraph in the revised manuscript to further elaborate on these crucial architectural distinctions.\n\n---\n\n> **2. On Deeper Failure Case Analysis**\n\nWe agree that a detailed failure analysis is essential. In the revised appendix, we will detail two primary failure modes with qualitative examples:\n\n*   **Limited Realism in Niche Domains:** The model struggles with highly specialized or rare sounds (e.g., specific industrial machinery and game sounds) where training data is scarce. The generated audio can sound generic or miss key acoustic characteristics, pointing to a need for more diverse training data.\n\n*   **Imprecise Spatialization for Dynamic Objects:** While our model generates stereo audio, since it is not specifically trained to constrain spatial information, it sometimes fails to accurately spatialize fast-moving or occluded objects. This is a limitation of the MLLM's ability to conduct fine-grained spatial reasoning, leading to less immersive results in complex scenes.\n\n---\n\n> **3. CoT Caption for MovieGen and $\\text{CLAP}_{\\text{cap}}$ Scores**\n\nThank you for this important question regarding our evaluation protocol. For the MovieGen Audio Bench evaluation (**Table 2**), we generated new CoT texts for each sample using our fine-tuned MLLM. The reported $\\text{CLAP}_{\\text{CoT}}$ scores are based on these generated texts.\n\n**For reference, $\\text{CLAP}\\_{\\text{cap}}$ scores are (see Supplementary D.1, line 156): MMAudio 0.43, MovieGen 0.44, ThinkSound 0.49**. This shows that ThinkSound outperforms other methods with **both original and CoT-based captions**, highlighting the strength of our audio foundation model. We will clarify this in Table 2 to make it self-contained.\n\n---\n\n> **4. Verbosity/Redundancy in CoT Reasoning**\n\nThis is a crucial point. We address this risk of verbose or hallucinated CoT text through a **two-pronged strategy**: (1) rigorous, multi-stage quality control during the creation of our AudioCoT training data, and (2) direct constraints on CoT generation during inference.\n\n**First, we ensure our MLLM is trained on high-quality, concise, and factually grounded reasoning chains** through a comprehensive quality control pipeline (**Supp. A.2**), which includes automated CLAP-based filtering, LLM-based validation, and rigorous human verification, specifically:\n*   **Automated Filtering:** We use CLAP scores to automatically identify and discard audio-CoT pairs with poor semantic alignment, ensuring the MLLM learns from relevant examples.\n*   **LLM-based Validation:** We leverage GPT-4.1-nano as a validation agent to check the semantic plausibility and logical consistency of generated editing instructions, preventing illogical or \"hallucinated\" steps from entering our dataset.\n    \n*   **Human Verification:** At each stage, a 5% random sample of the data is manually reviewed by our team. If the rejection rate exceeds a 5% threshold, we recalibrate our automated filters and reprocess the entire batch.\n\n**Second, during inference, we apply direct constraints** to the CoT generation process. Using prompts that limit reasoning length (≤ 3 sentences, ≤ 77 tokens)and focus the output strictly on audible events and their temporal relations.\n\n**Finally, to empirically validate this entire approach**, we conducted a **new ablation study** quantifying the impact of verbosity. We removed our length constraints to generate \"Overly Detailed\" CoT prompts and compared them against our proposed \"Concise CoT\".The results demonstrate a clear performance degradation with verbose reasoning:\n\n| Setting | FD↓ | **$\\text{KL}_{\\text{PaSST}}$**↓ | **$\\text{KL}_{\\text{PaNNs}}$**↓ | DeSync↓ | **$\\text{CLAP}_{\\text{CoT}}$**↑ |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| Overly-detailed CoT | 43.56 | 1.61 | 1.55 | 0.54 | 0.35 |\n| **Concise CoT (Ours)** | **34.56** | **1.52** | **1.32** | **0.46** | **0.46** |\n\nThese results provide strong empirical evidence that **our dual strategy—proactive data curation and reactive inference constraints—is effective**. The concise reasoning fostered by our approach is critical for achieving both high objective quality and superior semantic alignment. We will add this comprehensive explanation and the new ablation study to the revised manuscript.\n\n---\n\n> **5. Evaluation Splits and Object-Focused Benchmark**\n\nThank you for requesting clarification. **All evaluation benchmarks are derived from the VGGSound test set but are carefully curated for different purposes**:\n\n*   **Stage 1 (Table 1 - Foundational Generation):** ~15,000 general video-audio pairs from the standard VGGSound test set.\n*   **Stage 2 (Table 3 - Object-Focused Generation):** ~2,000 samples with persistent, clearly identifiable sound-emitting objects for ROI-based evaluation.\n*   **Stage 3 (Table 4 - Audio Editing):** ~2,000 samples constructed by combining clips from distinct videos to test instruction-based editing (e.g., adding sound from one context to another). This set is distinct from the Stage 2 set.\n\nWe will clarify this in the main paper, point to detailed procedures in **Supp. A.3**, and release all splits with our data.\n\n---\n\n> **6. Model Parameter and Inference Time Reporting**\n\nThank you for this crucial question. For fair comparison, we follow the standard protocol in prior works [1, 2, 3]. The parameter count in **Table 1 (1.30B)** refers **exclusively to the trainable parameters of our audio foundation model**, excluding all frozen components (MLLM, VAE, encoders). Correspondingly, the reported inference time **(1.07s)** measures **only the core audio synthesis pipeline** (flow matching + VAE decoding). All conditioning features (including CoT) are pre-computed offline for all methods, ensuring a direct and fair comparison of generation speed. We will make this protocol explicit in the revision.\n\n---\n\n> **7. Why 3× Faster Inference With More Parameters?**\n\nThis is an excellent observation that highlights a key strength of our architectural design. The speedup is primarily due to ThinkSound’s **efficient VAE-based audio representation (32× compression) and vocoder-free architecture**, yielding much shorter latent sequences and removing the need for a separate vocoder (which adds ~0.45s/10s audio in MMAudio). Our end-to-end pipeline thus achieves **significantly lower inference time despite comparable parameter count**. The timing breakdown below isolates this efficiency gain to the core generative steps, clearly demonstrating how our design choices lead to a substantial speedup.\n\n| Method | Flow Matching Sampling (s) | VAE/Vocoder (s) | **Total Timed (s)** |\n| :--- | :---: | :---: | :---: |\n| MMAudio | ~2.56 | ~0.45 | ~3.01 |\n| ThinkSound | ~1.06 | ~0.01 | **~1.07** |\n\n---\n### [Reference]\n[1] Luo, Simian, et al. \"Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models.\" Advances in Neural Information Processing Systems 36 (2023): 48855-48876.\n\n[2] Wang, Yongqi, et al. \"Frieren: Efficient video-to-audio generation with rectified flow matching.\" arXiv e-prints (2024): arXiv-2406.\n\n[3] Cheng, Ho Kei, et al. \"Taming multimodal joint training for high-quality video-to-audio synthesis.\" arXiv e-prints (2024): arXiv-2412."}, {"review_id": "SoaybeMY4j", "reviewer": "Reviewer_2MyN", "summary": "This paper introduces ​ThinkSound, a novel framework for video-to-audio (V2A) generation that leverages ​Chain-of-Thought (CoT) reasoning​ to address the limitations of existing end-to-end approaches in capturing nuanced audio-visual relationships. Key innovations include ​AudioCoT—the first dataset with structured audio reasoning annotations —and a ​unified audio foundation model​ using flow matching with gated multimodal fusion.", "questions": "The reported inference speed is impressive, particularly its advantage over MMAudio. However, this result seems somewhat counter-intuitive, as the ThinkSound pipeline involves two distinct models: an MLLM for CoT generation and the flow matching model for audio synthesis. Could the authors provide a more detailed breakdown of this latency measurement? Specifically:\n1. How was the total inference time calculated? Does it include the MLLM's generation time?\n2. What are the primary factors contributing to this high efficiency? A more in-depth analysis explaining why the proposed framework is significantly faster than a diffusion-based model like MMAudio, despite its two-component architecture, would be highly beneficial for the reader.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The motivation is strong and timely. Leveraging the reasoning capabilities of MLLMs to enhance downstream generative models is a critical and underexplored direction in the V2A field. The authors convincingly demonstrate that their approach achieves state-of-the-art (SOTA) performance, validating the effectiveness of their core idea.\n2. The unification of audio generation and editing within a single framework is a significant strength. By exploring two intuitive editing modalities—Region-of-Interest (ROI) based refinement and natural language instruction-based editing—the paper naturally showcases the practical benefits and versatility of incorporating an MLLM.\n3. The introduction of the AudioCoT dataset is a valuable contribution in its own right. The detailed construction process and the resulting dataset will likely serve as a crucial resource for future research in reasoning-based audio generation and editing.\n\nWeaknesses:\n1. The paper lacks essential mathematical formulations. For instance, including the objective functions for fine-tuning the MLLM and for training the flow matching model would significantly improve the paper's clarity and reproducibility.\n2. A more significant weakness is the lack of a dedicated performance evaluation for the MLLM's reasoning capabilities in the main text. The MLLM's ability to generate high-quality CoT is central to the entire framework's success. Therefore, its effectiveness and reliability should be explicitly demonstrated and analyzed, rather than being treated as a given. A standalone analysis of the MLLM's performance would substantially bolster the paper's core claims.", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 2MyN\n\nWe sincerely thank the reviewer for your positive feedback and recognizing our work as `the motivation is strong and timely`, `a unified framework for audio generation and editing is a significant strength`, and `AudioCoT dataset is a valuable contribution`. Your suggestions for improving the paper's clarity and rigor are particularly valuable. Below, we hope the following point-to-point response could completely resolve your concerns. We are fully committed to open-sourcing all code, data, and models upon acceptance.\n\n---\n\n> **1. On Mathematical Formulation**\n\nThank you for this suggestion. We agree that providing explicit mathematical formulations is crucial for clarity and reproducibility. In the revised manuscript, we will add a **new \"Preliminaries\" section**. This section will formally define the problems of MLLM fine-tuning and conditional flow matching, and will present their respective loss functions and training objectives in clear mathematical terms.\n\n---\n\n> **2. Dedicated MLLM Reasoning Evaluation**\n\nWe thank the reviewer for raising this valuable suggestion. The quality of the CoT reasoning is indeed the linchpin of our framework. To further substantiate our claims and in response to your valuable suggestion, we have conducted comprehensive evaluations using both expert human annotators and automated (LLM-based) metrics:\n\n*   **Human Evaluation:**  \n We randomly sampled 100 VGGSound test set cases, covering diverse audio-visual scenarios. Two expert annotators independently rated each generated CoT across five dimensions crucial to audio-visual reasoning: (1) multimodal integration, (2) specificity of audio details, (3) feasibility for audio generation, (4) logical consistency, and (5) brevity and format compliance (≤3 sentences, ≤77 tokens). Each dimension is rated from 0 to 1 (total max 5). Disagreements >1 point were resolved by a third annotator.\n    \n*   **LLM-Based Similarity Evaluation:**  \n For large-scale, objective analysis, we compared the generated CoTs to ground-truth CoTs (automatically constructed as described in Section 3.2) using GPT-4-nano, which was prompted to focus specifically on reasoning structure, causal/temporal relationships, and object-sound associations. The similarity score is on a 0–5 scale:\n    \n    * 5 = Nearly identical in structure, content, and reasoning\n        \n    * 4 = Mostly similar, minor omissions in details or stepwise logic\n        \n    * 3 = Partially similar, significant reasoning elements missing\n        \n    * 2 = Little overlap in reasoning structure/content\n        \n    * 1 = Barely any overlap\n        \n    * 0 = Completely unrelated\n        \n\n**Results**:\n\n| Model | Human Score (0-5) | LLM CoT Similarity (0-5) |\n| :--- | :---: | :---: |\n| Qwen2.5-VL-7B | 3.78 | 3.95 |\n| Qwen2-Audio-7B | 3.82 | 4.09 |\n| **ThinkSound (VideoLLaMA2)** | **4.13** | **4.31** |\n\nThese results provide **strong empirical evidence** that our fine-tuned MLLM greatly outperforms other powerful MLLMs in generating high-quality, structured CoT for V2A reasoning. This new section will be prominently featured in our revised paper.\n\n---\n\n> **3. Inference Time Breakdown and Efficiency**\n\nWe appreciate your interest in the efficiency of our framework and agree that reporting the breakdown is important for transparency and fair comparison.\n\n*   **Timing Protocol:** The reported inference time in Table 1 measures **only the online audio synthesis stage**, which is the standard practice for fair comparison in V2A generation [1, 2]. The MLLM-based CoT generation is treated as a one-time, offline pre-processing step for the entire test set, akin to feature extraction. This protocol is applied consistently across all methods. The total time for our pipeline is the sum of offline and online steps, but only the online portion is used for latency comparison. We will clarify it in the revised version.\n    \n*   **Why ThinkSound is Faster:** The primary reason for ThinkSound’s superior speed is **its highly compressed VAE-based audio latents and vocoder-free architecture**, enabling much shorter sequence modeling and eliminating the heavy computational overhead of flow-based sampling and external vocoder reconstruction used in models like MMAudio.\nThe detailed breakdown below clearly shows that our fast flow matching sampling and near-instant VAE decoding are the sources of our efficiency:\n\n| Method | MLLM (s) | Feature Extraction (s) | VAE/Vocoder (s) | Flow Matching Sampling (s) |\n| :--- | :---: | :---: | :---: | :---: |\n| MMAudio | – | ~1.01 | ~0.45 | ~2.56 |\n| ThinkSound | ~2.44 | ~1.20 | ~0.01 | ~1.06 |\n\nThis analysis shows our pipeline separates expensive reasoning into an offline step, enabling a fast online synthesis process. We will add this breakdown to the appendix.\n\n---\n### [Reference]\n\n[1] Luo, Simian, et al. \"Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models.\" Advances in Neural Information Processing Systems 36 (2023): 48855-48876.\n\n[2] Wang, Yongqi, et al. \"Frieren: Efficient video-to-audio generation with rectified flow matching.\" arXiv e-prints (2024): arXiv-2406."}, {"review_id": "nSRvrm5amk", "reviewer": "Reviewer_2uba", "summary": "The paper introduces ThinkSound, a three-stage framework that brings chain-of-thought (CoT) reasoning into video-to-audio generation. A fine-tuned VideoLLaMA 2 model first produces explicit, step-by-step CoT instructions; these then guide a single, multimodal flow-matching audio foundation model that can (1) create an initial “foley” soundtrack, (2) refine sounds for user-clicked regions, and (3) perform text-guided edits such as adding or removing events. To train and evaluate this pipeline, the authors curate AudioCoT, a large dataset pairing videos, audio/text clips, and structured CoT annotations. Experiments on VGGSound show clear gains over recent V2A baselines (e.g., MMAudio) in fidelity, semantic alignment, and human MOS, while an out-of-distribution test on the MovieGen Audio Bench confirms strong generalisation. Ablations reveal large drops when CoT reasoning or the gated video-audio fusion is removed, underscoring the importance of both components.", "questions": "My questions are listed in Strengths And Weaknesses. Specifically:\n\n- I may have missed where the editing training data is coming from. Can the authors highlight that part?\n- Have the authors tried verifying how clean or noisy the generated data is? Any data verification step?\n- Stage 2: Interactive Object-Focused Audio Generation is supposed to be click-based and done by humans -- how did the authors manage to do an objective evaluation? What did I miss?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths:\n\n- The paper is nicely written and well presented.\n- The paper tries to solve a relevant problem of video-audio generation, which is hot now.\n- The paper is well structured and thought out.\n- The results look promising.\n\n### Weaknesses:\n\n- The title is a bit misleading. The title made me think that somehow the authors propose CoT in the audio generation process (also the paper is video-to-audio and not just audio, which should also be highlighted) -- which ideally means that the model thinks step by step to generate audio. However, the paper generates thinking in text, which is input to a DiT-like model by conditioning -- so ideally, the generation process is not what is thinking. The step-by-step in the process also comes through additional steps in generation (which are kind of optional) -- but thinking steps are not happening while generating audio -- which is what CoT ideally is.\n- ⁠The Cost is just conditioned to the DiT with a T5. I am doubtful if any model (DiT in this paper) would understand thought processes through T5 embeddings. T5 embeddings, in prior literature, is good at passing information about objects, attributes, etc -- the fact that it can pass knowledge about thought processes is something I am not being able to comprehend (maybe some ablation is needed to justify this) -- and it is important as the paper is based on this.\n- ⁠Figure 3 left is a bit misleading in the sense that the DiT should not be on top of the Decoder -- this ideally indicates a different form of architecture -- the DiT gets the output of MLLM in a cascaded fashion, and it's not joint.\n- About Section 4.4:\n  - Object-focused audio Generation is not new. See [this paper](https://arxiv.org/abs/2506.04214) for example. \n  - I may have missed where the editing training data is coming from. Can the authors highlight that part?\n- About Section 3.2\n  - Have the authors tried verifying how clean or noisy the generated data is? Any data verification step?", "was_revised": true, "final_justification": "The authors have made efforts. The method has some flaws and not very robust. I am borderline.", "rebuttal": "## Response to Reviewer 2uba\n\nWe thank the reviewer for the constructive feedback, as well as for recognizing our work as `nicely written and well presented`, `well structured and thought out`, and with `promising results`. We are fully committed to open-sourcing all code, data, and benchmarks upon acceptance. We hope the following point-by-point responses will completely address all remaining concerns.\n\n---\n\n> **1. On the Title and the Nature of \"Thinking\"**\n\nThank you for the opportunity to clarify our definition and the specific role of \"Chain-of-Thought\" (CoT) in our framework, particularly regarding the title.\n\n*   **Our Contribution:** The core innovation of ThinkSound is **not** to embed a step-by-step generative process within the audio synthesis model itself. Rather, as described in the **Abstract (L10-12)**, our contribution is the introduction of an **explicit, interpretable, and controllable reasoning stage**—powered by an MLLM—that ***precedes and guides*** the audio foundation model. We expect this would be similar to the text-form reasoning LLM, which begins with a pure thinking process and then generates the whole final response. This MLLM-generated textual CoT serves as a detailed blueprint, decomposing the complex video-to-audio task into manageable sub-problems, much like a planning module.\n\n*   **Motivation and Analogy:** Our approach is deliberately designed to mimic the workflow of professional sound designers. They first analyze a visual scene, reason about the sequence of events and acoustic properties, and *only then* proceed to synthesize and layer sounds. Existing end-to-end models bypass this critical reasoning step, treating generation as an opaque transformation. By making the reasoning explicit, ThinkSound gains significant advantages in interpretability, controllability, and fidelity, as our results demonstrate.\n\n*   **Consistency with an Emerging Paradigm:** This use of CoT—where a large model generates a textual reasoning plan to steer a specialized downstream model—is a powerful and emerging paradigm in multimodal AI, as acknowledged by Reviewer Ajam and 2MyN. It is methodologically consistent with recent works such as **ImageGen-CoT [1], DeepSound-V1 [2], and MINT [3]**, which also leverage LLM-generated text-based reasoning to guide complex generative tasks.\n\nWe will revise the introduction and methodology sections to make this architectural choice and our motivation crystal clear.\n\n---\n\n> **2. On the Use of T5 Embeddings for Encoding Reasoning**\n\nThank you for this insightful question. We address it with both a theoretical rationale and targeted empirical evidence.\n\n*   **Theoretical Rationale:** While T5 is pre-trained on general web text (C4), a massive English-language dataset containing approximately 750GB of clean web text, its Transformer architecture with pre-training paradigm is fundamentally designed to capture long-range dependencies and compositional structure within sequential data. The reasoning steps in our CoT (like \"first do A, then B happens, which causes C\") are expressed through natural language syntax and logical connectives. **T5's contextual embeddings are suitable to capture these structural and causal cues.** Furthermore, the goal is not for T5 to \"understand\" reasoning in a human sense, but to **produce a rich, structured conditioning signal** that the downstream diffusion transformer can effectively leverage.\n\n*   **Empirical Validation:** To prove that the T5 encoder effectively utilizes the *structure* of the CoT, we conducted an ablation study to isolate the contribution of logical structure versus mere token content. We compared our full model against two degraded variants:\n    1.  `Tags Only`: Provided the model with only extracted keywords (e.g., \"car,\" \"rain,\" \"dog bark\"), removing all reasoning structure.\n    2.  `Randomized CoT`: Randomly shuffled the sentences within the CoT, preserving all keywords and sentence content but destroying the logical flow.\n\nThe results unequivocally show that the ordered, logical structure is critical for performance:\n\n| **Setting**             | FD↓ | **$\\text{KL}_{\\text{PaSST}}$**↓ | **$\\text{KL}_{\\text{PaNNs}}$**↓ | DeSync↓ | **$\\text{CLAP}_{\\text{CoT}}$**↑ |\n| ----------------------- | ---------------- | ---------------- | ---------------- | ---------------- | ---------------- |\n| Randomized CoT          | 40.52            | 1.56             | 1.35             | 0.51             | 0.43             |\n| Tags Only               | 39.35            | 1.60             | 1.38             | 0.50             | 0.42             |\n| **Ours (Ordered CoT)**  | **34.56**        | **1.52**         | **1.32**         | **0.46**         | **0.46**         |\n\nAs shown, both ablations lead to a significant performance drop across key metrics. This strongly indicates that the performance gain is attributable to the **stepwise logical structure of the CoT—as encoded by T5—and not merely the presence of more keywords or longer context.**\n\n---\n\n> **3. On the Architectural Clarity of Figure 3**\n\nWe thank the reviewer for this feedback. The reviewer's understanding of our architecture as a **cascaded pipeline** is perfectly correct, as we describe in the manuscript (e.g., Sec. 4.1, L167-168). We agree that the diagram could better reflect this flow, and we will revise Figure 3 in the final version to be more explicit and self-contained.\n\n---\n\n> **4. On the Novelty of Object-Focused Generation**\n\nThank you for bringing this concurrent work (`arXiv:2406.04214`) to our attention. As this paper appeared on arXiv after the NeurIPS submission deadline, we were unaware of it. We will gladly cite and discuss it in our related work section. However, our work remains clearly distinct in several critical aspects:\n\n*   **Input Modality:** Our method operates on **videos**, not static images. This requires complex spatio-temporal reasoning about events and motion that is fundamentally absent in image-based generation.\n*   **Framework Scope:** Our object-focused stage is one component of a comprehensive, three-stage interactive pipeline that also includes foundational foley generation and instruction-based editing, all unified under a single model architecture.\n*   **Core Method:** Our primary novelty is the **explicit CoT reasoning** that guides the entire generation process, a fundamentally different mechanism from the cited work.\n\n---\n\n> **5. On Data and Evaluation Details**\n\nWe apologize for any lack of clarity and provide the requested details below, with pointers to the supplementary.\n\n*   **(a) Editing Data Source:** The training data for audio editing was programmatically generated from our AudioCoT dataset, as detailed in **Supplementary A.2 (Stage 3)**. Specifically, we used a powerful LLM (GPT-4) to semantically select and pair source/target audio segments from our dataset. The LLM then generated a corresponding natural language instruction (e.g., *\"Add the sound of rain to the existing birdsong\"*). This automated procedure creates high-quality and contextually plausible `(instruction, input_audio, output_audio)` triplets, which are essential for training a robust editing model.\n\n*   **(b) Data Quality Verification:** This is a crucial point. We implemented a rigorous, multi-step quality control pipeline for AudioCoT, detailed in **Supplementary A.2 (Quality Control)**. This process includes:\n     1. **Automated Filtering:** We use CLAP scores to automatically identify and discard audio-CoT pairs with poor semantic alignment, ensuring the MLLM learns from relevant examples.\n     2. **LLM-based Validation:** We leverage GPT-4.1-nano as a validation agent to check the semantic plausibility and logical consistency of generated editing instructions, preventing illogical or \"hallucinated\" steps from entering our dataset.\n    3. **Human Verification:** At each stage, a 5% random sample of the data is manually reviewed by our team. If the rejection rate exceeds a 5% threshold, we recalibrate our automated filters and reprocess the entire batch.\n    \n    This process ensures the high quality of our dataset. Furthermore, as shown in our **response to Reviewer 2MyN (Q2)**, we also objectively and subjectively evaluated the CoT data generated by our fine-tuned MLLM, confirming it is high-fidelity and suitable for downstream conditioning.\n\n*   **(c) Interactive Stage Evaluation:** We thank the reviewer for this feedback. For our quantitative experiments (Table 3), we adopted a standard and reproducible methodology: **simulating user interaction**. As detailed in **Supplementary A.3**, we used ground-truth object bounding boxes from the videos to serve as automated \"user clicks.\" This allowed us to generate audio for a targeted region and then objectively compare it against the corresponding ground-truth audio for that same region using metrics like FD and KL. This approach provides a repeatable and fair benchmark for the interactive generation capability.\n\n---\n\n### **References**\n\n[1] Liao, J., et al. \"Imagegen-cot: Enhancing text-to-image in-context learning with chain-of-thought reasoning.\" arXiv preprint arXiv:2403.19312 (2024).\n\n[2] Liang, Y., et al. \"DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videos.\" arXiv preprint arXiv:2403.22208 (2024).\n\n[3] Wang, Y., et al. \"Mint: Multi-modal chain of thought in unified generative models for enhanced image generation.\" arXiv preprint arXiv:2403.01298 (2024)."}, {"review_id": "p6ZuPJXCmz", "reviewer": "Reviewer_MPd8", "summary": "This paper proposes ThinkSound, a novel framework for interactive audio generation and editing for videos wit CoT reasoning. The methods includes three stages, with foundational foley generation, interactive object-centric refinement, and targeted editing guided by instructions. Besides, a dataset with structural reasoning annotations AudioCoT is proposed. Experiments show the capabilities in V2A generation in audio metrics, CoT metrics, and Movie Gen Audio benchmark.", "questions": "1. L155-160, how is instruction-based audio constructed? How to get paired audio 1 and audio 2? Is it a random process or meticulous one?\n\n2. Why use CoT in audio generation? Better to provide qualitative comparison of w/wo CoT process (for example, in Fig.2, what will the integrated analysis be if wo CoT)\n\n3. Why adopt such an architecture of fig. 3 right part? Can the design of the MM-DiT be validated effectively? (e.g. the ablation study of adaptive fusion)", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n\n1.The paper proposes a novel three-stage interactive framework for V2A for soundscapes through CoT reasoning.\n\n2.A large-scale multimodal dataset AudioCoT with detailed CoT annotation for visual, textual, and audio description is proposed.\n\n3.The experiments validate the proposed method’s performance in audio generation.\n\n\nWeakness (details see “Questions”):\n\n1.The clarity of dataset construction and training architecture needs to improve.\n\n2.The effectiveness and necessity of incorporating CoT reasoning need to be validated qualitatively\n\n3.The design module of MM-DiT architecture needs more experiments to validate.", "was_revised": true, "final_justification": "After reviewing the detailed response from the authors, I find that my initial concerns have been addressed. However, I still believe that the qualitative comparison between the method with and without the CoT process, which is not included in the current submission and cannot be incorporated in the rebuttal, needs further validation to demonstrate the full effectiveness of the method.\n\nAdditionally, I share the concerns raised by Reviewer 2uba regarding the robustness of the method. There appear to be certain limitations and flaws that may affect its generalizability or practical application. Given these issues, I am still on the borderline about my recommendation.", "rebuttal": "## Response to Reviewer MPd8\n\nWe thank you for the reviewer's constructive feedback and for `recognizing the novelty of our three-stage framework` and `the contribution of the AudioCoT dataset`. We hope the following point-by-point response will fully address your concerns. We are committed to open-sourcing all code, data, and models upon acceptance.\n\n---\n\n> **1. On the Clarity of the Data Construction Pipeline for Audio Editing**\n\nThank you for this critical question. We wish to emphasize that our editing dataset is constructed through a **meticulous, multi-stage pipeline, not a random process**. This systematic approach, detailed in **Supplementary A.2**, is designed to produce high-quality, semantically coherent, and contextually plausible training data.\n\nHere is a step-by-step breakdown of our process for creating each `(instruction, input_audio, target_audio)` triplet:\n\n1.  **Semantic Pairing and Filtering:** We begin by programmatically identifying audio components for editing (e.g., for an addition or removal task). To ensure the resulting instruction is unambiguous, we perform semantic filtering based on audio event tags. For example, we exclude pairs with overlapping sounds (e.g., avoiding a \"remove engine\" instruction in a scene also tagged with \"passing car\"). We then score potential pairs for contextual plausibility, ensuring that combined sounds are logical within the scene's context.\n\n2.  **Instruction-CoT Generation:** Once a valid pair of audio components is identified, we use GPT-4 to generate a natural language instruction and its corresponding CoT. This instruction is based on the desired editing operation and is grounded in the video's content by leveraging the original CoT from Stage 1.\n\n3.  **Paired Audio Synthesis:** We then programmatically synthesize the input_audio and arget_audio. For an \"addition\" task, we might combine two clips (birdsong + rain) to create the target_audio, while the original birdsong clip serves as the input_audio. This creates a perfectly aligned training pair for the editing model.\n\n4.  **Rigorous Human Verification:** To guarantee quality, we implement a **human-in-the-loop verification protocol**. A random 5% sample of the generated data is manually reviewed at each stage. If the human rejection rate exceeds a **5% threshold**, we recalibrate our automated filters and re-process the entire batch. This iterative feedback loop ensures the perceptual quality and semantic integrity of our final training data.\n\nWe will integrate this more detailed description directly into the main body of our paper (Section 3.2) to fully articulate the rigor of our data generation methodology.\n\n---\n\n> **2. On the Qualitative Necessity and Impact of CoT Reasoning**\n\nThank you for your feedback. While our quantitative results in Tables 1, 3, 4, and 5 consistently show that CoT provides significant gains, qualitative evidence is helpful to understanding *why*.\n\nCoT enables our model to move beyond generating a generic \"sound blob\" and instead synthesize a **structured soundscape with correct temporal and causal relationships**. For example, given a video of a person walking to a door and knocking:\n\n*   **Without CoT**, a model might generate a temporally confused mixture of \"footsteps\" and \"knocking\" sounds, potentially out of sync or overlapping incorrectly.\n*   **With CoT**, the MLLM generates a plan like: \"_First, there is the sound of footsteps on a wooden floor, which grow louder. Then, after the footsteps stop, there is a sharp knocking sound on the door._\" This explicit plan guides the audio model to generate the events in the correct sequence and with the correct acoustic properties.\n\nWe fully agree that a visual comparison is the best way to show this. **We will add a new qualitative comparison figure to the appendix.** Similar to Figure 4, it will show side-by-side spectrograms of audio generated from the same video clip, with and without CoT, directly illustrating how CoT enables superior temporal structure and event accuracy.\n\n---\n\n> **3. On the Design and Validation of the MM-DiT Architecture**\n\nThank you for your suggestion. Our enhanced MM-DiT is specifically designed to effectively process and fuse the complex, multimodal streams inherent to our task. We validate its design with both existing ablations from the paper and a new targeted experiment.\n\n**Validation of Key Components:** We provide three key pieces of empirical evidence:\n\n1.  **Ablation on Gated Fusion (Existing):** As the reviewer noted, we investigated the importance of our adaptive fusion module. **Table 6 in our paper already provides this ablation.** It shows that our gated fusion mechanism greatly outperforms simpler linear fusion or using audio features alone, confirming the value of this specific design choice.\n    \n2.  **Ablation on Dual Text-Encoder Strategy (Existing):** We validated our choice to use two complementary text encoders. As shown in **Table 5 of our paper**, using both CLIP-encoded captions (for global context) and T5-encoded CoT (for CoT texts) outperforms using either one alone.This result confirms that both conditioning signals are necessary and complementary for achieving the best performance.\n    \n3.  **Ablation on the Full MM-DiT Architecture (New):** To validate the overall multimodal architecture against a simpler baseline, we conducted a new experiment comparing our full MM-DiT to a standard DiT model conditioned only on text (CoT) and global CLIP embeddings, without the multi-stream transformer blocks.\n    \n\n| **Method** | FD↓ | **$\\text{KL}_{\\text{PaSST}}$**↓ | **$\\text{KL}_{\\text{PaNNs}}$**↓ | DeSync↓ | **$\\text{CLAP}_{\\text{CoT}}$**↑ |\n| --- | --- | --- | --- | --- | --- |\n| DiT | 45.20 | 1.64 | 1.45 | 0.56 | 0.39 |\n| MM-DiT | **34.56** | **1.52** | **1.32** | **0.46** | **0.46** |\n\nThe results show a dramatic improvement across all metrics, which confirms that the multi-stream design is crucial for multimodal audio generation."}]

Author rebuttal:

Meta-review: **Scientific claims:**  
The paper introduces a reasoning-driven framework for video-to-audio (V2A) generation based on Chain-of-Thought (CoT) reasoning from an MLLM. It proposes object-centric refinement and instruction-based editing. The authors also contribute a dataset  (AudioCoT dataset) of structured reasoning annotations. Experiments on VGGSound and MovieGen show strong results. 


**Weaknesses:**   Concerns about robustness and qualitative evaluation depth.  

**Decision:** Accept (poster). Authors adequately addressed most concerns; reviewers agreed, as evidenced by raised scores confirming resolution.

**During the discussion**, The Authors added key ablations to clarify concerns: the T5 encoder uses CoT order, not just keywords (Ajam, 2uba); human + automated MLLM reasoning evaluations show clear gains (2MyN); and MM-DiT fusion ablation confirms its role (MPd8).
Reviewers acknowledged that concerns were resolved.
Decision: Accept (poster)