Title: ThinkSound: Chain-of-Thought Reasoning in Multimodal LLMs for Audio Generation and Editing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Any-to-Audio Generation

Chain-of-thought

Multimodal Large Language Model
Abstract: While end-to-end video-to-audio generation has greatly improved, producing high-fidelity audio that authentically captures the nuances of visual content remains challenging. Like professionals in the creative industries, this generation requires sophisticated reasoning about items such as visual dynamics, acoustic environments, and temporal relationships. We present **ThinkSound**, a novel framework that leverages Chain-of-Thought (CoT) reasoning to enable stepwise, interactive audio generation and editing for videos. Our approach decomposes the process into three complementary stages: foundational foley generation that creates semantically coherent soundscapes, interactive object-centric refinement through precise user interactions, and targeted editing guided by natural language instructions. At each stage, a multimodal large language model generates contextually aligned CoT reasoning that guides a unified audio foundation model. Furthermore, we introduce **AudioCoT**, a comprehensive dataset with structured reasoning annotations that establishes connections between visual content, textual descriptions, and sound synthesis.  Experiments demonstrate that ThinkSound achieves state-of-the-art performance in video-to-audio generation across both audio metrics and CoT metrics, and excels in the out-of-distribution Movie Gen Audio benchmark. The project page is available at https://ThinkSound-Project.github.io.


Peer reviews:
comments: SUMMARY:
This paper proposes ThinkSound, a novel framework for interactive audio generation and editing for videos wit CoT reasoning. The methods includes three stages, with foundational foley generation, interactive object-centric refinement, and targeted editing guided by instructions. Besides, a dataset with structural reasoning annotations AudioCoT is proposed. Experiments show the capabilities in V2A generation in audio metrics, CoT metrics, and Movie Gen Audio benchmark.

STRENGTHS AND WEAKNESSES:
Strength:

1.The paper proposes a novel three-stage interactive framework for V2A for soundscapes through CoT reasoning.

2.A large-scale multimodal dataset AudioCoT with detailed CoT annotation for visual, textual, and audio description is proposed.

3.The experiments validate the proposed method’s performance in audio generation.


Weakness (details see “Questions”):

1.The clarity of dataset construction and training architecture needs to improve.

2.The effectiveness and necessity of incorporating CoT reasoning need to be validated qualitatively

3.The design module of MM-DiT architecture needs more experiments to validate.

QUESTIONS:
1. L155-160, how is instruction-based audio constructed? How to get paired audio 1 and audio 2? Is it a random process or meticulous one?

2. Why use CoT in audio generation? Better to provide qualitative comparison of w/wo CoT process (for example, in Fig.2, what will the integrated analysis be if wo CoT)

3. Why adopt such an architecture of fig. 3 right part? Can the design of the MM-DiT be validated effectively? (e.g. the ablation study of adaptive fusion)

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ThinkSound, a three-stage framework that brings chain-of-thought (CoT) reasoning into video-to-audio generation. A fine-tuned VideoLLaMA 2 model first produces explicit, step-by-step CoT instructions; these then guide a single, multimodal flow-matching audio foundation model that can (1) create an initial “foley” soundtrack, (2) refine sounds for user-clicked regions, and (3) perform text-guided edits such as adding or removing events. To train and evaluate this pipeline, the authors curate AudioCoT, a large dataset pairing videos, audio/text clips, and structured CoT annotations. Experiments on VGGSound show clear gains over recent V2A baselines (e.g., MMAudio) in fidelity, semantic alignment, and human MOS, while an out-of-distribution test on the MovieGen Audio Bench confirms strong generalisation. Ablations reveal large drops when CoT reasoning or the gated video-audio fusion is removed, underscoring the importance of both components.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths:\n\n- The paper is nicely written and well presented.\n- The paper tries to solve a relevant problem of video-audio generation, which is hot now.\n- The paper is well structured and thought out.\n- The results look promising.\n\n### Weaknesses:\n\n- The title is a bit misleading. The title made me think that somehow the authors propose CoT in the audio generation process (also the paper is video-to-audio and not just audio, which should also be highlighted) -- which ideally means that the model thinks step by step to generate audio. However, the paper generates thinking in text, which is input to a DiT-like model by conditioning -- so ideally, the generation process is not what is thinking. The step-by-step in the process also comes through additional steps in generation (which are kind of optional) -- but thinking steps are not happening while generating audio -- which is what CoT ideally is.\n- ⁠The Cost is just conditioned to the DiT with a T5. I am doubtful if any model (DiT in this paper) would understand thought processes through T5 embeddings. T5 embeddings, in prior literature, is good at passing information about objects, attributes, etc -- the fact that it can pass knowledge about thought processes is something I am not being able to comprehend (maybe some ablation is needed to justify this) -- and it is important as the paper is based on this.\n- ⁠Figure 3 left is a bit misleading in the sense that the DiT should not be on top of the Decoder -- this ideally indicates a different form of architecture -- the DiT gets the output of MLLM in a cascaded fashion, and it's not joint.\n- About Section 4.4:\n  - Object-focused audio Generation is not new. See [this paper](https://arxiv.org/abs/2506.04214) for example. \n  - I may have missed where the editing training data is coming from. Can the authors highlight that part?\n- About Section 3.2\n  - Have the authors tried verifying how clean or noisy the generated data is? Any data verification step?\n\nQUESTIONS:\nMy questions are listed in Strengths And Weaknesses. Specifically:\n\n- I may have missed where the editing training data is coming from. Can the authors highlight that part?\n- Have the authors tried verifying how clean or noisy the generated data is? Any data verification step?\n- Stage 2: Interactive Object-Focused Audio Generation is supposed to be click-based and done by humans -- how did the authors manage to do an objective evaluation? What did I miss?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ​ThinkSound, a novel framework for video-to-audio (V2A) generation that leverages ​Chain-of-Thought (CoT) reasoning​ to address the limitations of existing end-to-end approaches in capturing nuanced audio-visual relationships. Key innovations include ​AudioCoT—the first dataset with structured audio reasoning annotations —and a ​unified audio foundation model​ using flow matching with gated multimodal fusion.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The motivation is strong and timely. Leveraging the reasoning capabilities of MLLMs to enhance downstream generative models is a critical and underexplored direction in the V2A field. The authors convincingly demonstrate that their approach achieves state-of-the-art (SOTA) performance, validating the effectiveness of their core idea.\n2. The unification of audio generation and editing within a single framework is a significant strength. By exploring two intuitive editing modalities—Region-of-Interest (ROI) based refinement and natural language instruction-based editing—the paper naturally showcases the practical benefits and versatility of incorporating an MLLM.\n3. The introduction of the AudioCoT dataset is a valuable contribution in its own right. The detailed construction process and the resulting dataset will likely serve as a crucial resource for future research in reasoning-based audio generation and editing.\n\nWeaknesses:\n1. The paper lacks essential mathematical formulations. For instance, including the objective functions for fine-tuning the MLLM and for training the flow matching model would significantly improve the paper's clarity and reproducibility.\n2. A more significant weakness is the lack of a dedicated performance evaluation for the MLLM's reasoning capabilities in the main text. The MLLM's ability to generate high-quality CoT is central to the entire framework's success. Therefore, its effectiveness and reliability should be explicitly demonstrated and analyzed, rather than being treated as a given. A standalone analysis of the MLLM's performance would substantially bolster the paper's core claims.\n\nQUESTIONS:\nThe reported inference speed is impressive, particularly its advantage over MMAudio. However, this result seems somewhat counter-intuitive, as the ThinkSound pipeline involves two distinct models: an MLLM for CoT generation and the flow matching model for audio synthesis. Could the authors provide a more detailed breakdown of this latency measurement? Specifically:\n1. How was the total inference time calculated? Does it include the MLLM's generation time?\n2. What are the primary factors contributing to this high efficiency? A more in-depth analysis explaining why the proposed framework is significantly faster than a diffusion-based model like MMAudio, despite its two-component architecture, would be highly beneficial for the reader.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ThinkSound, which is a multimodal large language model based on chain-of-thought reasoning for text,video-guided sound generation.\nThrough a 3-step pipeline, it can generate and edit customized sounds by borrowing knowledge from MLLM.\nMoreover, it proposes AudioCoT benchmark, carefully collecting structured reasoning annotations.\nIt shows remarkable performance compared to the current SOTA (MMAudio) in both in-domain and out-of-distribution experiments.\n\nSTRENGTHS AND WEAKNESSES:\nS1. It shows remarkable performance for TV2A.\n\nS2. The AudioCoT benchmark would be valuable in this community.\n\nS3. The authors provide comprehensive experiments to show the robustness and effectiveness of CoT reasoning in the TV2A task.\n\nW1. Given the structural similarity to MMAudio, it needs to clarify the architectural advantages of the proposed method.\n\nW2. Some parts are unclear in the experiments.\n\nW3. More deeply failure case analysis is needed.\n\nQUESTIONS:\n- The paper reports CLAP<sub>CoT</sub> scores on MovieGen Audio Bench (Table 2), suggesting that CoT reasoning captions were used. However, it is unclear whether these CoT captions were newly generated for MovieGen samples. Could the authors clarify whether reasoning-based captions were generated for MovieGen, and if so, what CLAP scores are for original captions?\n\n- While the use of CoT reasoning appears to improve performance across metrics such as CLAP<sub>CoT</sub> and MOS, it remains unclear whether verbose or overly detailed reasoning could lead to redundant or hallucinated audio content. I suggest the authors discuss how they handle potential over-generation and whether any such cases were observed during evaluation.\n\n- The benchmark used in Table 3 for object-focused audio generation is not clearly described. Is it based on a specific split of the AudioCoT dataset? If so, how does it differ from the test set used in Table 4? Please clarify the evaluation setup and explain any factors that may account for the differences in results.\n\n\n- Are the parameter and inference time numbers in Table 1 inclusive of the VideoLLaMA2 reasoning module?\n\n\n- Why does ThinkSound achieve 3× faster inference time than MMAudio, despite having slightly more parameters?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}