Audio-Sync Video Generation with Multi-Stream Temporal Control
Authors: Shuchen Weng
Haojie Zheng
Zheng Chang
Si Li
Boxin Shi
Xinlong Wang
Venue: NeurIPS 2025 poster ()
Keywords: Video generation
Primary area: applications

Abstract: Audio is inherently temporal and closely synchronized with the visual world, making it a naturally aligned and expressive control signal for controllable video generation (e.g., movies).
Beyond control, directly translating audio into video is essential for understanding and visualizing rich audio narratives (e.g., Podcasts or historical recordings).
However, existing approaches fall short in generating high-quality videos with precise audio-visual synchronization, especially across diverse and complex audio types.
In this work, we introduce MTV, a versatile framework for audio-sync video generation. MTV explicitly separates audios into speech, effects, and music tracks, enabling disentangled control over lip motion, event timing, and visual mood, respectively—resulting in fine-grained and semantically aligned video generation.
To support the framework, we additionally present DEMIX, a dataset comprising high-quality cinematic videos and demixed audio tracks. DEMIX is structured into five overlapped subsets, enabling scalable multi-stage training for diverse generation scenarios.
Extensive experiments demonstrate that MTV achieves state-of-the-art performance across six standard metrics spanning video quality, text-video consistency, and audio-video alignment.

Audio-Sync Video Generation with Multi-Stream Temporal Control
Shuchen Weng Haojie Zheng 1.2f Zheng Chang Li Boxin Shi Xinlong Wang
*Beijing Academy of Artificial Intelligence
School of Software and Microelectronics, Peking University
Schooo of Artificial Intelligence, Beijing University of Posts and Telecommunication
State Key Lab of Multimedia Info. Processing, School of Computer Science, Peking University Nat'l Eng. Research Ctr. of Visual Tech., School of Computer Science, Peking University (scweng, wangxinlong @ baaaiaaa cnn, suimu @ stw.pku.eee...... www.pk..eec.. (zhengchang98,) lisi) @@ bupt.edu.cn, shiboxin @ @eciii@@ku..eee pku.edu.cn
Abstract
Audio is inherently temporal and closely synchronized with the visual world, making it a naturally aligned and expressive control signal for controllable video generation (e.g., movies). Beyond control, directly translating audio into video is essential for understanding and visualizing rich audio narratives (e.g., Podcasts historical recordings). However, existing approaches fall short in generating high-quality videos with precise andio-visual synchronization, especially across diverse and complex audio types. this work, we introduce MTV, a versatile framework for audio-sync video generation. MTV explicitly separates audios into speech, effects, and music tracks, enabling disentangled control over lip mo- tion, event timing, and visual mood, semantically aligned video generation. To support the framework, we additionally present DEMIX, a dataset comprising high-quality cinematic videos and demixed audio tracks. DEMIX structured into five overlapped subsets, enabling scal- able multi-stage training for diverse generation scenarios. Extensive experiments demonstrate that MTV achieves kate-of-the-an performance across six standard metrics spanning video quality, text-video consistency, and audio-vides alignment. Project page: https: sttpp://bjzheng net bet/projects/MTV/
resuultnn in fine-grained and
Introduction
Audio fundamental medium in daily life, crucial for both information delivery (e.g., commu- nication, notifications, and education) and immersive experiences (e.g., enhancing the impact of film visuals). Despite the prevalence of audio-centric platforms (e.g., Podcasts), content presented solely through audio lacks the visual dimension needed to fully convey the richness of events. Since audio is naturally temporal and inherently synchronized with the visual world, researchers [ have devoted considerable attention to translating audios into corresponding videos to enhance audience understanding of rich audio narratives (e.g., historical recordings).
Despite great progress, existing methods face practical limitations in generating high-fidelity cine- matic videos with precise synchronization (e.g., pouring water into the transparent cup), primarily due to: (i) Under-specifeed audio-visual mapping. Current approaches handle a wide spectrum of audios and map them to various target scenes (e.g.. landscapes [4], dancing (5). music performances (6)). This broad representation scope potentially leads to ambiguous mappings lacking specificity between audio and visual features. (ūi) Inaccurate temporal alignment. Existing methods primarily focus on
Equal contributions. Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Character-centric narrative
brown jacket and blue shirt talking mobile phone
Multi-character interaction
A male on left, wearing a black suit woman on the right, wearing a red
Sound-triggered events
The focus is on the water's 's movement as it is poured into the glass from above.
Music-shaped ambiance
young woman ... flowers purple and yellow dressedd jocket
www.:///WWWAAAA/A///AAAA/. WWW WWW.MM/MMM/WW//MMMMMMMMM WWWM: www.w/////eeooeeeee.e.oo 4ay Araper peperppeneneeeeee www.aa/wweeeweee/ www./////////aa... www. www.:///wwwww/ww.////.... eeeeee
Camera movement
An old, rusted car driving on a suburban street with a lidedd white paint job
www.a//www...eeeeeeeeeoooeeeeeee www.wn www.hhutuwwttttlllll wwwl pa wy wwwpla/wwiiiieeeeeeee wwtp://wwwwwwwwwww////////......
Figure 1: MTV demonstrates versatile audio-syne video generation capabilities following user- provided text descriptions specifying scenes and subjects. Capabilities shown include producing videos centered targeted characters (Ist and 2nd rows) while triggering events with sound effeets (3rd row), generating visual mood with accompanying music (4th row), and adaptively handling camera movement (5th row). We present these generated videos in the supplementary materials.
building scene-level semantic consistency (e.g., translating engine sound to a car-centered video), struggling with accurate timing correspondence between individual audio events and their visual features (e.g., speech 7 motion [8], and visual mood
In this paper, we propose the MTV framework, enabling Multi-stream Temporal control for audio- syne Video generation overcome aforementioned issues, with versatile capabilities across scenarios illustrated in Fig. 1 Instead of attempting direct mapping from composite audios, we explicitly separate audios into distinct controlling tracks (i.e., speech, effects, and music), inspired by CDX'23 To provide sufficient high-quality video clips with demixed audio tracks, we contribute a large-scale DEMIX dataset with tailored data processing, including 392K video clips with 1.2K hours. These tracks enable the model to precisely control lip motion, event timing, and visual mood, resolving the ambiguous mapping. To further incorporate rich visual semanties beyond direct audio cues, we leverage features (e.g., subject gesture, scene appearance, camera movement) initially derived from a pretrained text-to-video model [10], and subsequently finetuned using video clips from the DEMIX dataset. To enable the progressive extension of learned high-level video semantic features stage-by-stage. this dataset is structured into five overlapped subsets. multi-stage training strategy introduced to leam concrete and localized controls (e.g., lip motion) towards more abstract and global influences (e.g., visual mood), leading to clear audio-visual relationships.

---

To achieve accurate temporal alignment, we propose the Multi-Stream Temporal ControlNet (MST- ControlNet) within the MTV framework. The interval stream designed for specific feature synchronization, which extracts features from the speech and effects tracks. It employs interval interaction blocks to understand each track individually and construct their interplay, maintaining the coherence with inferred semantic features. After that, interval feature injection module inserts features of each track into corresponding time intervals to drive lip motion and event timing. Since visual mood typically covers the entire video clip, the holistic stream is designed for overall aesthetic presentation, which extracts features from the music track using the holistic context encoder. These features then serve as style embeddings, applied uniformly to all frames through global style injection, controlling the visual mood.
We summarize our contributions as follows:
We present MTV, versatile audio-syne video generation framework by demixing audio inputs, achieving precise audio-visual mapping and accurate temporal alignment.
We introduce an audio-syne video generation dataset structured into five overlapped subsets, presenting the multi-stage training strategy for learning audio-visual relationships.
We propose the multi-stream temporal ControlNet to distinctively process demixed audio tracks and precisely control lip motion, event timing, and visual mood, respectively.
2 Related Works
2.1
Video Diffusion Model
The field of video generation has made significant progress with the adoption of diffusion models. Early approaches [1 extend the dynamic modeling capabilities of pretrained text-to-image diffusion models 14 by incorporating temporal layers (e.g., 3D convolutions [15] and temporal attention (16). However, these methods face inherent challenges in capturing long-range spatial- temporal dependencies due to the convolutional architectures of their backbone (e.g.. UNet (17]). To overcome this limitation, Sora report [18] presents the potential of the diffusion transformer (DiT) [99 architecture, prompting shift towards integrating 3D VAE (20) for spatial-temporal compression and scaling up to train the entire DiT-based model. Further improvement has been achieved by recent foundation models through adaptive layernorm modules [TO]. progressive scaling 21 [22, and post-training techniques [3 These advancements in text-to-video models provide strong foundation and powerful generative priors that could potentially be leveraged for related cross-modal tasks, such as high-quality audio-sync video generation.
2.2 Audio-driven Image Animation
Audio-driven image animation aims to generate dynamic visuals from static image, synchronized with user-provided audios. Several previous works animate general objects or scenes while maintain- ing audio-visual consistency. Sound2Sight (24) and CCVS (55 leverage the context of preceding frames to achieve audio-driven subsequent frames generation. TPOS (26) uses audios with variable temporal semantics and amplitude to guide the denoising process. ASVA (27] incorporates temporal audio control module for effective audio synchronization. Other works concentrate on audio-driven human animation. Talking head (7, 28 focus animating human face images to produce lip motion that synchronize with the speech. Recent works extend animation beyond the head to include half-body movements [3I and introduce pose control for full-body animation [32] Another specific application is music-to-dance music. Despite the audio-visua synchronization of these methods, their reliance on static images restricts models' capability to generate dynamic scenes required for cinematic videos.
[3 34], which generates human dance according to the beat of the
2.3 Audio-syne Video Generation
Audio-sync video generation does not require additional images for reference, offering the potential for free scene creation. Early works are designed based on VQGAN 355 and StyleGAN [36], achiev- ing audio control through multi-modal autoregressive transformers 2 and style code alignment 37. Recently, following the success of diffusion models demonstrating effectiveness in general video generation, researchers have turned their attention. Highlighting the benefit of multi modal
3

---

Table 1: Comparison of DEMIX dataset and previous datasets.
Modality
Scene
Audio component
Specifications
Method
Year
Text Audio People Objects
Cinematic
Speech Effects
Music
Demix
Clips
Hours
UCF-101 38
2012
13K
27
HIMV-200K
39
2017
V
200K
AudioSet [40
2017
V
2.1M
5.8K
VoxCeleb2 [4]
2018
150K
2.4K
VGGSound 42
2020
V
200K
550
WehVid- 10M
43
2021
10.7M
52K
Landscupe 4
2022
9K
26
Intern Vid
2024
V
7.1M
760K
Ours (DEMIX)
2025
392K
1.2K
conditions, TA2V [6] demonstrates that conditioning on both text descriptions and audio inputs sig- nificantly enhances the quality of generated videos. To achieve audio-visual alignment at both global and temporal levels, Tempo [ designs lightweight adapter for text-to-video generation model. Introducing a unified diffusion architecture, MM-Diffusion [5] enables both joint audio-video generation and zero-shot audio-syne video generation. Leveraging diffusion-basee latent aligners for open-domain audio-visuse generation. Xing et al. [3] achieve the audio-sync video editing and open- domain content creation. Although great progress has been made, audio-syne video generation still faces under-specific audio-visual mapping and inaccurate temporal alignment. Therefore, achieving cinematic quality remains challenging.
3 Dataset
We introduce the DEMIX dataset, tailored for training demixed audio-sync video generation models.
Data source. The training data is sourced from three aspects: (i) 65 hours of talking head videos from CelebV-HQ [45]; (ii) 4,923 hours of cinematic vidcos from MovieBench [46) (69h), Condensed Movies 47 (1,270h). and Short-Films 20K [48] (3,584h); and (iii) 8,903 hours film-related videos from Tube. All collected vidcos include their accompanying audio tracks.
Video filtering. Following previous video generation models 10 12 49], we use PySceneDetect [50]
segment video into single-shot clips. aaddiobox-aesthette (1 is further used assess the quality of accompanying audio, removing clips with low scores. For the left video clips, we annotate each
one with text descriptions using LLaVA- Video [52].
Demixing filtering. To improve audio demixing reliability, we employ dual-demixing comparison strategy, comparing demixing outputs from MVSEP (53) (speech, effects, music) and Spleeter [54] (speech, others). After that, we calculate the LI distance between the speech tracks. Next, the 'others' track from Spleeter conditionally compared: to the effects track from MVSEP if music silent (below -45dB), and to the music track if effects are silent. Clips are discarded only if high LI distances are found on any of the comparable pairs.
Voice-over filtering. To build clear audio-visual relationships for cinematic videos, we first detect whether people present in the videos using YOLO [55]. Next, perform speaker diarization for the accompanying audio using Scribe (56) to identify active speaker segments and count the number of speakers. After that, we detect the active speaker from videos for each frame using TalkNet (57). As a result, we can discard clips where speech occurs in the audio but the video analysis detects neither a visible person nor an active speaker in the corresponding frames.
Subset division. To facilitate multi-stage training for versatile audio-sync vidco generation models, the filtered DEMIX data is structured into five overlapped subsets. The basic face subset comprises all talking head videas. The remaining cinematic and film-related videos are then categorized to form the other subsets: assignment single character multiple characters depends on the annotated human count, while assignment to sound event or visual mood occurs if the respective effects or music track is non-silent.
Data statistics. After data collection and filtering, our DEMIX dataset includes 18K basic face, 54K single character, 39K multiple characters, 166K sound event, and 195K visual mood data, tailored for

---

(a) Input Video
[b)
Denoising Network
(c) Output Video
DIT block
X1.
Omndos procese
VIEE enc030
Adding room
Latari
Canaat
Imanaf
1100000
Gam
J00000 3uh
(d) Multi-stream Temporal ControlNet
(e) Text Description
Vinr Infordtion Broc
parson conversation. yourg
E
Sprwch
Wanzee urnoddrr
formal
dark
black vest. and a striped ti.
E
Person2: An oider man the right with grayying hair. weoring a casual plaid shirt. Person2 is speakknng Scone Description: men soricus Persont
Torneo conted encader
P
The pipeline of our MTV framework. (a-c) MTV is built pretrained text-to-video model [10] that provides strong generative priors for synthesizing diverse visual scenarios. (d)
Figure 2:
Explicitly separated audio tracks (i.e., speech, effects, music) are fed into our proposed multi-stream temporal ControlNet to ensure synchronization for lip motion, event timing, and visual mood. (e)
The MTV framework trained our contributed DEMIX dataset with five overlapped subsets and tailored text structures, enabling multi-stage training strategy for audio-sync video generation.
einematic videos, totaling non-overlapped 392K clips with 1.2K hours, accompanied by demixed audio tracks] For comprehensive evaluation, we hold out 1K video clips from the dataset to form the testing set. We provide an additional comparison with existing audio-related datasets 33 in Tab. [ highlighting that ours is tailored for versatile audio-sync video generation using demixed audio tracks, while robustly covering scenarios with people, objects, and cinematic visuals.
Method
This section begins with an overview of our MTV framework for audio-sync video generation (Sec. ). Next, we detail the Multi-stream Temporal ControlNet (MST-ControlNett) including the interval stream for specific feature synchronization, and the holistic stream for overall aesthetic presentation (Sec. 4.2 Finally, we present the multi-stage training strategy for effectively learning audio-visual relationships (Sec. 4.3)
4.1 Overview
MTV generates audio-sync videos based on user-provye user-provided text descriptions, (specifying the scenes and subjects) and demixed audio tracks a {a", a", a.") (representing speech, effects, and music) to respectively drive the lip motion, event timing, and visual mood. The pipeline is illustrated in Fig. 2
Video compression. As presented in Fig. (a), MTV is equipped with a pretrained spatio-temporal variational autoencoder (VAE) encoder to map video clips x into latent code 20 E(x). After that, its corresponding VAE decoder is used reconstruct video clips from the latent code x D(zo).
Denoising network. As presented in Fig. 2 (b), we concatenate the text embeddings f and noised latent code 2, before feeding them into the network to ensure the video-text correspondence. The expert Adaptive LayerNorm (AdaLN) [00 then independently processes text and video features within this unified sequence. Next, 3D full-ateention is used to interact semantics of text embeddings with corresponding video features. After being extracted by MST-ControlNet, audio cues are integrated via the interval feature injection and holistic style injection mechanisms. Finally, feed-forward network (FFN) is used to refine the resulting video features.
Denoising process. As presented in Fig. (c). MTV finally generates audio-sync videos by iteratively denoising latent codes. During training, at each time step {O....,T), T). Gaussian noise é; ~
Dataset samples visualized in the supplementary materials.
5

---

(0,01) is added to the clean latent code 2n to produce noised latent code = Vār + vI azeg diffusion transformer Eg is trained to predict the noise c.. given the noised latent code 2t, demixed audio tracks a, denoising time step 1, and text descriptions y. The diffusion transformer is trained by
minimizing the loss:
am = -1 [lke co(=e, a, 1)
For inference, we iteratively denoise a randomly sampled noise 2r N(.0 1) to obtain the latent code z to generate video clips with the VAE decoder x' = D(z)).
4.2 Multi-stream Temporal ControlNet
After explicitly separating audios into speech, effects, and music tracks, we propose the MST- ControlNet achieve accurate temporal alignment by respectively controlling lip motion, event timing, and visual mood. As presented in Fig. (d). the architecture consists of an audio encoding module followed by two specialized streams.
Audio encoding. Given demixed audio tracks a = fa", a", .,"), we initially extract their correspond- ing features { f fmm) from the demixed tracks using wav2vec [58]. After that, speech and effect features are fed into the interval stream for specific feature synchronization. Instead, music features are fed into the holistic stream for overall aesthetic presentation.
Interval stream. We design the interval stream to interval- wise control the lip motion and event timing. Specifically, we separately process speech features f* and effect features fr with stack of linear layers and concatenate them before feeding them into N interval interaction blocks. Within each block, these features are processed independently (via AdaLN, Gate, and FFN) to refine per-track understanding. To model their interplay at each time interval i, the corresponding speech features ff and effects features J are jointly processed by a self-attention | = SelfAttm[[/,, f())) This interaction also maintains the coherence with inferred semantic features. Finally, interacted speech features f and effects features f" are integrated into their corresponding time intervals via the interval feature injection mechanism:
h = fro h = CCosssAtt(hh, f.),
(2)
where h2 represents the video latent code i-th interval. CrossAttn(:;)) -) means a cross-attention. where the latent code serves as the query and the audio features as the key and value. Let M be the number of intervals, the resulting latent code is then updated h (hę h h
Holistic stream. The holistic stream is designed to control the visual mood for the entire video clip. Specifically, we process the music features f through a holistic context encoder, comprising three linear layers and a ID convolutional layer to extract features representing the visual mood. Since the environmental ambiance typically covers the entire video clip, an average pooling is applied to merge all the intervals and transform them into holistic music features f' Next, these features are regarded as style embeddings. By independently transforming these features into scale factor nn = Linear( (m)) and shift factor 3Tm = Linear( (fmm) we modulate the video latent code h uniformly across all intervals via the holistic style injection:
ht = h @ (7 + +1) + 3
(3)
where hm is the modulated latent code, fed into the denoising network to refine video features.
4.3 Multi-stage training strategy
As the dataset is structured as five overlapped subsets, we introduce the multi-stage training strategy to progressively scale up the model stage-by-stages
Text structure. As presented in Fig. (e). we create template to structure text descriptions, enabling our MTV framework to be compatible with these distinct training subsets. Specifically, this template begins with sentence indicating the number of participants (e.g., "Two person conversation"), based on Scribe [56] speaker counts. It then consists of subsequent entries for each individual, starting with unique identifier (e.g., Personl, Person2) followed by their respective appearance description. Following these individual entries, explicit identifier for the currently active speaker is specified. Finally, a sentence provides an overall description of the scene. Notably, when there no active speaker in the video, only the overall description will be provided.

---

Table 2: Quantitative experiment results of comparison and ablation. (4) means higher (lower) is better. Throughout the paper, best performances are highlighted bold.
Method
FVD /
Temp-C (%)
Text-C (%) f
Audio-C (%)
Syne-C
Syne-D +
Comparison with state-of state-of-the-art methods
MM--iifusiin
5
879.77
94.15
15.61
5.43
1.53
11.21
TempoTokees
795.88
93.13
24.68
6.71
1.45
10.48
Xing el al. [
805.23
93.30
24.51
7.30
1.55
10.50
Ours (MTV)
626.06
95.40
26.55
26.22
3.17
9.43
Ablation study
W/o
667.81
95.30
26.49
24.68
2.46
9.55
W/o SI
626.46
94.84
25.50
19.64
2.53
9.76
Wo TB
698.36
95. 14
26.37
24.50
2.31
9.78
Training schedule. We train the model from concrete and localized controls towards more abstract and global influences. Initially, we train the model to learn lip motion using the basic face subset. then learns human pose, scene appearance, and camera movement on the single character subset. To handle scenarios with multiple speakers, we subsequently train the model on the multiple characters subset. Following this, our training focus shifts to event timing and extending subject understanding from humans to objects using the sound event subset. Finally, we train the model on the environmental ambiance subset to improve its representation of visual mood.
Training details. We initialize our spatial-temporal VAE and DiT backbone with pretrained weights from Cog VideoX [10] and train our model to generate audio-sync videos at a 480 x 720 resolution. For each stage, we train our model for 40K steps on 24 NVIDIA A800 GPUs using the Adam-based optimizer [59] with a learning rate of 1 x 10 where CSST-ContollNet and attention layers of the backbone are trainable. For inference, our model requires 280s to generate 49-frame audio-sync video on a NVIDIA A100 GPU.
5
Experiments
5.1 Comparison with state-of-the-art methods
As audio-sync video generation is an emerging task, the relevant comparison methods are still developing. We compare our method with three recent state-of-the-art approaches our DEMIX dataset. For TempoTokens [ and Xing et al. [3], evaluate them using both text descriptions and corresponding audios as their original configuration. Since MM-Diffusion (5) can only support audio inputs and its training focuses specific landscape and dancing, finetune it to ensure fair comparison. 50 videos are randomly selected from the testing set for evaluation.
Quantitative comparisons. As presented in Tab. we quantitatively evaluate performance across three main aspects: (i) Visual quality is assessed using Frechét Video Distance (FVD) [60]. (ii) Temporal consistency (Temp-C) is measured by calculating similarity between consecutive frames using CLIP 61]. (iii) We examine text-video alignment via Text Consistency (Text-C) 622, audio- video alignment using Audio Consistency (Audio-C) (63) and specifically lip motion synchronization with Sync-C and Sync-D [64]. As a result, our framework outperforms state-of oftte-of-theann art methods across all six quantitative metrics. These metric details are provided in the supplementary materials.
Qualitative comparisons. As presented in Fig. 3 qualitative comparisons with state-of-the-at methods I 5 highlight the advantages of our framework. For instance, even after finetuning MM-Diffusion S for over 320K steps using the official code on NVIDIA A100 GPUs, still struggles with generating cinematic videos. TempoTokenss [I] struggles to generate cinematic videos for complex text-specified scenarios, resulting in unrealistic human expressions (Fig. 3 leftt. Xing et al. 3 find it difficult to effectively achieve audio synchronization for specific event timing, leading incorrect rendering of human gestures for guitar performance (Fig. 3 right). contrast, our MTV framework faithfully generates audio-syne videos with cinematic quality.

---

man gray hair The woman wearing denim shirt
personn ... playing the guitar red and parple stage lights
MM MiN-Diffulon
Tempo ToloTo
Xing
Ours
Figure 3: Visual comparison results with state-of-the-ar methods for audio-sync video generation.
An older man wearing a fauled blue cont photegraph
woman with dark a long-sleeved. t-necked dress
www.
Wio SE
TRU BEEN
EE
Wil SI
Weo TB
Qurs
Figure 4: Ablation study results of different MST-ControlNet variants.
5.2
Ablation Study
To evaluate the effectiveness key components within MST-ControlNet, we conduct ablation studies against three baseline configurations, as shown in Fig. aand Tab. 2
W/o SE (Separate Extraction). We extract all features from demixed audio tracks using interval interaction blocks. This prevents music features from shaping the overall aesthetic presentation, leading to reduced visual mood (Fig. left, degraded FVD and Temp-C).
W/o SI (Separate Injection). We extract features from demixed audio tracks by their respective encoders. These features are then concatenated and injected into the denoising network via shared cross-attention. This reduces conditional consistency (Fig. 4 left, decreased Text-C and Audio-C).
W/o TB (Training Backbone). We freeze all weights of DiT backbone and only train our proposed MST-ControlNet to preserve more generative priors. This impairs the specific feature synchronization, especially the lip motion synchronization (Fig. 4 right, reduced Sync-C and Sync-D).

---

Table 3: User study results. Ours (MTV) clearly produces higher score than state-of-ther the-tartt methods.
Subjective criteria
MM- CMM-Diffusion [
Tekeo'Tokens
Xing et al. [
Ours (MTV)
Semantic consistency
0.96%
13.60%
11.28%
74.16% %
Motion fluency
0.64%
8.96%
12.56%
77.84
Overall preference
0.72%
12.00%
12.40%
74.88 %
Table 4: Quantitative experiment results with alternative pre-trained components.
Method
FVD 4
Temp-C (%) f Text-C (%) f
Audio-C (%)
Sync-C f
Sync-D +
Cog VideoX+ Vec
626.06
95.40
26.55
26.22
3.17
9.43
Cog VideoX+ Beaats
598.53
95.91
26.25
25.28
3.02
9.52
Wan14B+We Wev2Vee
353.61
96.36
27.23
26.49
3.08
9.56
5.3
User Study
To better evaluate our method from a human perception perspective, we conduct three subjective user study experiments Tab. 3 We present videos generated by our method and all baselines to participants and ask them to choose the best one based on the following criteria: (i) Semantic consistency. How well the video content aligns with the text description. (ii) Motion fluency. The realism and temporal coherence of the motion. (ii)) Overall preference. How good the holistic quality the video is. For each study, we randomly select text descriptions from the test set, and the evaluations are conducted by 25 volunteers. The table below shows the percentage of times each method chosen as the winner. Our method consistently favored by human observers and has achieved the highest scores across all three subjective criteria.
5.4 Analysis of Pre-trained Components
We evaluate the robustness of our proposed method by integrating it with alternative pre-trained components. Specificallyy we test replacing the audio encoder (Waw2Vec/BEATs) and the video backbone CoggVideo//Wan44B in Tab.
BEATs. Since Wav2Vec 58 is common setting for speech encoding (e.g., Hallo3 (7)), this baseline only replaces it with BEATs (65] for both the effects and music tracks. As shown in Tab. 4 this baseline achieves comparable (or slightly better) video-related metrics (i.e., FVD and Temp-Co C) but shows a slight degradation on audio-related metrics (i.e., Audio-C, Sync-C, and Sync-D), suggesting that our current choice of Wav2Vec (58) is robust and effective one for this task.
Wan14B, Since Wan 4B 21 shares similar DiT-based structure with Cog VideoX 11 we can integrate our proposed MST-ControlNet into without architectural changes. Specifically, interval feature injection and holistic style injection modules are added after each text cross-attention layer. The quantitative results below show this baseline achieves better performance on video- and text-related metrics (i.e., FVD, Temp-C, and Text-C) due to the stronger capabilities of the Wan14B (2I), while achieving comparable performance on all audio-related metrics (i.e., Audio-C, Sync-C, and Sync-D).
5.5 Application
As presented in Fig. 5 our model support four typical scenarios: (i) By integrating text-to-video generative priors and learned audio-visual synchronized capabilities, our model can create vivid virtual characters. (ii) Given user- provided images and taking them as arbitrary keyframes, our model can drive the image according to the given audios. (iāi) Although our model generates video segments of 49 frames, it can achieve long video generation by using the generated frame to initialize the next segment. (ïv) Following training-free approaches (66), our model can generate scene transitions guided by providing viry-narying text descriptions.

---

Character creation
Hidf' wrirring sunglaasses and and wvvoring zur
Keyframe guidance
dracce with
white for
Long video generation
womaon wilh shonider gegghh slack Rutic
Hight--cooorsi blazer dark with flowers
Scene transitions
dark kmight rides back horse gullop grasslande
flchdl..
Figure 5: Examples of versatile application scenarios for our proposed MTV framework.
Apptarance
hatr
Black rap, moddie
Grogy
Event Eventtiming
Vitual
Lift pulling
Chonging hunde
Righo prensing
Peangfil
Happy
Figure 6: Examples of controllability study for text descriptions and demixed audios.
5.6 Controllability
As shown in Fig. 6 leveraging control from both text descriptions and the three demixed audio tracks (i.e., speech. effects. music), our model can offer controllability across following four key aspects: (i) Modifying the text descriptions while keeping all audio tracks fixed allows the visual scene appearance
edited without affecting the audio synchronization. (ii) Given demixed speech track, the model enables precise control over the synchronized lip motion of the generated character. (iii) Similarly, with demixed effects track, the model accurately synchronizes event timing with the sound effects. (iv) By changing the demixed music track, the model creates different visual moods for the generated video.
Conclusion
In this work, presented MTV, versatile framework for audio-syne video generation. MTV leverages generative priors from pretrained text-to-video models [10] and is trained on our contributed DEMIX dataset that provides sufficient cinematic videos with demixed audio tracks. Equipped with our proposed MST-ControlNet, MTV is able to independently control lip motion, event timing, and visual mood. Combined with a multi-stage training strategy for effective learning of complex audio-visual relationships, MTV achieves state-of-the-an performance across six evaluation metrics.
Limitation. Although our approach demonstrates the potential of using demixed audio tracks for precise video control, is fundamentally limited by the scope of categories provided by upstream audio demixing techniques [53] (54). We believe the capabilities of audio-sync video generation methods will further progress with advancements in audio demixing methods.
Acknowledgement. This work is supported by National Natural Science Foundation of China (Grant No. 62 621301. We thank all the insightful reviewers for the helpful suggestions, and the colleagues at Beijing Academy of Artificial Intelligence for their support throughout this project.
10

---

References
[1] G. Yariv, I. Gat, S. Benaim, L. Wolf, I. Schwartz, and Y. Adi, "Diverse and aligned audio-to- video generation via text-to-video model adaptation," in AAAI, 2024.
[2] Ge, T. Hayes, Yang, X. Yin, G. Pang, D. Jacobs, J.-B. Huang, and D. Parikh, "Long video generation with time-agnostic vqgan and time-sensitive transformer," in ECCV, 2022.
[3] Y. Xing, Y. He, Z. Tian, X. Wang, and Q Chen, "Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners," in CVPR, 2024.
[4] S. H. Lee, G. Oh, W. Byeon, J. Bae, C. Kim, W. J. Ryoo, S. H. Yoon, J. Kim, and S. Kim, "Sound-guided semantic video generation," " in ECCV, 2022.
[5] L. Ruan, Y. Ma, H. Yang, H. He, B. Liu, J. Fu, N. J. Yuan, Q. Jin, and B. Guo, MMM-Diffusion: Learning modai diffusion models for joint audio and video generation." in CVPR, 2023.
[6] M. Zhao, W. Wang, T. Chen, R. Zhang, and R. Li, "TA2V: Text-audio guided video generation." TMM, 2024.
[7] Cui, H. Li, Zhan, H. Shang, K. Cheng, Y. Ma, S. Mu, H. Zhou, Wang, and S. Zhu, "Hallo3: Highly dynamic and realistic portrait image animation with video diffusion transformer," CVPR, 2025.
[8] S. Qian, Z. Tu, Y. Zhi, W. Liu, and S. Gao, "Speech drives templates: Co-speech gesture synthesis with learned templates," in ICCV, 2021.
[9] B. M.-K. Ng. S. R. Sudhoff, H. Li, J. Kamphuis, T. Nadolsky, Y. Chen, K. Y.-J. Yun, and Y.-H. Lu, "Visualize music using generative arts," in CAI, 2024.
[10] Yang, J. Teng, W. Zheng, M. Ding, S. Huang. Xu, Yang, W. Hong. X. Zhang, G. Feng, et al., "CogVideox: Text-to-video diffusion models with expert transformer." in ICLR, 2025.
[11] P. Esser, Chiu, P. Atighehchian, Granskog, and A. Germanidis, "Structure and content- guided video synthesis with diffusion models," in ICCV, 2023.
[12] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, Voleti, A. Letts, et al., "Stable video diffusion: Scaling latent video diffusion models to large datasets." arXiv preprint arXiv:2311.15127 2023.
[13] Y. He, T. Yang, Zhang, Shan, and Q. Chen, "Latent video diffusion models for high-fidelity long video generation." arXiv preprint 2rXiv:2211.3222 2022.
[14] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High-resolution image synthesis with latent diffusion models," in CVPR, 2022.
[15] T. Soo Kim and A. Reiter, "Interpretable 3D human action analysis with temporal convolutional networks," in CVPR workshops, 2017.
[16] G. Bertasius, H. Wang, and L. Torresani, "Is space-time attention all you need for video understanding?." in ICML, 2021.
[17] O. Ronneberger, P. Fischer, and T. Brox, "U-Net: Convolutional networks for biomedical image segmentation," in MICCAI, 2015.
[18] Y. Liu, K. Zhang, Y. Li, Z. Yan, C. Gao, R. Chen, Z. Yuan, Huang, H. Sun, J. Gao, al., "Sora: A review on background, technology, limitations, and opportunities of large vision models," " arXiv preprint 17Xiv:2402.11177, 2024.
[19] W. Peebles and S. Xie, "Scalable diffusion models with transformers," in ICCV, 2023.
[20] I. Yu, J. Lezama, N. B. Gundavarapu, I. Versari, K. Sohn, D. Minnen, Y. Cheng, V. Birodkar, A. Gupta, Gu, A. G. Hauptmann, B. Gong, M.-H. Yang, Essa, A. Ross, and I.. Jiang, "Language model beats diffusion tokenizer is key to visual generation," in ICLR, 2024.
11

---

[21] A. Wang, B. Ai. B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao. J. Yang, J. Zeng. et al., "Wan: Open and advanced large-scale video generative models," arXiv preprint 2025.
[22] W. Kong, Q Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. J., Zhang, et al., "Hunyuanvideo: A systematic framework for large video generative models," " arXiv preprint 2024.
arXiv:2412.033
[23] G. Ma, H. Huang, K. Yan, L. Chen, N. Duan, S. Yin, C. Wan, R. Ming, X. Song, X. Chen, et al., "Step-Videe VVV technical report: The practice, challenges, and future of video foundation model," arXiv preprint W5X::2502.112488, 2025.
[24] M. Chatterjee and A. Cherian, "Sound2Sight: Generating visual dynamics from sound and context," in ECCV, 2020.
[25] G. Le Moing, J. Ponce, and C. Schmid, "CCVS: Context-aware controllable video synthesis," " in NeurlPS, 2021.
[26] Y. Jeong, W. Ryoo, S. Lee, D. Seo, W. Byeon, S. Kim, and J. Kim, "The power of sound (TPoS): Audio reactive video generation with stable diffusion," in ICCV, 2023.
[27] L. Zhang, S. Mo, Y. Zhang, and P. Morgado, "Audio-synchronized visual animation," in ECCV, 2024.
[28] Jiang, C. Liang, Yang, G. Lin, T. Zhong, and Y. Zheng, "Loopy: Taming audio-driven portrait avatar with long-term motion dependency," in ICLR, 2025.
[29] H. Wei, Z. Yang. and Z. Wang, "AniPortrait: Audio-driven synthesis of photorealistic portrait animation," arXiv preprint arXiv:2403.17694, 2024.
[30] W. Zhang, X. Cun, Wang, Y. Zhang, X. Shen, Y. Guo, Shan, and F. Wang, "SadTalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation," in CVPR, 2023.
[31] G. Lin, J. Jiang, C. Liang. T. Zhong, J. Yang, and Y. Zheng, "CyberHost: A one-stage diffusion framework for audio-driven talking body generation," in ICLR, 2025.
[32] G. Lin, J. Jiang, J. Yang. Z. Zheng, and C. Liang, "OmniHuman-I: Rethinking the scaling-up of one-stage conditioned human animation models," " arXiv preprint arXiv:2502.01061, 2025.
[33] W. Xuanchen, W. Heng, L. Dongnan, and W. Cai, "Dance any beat: Blending beats with visuals in dance video generation," in WACV, 2025.
[34] Z. Chen, H. Xu, G. Song, Y. Xie, C. Zhang, X. Chen, C. Wang, D. Chang, and L. Luo, "X- dancer: Expressive music to human dance video generation," arXiv preprint 2xXv:2502.17414 2025.
[35] P. Esser, R. Rombach, and B. Ommer, "Taming transformers for high-resolution image synthesis," in CVPR, 2021.
[36] T. Karras, S. Laine, and T. Aila, "A style-based generator architecture for generative adversarial networks," in CVPR, 2019.
[37] D. Jeong, S. Doh, and T. Kwon, "Träumerai: Dreaming music with stylegan," " arxiv preprint arXiv:2102.04680. 2021.
[38] K. Soomro, A. R. Zamir, and M. Shah, "Ucf101: A dataset of 101 human actions classes from videos in the wild," arXiv preprint arXiv: 1212.002, 2012.
[39] S. Hong, W. Im, and S. Yang, "Content-based video- music retrieval using soft intra-modal structure constraint," arXiv preprint arXiv 2017.
[40] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, "AudioSet: An ontology and haman-labeled dataset for audio events," in ICASSP, 2017.
12

---

[41] J. S. Chung, A. Nagrani, and A. Zisserman, "VoxCeleb2: Deep speaker recognition," arXiv preprint 1806. 05622, 2018.
[42] H. Chen, W. Xie, A. Vedaldi, and A. Zisserman, "VggSound: A large-scale audio-visual dataset," in ICASSP,2000
[43] M. Bain, A. Nagrani, G. Varol, and A. Zisserman, "Frozen time: A joint video and image encoder for end-to-end retrieval," in ICCV, 2021.
[44] Y. Wang, Y. He, Y. Li, K. J. Yu, X. Ma. X. Li, G. Chen, X. Chen, Y. Wang, al., "Intern A large-scale video-text dataset for multimodal understanding and generation," in ICLR, 2024.
[45] H. Zhu, W. Wu, W. Zhu, L. Jiang, Tang, L. Zhang, Z. Liu, and C. C. Loy, "CelebV-HQ: A large-scale video facial attributes dataset," in ECCV, 2022.
[46] W. Wu, M. Liu, Z. Zhu, X. Xia, H. Feng, W. Wang, K. Q. Lin, C. Shen, and M. Z. Shou, "MovieBench: A hierarchical movie level dataset for long video generation," arXiv preprint arXiv:2411.1562, 2024.
[47] M. Bain, A. Nagrani, A. Brown, and A. Zisserman, "Condensed movies: Story based retrieval with contextual embeddings," ACCV, 2020.
[48] R. Ghermi, X. Wang, Kalogeiton, and Laptev, "Short film dataset (SFD): A benchmark for story-level video understanding." arXiv preprint arXiv:2406. 10221, 2024.
[49] H. Chen, Y. Zhang. X. Cun, M. Xia, X. Wang. C. Weng, and Y. Shan, "VideoCrafter2: Over- coming data limitations for high-quality video diffusion models," in CVPR, 2024.
[50] B.
Castellano. "Video
detection
and
analysis
tool."
[51] A. Tjandra, Y.C.. Wu, B. Guo, J. Hoffman, B. Ellis, A. Vyas, B. Shi, S. Chen, M. Le, N. Zacharov, al., "Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound," arXiv preprint arXiv:2502.05 2025.
[52] Y. Zhang, J. Wu, W. Li, B. Li, Z. Ma, Z. Liu, and C. Li, "Video instruction tuning with synthetic data," arXiv preprint arXiv:2410.0277 2024.
[53] R. Solovyev, "Cinematic sound demixing." http://giinbccomm////tttt/sssss Sinematic-Sound GoonddDemixxin.
[54] R. Hennequin, Khlif, Voituret, and M. Moussallam, "Spleeter: a fast and efficient music source separation tool with pre-trained models," Journal of Open Source Software, 2020.
[55] G.
Jocher, J. Qiu, and A. Chaurasia, http://ggthak.oommmllaaaayciiias
"Ultralytics
YOLO."
[56]
Elevenlabs, "Meet scribe."
[57] S. Beliaev and B. Ginsburg, "TalkNet 2: Non-autoregressive depth-wise separable convolu- tional model for speech synthesis with explicit pitch and duration prediction," arXiv preprint wxXiv:2104008183 2021.
[58] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: a framework for self sulf-super learning of speech representations," in NewrIPS, 2020.
[59] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint 11X 2014.
[60] T. Unterthiner, S. Van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly, "To- wards accurate generative models video: new metric & challenges," " arXiv preprint arXiv:1812.01717, 2018.
[61] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark, et al., "I "Learning transferable visual models from natural language supervi- sion," in ICML, 2021.
13

---

[62] J. Wang, C. Wang, K. Huang, J. Huang, and Jin, "VideoCLIP-XL: Advanciin long description understanding for video clip models," arXiv preprint arXiv:2410.00741, 2024.
[63] R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and Misra, "ImageBind: One embedding space to bind them all," in CVPR, 2023.
[64] J. S. Chung and A. Zisserman, "Out of time: automated lip syne in the wild," in ACCV Workshops, 2017.
[65] S. Chen, Y. Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and Wei, "BEATs: Audio pre-training with acoustic tokenizers," in ICML, 2023.
[66] M. Cai, Cun, X. Li. W. Liu, Z. Zhang, Y. Zhang, Shan, and X. Yue, "DitCtrl: Exploring attention control in multi modil diffusion transformer for tuning- free multi-prompp longer video generation," " arXiv preprint arXiv:2412. 18597, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction part reflect the main contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Sec. 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The model framework and experiment settings are described in detail.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer:
[No]
Justification:
Both the dataset and the codes will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
connnooolcyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuii wwwiiicgguie///eeeeeeeeic CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experiments are presented in detail.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: These can be found in Sec. 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //te:::///uuis..eeeee http:///.ccppppcccctcccccicieee.
Answer: [Yes]
Justification:
The authors fully aware of the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper has societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
18

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: They are cited clearly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworeewwthhhe....cm com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
19

---

Answer: [Yes]
Justification: Codes and the dataset will be released upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
20

---

Justification:
We clearly describe the usage of LLMs to annotate our dataset.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy http:ees//eerrppcoccccoooorrrere for what should or should not be described.
21


Reviews:
[{"review_id": "M1Cj8hcVKJ", "reviewer": "Reviewer_453Q", "summary": "This paper introduces MTV framework (Multi-stream Temporal control for audio-sync Video generation), a framework for generating high-fidelity cinematic videos with precise audio-visual synchronization across diverse audio types. Existing methods struggle with under-specified audio-visual mapping and inaccurate temporal alignment. MTV addresses these challenges by separating audio inputs into speech, effects and music tracks, for disentangled control over lip motion, event timing, and visual mood, enabling fine-grained and semantically aligned video generation. To support this framework, the authors present an audio-sync video generation dataset, DEMIX, structured into five overlapped subsets and introduce multi-stage training strategy. Additionally, the paper proposes MST-ControlNet (Multi-Stream Temporal ControlNet) to distinctively process demixed audio stream and inject audio features of each track into corresponding time intervals, achieving fine-grained synchronization. The performance of MTV for video quality, text-video consistency, and audio-video alignment outperforms 3 baseline methods across six standard metrics.", "questions": "- Robustness to imperfect demixing isn't evaluated. Have you evaluated synchronization quality under varying levels of demixing noise?\n- Can MTV maintain its performance gain when it is applied on various audio-visual datasets or other state-of-the-art video-diffusion models? (See weakness 2)", "limitations": "- MTV generates audio-sync videos only at a 480 x 720 resolution and 49-frame segments, limiting its scalability to high-resolution or long-form video generation.\n- The paper lacks ablations or empirical justification for key design choices (e.g., stream interaction blocks, training schedule).", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- MTV method is directly aligned with the two core motivations--under-specified audio-visual mapping and inaccurate temporal alignment--by explicitly demixing audio into distinct controlling tracks with dedicated temporal control mechanisms.\n- The authors contribute DEMIX, a large-scale cinematic video clips with demixed audio tracks, structured into five overlapping subsets, which facilitates multi-stage training.\n\nWeaknesses\n- MTV's performance depends on the accuracy of upstream demixerrs (MVSEP, Spleeter). Errors in source separation can propagate and degrade synchronization.\n- Quantitative results are reported only on DEMIX dataset and CogVideoX baseline model. It remains unclear how well MTV generalizes to other T2V models and different datasets (e.g., non-cinematic content)\n- The paper relies solely on automated metrics; incorporating qualitative human assessments would better validate diverse aspects including detailed semantic consistency, narrative fluency of video, and user preferences.", "was_revised": true, "final_justification": "After thoroughly reviewing the authors' rebuttal, I find that they have sufficiently addressed most of the key concerns raised in the initial review. They demonstrated the generalizability of the proposed MTV framework through additional results on new datasets (e.g., Landscape, Audioset-Drum) and a different backbone (Wan14B). They also validated robustness to varying demixing quality and clarified concerns on resolution and video length by providing detailed qualitative examples. In addition, they presented ablation studies and justifications for major design choices, making the method more convincing. I would kindly encourage the authors to reflect these additional experiments and clarifications in the camera-ready version to further strengthen the quality of the paper. Plus, I am happy to increase my rating to a more positive one.", "rebuttal": "We thank the reviewers for their insightful and constructive feedback.\n\nIn accordance with the rebuttal policy, we are unable to include new qualitative results (e.g., generated videos) or external links. We assure the reviewers that, should the paper be accepted, the camera-ready version will be thoroughly updated to include:\n\n1. All additional experiments and qualitative results are discussed in this rebuttal.\n\n2. A public link to an open-source demo showcasing our MTV framework as a practical video generation tool.\n\nWe now address the specific points raised by the reviewers.\n\n## ***Weakness***\n\n**W1: Errors in source separation**\n\nWe agree that the upstream source separation is not always perfect. However, all of our reported results are generated from these imperfectly separated audio tracks, not from idealized clean inputs. As shown in Tab. 2 and Fig. 3, our framework achieves state-of-the-art performance across six standard metrics, directly demonstrating its inherent robustness in real-world scenarios. This stems from our demixing filtering strategy (Lines 117-122). We explicitly filter out extremely poor-quality samples during training, which enables our model to learn precise audio-visual synchronization,  while the remaining unclear separations serve as a form of data augmentation that enhances robustness.\n\nTo completely bypass this separation challenge, we further developed a demo. It uses Qwen3 to interpret user prompts into audio descriptions, synthesizes them into perfectly clean audio with ElevenLabs, and then generates a synchronized video with our MTV framework. This presents a complete text-to-video pipeline to avoid potential source separation errors entirely.\n\n**W2: Additional datasets and backbone**\n\nWe thank the reviewer for this valuable suggestion.\n\n1. Evaluation on new datasets\n\nTo demonstrate the generalization capabilities of our MTV framework, we follow TempoTokens to conduct additional experiments on both the Landscape and AudioSet-Drum datasets. To make a fair comparison, we fine-tune all baseline methods (MM-Diffusion, TempoTokens, and Xing et al.) on our DEMIX dataset using their official training schedules, and evaluate them on these separate datasets. As shown in the tables below, our MTV framework still achieves significantly better performance.  Since neither dataset includes human talking, the lip synchronization metrics (Sync-C and Sync-D) are not applicable for this evaluation.\n\n#### **Comparison on Landscape datasets**\n\n| Method | FVD ↓ | Temp-C (%) ↑ | Text-C (%) ↑ | Audio-C (%) ↑ | \n| :--- | :---: | :---: | :---: | :---: |\n| MM-Diffusion | 807.65 | 94.74 |14.66| 16.59 | \n| TempoTokens | 797.33 | 94.67 | 21.73 | 18.86 |\n| Xing et al. | 838.03 | 94.71 | 21.04 | 18.70 | \n| Ours (MTV) | **697.51** | **96.98** | **25.35** | **23.37** | \n\n#### **Comparison on AudioSet-Drum dataset**\n\n| Method | FVD ↓ | Temp-C (%) ↑ | Text-C (%) ↑ | Audio-C (%) ↑ | \n| :--- | :---: | :---: | :---: | :---: |\n| MM-Diffusion | 1520.09 | 94.59 | 14.90 | 14.11 |\n| TempoTokens  | 1512.97 | 94.28 | 23.18 | 15.59 | \n| Xing et al. | 1589.46 | 94.49 | 23.73 | 17.84 | \n| Ours (MTV) | **1511.53** | **97.50** | **25.62** | **39.61** |\n\n2. Adaptability to additional T2V backbone\n\nOur proposed MSTControlNet can be seamlessly integrated into various T2V backbones (e.g., OpenSora and Wan14B) without architectural changes, as they share a similar DiT-based structure. Due to time limitations, we only implement and test the integration with Wan14B. Specifically, our interval feature injection and holistic style injection modules are added after each text cross-attention layer. \n\nFor inference, MTV (Wan14B backbone) requires 180s to generate an 81-frame, audio-synchronized video at a 480P resolution on 8 NVIDIA A100 GPUs. We report the new quantitative results below.\n\n| Method | FVD ↓ | Temp-C (%) ↑ | Text-C (%) ↑ | Audio-C (%) ↑ | Sync-C ↑ | Sync-D ↓ |\n| :--- | :---: | :---: | :---: | :---: |:---: |:---: |\n| CogVideoX |626.06 |95.40 |26.55 |26.22 | **3.17** | **9.43** |\n| Wan14B | **353.61** | **96.36** | **27.23** | **26.49** | 3.08 | 9.56 |\n\n**W3: Human assessments**\n\nTo better evaluate our method from a human perception perspective, we conducted three subjective user study experiments as requested. We presented videos generated by our method and all baselines to participants and asked them to choose the best one based on the following criteria:\n\n* Semantic consistency: \"Which video best matches the text description?\"\n\n* Motion fluency: \"Which video displays more realistic and temporally coherent motion?\"\n\n* Overall preference: \"Which video do you prefer overall, considering all aspects?\"\n\nFor each study, we randomly select 50 text descriptions from the test set, and the evaluations are conducted by 25 volunteers. The table below shows the percentage of times each method is chosen as the winner. Our method is consistently favored by human observers and has achieved the highest scores across all three subjective criteria.\n\nWe hope that our new experimental results have fully addressed the reviewer's concerns, and we would be happy to provide further clarification on any remaining points during the author-reviewer discussion period.\n\n| method \\ metrics | Semantic consistency ↑ | Motion fluency ↑ | Overall preference ↑ |\n| ---------------- | -------------------- | -------------- | ------------------ |\n| MM-Diffusion     | 0.96%   | 0.64%   | 0.72%  | \n| TempoTokens      | 13.60%     | 8.96%   | 12.00% |\n| seeing and hearing| 11.28%     | 12.56%   | 12.40% |\n| Ours(MTV)        | **74.16%**     | **77.84%** | **74.88%**  |\n\n\n**Q1: Varying levels of demixing quality**\n\nWe follow the reviewer's suggestion to conduct an analysis of our model's performance under varying levels of demixing quality. We create three quality-based subsets (Level 1 being the highest quality, Level 3 the lowest), with each subset containing 20 samples. The quality level for each sample is determined by a consensus rating from 5 human volunteers. We also evaluate a \"Random\" subset, which is randomly sampled from the entire test set for baseline comparison.\n\nAs the results below show, our model is robust to the quality of the demixed audio. Its performance on metrics like FVD and Text-C shows only minor fluctuations across the quality levels, while audio-visual correlation metrics (Audio-C and Sync-C) are even slightly better on lower-quality inputs. This may suggest that our model effectively focuses on audio cues that persist even when the demixed audio is not perfectly clean. Notably,  with any-level quality inputs, our model remains superior to all baselines evaluated on the Random subset. Therefore, our multi-stream approach provides robust gains overall.\n\n| Demix quality | FVD ↓ | Temp-C (%) ↑ | Text-C (%) ↑ | Audio-C (%) ↑ | Sync-C ↑ | Sync-D ↓ |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| MTV (Level1) | **617.23** | 95.14 | **27.03** | 24.81 | 2.88  | **8.99** |\n| MTV (Level2) | 665.18 |95.60 |26.66 | 26.01 | 3.13 | 9.29 |\n| MTV (Level3) | 638.92 | **95.88** | 26.52 | **26.87**| **3.46** | 9.41 |\n| MTV (Random) | 626.06 | 95.40 | 26.55 | 26.22 | 3.17 | 9.43 |\n| MM-Diffusion (Random) | 879.77 | 94.15 | 15.61 | 5.43 | 1.53 | 11.21 |\n| TempoTokens (Random) | 795.88 | 93.13 | 24.68 | 6.71 | 1.45 | 10.48 |\n| Xing et al. (Random) | 805.23 | 93.30 | 24.51 | 7.30 | 1.55 | 10.50 |\n\n**Q2:  Additional datasets and backbone**\n\nSee Reviewer 453Q-W1.\n\n**L1: Resolution and duration**\n\nIn fact, the resolution of our MTV framework ($480 \\times 720$) is the highest among all comparison methods. For instance, MM-Diffusion is limited to a $256 \\times 256$ resolution, while TempoTokens and Xing et al. can only generate $384 \\times 384$ videos. Additionally, as demonstrated in our previous response (W2), we have adapted our MSTControlNet to the Wan14B backbone, enabling our framework to generate high-resolution $720 \\times 1280$ videos.\n\nOur framework is fully capable of generating long videos. Numerous examples of long videos are already present in our submission. For instance:\n\n* In the main paper, the video in the 2nd row of Fig. 1 is 132 frames, and the video in the 2nd row of Fig. 5 is 146 frames.\n\n* The supplementary material contains even longer examples, including videos of 110, 116, 156, and up to 234 frames.\n\n**L2: Key design choices**\n\nThe architecture of our stream interaction block is inspired by the dual-stream DiT block of the HunyuanVideo, a proven effective architecture. Beyond this, we already provide extensive ablation studies in the paper:\n\n* We analyze the architecture of our MTSControlNet in Sec. 5.2 (Lines 250-255).\n\n* We study the optimal number of blocks in Appendix A.2.\n\nWe further conduct an additional ablation study (\"Linear\"), where we replace our interval interaction block with a stack of linear layers (following Hallo3). As the results shown below, our proposed interval interaction block achieves superior performance on this task.\n\nWe also conduct an additional ablation study (\"Mix\") to validate our training strategy, where we train our MTV framework on the full DEMIX dataset, without the multi-stage training strategy.\n\nAs the results below show, this ablation significantly degrades lip synchronization performance. We believe this is because lip-sync is a highly sensitive task that requires precisely aligning audio with a small visual region (i.e., the mouth). Without first learning this fundamental alignment on a simpler subset of facial data, the model struggles to learn this correspondence when faced with the complexity of large motions and multiple people. \n\n| Demix quality | FVD ↓ | Temp-C (%) ↑ | Text-C (%) ↑ | Audio-C (%) ↑ | Sync-C ↑ | Sync-D ↓ |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| MTV (Linear)  | 644.72 | 95.15 | 26.51 | 26.14 | 2.19 | 9.56 |\n| MTV (Mix) | 653.14 | 94.63 | 26.27 | 24.30 | 2.16 | 9.59 |\n| MTV | **626.06** | **95.40** | **26.55** | **26.22** | **3.17** | **9.43** |"}, {"review_id": "9oagWofTc2", "reviewer": "Reviewer_uen2", "summary": "This paper presents MTV, a unified and controllable framework for audio-synchronized video generation. The framework is supported by a newly introduced dataset, DEMIX, which includes high-quality cinematic videos with corresponding audio tracks and is structured to support scalable training across varied video generation tasks. \nThe proposed approach demonstrates state-of-the-art performance in terms of video quality, text-video consistency, and audio-video alignment.", "questions": "1. Is it reasonable to use the same Wav2vec encoder to process all types of audio signals (speech, sound effects, and music)?\nWav2vec is primarily designed to extract semantic representations from speech. While this may be effective for speech, the authors should consider whether using alternative encoders for sound effects and music could yield better performance or more meaningful features.\n\n2. In the proposed \"multi-stream temporal ControlNet\", I am particularly curious why speech and sound effects are mixed as input.\nIn clean talking-head scenarios, sound effects may interfere with accurate lip synchronization.\nThe authors should validate this design choice and provide a reasonable explanation for combining these audio streams.\n\n3. How does the method handle multiple speakers? (Figure 1, second row). How are different speech signals distinguished and mapped to corresponding face regions? Would this setup lead to interference in visual information when speech overlaps or when there is temporal misalignment?\n\n4. Is it sufficient to divide the input audio signal into only three components: speech, sound effects, and music?\nThe current categorization overlooks other potentially important audio modalities such as ambient sounds and reverb/spatial cues. \n\nIf the authors can provide clear and detailed responses to the concerns raised, I would be willing to reconsider and potentially raise my overall score.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths \n\n- The work is overall very clearly written and expressed. \n\n- The author provides a new dataset DEMIX, and the corresponding view and experiment is supported\n\nWeaknesses\n\n- The embeddings are uniformly applied to all frames via global style injection to control visual emotion (Lines 56–57).\nIs such global control reasonable? For long-sequence video modeling, mood distribution may not be uniform over time.\nThe authors are advised to provide a proper reference for the statement \"Since visual mood typically covers the entire video clip\" instead of relying on subjective assumptions. \n\n- The \"multi-stream temporal ControlNet\" essentially injects audio information into the DiT via cross-attention. This approach appears overly simplistic and lacks novelty.\n\n- The paper emphasizes the extraction of features from the music track to enhance visual mood. However, the experimental section does not include any objective evaluation metrics related to emotion or mood to support this claim.", "was_revised": true, "final_justification": "Although the model design of this paper is somewhat blurred and lacks novelty, the authors' well-reasoned explanation clearly addresses these issues. \n\nBesides, the choice of metrics requires careful consideration (including emotion/mood evaluation and multi-speaker AV-sync). \nThe evaluation of Emotional aligning is still a bit far-fetched by choosing FVD whether close to GT.\n\nOverall, this paper is very meaningful for community, and I'm pleased to have improved my score to \"4 Borderline Accept.\" More importantly, I hope the authors can public the source code to facilitate better community evaluation and development.", "rebuttal": "We thank the reviewers for their insightful and constructive feedback.\n\nIn accordance with the rebuttal policy, we are unable to include new qualitative results (e.g., generated videos) or external links. We assure the reviewers that, should the paper be accepted, the camera-ready version will be thoroughly updated to include all additional experiments and qualitative results discussed in this rebuttal.\n\nWe now address the specific points raised by the reviewers.\n\n**W1: Global style control**\n\nWe agree that for long-sequence videos, mood distribution may not be uniform. However, we would like to clarify that our approach follows the common and well-justified practices of both affective computing and video generation.\n\n* Affective computing\n\nOur work focuses on generating video clips, typically under 10 seconds. In such durations, assuming a consistent visual mood is a standard approach. For example,  the LIRIS-ACCEDE [1] dataset (a general video dataset for affective content analysis) consists of video clips between 8-12 seconds, each annotated with a global Arousal-Valence value. The authors explicitly justify this choice, stating that using these durations \"greatly minimizes the probability that annotations are a weighted average of consecutive emotions\".\n\nFurthermore, this principle of applying a global label is also adopted in the highly influential DEAP dataset [2] for music videos, where even one-minute clips are assigned a single set of ratings (Valence, Arousal, Dominance, etc.) for the entire duration.\n\n* Video generation\n\nIn our MTV framework, music features serve as a representation of the overall aesthetic, which is analogous to a global style for the generated video. Enforcing such global style consistency is a central challenge and a standard practice in recent video editing models to ensure temporal coherence. \n\nFor example, FateZero [3] \"to warp the middle frame for attribute and style editing\" to guide the entire sequence, while Render-A-Video [4] applies \"cross-frame attention to all sampling steps for global style consistency\". Notably, both methods utilize a single text description as a global style command for all frames, which is conceptually similar to how our framework processes a single music feature stream as a global guide.\n\n[1] Deap: A database for emotion analysis; using physiological signals. IEEE transactions on affective computing, 2011.\n\n[2] LIRIS-ACCEDE: A video database for affective content analysis. IEEE Transactions on Affective Computing, 2015.\n\n[3] Fatezero: Fusing attentions for zero-shot text-based video editing. ICCV, 2023.\n\n[4] Rerender a video: Zero-shot text-guided video-to-video translation. SIGGRAPH Asia, 2023.\n\n**W2: Cross-attention**\n\nThe core novelty of our multi-stream temporal ControlNet lies in demixing the audio into speech, effects, and music tracks. This allows user-provided audio to precisely control the generated video's lip motion, event timing, and visual mood, respectively. To achieve this, we design an interval stream for specific feature synchronization and a holistic stream for overall aesthetic presentation. As a result, our MTV framework achieves significantly better performance than the comparison methods (Tab. 2)\n\nAdditionally, the injection module is more sophisticated:\n\n* The interval stream employs interval interaction blocks to model the interplay of speech and effects within each time interval using self-attention, which maintains coherence with inferred semantic features. These interacted features are then injected via the interval feature injection mechanism. Unlike standard cross-attention that applies a condition to all frames, our approach injects each audio feature only into its corresponding video time interval, enabling precise temporal synchronization.\n\n* The holistic stream uses a holistic context encoder to extract music features as video style embeddings. To ensure visual consistency, an average pooling is applied to merge all intervals. Subsequently, we estimate scale and shift factors from these holistic music features and inject them into all intervals via the holistic style injection formula (Eq. 3). Notably, no cross-attention is applied in this stream.\n\n**W3: Visual mood**\n\nWe have adopted the Audio Consistency (Audio-C) metric, which uses ImageBind embeddings to calculate the cosine similarity between video and audio features. A higher alignment score indicates the generated video features are more semantically consistent with the audio. We further consider the FVD as an indirect indicator, which measures the distribution gap between generated videos and ground truth videos. A significant mismatched mood would result in a poorer FVD score.\n\nWe further conduct an additional ablation study, where we remove the music track. As shown in the table below, this leads to a degradation in both FVD and Audio-C scores, demonstrating the important role of the demixed music track.\n\nIf the reviewer finds any appropriate metric, we would be glad to add its quantitative results.\n\n| Method | FVD ↓ | Temp-C (%) ↑ | Text-C (%) ↑ | Audio-C (%) ↑ | Sync-C ↑ | Sync-D ↓ |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| W/o music track |695.88 |95.17 | 26.40 | 25.69 | 3.07 | **9.42** |\n| MTV | **626.06** | **95.40** | **26.55** | **26.22** | **3.17** | 9.43 |\n\n**Q1: Alternative audio encoders**\n\nIn our empirical experiments, we find that using different audio encoders produces similar performance. This finding is consistent with previous Visual-Animation [6] (see its Appendix 6.7).\n\nWe conduct an additional ablation experiment where we replace Wav2Vec with Beats for both the effects and music tracks. As shown in the table below, this ablation achieves comparable performance, suggesting that our current choice of Wav2Vec is a robust and effective one for this task.\n\n| Method | FVD ↓ | Temp-C (%) ↑ | Text-C (%) ↑ | Audio-C (%) ↑ | Sync-C ↑ | Sync-D ↓ |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| Beats | **598.53** | **95.91** | 26.25 | 25.28 | 3.02 | 9.52 |\n| Wav2Vec |  626.06 | 95.40 | **26.55** | **26.22** | **3.17** | **9.43** |\n\n[6] Audio-Synchronized Visual Animation. ECCV, 2024.\n\n**Q2: Mixed speech and sound effects**\n\nThe speech and effect tracks are not mixed into a single input. Specifically, we apply a self-attention mechanism across the features of both the speech and effect tracks. This allows them to model their interplay at each time interval. However, they remain as two separate tracks after this step. Subsequently, these two tracks are injected into the video features independently using two separate masked cross-attention modules.\n\nWe recognize that if two audio control signals are applied without any interaction, their independent features could create conflicting instructions and interfere with sensitive tasks like lip synchronization. Therefore, our goal is to construct a coherent control representation while retaining the distinct nature of each track. This is inspired by real-world scenarios where visual events are driven by a combination of sounds (e.g., a person speaking while their footsteps are heard). By modeling the interplay, our network can learn to produce a unified semantic feature that guides the larger video context (e.g., both lip movements and body posture).\n\nWe further conduct an additional ablation study, where we remove the shared self-attention module that models the interplay and instead apply self-attention to each track individually. As shown in the table below, this ablation leads to a degradation in both lip-sync scores. Therefore, our interplay self-attention module is the solution to potential interference, instead of a source of it.\n\n| Method | FVD ↓ | Temp-C (%) ↑ | Text-C (%) ↑ | Audio-C (%) ↑ | Sync-C ↑ | Sync-D ↓ |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| W/o Share | 683.10 | 94.96 | 26.43 | 25.64 | 2.64 | 9.52 |\n| MTV | **626.06** | **95.40** | **26.55** | **26.22** | **3.17** | **9.43** |\n\n**Q3: Multiple speakers**\n\nIn our specific implementation, we introduce an additional speech track for a second speaker during the multi-character training stage. The weights of the second speech track are initialized with those of the first speech track, which is pre-trained on a single-character subset. To handle multiple speakers, we perform speaker diarization using Scribe to distinguish active speaker segments during data processing (Line 124-126). After that, we crop the segments for the first and second speakers and feed the cropped audio into two individual speech tracks. This allows our MTV framework to handle speaking overlaps between multiple speakers.\n\nTo map a speech track to its corresponding face region, we create a template to structure text descriptions so that the correspondence is adaptively learned (Line 203-210). These text descriptions explicitly state which speaker is active and their turn-taking order.\n\nAs our model uses audio to control the video generation, the generated video is inherently aligned with the provided audio. Therefore, temporal misalignment does not occur.\n\n**Q4:  Three components**\n\nWe demix audio into three components: speech, sound effects, and music. This methodology aligns with the standards of the CDX'23 challenge, and we adopt its leading models. In practice, this framework separates human voice into the speech track and instrumental accompaniment into the music track. All remaining audio components, including crucial modalities like ambient sounds and reverb, are thereby categorized as the effects track.\n\nTo evaluate the information integrity of this process, we sum the three demixed audio tracks and calculate the L1 distance against the original audio. The table below presents the mean L1 distance, which demonstrates the high fidelity and information completeness of our separation.\n\n| Speech | Effects | Music | Speech+Effects+Music | \n| :--- | :---: | :---: | :---: | \n|0.008156 |0.019122 |0.017116 |**0.000863** |"}, {"review_id": "BX2T4KkKRL", "reviewer": "Reviewer_3d1R", "summary": "This paper introduces MTV for generating video synchronized with audio input by explicitly separating audio into three tracks: speech, effects, and music. This separation allows precise control over lip motion, event timing, and visual mood. \nIn addition, the authors introduce DEMIX, a new large-scale dataset composed of cinematic video clips paired with demixed audio. \nThe paper also proposes a new component, the Multi-Stream Temporal ControlNet (MST-ControlNet), designed to handle these separate audio streams and integrate their signals into a text-to-video diffusion model. The paper reports state-of-the-art results across several metrics on audio-sync video generation, including video quality, audio-video synchronization, and consistency with text descriptions. The authors also provide extensive ablations.", "questions": "1. How are the other 2 baselines (TempoTokens and Xing et al.) evaluated? Are they retrained on the DEMIX dataset?\n2. Could the authors provide more details on the fine-tuning of MM-Diffusion? Does it follow the same multi-stage training process on all five subsets of DEMIX? \n3. Could the authors provide examples where the audio demixing is not perfect and show how this negatively impacts the final video?\n4. Could the authors provide examples with multiple people and complicated scenarios for text descriptions?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This paper targets a meaningful and useful gap in video generation and audio-video synchronization, especially in the field of cinematic video.\n2. This paper builds a new dataset, DEMIX, with a large-scale and careful filtering, which is a solid contribution to the community. \n3. The paper provides remarkable quantitative and qualitative results over three baseline models, indicating the effectiveness of the proposed method.\n4. Paper writing is clear and easy to understand.\n\nWeaknesses:\n1. The proposed model is heavily dependent on a list of existing models. For example: \n(1) Audio track separation has a significant impact on the model's overall performance (Tab 2 ablation study), while it is heavily bottlenecked by the demixing tools (MVSEP, Spleeter). The authors didn't provide failure cases when audio is noisy and is not cleanly separable, which is quite common in real-world scenarios. \n(2) The proposed model applies CogVideoX as base model, wav2vec for audio encoding, and T5 for text embedding. I believe the novelty of architecture mainly lies in module assembling. Given that, it is unclear how much of the performance gain comes from the proposed method instead of pretrained models. More ablations on choosing different pre-existing models are suggested.\n(3) Given the complicated pipeline that includes multiple external models, I have concerns on model's reproducibility.\n2. From my understanding, the text template is fixed (Sec 4.3), which lacks generalizability when the scenario is complicated to describe. There lacks such examples (e.g, 3 or more people) discussed in the paper. There lack of discussion on limitations of text descriptions, which I believe has a high impact on the overall performance and control quality. \n3. Concerns about comparison fairness. It is unclear to me if the other 2 baselines (TempoTokens and Xing et al.) are retrained on the DEMIX dataset or not. Without consistent training and evaluation approaches, the fairness of performance comparisons is questionable. Experiments on public standard benchmark are suggested.", "was_revised": true, "final_justification": "I have raised my score for this paper from 2 to 4.\n\nMy initial score (2: Reject) was due to significant concerns about comparison fairness, the novelty of the contributions, the model's reproducibility, and the lack of experiments to show the model's robustness.\n\nThe authors' rebuttal was convincing. They addressed every major issue by providing extensive new results, which included: (1) Experiments on different source separation levels. (2) Evaluating on public benchmarks to show generalization. (3) Conducting new ablations with a different T2V backbone (Wan14B) and different pre-existing models. (4) Adding a human evaluation study. (5) Experiments on comparison fairness. \n\nAlso, my follow-up questions and concerns (training schedule, data augmentation, and training on noisy separations) were clearly addressed. \n\nThe author also claimed that all source code and pre-trained models will be released.\n\nBased on these, I am now much more positive about this work, and therefore raised my score from 2 to 4.", "rebuttal": "We thank the reviewers for their insightful and constructive feedback.\n\nIn accordance with the rebuttal policy, we are unable to include new qualitative results (e.g., generated videos) or external links. We assure the reviewers that, should the paper be accepted, the camera-ready version will be thoroughly updated to include:\n\n1. All additional experiments and qualitative results are discussed in this rebuttal.\n\n2. A public link to an open-source demo showcasing our MTV framework as a practical video generation tool.\n\nWe now address the specific points raised by the reviewers.\n\n## ***Weakness***\n\n**W1-1: Unclear source separation**\n\nWe agree that audio demixing tools are not always perfect. However, we would like to clarify that all of our reported results are generated from these imperfectly separated audio tracks, not from idealized clean inputs. As shown in Tab. 2 and Fig. 3, our framework achieves state-of-the-art performance across six standard metrics, directly demonstrating its inherent robustness in real-world scenarios. This stems from our demixing filtering strategy (Lines 117-122). We explicitly filter out extremely poor-quality samples during training, which enables our model to learn precise audio-visual synchronization,  while the remaining unclear separations serve as a form of data augmentation that enhances robustness.\n\nWe additionally create three quality-based subsets (Level 1 being the highest quality, Level 3 the lowest), with each subset containing 20 samples. The quality level for each sample is determined by a consensus rating from 5 human volunteers. We also evaluate a \"Random\" subset, which is randomly sampled from the entire test set for baseline comparison. \n\nAs the results below show, our model is robust to the quality of the demixed audio. Its performance on metrics like FVD and Text-C shows only minor fluctuations across the quality levels, while audio-visual correlation metrics (Audio-C and Sync-C) are even slightly better on lower-quality input. This may suggest that our model effectively focuses on audio cues that persist even when the demixed audio is not perfectly clean. Notably,  with any-level quality inputs, our model remains superior to all baselines evaluated on the Random subset. Therefore, our multi-stream approach provides robust gains overall.\n\n| Demix quality | FVD ↓ | Temp-C (%) ↑ | Text-C (%) ↑ | Audio-C (%) ↑ | Sync-C ↑ | Sync-D ↓ |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| MTV (Level1) | **617.23** | 95.14 | **27.03** | 24.81 | 2.88  | **8.99** |\n| MTV (Level2) | 665.18 | 95.60 | 26.66 | 26.01 | 3.13 | 9.29 |\n| MTV (Level3) | 638.92 | **95.88** | 26.52 | **26.87**| **3.46** | 9.41 |\n| MTV (Random) |  626.06 | 95.40 | 26.55 | 26.22 | 3.17 | 9.43 |\n| MM-Diffusion (Random) | 879.77 | 94.15 | 15.61 | 5.43 | 1.53 | 11.21 |\n| TempoTokens (Random) | 795.88 | 93.13 | 24.68 | 6.71 | 1.45 | 10.48 |\n| Xing et al. (Random) | 805.23 | 93.30 | 24.51 | 7.30 | 1.55 | 10.50 |\n\nFinally, to completely bypass this separation challenge, we further developed a demo. It uses Qwen3 to interpret user prompts into audio descriptions, synthesizes them into perfectly clean audio with ElevenLabs, and then generates a synchronized video with our MTV framework. This presents a complete text-to-video pipeline to avoid potential source separation errors entirely.\n\n**W1-2: Different pre-existing models**\n\nWe create two new baselines by swapping key pre-trained components to evaluate how much of the performance gain comes from our proposed method.\n\n1. Wan14B with umT5. Since Wan14B shares a similar DiT-based structure with CogVideoX, we can integrate our proposed MTSControlNet into it without architectural changes. Specifically, our interval feature injection and holistic style injection modules are added after each text cross-attention layer. The quantitative results below show this baseline achieves better performance on video and text metrics (FVD, Temp-C, Text-C) due to the stronger capabilities of the Wan14B and umT5 models, while achieving comparable performance on all audio-related metrics (Audio-C, Sync-C, and Sync-D). This demonstrates that our MTSControlNet maintains its high performance on audio alignment and lip synchronization, independent of the specific T2V backbone.\n\n2. Beats. Since Wav2Vec is a common setting for speech encoding (e.g., Hallo3), this baseline only replaces Wav2Vec with Beats for both the effects and music tracks. As shown in the table below, this baseline achieves overall comparable performance, suggesting that our current choice of Wav2Vec is a robust and effective one for this task. This finding is consistent with previous work in Visual-Animation [1] (see its Appendix 6.7).\n\n| Method | FVD ↓ | Temp-C (%) ↑ | Text-C (%) ↑ | Audio-C (%) ↑ | Sync-C ↑ | Sync-D ↓ |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| CogVideoX+T5+Wav2Vec  |626.06 |95.40 | 26.55 | 26.22 | **3.17** | **9.43** |\n| Wan+umT5+Wav2Vec |**353.61** | **96.36** | **27.23** | **26.49** | 3.08 | 9.56 |\n| CogVideoX+T5+Beats | 598.53 | 95.91 | 26.25 | 25.28 | 3.02 | 9.52 |\n\nNotably, pre-trained T2V generators are typically tightly coupled with their specific text encoders. Therefore, replacing only the video generator or its text encoder separately would require fine-tuning the entire backbone, which requires prohibitive computational resources and is not feasible within the rebuttal period.\n\n[1] Audio-Synchronized Visual Animation. ECCV, 2024.\n\n**W1-3: Reproducibility**\n\nWe will release the source code and pre-trained models upon acceptance.  Should the reviewer have questions about any specific implementation details during the rebuttal period, we are glad to post the corresponding code directly in the rebuttal response.\n\n**W2: Complicated scenario**\n\nThe primary purpose of our text template is not to describe every detail of the scene, but to effectively specify the active speakers, their appearance, and the scene in which they are present. Based on our empirical observation that most one-shot scenarios have fewer than three active speakers, we initially set the upper bound of active speakers to two.\n\nWhile this number can be increased to three or more with fine-tuning, the true bottleneck for complex scenes lies in the token limit of the text encoder (i.e., T5's 512-token limit). This inherent constraint of the T2V backbone prevents users from providing long descriptions for every single person, regardless of whether a structured template is used. As a result, for scenarios with complex backgrounds (e.g., a crowd of people), these details can be concisely described and included in the \"scene description\" portion of our template (e.g., in a bustling city square full of people). We will include such cases in the final version.\n\nWe further conduct an additional ablation study by fine-tuning our model using only free-form text descriptions instead of our structured template. As the results below show, the video quality is degraded, demonstrating the effectiveness of our text template design.\n\n| Method | FVD ↓ | Temp-C (%) ↑ | Text-C (%) ↑ | Audio-C (%) ↑ | Sync-C ↑ | Sync-D ↓ |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| Free form | 638.89 | 95.42 | 26.45 | 25.17 | 2.92 | 9.45 |\n| Text template | **626.06** | **95.40** | **26.55** | **26.22** | **3.17** | **9.43** |\n\n**W3: Comparison fairness**\n\nIn our initial submission, we evaluate TempoTokens and Xing et al. using their publicly released weights. To make a fair comparison, we fine-tune all baseline methods (MM-Diffusion, TempoTokens, and Xing et al.) on our DEMIX dataset using their official training schedules, and re-evaluate them on our DEMIX dataset.  The updated results are presented below. Our proposed MTV framework still maintains a significant lead, although TempoTokens and Xing et al. show notable improvement in lip synchronization after fine-tuning.\n\n| Method | FVD ↓ | Temp-C (%) ↑ | Text-C (%) ↑ | Audio-C (%) ↑ | Sync-C ↑ | Sync-D ↓ |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| MM-Diffusion |879.77|94.15|15.61 | 5.43 | 1.53 | 11.21 |\n| TempoTokens | 883.35 | 92.61 | 24.21 | 6.06 | 1.85 | 10.26 |\n| Xing et al. | 847.85 | 92.79 | 24.69 | 6.11 | 1.78 | 10.03 |\n| Ours (MTV) |  **626.06** | **95.40** | **26.55** | **26.22** | **3.17** | **9.43** |\n\nWe further follow TempoTokens to conduct evaluations on both the Landscape and AudioSet-Drum datasets.  As shown in the tables below, our MTV framework still achieves significantly better performance.  Since neither dataset includes human talking, the lip synchronization metrics (Sync-C and Sync-D) are not applicable for these evaluations.\n\n#### **Comparison on Landscape datasets**\n\n| Method | FVD ↓ | Temp-C (%) ↑ | Text-C (%) ↑ | Audio-C (%) ↑ | \n| :--- | :---: | :---: | :---: | :---: |\n| MM-Diffusion | 807.65 | 94.74 |14.66| 16.59 | \n| TempoTokens | 797.33 | 94.67 | 21.73 | 18.86 |\n| Xing et al. | 838.03 | 94.71 | 21.04 | 18.70 | \n| Ours (MTV) | **697.51** | **96.98** | **25.35** | **23.37** | \n\n#### **Comparison on AudioSet-Drum dataset**\n\n| Method | FVD ↓ | Temp-C (%) ↑ | Text-C (%) ↑ | Audio-C (%) ↑ | \n| :--- | :---: | :---: | :---: | :---: |\n| MM-Diffusion | 1520.09 | 94.59 | 14.90 | 14.11 |\n| TempoTokens  | 1512.97 | 94.28 | 23.18 | 15.59 | \n| Xing et al. | 1589.46 | 94.49 | 23.73 | 17.84 | \n| Ours (MTV) | **1511.53** | **97.50** | **25.62** | **39.61** |\n\n**Q1: Retrained baselines**\n\nSee W3 please.\n\n**Q2: Finetuning MM-Diffusion**\n\nFor fine-tuning MM-Diffusion, we strictly follow the official training schedule: We first train the coupled U-Net, and subsequently train its super-resolution model. Both stages are performed on our full DEMIX dataset.\n\nWe do not apply our training schedule to comparison methods. As this is the contribution of our MTV framework, applying it to other methods would prevent a fair evaluation of our improvement.\n\n**Q3: Imperft audio demixing**\n\nSee W1-1 please.\n\n**Q4: Complicated scenarios**\n\nSee W2 please."}, {"review_id": "M12YJlZhJo", "reviewer": "Reviewer_1iMS", "summary": "This paper presents a new method to adapt video generation models for audio-to-video synthesis. It assumes that conditional audio is given as three separate tracks: speech, sound effects, and music. Under this setting, the proposed model extracts features from the three tracks via Multi-stream Temporal ControlNet and feeds them into the pretrained video generation model through additional cross-attention and feature modulation depending on the type of track. To train and evaluate the model, the authors created a new dataset comprising video, text, and demixed audio tracks. In the experiments, the proposed model outperforms prior audio-to-video models in terms of both video quality and audio-visual alignment.", "questions": "- Did the authors conduct any filtering process regarding license or copyright of the video when constructing the dataset?", "limitations": "The limitations of the proposed method are briefly discussed in Section 6.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The idea of using demixed audio for audio-to-video synthesis is interesting and convincing. \n  - Explicitly handling such separate audios should enable fine-grained controls to the video generation process, because how conditioning audio impacts on the generated videos should vary depending on the type of the audio.\n- The dataset construction process seems to be thorough, and the release of this dataset would benefit a lot the community in the field of audio-visual generation.\n- The proposed model empirically outperforms prior audio-to-video models significantly. I have checked the supplementary video and the performance gain is also obvious in perceptual quality.\n- The manuscript is well-written and easy to follow.\n\n## Weaknesses\n\n- I have a concern regarding license or copyright issues on the film-related videos mentioned in line 111.\n  - This is because the dataset construction process shown in Section 3 seems not to contain a filtering process to eliminate this issue.\n- I do not find crucial weaknesses regarding the design of the proposed method and the experiments. Shown below are just a few suggestions to strengthen the significance of this study rather than weaknesses. It is completely fine to ignore them during the rebuttal phase.\n  - In the proposed model, the extracted features \\tilde{f} are commonly used for every layer of DiT, and this design is essentially different from the standard setting of ControlNet for DiT adopted in popular models such as Stable Diffusion 3.5 and PixArt-delta, where the features are extracted in a layer-wise manner. I understand that the standard design cannot be directly applied to the case in this study due to the domain gap between video and conditional audio, but it would be worth trying layer-adaptive feature extraction also in the proposed model. I personally suspect that the above-mentioned design could contribute the trade-off mentioned in Section A.2.\n  - Regarding the generalizability, it would be great if the proposed method also works with more recent video diffusion models such as Wan, but I understand that trying this could require prohibitively large computational cost for training.", "was_revised": true, "final_justification": "I maintain my rating to be 5.\n\nI raised a concern on the copyright issue of the dataset. This was addressed in the rebuttal and resolved by the ethics review. Thus, I do not have any reason to downgrade my rating.", "rebuttal": "We thank the reviewers for their insightful and constructive feedback.\n\nIn accordance with the rebuttal policy, we are unable to include new qualitative results (e.g., generated videos) or external links. We assure the reviewers that, should the paper be accepted, the camera-ready version will be thoroughly updated to include:\n\n1. All additional experiments and qualitative results are discussed in this rebuttal.\n\n2. A public link to an open-source demo showcasing our MTV framework as a practical video generation tool.\n\nWe now address the specific points raised by the reviewers.\n\n\n\n**W1: Copyright**\n\nWe thank the reviewer for this important question. We take data sourcing and copyright considerations very seriously and have followed established academic practices.\n\n1. Data sources:\n\nAs detailed in our paper (Sec. 3, Lines 109-112), our DEMIX dataset is constructed from two main sources:\n\nExisting Public Datasets: We use several well-established benchmarks, including CelebV-HQ, MovieBench, Condensed Movies, and Short-Films 20K.\n\nPublicly Available Videos: We also collect data from YouTube, mirroring the widely-used academic datasets like VGGSound.\n\n2. Release justification:\n\nFor videos from existing datasets, we will follow their original protocols by providing the video IDs and other necessary metadata for users to retrieve the data from the original sources.\n\nFor videos we collected from YouTube, we will provide a list of the corresponding URLs and timestamps.\n\nFor our contributions, we will directly provide our generated annotations, including text descriptions and the demixed audio tracks.\n\nWe do not redistribute the original video/audio content. The dataset is compiled strictly for academic research.\n\n3. Licensing:\n\nTo ensure clarity, we will adopt a dual-licensing approach similar to VGGSound:\n\nOur code, annotations, and data-processing scripts will be released under the Apache 2.0 License.\n\nWe will explicitly state that the copyright of the underlying video and audio content remains with their original owners.\n\n\n**W2: Layer-adaptive feature extraction**\n\nWe sincerely thank the reviewer for this insightful suggestion and for proposing an interesting alternative design.\n\nOur current design (audio features are injected into all DiT blocks) follows the established practice in state-of-the-art talking face generation models (e.g., FantasyTalking). Since this approach has been proven effective and robust, we adopt it as our primary design.\n\nTo explore the reviewer's suggestion, we are implementing a layer-wise approach. We find that directly adapting the standard DiT blocks of CogVideoX results in out-of-memory errors, even using eight A800 80G GPUs. Therefore, we are proceeding by modifying our own MTSControlNet to perform layer-wise injection, where specific blocks of audio features are injected into corresponding blocks of the DiT backbone.\n\nDue to the time constraints of the rebuttal period, this experiment is currently in progress. We will post the results and our analysis as soon as they are available.\n\n**W3: Adaptability to additional T2V backbone**\n\nOur proposed MSTControlNet can be seamlessly integrated into various T2V backbones (e.g., OpenSora and Wan) without architectural changes, as they share a similar DiT-based structure. Due to time limitations, we only implement and test the integration with Wan14B. Specifically, our interval feature injection and holistic style injection modules are added after each text cross-attention layer. \n\nFor inference, MTV (Wan14B backbone) requires 180s to generate an 81-frame, audio-synchronized video at a 480P resolution on 8 NVIDIA A100 GPUs. We report the new quantitative results below.\n\n| Method | FVD ↓ | Temp-C (%) ↑ | Text-C (%) ↑ | Audio-C (%) ↑ | Sync-C ↑ | Sync-D ↓ |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: |\n| CogVideoX | 626.06 | 95.40 | 26.55 | 26.22 | **3.17** | **9.43** |\n| Wan14B | **353.61** | **96.36** | **27.23** | **26.49** | 3.08 | 9.56 |\n\n**Q1: Copyright**\n\nSee W1 please."}]

Author rebuttal:

Meta-review: Strengths
* DEMIX dataset pairing video and separated audio tracks (speech, effects, and music)
* Demixing and curation strategy for the dataset
* (debatable - see Weaknesses) Multi-Stream Temporal ControlNet with time interval injection for speech and effects, holistic style injection for music
* Results are convincing

Weaknesses
* Condition injection with a ControlNet-like architecture is common

During the rebuttal, the authors conducted many additional experiments, such as using Wan instead of CogVideoX as a T2V backbone and varying demixing quality, based on the reviewer's comments. These additional experiments will strengthen the paper with additional insights for the community.

Therefore, I recommend accepting the paper.
Decision: Accept (poster)