Aligning What Matters: Masked Latent Adaptation for Text-to-Audio-Video Generation
Authors: Jiyang Zheng
Siqi Pan
Yu Yao
Zhaoqing Wang
Dadong Wang
Tongliang Liu
Venue: NeurIPS 2025 poster ()
Keywords: Cross-modal Alignment
Text-to-Audio-Video Generation
Primary area: applications

Abstract: Text-to-Audio-Video (T2AV) generation aims to produce temporally and semantically aligned visual and auditory content from natural language descriptions. While recent progress in text-to-audio and text-to-video models has improved generation quality within each modality, jointly modeling them remains challenging due to incomplete and asymmetric correspondence: audio often reflects only a subset of the visual scene, and vice versa. Naively enforcing full alignment introduces semantic noise and temporal mismatches. To address this, we propose a novel framework that performs selective cross-modal alignment through a learnable masking mechanism, enabling the model to isolate and align only the shared latent components relevant to both modalities. This mechanism is integrated into an adaptation module that interfaces with pretrained encoders and decoders from latent video and audio diffusion models, preserving their generative capacity with reduced training overhead. Theoretically, we show that our masked objective provably recovers the minimal set of shared latent variables across modalities. Empirically, our method achieves state-of-the-art performance on standard T2AV benchmarks, demonstrating significant improvements in audiovisual synchronization and semantic consistency.

Aligning What Matters: Masked Latent Adaptation for Text-to-Audio-Vidd Video Generation
Jiyang Zheng Siqi Pan", Yu Yao', Zhaoqing Wang', Dadong Wang', Tongliang Liu" Syyney AI Center, The University of Sydney CSIRO. Data61 DDolby Laboratories
{jzhe5740, zwan6779)@umi.syde www.ii..dddeyeee.e edu. au {siqi. panpadolbyy....aoooaoooooaamaa {dadong wang) @data61. csiro. yyu.yao, tongliang. liul@ayddney edu.au au
Abstract
Text-to-Audio-iin Video (T2AV) generation aims to produce temporally and semanti- cally aligned visual and auditory content from natural language descriptions. While recent progress text-to-audio and texi-to-video models has improved generation quality within each modality, jointly modeling them remains challenging due to in- complete and asymmetric correspondence: audio often reflects only subset of the visual scene, and vice versa. Naively enforcing full alignment introduces semantic noise and temporal mismatches. To address this, we propose novel framework that performs selective cross modal alignment through a learnable masking mechanism, enabling the model isolate and align only the shared latent components relevant to both modalities. This mechanism is integrated into an adaptation module that interfaces with pretrained encoders and decoders from latent video and audio diffu- sion models, preserving their generative capacity with reduced training overhead. Theoretically, we show that our masked objective provably recovers minimal set of shared latent variables across modalities. Empirically, our method achieves state- of f--th--art performance on standard T2AV benchmarks, demonstrating significant improvements in audiovisual synchronization and semantic consistency.
Video
Audio
Visual
Auditory Features
Figure 1: Visual-auditory feature alignment essential in cetttto-audioviidee (T2AV) generation, yet assuming full correspondence between audio and visual modalities is often problematic. For example, visual elements like roads or buildings may not produce sound, while audio events such wind may lack visual presence. Aligning such mismatched features introduces semantic noise, resulting in reduced cross-modal consistency and temporal mismatch in the generated outputs.
'Work done during an internship Dolby, 'Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Recent advances in multimodal generative models 24, 28, 37, 39, 48,48, 42] have enabled high-quality content creation across text, image, audio, and video modalities. While notable progress has been made in text-to- video [3. 10, 21, 13, 38] and text-to-audio [6, 26, 27, 32] generation individually, they are typically studied in isolation, leaving joint audiovisual generation from text largely underexplored. Text-to-Aude Audio- Video (T2AV) generation addresses this gap by aiming to synthesize audio and video streams that are both semantically and temporally aligned, conditioned on single text prompt. This involves not only generating high-quality content for each modality, but also ensuring that the output audio and video remain contextually consistent and synchronized.
Achieving this requires modeling cross-modal alignment, where both audio and visual representations capture the informative content conveyed by the other modality. To facilitate such alignment, existing approaches often project multimodal features into a shared embedding space [33, 41, 52]. This facilitates the model to capture joint semantics across modalities. However, forcing all audio and visual features to align can be problematic. In real-world settings, audio and visual streams may exhibit only partial alignment: audio may describe only parts of visual scene, visual frames may contain elements absent from the audio (See Figure 1). Enforcing full alignment under such conditions introduces mismatched information into the joint representations, resulting in semantically inconsistent or temporally desynchronized outputs during T2AV generation.
To address the challenge of partial correspondence between modalities, we introduce SAVA, a framework for Selective Audio-Visuaa Aligmment in texi-to-audio-vider generation. Comparing with existing approaches [31, 33, 41, 45, 52] that assume full alignment between audio and visual features, SAVA identifies and aligns only those latent components that are jointly predictive across modalities, while disregarding modality-specific information that could otherwise introduce noise or conflict. The overall pipeline, as shown in Figure 3. proceeds three stages: Align and Fine-ture, learns to map multimodal latents by selectively filtering out irrelevant dimensions in the latent space using a learnable mask, allowing the model to focus only on features that contribute meaningfully to both modalities. The alignment is learned through adapter networks applied to pretrained encoders. Then, we fine-tune the generator using the aligned multi-condition inputs. Inference, operates in a cascaded manner, projecting features from video and audio into an aligned subspace, and conditioning the corresponding generator on both the text and the aligned video/audio signals. This design enables synchronized and consistent audio-visud generation while preserving efficiency and modularity.
SAVA grounded in a causal view of multimodal generation, where audio and visual signals are generated from a mixture of shared and distinct latent factors. We provably show that the masked alignment objective recovers the minimal set of shared latent variables (those which constitute the true semantic interface between modalities). This not only ensures interpretability and robustness but also mitigates the entanglement issues observed prior alignment-hase based models. Our empirical results across diverse benchmarks confirm that SAVA significantly improves semantic alignment and temporal synchronization in T2AV generation, outperforming existing baselines. A brief review of related works is provided in Appendix A.
2 Problem Formulation
Auttto-Audoo--iiee (T2AV) generation requires accu- rate alignment between audio-visual representations to preserve meaningful cross-modal correspondence. In par- ticular, semantic misalignment, where visual and auditory components do not reflect the same underlying content, can mislead generative models and degrade the consis- tency of the resulting outputs. Our objective is to enable selective and reliable alignment by identifying and pre- serving only the semantically relevant components across modalities during training. To this end, we begin by re- viewing the text-to-audio-video generative process.
XyA
Figure 2: The data generative process of audiovisual data. Audio features ZA selectively derived from visual features 7, guided by learned mask m. Each modality-specife latent combines with residual noise to produce the outputs.
Data Generative Process As illustrated in Figure 2, we model the audiovisual data generation process using

---

structured causal model composed of three latent variable sets: video-specific latent variables Zy. audio-specific latent variables Z,, and cross-modal latent variables Zc. which encode shared content factors underlying both modalities. Z may include semantically grounded, temporally evolving entities that manifest both the visual and auditory domains (e.g., a barking dog or moving vehicle). In contrast, the poddalit--specifi latents Zy and A capture factors that are unique to the video and audio domains, respectively. The latent variables are causally connected and evolve over time, with each group at time step é potentially influenced by their own past states and the past states of other groups, Formally, the evolution of these latent variables follows:
Zi - Z 2z ) Z - ( 7 26
where the superscript "past" denotes historical latent states (e.g.. from time 1). and the arrows represent causal influence. These relationships reflect the potential bidirectional statistical and causal dependencies [44] across modalities.
The observable variables: video Xy and audio XA, are generated from their corresponding modality- specific latent variables in conjunction with the cross-modal latent:
Xv + {Zv, Zc}, XA (ZA, Zc).
This formulation reflects that while Zc captures semantically aligned and temporally correlated content, Zy and ZA may contain orthogonal information that should not be foreed into alignment. Therefore, when attempting to recover cross-modal structure, it is critical to distinguish shared factors from modality-specifff ones.
3 Methods
Selective Latent Alignment Let I2 E Xv, Xv S RTxHXWx3 be video clip and za XA. XA R its corresponding audio spectrogram. We extract frozen modality-specifff embeddings & = fxx)),, &a = (z where fo is a pretrained video VAE encoder [10] and fa is pretrained audio diffusion encoder [49, 26]. These raw embeddings may contain modality-speciffe noise and are not guaranteed to lie in a common semantic subspace. To expose the shared latent structure Zc C Rdd apply learnable reparameterizations. (i.e., adapter networks):
2g = (2x),
La = qA(a))
(3)
This projection step adapts the output of each frozen encoder and is subsequently trained to isolate cross-modal features. We then introduce two mask networks:
Mr, MA 2d + 10.
each taking both ã, and E, as input. Due to the semantic ambiguity and contextual diversity in audio-visual alignment, the relevant latent dimensions within the visual representation can vary depending on the specific context. For example, a single video clip may be paired with different types of audio, such background music or voiceover narration, each requiring attention distinct visual regions or semantic features. Accordingly, the masking function should be conditioned on both video and audio inputs. Conditioning on only one modality impairs the model's ability to disambiguate cross- cross-modal variations, leading to suboptimal or unstable mask learning.
Cross-Modal Reconstruction Let Sy S [d] and SA S [d] represent the indices of dimensions selected by the soft masks. After applying a thresholding operation, we obtain binary supports Sy S [d] and SA S [d], which indicate the dimensions where the mask value equals 1 (i.e., Sy = (i E [4] Mv æ.); = 1,, S. E [d] Ma 2a); = 1}). These binary masks are then used to construct the masked latent representations by retaining only the selected dimensions:
- Mv (ž., 2.)) 3 ž.,
MA ā.) Zas
(5)
and decode each using the corresponding latent diffusion decoders Aa and gv:
a. = -Z,),.)
2. f (ž..).
(6)
To ensure that the masked latent remains informative, we reconstruct each modality from the masked latent of the other. This reconstruction objective acts as constraint that prevents degenerate masking
3

---

Mese Blockers
Lacent Adaycation
Alignment
Video Segments
Alticer
Audio
Audio Segments
Aligm
mullicondiiion Rineftnting
Miceo
T2V
Encosen
Eheett
Decoder
T2V Duitusion
Xlipter
TVZA DiPfusion
TAZV Oiifusion
Video Decoder
Saape Ei Cascaded
Tost Ennodwe
tooingg
Video
Ausio
Denalding
TV24 Diffusion
Audin Decoder
Diflusion
Ensoide
T2A
Audoo
Enscall
Abaoda salg
Figure 3: Overview of our proposed T2AV framework. The system involves training a learnable mask that selectively aligns the latents of each modality. filtering irrelevant visual content (e.g., tree trunks) while preserving meaningful cues (e.g.. bamboo being eaten). The aligned representations are then used fine-tune the generator, adapting the multimodal conditions alongside the text condition, followed by generation through latent diffusion model.
solutions. Without it, the sparsity loss alone would encourage all-zero masks, as they trivially minimize the LI penalty while discarding all information [23, 50J. We reconstruct across modalities (i.e., audio from masked video latent and video from masked audio latent) rather than within the same modality. This cross-moss modal reconstruction forces the mask to preserve only the latent dimensions that predictive of the other modality, thereby isolating the shared semantic structure. As result, the model learns compact and meaningful latent supports that are truly cross-mas cross-modally informative.
3.1 SAVA-Diffusion
In this section, we present the implementation of the proposed SAVA-Diffusion framework for Text-to-Audio- Video (T2AV) generation (See Figure 3). The framework consists of three stages: (1) a masked latent adaptation stage in which we train aligned projections of video-audio latents via selective masking, and (2) fine-tuning stage that adapts the aligned latents as a joint condition alongside the text condition. (3) cascaded diffusion generation stage, where high-fidelity audio and video outputs are synthesized using latent diffusion models sequential orders.
Stage I: Masked Latent Adaptation In first stage, our method implements selective cross-modal alignment by learning to isolate the latent dimensions that are predictive of the other modality. We first obtain reparameterized embeddings ž, and &, from the frozen encoders [53, 26] and adapters. These are passed the coddalty--ppeciff mask functions, each conditioned on both modalities, to produce binary masks that filter the latent features. The masked latents are then decoded to reconstruct opposite modality. encourage a consistent embedding geometry between modalities [51], we further include direct alignment loss between ž, and E, prior to masking. This stabilizes training and promotes representational coherence across modalities. The total objective is:
Listal
1 N
Σ 1=1
.A @ =(-)) ev(a! oV (MA & 2(1)) + a Latign ( :0!2)
(7)
+1 + + ||alll))
where CA and by are cross-modal reconstruction losses, and the & regularization encourages sparsity the learned supports. Lalign measures the distance (e.g., normalized f2) between the unmasked latent representations, and a controls the alignment strength.

---

Stage II: Multi-condition Fine-tuning In Stage II, the TV2A and TA2V diffusion backbones are fine-tuned separately, each use the cross-modal latent learned in Stage I, while keeping the decoders .v.9A frozen. For audio, given the text embedding o fx())) and the aligned visual latent & = qv (2 we form CΛ = 24; Φ (A())) and adapt the conditioning pathway via LoRA [12], similarly, for video we form cv = 263 Φv from the aligned audio latent En (A ) the
objective is the diffusion loss:
LFT. = Ex Ex - (0 4 1
(ar 1 cv) !
CFTV = En,,.. 6 6
(8)
with LoRA parameterization W = W + BA on selected cross-attention or FiLM layers. The total objective is not coupled during optimization, instead, we run distinct trainings. LoRA only on conditioning layers (mid-block and a few down/up blocks), and all backbone convolutions and decoders frozen preserve pretrained priors while teaching each model, in isolation, respond its new cross-modal condition. In addition, we fine-tune the individual T2A and T2V models to further enhance generation quality across modalities for the inference pipeline.
Stage III: Cascaded Diffusion Inference Building on the aligned latent representations from stage I and fine-tuned diffusion models from stage II, we generate video and audio in a cascaded manner using independently finetuned single-modal diffusion models (T2A, T2V) [53, 26] and multi-model diffusion models (TA2V, TV2A). As illustrated in Figure 3, the process begins by generating a video from a text prompt using a T2V diffusion model. The resulting visual latent = is then adapted through a lightweight projection network Pa. producing an audio-guiding latent &, that encodes visually grounded cues. This latent conditions the subsequent audio generation, serves as a supervision signal finetune the audio diffusion model for improved semantic coherence. Similarly, we can generate video conditioned on both audio and text latents. By structuring the process in this cascaded fashion, we ensure that the audio is aligned with the generated visual/audible content. The detailed formulation of the reverse diffusion process for both modalities is provided in Appendix C. Notably, the pretrained diffusion encoders remain frozen during Stage I, and only the adapters are updated; fine-tuning of the diffusion models is performed in Stage II to enhance generation quality while maintaining modularity and efficiency.
4 Theoretical Analysis
In this section we show that our masked cross-modal reconstruction with an li-penalty provably recovers exactly the shared latent factors between video and audio, i.e. the minimal Markov blankets on the pretrained features, even when those features are entangled. By faithfulness and d-separation on the latent DAG [35] over (Zv, Zy, Xv. XA), there exist unique index sets St. S [d] S S [d], which are the minimal Markov blankets of XA in Zv and of Xv in Z A. respectively. Equivalently, st is the smallest subset satisfying that conditioning on [vv i E St renders all other latent coordinates irrelevant XA- The analogous property holds for S)
To make precise what it means for two sets of latent factors to capture all and only the shared information, we introduce the following definition.
Definition 1 (Minimum Sufficient Latents). Given index sets Sv, SA S [d]. we say that the pairs
(55, 25t) are Minimum Sufficient Latents they satisfy
r( XA) I(Z XA), 1 I(v,; XA
2 0 Vj $ Sv.
1( A Xv) = I Xv), I(ZA;: Xv A = 0 vj $ SA
Key Assumptions We require four conditions (see Appendix B for formal definitions):
1. (DAG & d-Separation) There is a latent DAG over (Zv, AA, Xv, XA) whose minimal Markov blankets S correspond to the truly shared factors.
wiieck-wise Reparameterivation) The class of invertible maps ;v, QΛ is rich enough that there exists a reparameterization under which the shared block St (resp. Si)) becomes an axis-aligned subset St (resp. $t)) of the coordinates.
5

---

3. (Decoder Universality) The decoder families Qu. Qgy can approximate any conditional distri- bution, so that minimizing cross-entropy equivalent to minimizing true conditional entropy.
4. (Mask Universality & Penalty- The masks implement any support selection per example, and the sparsity weight À lies strictly between the smallest shared-factor contribution and the largest non-shared contribution (see Assumption 4).
Above assumptions are commonly used. First, a latent-variable DAG with faithfulness (Assump- tion 1) underlies most generative models in vision and audio, and the Markov blanket then exactly characterizes the shared information. This is the fundamental assumption in Causality [35]. Sec- ond, block-wise reparameterization (Assumption 2) merely requires that our invertible networks qv, QA have sufficient capacity to "whiten" or disentangle the small block of truly shared latents; in practice modern normalizing- miwoow and invertible-residua, architectures easily satisfy this. Third, decoder universality (Assumption 3) is standard in representation learning deep decoders with enough width and nonlinearity can approximate any conditional density arbitrarily well, so cross-ent eross-entropp minimization recovers true conditional entropy. Mask universality implies stipulate that mask networks are expressive enough to pick any subset of coordinates per example. All these universality have been supported by universal approximation theory of deep learning methods [14]. Finally, penalty-range requirement (Assumptions 4) implies that the sparsity weight À can be chosen (e.g. via cross- validaaiii to lie between the minimal utility of shared factor and the maximal spurious contribution of a non-shared factor. In practice, we can just make À be sufficiently small. Together, these common assumptions ensure our theoretical guarantees apply to many practical architectures.
Lemma 1 (Sufficientness of Reconstruction). Fix any invertible qy- Under Assumptions 1-4, any mask-decoder decoder pair (Mv.9A that minimizes E[- log Q9A (XA My @ Zv)l must satisfy, for every example, I 1z (zv (v))))); XA) = I(Zv: XA). In other words, the selected coordinates form sufficient statistic for XA-
This lemma shows that if we only optimize the reconstruction loss (cross-entropy) then the learned mask necessarily keeps all the information in Zv that is relevant to predicting XA. In other words, the selected subset of coordinates forms sufficient statistic for the audio modality, capturing every bit of shared information from the video embedding.
Lemma 2 (Sparsity- Induced Minimality). Fix any invertible qv. Under Assumptions 3-4, the joint minimizer (Mt. g%) = arg minMv.9. .94 [E| log Qoa (XA My Zv)] +A XB[I| [ satisfies. for every example,
St (Zv. 2A)= S4. 1(2vxiXA That is, the mask prunes away non-shared coordinates, recovering exactly the minimal shared block.
-03 ) St.
(9)
This lemma establishes that once we add sparsity penalty on the mask, the model discards every coordinate that does not uniquely contribute cross-modal reconstruction. The result the minimal subset of features which are precisely the shared latent block. Therefore no redundant or modality- specific information remains.
Theorem (Global Block- Alignmeet and Recovery). Under Assumptions 2. and 4 the global minimizer of Objective 7 yields (Z 5/5)) that satisfies Definition 2.
This theorem combines the sufficiency and minimality results in both directions (i.e., video and audio (oddoo)video) and shows that our simple mask-and-reconstrucc framework provably extracts exactly the shared latent variables and eliminates all modality-speciff components.
Experiments
5.1 Experiment Setup
Dataset We conduct experiments on two benchmark datassts:: VGGSound [5] and AudioCaps [22]. VGGSound comprises approximately 200K 10-second video clips spanning 310 sound classes, with strong audio-visual correspondence ensured by the presence of visible sound sources. Following the protocol in [52], we sample 5k and 3K clips from the train and test split, respectively, and annotate

---

Animatediff
AudioLDM
Diff-Foley
TAVDiffusion
Ours
that could represent wolf of "kept" borking
Temporal evolution
brighter that could the
better environmental context
complexity that would the
Screaming the wolf with white head and yellowish brown lay ground. golden leaf follen ground kept barking it really loud.
The white ambulance &s traveling quickly. buildings and parked automoblles the alongside the
Text Prompt
wolf
Figure 4: Text-to-Audio Cao--Audoooveeee Video generation results. We the same prompt in [33] for demonstration and compare our method against multiple baselines (Animatediff [8], AudioL.M [26]. Diff-Foley [31], and TAVDiffusion [33]). Compared to prior methods, our approach (unidirectional
setting as illustrated) produces higher quality and aligned video and audio content.
VGGSound+
AudioCaps
Method
FVD
FAD
AVHScore
CAVPSIM
FVD
FAD
AVHSccrre
CAVPSIM
Twoo-Straam.
768.5
6.29
0.058
0.104
961.4
7.36
0.041
0.165
CasC-Diff
708.5
7.53
0.141
0.126
961.1
9.51
0.092
0.192
TAVDid [33]
956.3
8.91
0.162
0.098
1131.9
8.43
0.105
0.182
CoDi [41]
709.4
8.36
0.108
0.149
902.5
9.07
0.098
0.211
JavisDiT [29]
697.4
6.17
0.153
0.140
801.2
7.55
0.104
0.207
Unidirectional
662.9
5.49
0.206
0.165
817.6
7.32
0.142
0.230
Bidirectional
701.4
0.217
0.183
852.4
0.157
0.242
Table 1: Quantitative comparison. Our method outperforms existing baselines in both generative quality metrics and alignment metrics, demonstrating improvements in fidelity as well as cross-modal consistency. For the unidirectional setting, directly adopt the fine-tuned T2V model for video generation. The generated audio for both the bidirectional and unidirectional settings identical.
them with text prompts using VideoBlip [54], adopted in [33]. AudioCaps consists of 46K audio clips paired with human-written captions sourced from AudioSet, and serves standard benchmark for audio-language grounding. We also sample 5K paired clips from the training split. To facilitate alignment learning and fine-tuning, we merge the training sets of both datasets, and perform evaluation separately on each test set.
Implementation Details To adapt the diffusion models the target data domains, we first fine-tune the video and audio diffusion models independently using the training set, respectively. For video generation, we employ the pretrained Cog V.d [53], and extract latent representations using its VAE encoder. For audio, we adopt AudioL.MM [26], which integrates pretrained CLAP encoder [49] for audio feature extraction. The latent dimensionality of aligned embeddings for audio generation is fixed at 512. Each generated sample has duration of 10 seconds, with video rendered at 16 frames per second and audio sampled 48 kHz. Our adapter and masking modules are implemented as multilayer perceptrons. For the masking mechanism, we evaluate both soft masks (sigmoid outputs as weights) and hard masks, obtained by thresholding 0.5. The loss weights and X2 are empirically set to 5 and O.I, respectively.

---

Vertical striations strumming
Text prompt: A mon playing guitar in the park.
Figure 5: Temporal alignment between visual motion and acoustic patters.. The strumming motion of the guitarist's hand aligns with vertical striations in the spectrogram, indicating synchronized transient audio events.
Evaluation Metrics We assess perceptual quality of the generated video and audio using Fréchet Video Distance (FVD) [43] and Fréchet Audio Distance (FAD) [37], respectively. Cross-modal semantic alignment measured by AVHScore [33], while CAVP similarity [31] evaluates temporal synchronization. For V2A performance, adopt the evaluation protocol from [52], including KI. divergence, Inception Score (ISc), Fréchet Distance (FD), and FAD.
Baselines We compare our method against two state-of-the-an art T2AV approaches: TAVDiffusion [33] and CoDi [41], using the same text prompts for all models. Additionally, we include: (1) a Cascaded
pipeline that uses Animatediff [8] for video generation followed by V2A-Mapper [45] for audio synthesis, and (2) a two-Stream approach in which video and audio are independently generated from the same text prompt using Animatediff and AudioLDM. For V2A generation, we also compare with the contrastive alignment method in [52] and SpecVQGAN [20]. a spectrogram-basse audio generator employing vector quantization.
5.2
Main Results
T2AV Generative Quality Table 1 presents the quantitative comparison of our proposed method against existing T2AV generation baselines on the text-labeled VGGSound and AudioCaps datasets. Our method consistently achieves the best performance across all reported metrics. On VGGSound, it reduces FVD and FAD to 662.9 and 5.49, respectively. reflecting significant improvements in both video and audio generation fidelity. Compared to the best- pest-performin baseline [41], our method achieve relative reduction of 6.5% in FVD and 16.0% in FAD. In terms of semantic alignment, our model achieves the highest AVHScore of 0.206 and 142 on VGGSound and AudioCaps, respectively, demonstrating improved correspondence between generated content and the input descriptions. A similar trend is observed in the qualitative results shown in Figure 4. For example, the wolf scenario, generated video better captures key semantic attributes such as the white head and vellow-brow body, while the visual background and ambient objects more faithfully reflect the textual prompt. Likewise, the ambulance scene displays correct object types, vehicle motion, and contextual elements like roadside greenery and traffic, showing high semantic fidelity across modalities.
Semantic and Temporal Alignment As shown in Table 1, our method achieves the highest CAVP- SIM scores on both VGGSound and AudioCaps, indicating improvement on cross-modal temporal alignment. This is further illustrated in Figure 4, where the temporal evolution of audio patterns (e.g., barking or sirens) closely corresponds with visual events. In the wolf example, distinct spectro- gram patterns align with repeated barking motions, while the ambulance scenario shows dynamic spectrogram textures matching siren intensity and vehicle motion. To further highlight this property, Figure 5 visualizes a man strumming a guitar, where the rhythmic hand motion aligns with vertically striated spectrogram features indicative of transient guitar strokes. These results collectively confirm that our method not only generates high-quality content but also preserves temporal synchronization across modalities. Additional examples are provided in Appendix D.
V2A Generation To further evaluate the effectiveness of cross- cross-model model alignment, we assess the video- to-audio (V2A) generation performance using subcomponent of our model. As shown in Table our approach outperforms existing V2A baselines: SpecVQGAN [20] and SeeHear [52] across most metrics, achieving better KI., ISc and FAD. These results indicate that our model effectively captures the shared semantic and temporal information between video and audio, enabling high-quality cross-
8

---

Method
K1.
ISct
FD!
FADL
Mask
FVD
FAD 4
AVHScore 1
CAVPSIM 1
SpecVQGAN [20]
3.290
5.108
37.269
7.736
SeeHear- Vani [52]
3.203
5.625
40.457
6.850
662.9
0.175
0.141
662.9
6.08
0.192
0.144
SccHcar-Full [52]
2.619
5.831
32,920
7.316
662.9
5.49
0.206
0.165
Ours
2.128
5.677
39,534
6,155
Table 3: Ablation study masking input modal- ities. D. no masking, direct alignment, O: only takes video modality embeddings the input, A: takes both video and audio modality embed- dings as the input.
Table 2: Video-to-Audio Generation Results. Our method outperforms existing V2A baselines across most evaluation metrics, demonstrating noticeable improvements in audio fidelity.
modal generation. The performance of this subcomponent further validates the robustness of our alignment strategy.
5.3 Ablation Study
Study on Mask Input We conduct an ablation study to assess the impact of different mask imput configurations on cross-modal generation quality in Table 3. We observe that when masking is applied (L), performance is sig- nificantly lower across all metrics, indicating that direct alignment without filtering introduces noise and misalign- ment. Conditioning the mask on video alone (O) yields moderate improvements, suggesting that video features contain partial cues for predicting shared content. How- ever, the best performance is achieved when the mask is conditioned on both video and audio embeddings (A), resulting in the lowest FAD and highest AVHScore and CAVPSIM. This confirms our hypothesis that observing both modalities enables the mask to more accurately iso- late cross- poos-modally relevant dimensions, thereby enhancing semantic consistency and temporal alignment in the gen- erated outputs.
KniScm CWTCGH Sagrarts
Effect of Temporal Segmentation We conduct abla- tion study to investigate the impact of different temporal segmentation strategies the performance of T2AV gen- eration. Specifically, given a 10-second vidco/audio clip, we divide the content into sub-clips of varying lengths (ranging from Is to 10s) and use the aligned segments for fine-tuning the pretrained diffusion models and training the alignment modules, including the masking functions and adapters. As illustrated in Figure 6, longer segment durations consistently improve generative quality, as measured by FAD and FVD, likely due to providing richer temporal context for fine-tuning the diffusion backbones. In contrast, shorter segments yield stronger performance in alignment metrics such as AVHScore and CAVPSIM, suggesting that temporally concise segments reduce misalignment and noise during cross-modal training. Based on this trade-off, we select 5-second segment length as balanced choice that supports both high generative fidelity and accurate audio-visual alignment.
Figure Ablation on different time seg- ment lengths. We find that longer seg- ments improve generative quality, while shorter segments benefit alignment.
6 Conclusion
We presented a multi-stage framework for text-to-audio-vidoo (T2AV) generation that addresses the challenge of semantic and temporal misalignment between modalities. Our method introduces a masked latent adaptation mechanism that selectively aligns video representations with audio embeddings using a learnable adapter and relevance mask. During inference, we leverage cascaded diffusion structure in which video is generated from text, and audio is subsequently synthesized conditioned on both text and the adapted video latent. This design ensures coherence across modalities while maintaining flexibility by reusing single-modal diffusion models. Extensive experiments demonstrate that our approach improves cross-modal consistency and achieves state-of-ther art results on multimodal generation benchmarks.

---

7 Acknowledgement
The authors would like thank Suqin Yuan and Muyang Li for their valuable feedback during the project. Jiyang Zheng is supported by the CSIRO Next Generation Graduates and Al for Missions PhD program. Tongliang Liu is partially supported by the following Australian Research Council projects: FT220100318, DP220102121, L LP220100527, LP220200949.
References
[1] Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. Musiclm: Generating music from text. arXiv preprint arXiv:2301.1 11325, 2023.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23733 2022.
[3] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 22563-22575, 2023.
[4] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing. Yaofang Liu, Qifeng Chen, Xintao Wang, al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprini arXiv:23 10. 2023.
[5] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2020.
[6] Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Text-to-audio generation using instruction guided latent diffusion model. In Proceedings of the 31st ACM International Conference on Multimedia, pages 3590-3598, 2023.
[7] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, 2023.
[8] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.0472. 2023.
[9] Jie Hong, Weihao Li, Junlin Han, Jiyang Zheng, Pengfei Fang, Mehrtash Harandi, and Lars Petersson. Goss: Towards generalized open-set semantic segmentation. The Visual Computer, 40(4):2391-2406 2024.
[10] Wenyi Hong. Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large- scale pretraining for text-to-video generation via transformers. In The Eleventh International Conference on Learning Representations, 2023.
[11] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In International conference on machine learning, pages 2790-2799. PMLR, 2019.
[12] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[13] Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEFF/VV Conference on Computer Vision and Pattern Recognition, pages 8153-8163, 2024.
10

---

[14] Guang-Bot Boang-Bin Huang, Lei Chen, and Chee-Kheone Siew. Universal approximation using incre- mental constructive feedforward networks with random hidden nodes. IEEE transactions on neural networks, 17(4):879-98) 892, 2024.
[15] Jiaxin Huang, Runnan Chen, Ziwen Li, Zhengqing Gao, Xiao He, Yandong Guo, Mingming Gong, and Tongliang Liu. Mllm-for3d: Adapting multimodal large language model for 3d reasoning segmentation. arXiv preprint wrXiv:2503181333 2025.
[16] Jiaxin Huang. Ziwen Li, Hanlve Zhang, Runnan Chen, Xiao He, Yandong Guo, Wenping Wang. Tongliang Liu, and Mingming Gong. Surprise3d: A dataset for spatial understanding and reasoning in complex 3d scenes. arXiv preprint arXiv:2507.077 00778, 2025.
[17] Rongjie Huang, Jiawei Huang, Dongchao Yang. Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin. and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pages 13916-13932. PMLR, 2023.
[18] Zhuo Huang, Chang Liu, Yinpeng Dong, Hang Su, Shibao Zheng, and Tongliang Liu. Machine vision therapy: Multimodal large language models can enhance visual robustness via denoising in-context learning. In Forty-first International Conference on Machine Learning, 2024.
[19] Zhuo Huang, Xiaobo Xia, Li Shen, Bo Han, Mingming Gong, Chen Gong, and Tongliang Liu. Harnessing oft-of-dine df-tsiibbbtttii examples via augmenting content and style. In The Eleventh International Conference on Learning Representations, 2023.
[20] Vladimir lashin and Esa Rahtu. Taming visually guided sound generation. arXiv preprint 1 2021.
[21] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang. Shant Navasardyan, and Humphrey Shi. Text2video-zer zero: Text-to-image diffusion models shot video generators. In Proceedings of the IEEE/CVF International Conference Computer Vision. pages 15954-15964, 2023.
[22] Chris Dongioo Kim, Byeongehang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generat- ing captions for audios in the wild. In NAACL-IIL,, 2019.
[23] Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial disentanglement for domain adaptation. In International conference on machine learning, pages 11455-11472. PMLR, 2022.
[24] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[25] Yexiong Lin, Yu Yao, and Tongliang Liu. Beyond optimal transport: Model-aligned coupling for flow matching. arXiv preprint wxXiv22555.2334 2025.
[26] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. In International Conference Machine Learning, pages 2 21450-21474. PMLR, 2023.
[27] Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang. and Mark D Plumbley. Audioldm 2: Learning holistic audio generation with supf-supe pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 2024.
[28) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
[29] Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Rongxin Jiang, Jiebo Luo, Hao Fei, and Tat-Seng Chua. Javisdit: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchronization. In arxiv, 2025.
[30] Run Luo, Renke Shan, Longze Chen, Ziqiang Liu, Lu Wang, Min Yang, and Xiaobo Xia. Vem: Vision concept modeling based on implicit contrastive learning with vision-language instruction fine-tuning arXiv preprint XXV::2504.19227. 2025.

---

[31] Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video- to-audio synthesis with latent diffusion models. Advances Neural Information Processing Systems, 36:48855 48876, 2023.
[32] Navonil Majumder, Chia- Yu Hung, Deepanway Ghosal, Wei-Ning Hsu, Rada Mihalcea, and Soujanya Poria. Tango 2: Aligning diffusion- text-to-audio generations through direct pref- erence optimization. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 564 572, 2024.
[33] Yuxin Mao, Xuyang Shen, Jing Zhang, Zhen Qin, Jinxing Zhou, Mochu Xiang, Yiran Zhong, and Yuchao Dai. Tavgbench: Benchmarking text to audible-vides generation. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 6607-6616 2024.
[34] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:21 2021.
[35] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: founda- lions and learning algorithms. The MIT Press, 2017.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8888888 8763. PmL.R, 2021.
[37] Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10219-10228, 2023.
[38] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, et al. Make-a- a- video: Text-to-video generation without text-video data. In The Eleventh International Conference on Learning Representations, 2023.
[39] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In The Twelfth International Conference on Learning Representations, 2024.
[40] Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi. Lei He, et al. Naturalspeech: End-to-end text-to-speech synthesis with human-level quality. IEFE Trunsactions Pattern Analysis and Machine Intelligence, 46(6):423444244 2024.
[41] Zineng Tang. Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 36:1 36:16083-16099, 2023.
[42] Gemini Team, Petko Georgiev. Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint 203 2024.
[43] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
[44] Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451-16467, 2021.
[45] Heng Wang. Jianbo Ma. Santiago Pascual, Richard Cartwright, and Weidong Cai. V2a-mapper: lightweight solution for vision-to-audio generation by connecting foundation models. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
12

---

[46] Zhaoqing Wang, Xiaobo Xia, Runnan Chen, Dongdong Yu, Changhu Wang, Mingming Gong. and Tongliang Liu. Lavin-dit: Large vision diffusion transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 20060-20070, 2025.
[47] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7623-7633, 2023.
[48] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. NEXI-GPT: Any-to-any multimodal LLM. In Forty-first International Conference on Machine Learning, 2024.
[49] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg- Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audin pretraining with feature fusion and keyword- to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
[50] Shaoan Xie. Lingjing Kong, Mingming Gong, and Kun Zhang. Multi- image generation and translation with identifiability guarantees. In The Fleventh International Conference on Learning Representations, 2023.
[51] Shaoan Xie, Lingjing Kong, Yujia Zheng. Yu Yao, Zeyu Tang, Erie P. Xing, Guangyi Chen, and Kun Zhang. Smartelip: Modular dision-language alignment with identification guarantees. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2025.
[52] Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7151-7161, 2024.
[53] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong. Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.00 00077, 2024.
[54] Keunwoo Yu, Zheyuan Zhang, Fengyuan Hu, Shane Storks, and Joyce Chai. Eliciting in-context learning in vision-language models for videos through curated data distributional properties. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20416-20433 Miami, Miami. Florida, USA, November 2024. Association for Computational Linguistics.
[55] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and
Qi Tian. Controlvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305. 13077, 2023.
[56] Jiyang Zheng, Weihao Li, Jie Hong, Lars Petersson, and Nick Barnes. Towards open-set object detection and discovery. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 3961-3970 2022.
[57] Jiyang Zheng, Jialiang Shen, Yu Yao, Min Wang, Yang Yang, Dadong Wang, and Tongliang Liu. Chain-of-focus prompting: Leveraging sequential visual cues to prompt large autoregressive vision models. The Thirteenth International Conference on Learning Representations, 2025.
[58] Jiyang Zheng, Yu Yao. Bo Han, Dadong Wang, and Tongliang Liu. Enhancing contrastive learning for ordinal regression via ordinal content preserved data augmentation. In The Twelfth International Conference on Learning Representations, 2024.
Appendix
A Related Works
Text-to-Audio- Video Generation Text-to-Audio- Video (T2AV) generation aims to synthesize audio and video streams that are semantically and temporally aligned, conditioned on single text
13

---

prompt. The task extends beyond text-to-video (T2V) and text-to-audio (T2A) generation by requiring consistency across modalities. Recent advances in T2V [4, 21, 47, 55, 25] and T2A [1. 17, 27, 32, 40] have enabled high-quality content generation in each modality. However, generating them independently often results in misaligned outputs, as the modalities are not conditioned on each other. A simple alternative is a cascaded approach, where one modality (e.g., video) is generated first and used to condition the other (e.g., audio). While this improves synchronization, it may propagate errors and lead to inconsistencies with the original text. To address these issues, recent T2AV methods propose joint modeling strategies. CoDi [41] unifies generation across multiple modalities in a single diffusion framework via aligning prompt encoders (text, image, video, audio) into shared input space using contrastive learning, with text as the central bridging modality. [52] aligns pretrained T2V and T2A models via shared semantic space using ImageBind. TAVDiffusion [33] adopts a two-stream latent diffusion model and addresses alignment via cross-attention and contrastive learning. Nevertheless, joint modeling of audio and visual modalities requires careful alignment of representations to preserve both semantic consistency and temporal synchronization. Our framework complements existing approaches by introducing targeted alignment mechanism that mitigates the impact of noisy or partial correspondences, leading to more faithful and consistent T2AV generation.
Cross-Modal Alignment Cross-modal alignment is crucial for integrating information from differ- ent modalities, facilitating tasks such as retrieval, recognition [56, 9I, and generation [19, 58, 57, 46). The goal is to project mpdelity-speciffc features into shared embedding space where semantically related inputs are closely aligned. In the vision-language domain [18, 15, 16, 30), CLIP [36] has become standard framework, while CLAP [49] and CAVP [31] extend contrastive alignment to audio-language and vision-audio pairs, respectively. ImageBind [7] further generalizes this approach to unify multiple modalities in a single embedding space. Such alignment modules are integral to conditional generative models [31]. While early approaches trained modality encoders from scratch, recent work shows that frozen foundation models can be effectively adapted using lightweight projec- tors [11, 34]. In jidoo-t--uddo generation, V2A-Mapper [45] learns a projection from CLIP to CLAP features using a simple MLP, enabling audio generation conditioned on vision without retraining large-scale models. Despite these advances, aligning the correct semantic content across modalities remains challenging. Representations often entangle modality-specifie and irrelevant information, leading to noisy alignment. SmarCLLIP [51] identifies this issue in vision-language models, showing that CLIP embeddings often entangle unrelated concepts due to coarse-grained alignment. These findings underscore broader challenge in multimodal generation: how to align information across modalities such that the learned representations do not introduce inconsistencies in the generated outputs. Our work addresses this by introducing masked adapter module that enables efficient and selective alignment between pretrained moodality-specifii encoders. By focusing aligmment on semantically relevant regions, our method mitigates noisy correspondence and improves consistency in T2AV generations.
B Theoretical Results and Proofs
B.1 Notations and Definitions
We introduce key notations and definitions as follows.
Let Xy E Ny (video) and XA E X' (audio) be two modalities. Pretrained encoders produce
Zv.a), We introduce learnable invertible reparameterizations
Żv = (Zv, (,, Ż = (ŽA. (2A.,...,AA..)) E R..
(10)
9V. : Rd + Rd,
(11)
and define
Zv = qv (2v),
Ż = (a(A))
(12)
We learn mask functions
My :Rd x Rd + fo, 1pd, MA : Rd x Rd + 10.
(13)
so that for each sample the model produces binary masks
(vv, ZA), Ma(Zv, 2A) E (0.
(14)
14

---

Denote their supports (or selected indices) by
Sy : Sv 2x) = {i: My (Z, = =1}.
(15)
SA = SA(Zv, ZA) {i: M MA, (Zv, -x)= =1]
(16)
Let z C Zy and 2 - ZA be the selected variables induced by learnable masks, i.e.,
ž (Zv,))išv.
zxA² = (ZAN))ESA.
Sv [d] \ Sv and S = [d] \ SA
(17)
for any set Sv - [d] and set SA S [d].
Write st _ [d] (resp. 5%22 for the true minimal Markov blanket of XA in Zy (resp. of Xv in Za). In other words, S is the smallest subset satisfying that conditioning on ZZ i E S renders all other latent coordinates irrelevant to XA. The analogous property holds for S
Reconstruction quality is measured via true conditional entropy:
H(Xa Z -Ellog )))) H(vv 2²²²) = =B p(vv [ Let Qos (Xx .) and QBv (Xv 1.) be decoder families. We optimize
(18)
Lv qv. gA) = BB-llgg Qas (XA Mv (Zv, ZA)= の 2v)] A (2v, (19) and symmetrically [a qv, MA) for audio-vvideo. Definition 2 (Minimum Sufficient Latents). Given index sets Sv, SA S [d]. we say that the pairs (Z AA) are Minimum Sufficient Latents if they satisfy
12 XA) 1( 5 XA), 1 (Zv,; Xa
2 Vj $ Sy:
1Z Xv) = 12xt, Xv), (Z2,s Xv ( = 0 vjj $ 5A
B.2 Assumptions
We introduce the assumptions required by our method as follows. Assumption 1 (DAG & d-Separation). The joint distribution of (Zv, ZA, Xv, x) according to DAG satisfying the global Markov property and faithfulness.
Hence for any Sy C [d].
TZZv, XA == Sv, and the same condition holds when V and A are interchanged. Assumption 2 (Block-wit wise Reparameterization), The joint function class for (qv, 9A) is rich enough that there exist invertible maps
X
3v VV
Sv
=
(20)
qv. Rd RR
(21)
5t S [d] satisfying and index-sets sets St.5t A
1( (Zv) XA) I (Z XA), 1 (Zv) XA at (2v) - Vj $ Sty
(22)
Assumption 3 (Decoder Universality). For any S C [d]. mingA E E[- log Qua (XA H(XA 2 and similarly for Xv 2
255) }
Assumption 4 (Mask Universality). The mask networks My, MA are sufficiently expressive to realize any mapping Rd x Rdd + {0, 1 i.e. choose any support S S [d] for each sample.
Assumption 5 (Penalty-Range). For any subset Sv C [d] and index i E [d], define
Av.(Sv) = 1(7v,,; XA 2(S0
which is the mutual information between Žv,; and XA conditioned the remaining variables 2(5v\10) =(vv : E S5 \ (i)). There exists constant X such that max A Av, ((d) < < < min Av.(5%) JESt
iESt
and the same condition holds when V and A are interchanged.
15

---

Note that above assumptions are common. Assumption 1 is fundamental assumption in causality [35]. Assumption merely requires that our networks qv. 94 have sufficient capacity "whiten" or disentangle the small block of truly shared latents. Assumption 3 assumes that deep decoders can approximate any conditional density arbitrarily well, so cross-entropy minimization recovers true conditional entropy. Assumption 4 implies stipulate that our mask networks are expressive enough to pick any subset of coordinates per example. All these assumptions 2, 3, 4 have been supported by universal approximation theory of deep learning methods [14]. Finally, Assumptions 5 implies that the sparsity weight À can be chosen to lie between the minimal utility of a shared factor and the maximal spurious contribution of non-shared factor. In practice, can just make À be sufficiently small.
B.3 Theoretical Results
The following lemma shows that, by minimizing the cross-entropy loss of decoder trained to reconstruct a short audio segments from the selected learned representations of video frames, one asymptotically recovers the conditional entropy of reconstructed short audio segments given those representations. The same result holds when swapping the roles of video V and the audio A.
Lemma 3 (Cross-Entropp Reduction to Conditional Entropy). Under Assumption 3, for any fixed mask function My and fixed qv, we have
min E E[-1ogg log Qun (Xx My ZA) [v)] DA
E 2[FT(XN =3
(23)
where Zy = qu (Zv) and Sv 3A) = support Mv (ăv, (AA).
Proof. Let
L(Mv 9A) E - Qga (Xa (2, 2v)
(24)
By the interchange of minima,
min L(Mv,9. = min min L(My ga)) My-sA
(25)
My 94
Fix any mask My. Then by Assumption (Decoder Universality),
min L(Mv gA) min E | 9^
[oc[-loge log Qua (XA 25y)
}
E HX 250).
(26)
Da
Note that (Xa (v 2x) の Zv z by definition, since the mask selects exactly those components. Therefore, the proof is complete.
The following lemma shows that any mask-decoder pair minimizing the cross-entropy reconstruction loss inevitably selects subset of video representations that retains the full mutual information with the audio segment, i.e., it forms a sufficient statistic for the audio segment. The same result holds when swapping the roles of video V and the audio A.
Lemma 4 (Sufficientness of Reconstruction) Fix any invertible qv. Under Assumptions 1-4, any mask- decoddr pair (Mv, .9A) that minimizes E1- Q9a (XA My @ Zv)] must satisfy, for every sample,
1 Sv :A))
I(Zv:XA)-
(27)
In other words, the selected coordinates form a sufficient statistic for X
Proof. For notational simplicity, we omit the arguments (2v, ZA) when writing (v ZA). Fix Qv and consider any minimizer (Mv, .A)) of the cross-entropy. By Lemma 3, this pair also minimizes E[ HXX 25r0 )]. Under Assumption 4, the mask My is expressive enough to choose the index set Sy arbitrarily for each sample. Hence the minimization decomposes per sample: for each (žv, EA).
we pick
Syv 2A) E arg min H H(AAl SC(d]
E 2
(28)
16

---

Recall that for any fixed Se.
H(X (/y) = H(XA) ;
xx),
(29)
By the causal faithfulness and causal Markov properties [35] (analogous to Assumption but applied to the variables produced by the neural network), it gives
1 XA) S X(v;
H(XA 25y) H(XA Zv).
(30)
Hence the unique minimizer of (XA ZZy) is any S satisfying
H(XA { H(A Zv),
(31)
which is equivalent to
T(Z XA) = I(Zv; XA)- Thus for each sample, T( T (Av. XA) = T(ăv; XA), completing the proof.
(32)
The following lemma shows that adding an E1peenll on the mask encourages sparsity: the optimal mask discards all non-shared coordinates and exactly recovers the minimal shared block of the variables produced by the neural network.
Lemma "Sparsity-Inducee Minimality). Fix any invertible qv. Under Assumptions 3-5, the joint minimizer
(M) arg min My
{E[- log Qus (Xx My @ 2v)] + XB[l
(33)
satisfies, for almost every sample,
(iv, BA) = St, I(2vjiX, Xx 2 Vj $ St.
(34)
That the mask prunes away all non-shared coordinates, recovering exactly the minimal shared block.
Proof of Lemma 5 (Sparsity- Induced Minimality). Fix qv. As before, by Decoder Universality
(Lemma 3) the joint minimization over (Mv, 9A) is equivalent
min E [X 2 A Svl]
(35)
My
Since My can choose Šv per sample (Assumption 4), we solve for each (žv, EA):
min SCId
f($)
where f($) = H 2 + A1S
(36)
For any j & 5, adding changes f by
f(š (i)) - - -(zrji 2500 + A.
(37)
By Assumption 5, ((2,, XA 2 (l.) < A, so f(S U ((}) > f(S) and no non-blanket
index is added. Similarly, for any i E S, dropping i changes by
S($ (1) (($) = (Zv. XA 2 21(0)) X.
(38)
and Assumption ensures this is positive for all i e St. Hence the unique minimizer is 5 5= =st and I [īvv;; XA 2 = 0 for j & St. as required.
The following theorem shows that, when jointly optimizing encoders, masks, and decoders with our bidirectional objective over both video and audio representations, the global minimizer precisely aligns and recovers the shared latent blocks-i.e. achieves exactly the block-alignment specified in Definition 2.
17

---

Theorem 2 (Global Block- Aoigmmetitm and Recovery). Under Assumptions 1, 2, 3, 4 and 5, the global
minimizer of Objective yields (Zy) A that satisfies Definition 2.
Proof. We decompose the total training objective into two symmetric parts,
L Lv-algv, Mv, Ca-v( MA.9v),
where, for instance,
=B[-log Lv-alqv,MM, Mv, = B log 10g (XA Mv (Zv, Za) @ qv (Zv)) +AB[[IMvli]]
1. Existence of an optimal block-aligned configuration. By Assumption 2, there exist ( M,) and 9 such that
(A) St, Cv-ald,, Mt (1 - (XA 23y + +118
The same argument applies to the audio-to-video term, yielding (g/1 M vtv).
2. Optimality of the shared supports. Fix any candidate (gv, ga). First, for fixed encoder Qv. Lemma 3 (Decoder Universality) shows that
min A Mv, = E{H(XA 2 + A
Lemma 4 then implies any minimizer My must satisfy
1 12 Sv (AA) = ((2v;
Next, we allow qv itself to vary. By Assumption 2, the encoder family contains some df that minimizes the conditional entropy [[ / )]. Lemma 5 ensures the unique sparsest choice of Sy is 8%. Altogether, when À is sufficiently small, the global minimizer of the video->audio term satisfy
ai at. MV(-) Sy.
An identical chain of reasoning on CA-VV yields
a = d/a-
M²( - S
Therefore, at the global optimum both pairs (Z x
thus coincide with the unique minimal
sufficient latents (Z ) so they satisfy Definition 1.
C Cascade Diffusion Model
Building on the aligned latent representations from Stage we generate video and audio in cascaded manner using independently finetuned single-modal diffusion models. As illustrated in Figure 3 (II), the process begins by generating a video from text prompt using video latent diffusion model. The resulting visual latent 2 is then adapted through lightweight projection network Pa. producing an audio-guiding latent én that encodes visually grounded cues. This latent, along with the original text embedding, conditions the subsequent audio generation. By structuring the process in this cascaded fashion, we ensure that the audio is temporally and semantically aligned with the generated video content. Notably, the pretrained diffusion backbones remain frozen during training, and only the adapters are updated, preserving modularity and enabling efficient adaptation to downstream multimodal generation tasks.
18

---

Animatediff
A
Diff-Foley
TAVDiffusion
Ours
As music in the bachground, be seen chatting and applouding the video, which various
people
pristine sky with few white clouds, white snow-capped birds chirping in the background.
Text Prompt
and attroctions from Disneyland. Visitors are shown enjoying rides, meeting characters, and exploring different themed lands throughout the park.
Figure 7: Additional Text-to-Audio- Video generation results compared with other baselines. We use the same text prompt as in [33] for our demonstration and compare the method against multiple baselines (Animatediff [8], AudioL.DM [26], Diff-Foley [31], and TAVDiffusion [33]).
Diffusion Formulation Let I, denote the input text prompt, which is encoded via pretrained text encoder fi(-) to obtain 2 = (2z). The video generation begins by sampling Gaussian noise 20 ~N(00, I), which is progressively denoised through the reverse diffusion process:
1 at āe ca. (0. 2) + +o, e~N(0,
2p
(39)
V1
Once the final video latent 2 is obtained, is projected into an audio-guiding latent &, Po(2,)) Audio generation is then conditioned on both &, and 2 using analogous denoising process:
- a₁ Ep, (2 Za; 20,1)) or e~ NO,
04
z
vI
(40)
The generated latents are subsequently decoded using pretrained decoders obtain the final outputs: i, = =g and a. ga (z..)
19

---

A black horse gallops Lhrough misty valley its hooves echoing loudly.
boy runs across frozen lake under the northern lights.
A marching banc parades down city street curing festival.
A cyclist races down mountain trail.
A lion roars in the middle the savennah.
A firefighter sprays water at burning building and fire alarm beeping.
Figure 8: Additional Text-to-Audio- Video generation results by our proposed framework.
20

---

A dog barking under the sun.
A dog barking in the snow.
man in blue suit playing guitar.
A man white suit playing guitar.
A panda eating bamboo.
An elephant eating bamboo.
Figure 9: Change of audible or non-audible attributes to the generative results.
D Additional Results
We present additional visualizations of our T2AV-generated results in Figure 7. Using the text prompts as those in [33]. we generate audio-video pairs and compare them against cxisting baselines. In the Disneyland scene, model produces vertically structured spectrogram features that plausibly correspond to discrete auditory events such as applause or exclamations, reflecting a diverse and dynamic soundscape. In the sky with bird chirping scene, observe consistent, rhythmic patterns in the spectrogram that align with natural bird calls, indicating accurate temporal grounding of audio events. These results highlight the model's ability synthesize semantically coherent and temporally aligned audio conditioned on visual content. Additional examples are provided in Figure 7.
E Sensitivity Test
We conduct a sensitivity analysis to probe how the system responds to prompt variations that (i) alter non-audible attributes (e.g.. background, static non-audible objects) while keeping the sound- causing event unchanged, and (fi) alter audible attributes (c.g., object/action that produces sound) while holding non-audible details fixed. As shown in Figure 9, for each prompt pair, we generate matched video-audio samples under identical random seeds. Qualitatively, we visualize spectrograms alongside key video frames to inspect whether non-audible edits leave the soundtrack invariant and whether audible edits induce commensurate changes in rhythm and energy. Ideally, von-audible edits should produce minimal shifts audio metrics, while audible edits should yield significant, directionally consistent changes. This analysis clarifies which aspects of the prompt our model treats as causally relevant for sound synthesis and highlights residual entanglements where visual-only edits still perturb the audio.
21

---

AVHT
CAVPT
FAD
AVH1
CAVPT
FAD1
0.001
0.174
0.150
5.52
1
0.208
0.161
7.03
0.01
0.197
0.159
5.60
0.206
0.165
5.49
0.1
0.206
0.165
5.49
10
0.174
0.150
7.42
0.199
0.154
5.31
100
0.172
0.141
6.97
Table 4: Grid search summary for
Table 5: Grid search summary for A.
Hyperparameter Studies
We study how the alignment weight a and sparsity weight À affect performance in Table 4 and 5. We sweep E {0.001, 0.01, and À E {1, 10, 100} on a validation split, measuring AVHScore and CAVP similarity (alignment), FAD (audio quality). In unidirectional setting the FVD is not affected by alignment. We Find that: Reducing a too much weakens the latent coupling. leading to semantic drift between modalities (e.g., misaligned motion and sound). Excessively high over- prunes the mask. Disabling sparsity results in overly dense masks, which fail to isolate cross-modal signals and slightly reduce performance in AVHScore and CAVP similarity.
G Limitations
While cascaded T2AV framework demonstrates strong performance in both generative quality and cross-modal alignment, several limitations remain. First, the reliance on sequential generation, where video is produced before audio, introduces unidirectional dependency that may constrain expressiveness in audio-visual so-synchronization, particularly for content requiring tight mutual feedback. Second, the alignment mechanism is trained temporally segmented clips, which may limit its ability to generalize to complex or highly dynamic temporal structures in longer sequences. Third, although we finetune the diffusion models better adapt to generated embeddings, the overall generation quality remains bounded by the capacity and resolution of the pretrained backbones, which not jointly optimized with the alignment modules. Finally, our framework assumes the availability of high-qualig paired data for training, extending it to settings with weak or noisy supervision remains an open challenge.
H
Social Impact and Safeguards
Our work focuses improving the quality and coherence of text-to-audioviieo (T2AV) generation through aligned latent representations and cascaded diffusion models. While such generative capabil- ities hold potential for positive applications in assistive technologies, content creation, and education, they also raise ethical concerns related to misinformation, deepfakes, and unauthorized content synthesis. In particular, the ability to generate realistic audio-visual content conditioned on arbitrary text inputs could be misused to fabricate misleading media or impersonate individuals. To mitigate these risks, recommend deploying the model with usage constraints such content watermarking, access control mechanisms, and -imman-n--the-log verification. Moreover, our system is not trained on or intended for real-person likeness reproduction, and safeguards should be established before applying it in socially sensitive domains. We also advocate for transparency in model provenance and responsible dataset curation, ensuring compliance with copyright, privacy, and faimness standards.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims made
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed limitation in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification:
The paper provide full set of assumptions and a complete and correct proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully disclose all the information needed reproduce the main experimental results
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [No]
Justification:
In accordance to the company policy.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
icl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccppppcccceeececcecccccctt / cenbbbissiii wwweiissiinoooooo (eeinnnyi for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes].
Justification: The paper specify all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA].
Justification:
Not applicable to our case.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
25

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes].
Justification:
The paper provide sufficient information on the computer resources
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp:///eeii..//eee
Answer: [Yes].
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification:
We include discussion about potential societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes].
Justification:
We include discussion about safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: All assets used are under CC-BY 4.0 license.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
28

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA].
Justification: The core method development in this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: : weeeur/ps.cccommfeeeeeeeeeeee/ee for what should or should not be described.
29


Reviews:
[{"review_id": "HkIhfssVOX", "reviewer": "Reviewer_iA4m", "summary": "This paper addresses the challenge of generating temporally and semantically aligned audio-visual content from text (Text-to-Audio-Video, T2AV). \nThe authors propose SAVA, a novel framework for Selective Audio-Visual Alignment. \nSAVA achieves state-of-the-art performance on standard T2AV benchmarks.", "questions": "1. The evaluation has limitations in baseline comparisons. In the T2AV experiment, the paper uses a cascaded text-to-video (T2V) and text-to-audio (T2A) approach but lacks a corresponding T2V+T2A baseline in Table 1 for direct comparison. Additionally, in the video-to-audio (V2A) experiment, the baselines are somewhat dated and limited in number. Including recent diffusion or flow-based models, as showcased in Figure 4 (e.g., Diff-Foley), would strengthen the evaluation’s comprehensiveness.\n\n2. The ablation studies lack clarity on critical details. For example, in Table 3, it is unclear whether adapters trained under different alignment strategies required further fine-tuning of the downstream video and audio diffusion models. Providing explicit details on the training pipeline and fine-tuning process would enhance reproducibility and understanding of the results.\n\n3. The paper does not include ablation studies exploring the impact of bypassing Stage 1’s Masked Latent Adaptation and directly training the adapter in Stage 2. Such an experiment could clarify whether the proposed masking strategy is essential or if a simpler adapter-based approach might yield comparable or better results, strengthening the justification for the two-stage design.\n\n\nIf the authors can provide clear and detailed responses to the concerns raised in **Weaknesses** and **Questions**, I would be willing to reconsider and potentially raise my overall score.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- Very good motivation. When audio and video are jointly generated, not all visual information and auditory information need to be interacted. \n\n- The motivation is reasonable and the explanation is clear.\n\n\n\n**Weaknesses**\n\n\n\n1. Generate a one-way relationship between audio and video in the framework. The framework actually uses a cascaded generation, first generating video and then conditionally generating audio. The author did not give a good reason or experimental analysis for why this one-way relationship was adopted instead of the mutual influence between the two modalities.\n\n2. The experimental design is unfair.\n\n- some baselines are missing. The CoDi model is relatively old, and there are relatively new text-to-audio-video models, JavisDiT and SVG_baseline, which can be compared, but without comparison, it is difficult to judge the performance of the model.\n\n- The existing baseline settings are not very reasonable. The video base model used by the author is CogVideoX, but the video model in Two-Streams and CasC-Diff uses AnimateDiff, so the comparison is not fair; the video-to-audio model does not use new ones such as DiffFoley MMAudio, and SpecVQGAN is relatively old.\n\n3. Some suggestions on presentation. In formula 3, it is better to change the sign of z_v after it passes through q_V to distinguish it, otherwise it will cause ambiguity.", "was_revised": true, "final_justification": "Recommended rating: \"5 Accept\"\nMost of my concerns have been addressed. \nIn general, I think this article is acceptable. \nIn addition, I hope the author will open source the later code to facilitate the development of the community.", "rebuttal": "We thank **Reviewer iA4m** for the thoughtful feedback and valuable suggestions, which have greatly helped us improve the clarity and depth of our work. Following are the responses to the questions and comments.\n\n---\n\n**Q1: Current one-way design and potential bidirectional extension.**\n\nWe thank the reviewer for raising this important point regarding the choice of a one-way relationship between video and audio in our framework.\n\nWe wish to first clarify that the unidirectional generation process does not effect mutual influence between the two modalities, we employ a masked latent alignment mechanism during training. This mechanism encourages the model to learn a shared cross-modal latent space by optimizing for both audio-to-video and video-to-audio reconstruction. As a result, although generation proceeds in a single direction at inference time, the model is trained to encode the mutual semantic structure of both modalities. This training design ensures that relevant information from both audio and video is captured in the shared latent space, thereby preserving semantic alignment without requiring bidirectional generation during deployment.\n\nFor practical consideration, visual content provides a more stable and structured prior for generating temporally aligned outputs. For example, the act of a drum being struck or a guitar being strummed is visually discrete and temporally grounded, providing strong cues for the expected acoustic outcome. Compared to audio, which may contain overlapping or ambiguous signals, visual representations often offer clearer event boundaries and object-specific dynamics. This structural clarity makes video a more reliable anchor modality when synthesizing aligned audio content. In addition, from a training perspective, finetuning the attention layers of a diffusion-based audio model to allow video-conditioning is substantially more efficient than modifying a large-scale video diffusion model, which further allows rapid adaptation to pretrained single-modal latent generative models.\n\nThat said, we appreciate the reviewer’s suggestion and have conducted an extended experiment on VGGSound+ to explore the bidirectional generation setup. In this setting, we reverse the pipeline for video generation (audio generation pipeline is unchanged): raw matching audio is first generated using a text-conditioned AudioLDM [1] model, then encoded into a latent representation via the audio VAE. This latent is projected into the shared space using a lightweight audio-to-video adapter. We then fine-tuned the CogVideo [2] architecture by Lora-tuning its cross-attention layers to support dual conditioning from both text and the projected audio latent. This allows the video generator to incorporate semantic signals from both modalities during generation.\n\n|        | FVD  $\\downarrow$     | FAV  $\\downarrow$     | AVHScore $\\uparrow$| CAVPSim $\\uparrow$|\n|----------------|----------------|----------------|----------------|----------------|\n| Uni-directional  | 662.9 | 5.49  |0.206|0.165|\n| Bi-directional  | 701.4 | 5.49  |0.217|0.183|\n\nOur findings show that this bidirectional setup results in improved cross-modal alignment, as measured by both AVHScore and CAVPSim. However, we also observe a slight degradation in video quality, likely due to the noisier and less temporally structured nature of audio signals being used as primary conditioning input. Nonetheless, this result highlights a promising direction for future work, and we are actively exploring ways to refine this dual-conditioning scheme to achieve better trade-offs between alignment and generative fidelity.\n\n---\n\n**Q2: Additional Baselines for T2AV and V2A Comparisons**\n\nThank you for this important and detailed feedback. \n\nRegarding the use of CogVideoX [1] for our framework versus AnimateDiff [3] in some baselines: our primary goal was to demonstrate that our proposed alignment method improves audio-visual consistency regardless of the underlying video backbone. We adopted CogVideoX [1] to ensure high-fidelity video generation in our main method, and used AnimateDiff as baselines to remain consistent with  prior works such as TAVDiffusion [4]. However, we acknowledge that this discrepancy introduces a potential source of variance. To address this, we have conducted control experiments with corresponding T2V+T2A baselines to isolate the impact of framework design choices. Specifically, we construct the Two-Streams and CasC-Diff baselines using the same fine-tuned CogVideoX as the video backbone.\n\n|        | FVD  $\\downarrow$     | FAV  $\\downarrow$     | AVHScore $\\uparrow$| CAVPSim $\\uparrow$|\n|----------------|----------------|----------------|----------------|----------------|\n| Two-Streams  | - | 6.29  |0.061|0.092|\n| CasC-Diff | - | 7.12  |0.154|0.117|\n|Ours|662.9|5.49|0.206|0.165|\n\nThe results demonstrate that the two-stream approach suffers from poor audio-visual alignment, while the vanilla cascade approach exhibits degradation in the quality of the generated audio conditioned on the generated video.\n\nRegarding the selected baselines, TAVDiffusion [4] and SeeingHearing [5] were both released in 2024 for T2AV and A2V generation. However, we recognize the importance of comparing against the most recent models. As suggested, we will include our concurrent work, JavisDiT [6], under a zero-shot setting on both the VGGSound+ and AudioCaps datasets. In addition, we compare the video-to-audio generative quality with Diff-Foley [7].\n\n|        | KL  $\\downarrow$     | ISc  $\\uparrow$     | FD $\\downarrow$| FAD $\\downarrow$|\n|----------------|----------------|----------------|----------------|----------------|\n| Diff-Foley  | 2.711 | 5.314  |37.262|6.861|\n| Ours  | 2.128 | 5.677  |39.534|6.155|\n\nIn summary, to address your concerns regarding evaluation, we:\n* Add direct comparisons with JavisDiT in the T2AV setting, the results will be reported once ready.\n* Incorporate the recent V2A method, Diff-Foley.\n* Include the corresponding T2V+T2A results.\n* Clarify that our improvements primarily stem from enhanced cross-modal alignment rather than from superior unimodal generation models.\n\n---\n\n**Q3: Ablation Study on Direct Alignment and Implementation Details**\n\nTable 3 in our manuscript reports the effect of bypassing Masked Latent Adaptation and directly training the adapter only. Our experimental results demonstrate that omitting the masking strategy significantly degrades audio generation quality and negatively impacts both the AVH score and CAVP similarity metrics. These findings underline the essential role of the masking strategy in effectively aligning the multimodal latents, thus justifying our proposed two-stage design. We will ensure this analysis and the corresponding insights are explicitly discussed in the revised manuscript.\n\nRegarding implementation details, our framework is built around a two-stage design. In the first stage, we train lightweight adapters to align latent representations between modalities. During this alignment stage, all components of the diffusion models, including the video and audio generators as well as their associated VAE encoders and decoders are kept frozen. The adapters are trained using the alignment objective introduced in Eq. 8 (Depending on the ablation, the input of mask varies. For direct alignment, the masks are set to all 1 to allow direct aligning, the sparsity losses become constant), to project one modality’s latent into the latent space of the other.\n\nOnce the adapters are trained, we proceed to the second stage, where we enable conditional generation based on the aligned representations. To make the diffusion model responsive to the new conditioning signals that include both text and projected audio/video features, we apply LoRA-based fine-tuning only to the cross-attention layers of the target diffusion model. All other parameters remain frozen during this stage as well. This selective fine-tuning ensures efficient adaptation while preserving the pretrained capabilities of the base models.\n\nIn the specific case of the ablation studies shown in Table 3, each row corresponds to a different alignment strategy applied to the adapter. For all these comparisons, the downstream diffusion models are held fixed after the initial LoRA-based tuning. That is, we evaluate each adapter variant in a zero-shot fashion without further training the diffusion model. This setup isolates the impact of the alignment strategy and allows for a clean comparison of adapter effectiveness.\n\n---\n\n**Q4: Presentation of Formula 3**\n\nThank you for pointing this out regarding Formula 3. We will adopt a distinct notation for this term in the revised manuscript to clearly distinguish its transformed representation.\n\n---\n\nReferences:\n\n[1] Liu, H., et al. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. In International Conference on Machine Learning. \n\n[2] Yang, Z., et al. CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. In The Thirteenth International Conference on Learning Representations.\n\n[3] Guo, Y., et al. AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning. In The Twelfth International Conference on Learning Representations.\n\n[4] Mao, Y., et al. Tavgbench: Benchmarking text to audible-video generation. In Proceedings of the 32nd ACM International Conference on Multimedia.\n\n[5] Xing, Y., et al. \"Seeing and hearing: Open-domain visual-audio generation with diffusion latent aligners.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[6] Liu, K., et al. \"Javisdit: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchronization.\" arXiv preprint arXiv:2503.23377 (2025).\n\n[7] Luo, S., et al. \"Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models.\" Advances in Neural Information Processing Systems 36 (2023)."}, {"review_id": "b0eWdfLOgI", "reviewer": "Reviewer_wB3p", "summary": "The paper introduces SAVA, a novel two-stage framework for text-to-audio-video (T2AV) generation that addresses the challenge of semantic and temporal misalignment between visual and auditory modalities. In real-world settings, not all visual elements produce corresponding sounds—objects like roads, buildings, or background scenery may be visually salient but silent—while some audio cues, such as sirens or wind, may lack clear visual counterparts. Existing approaches that attempt to align all features across modalities indiscriminately often introduce semantic noise and degrade cross-modal coherence. To overcome this, SAVA learns to selectively align only the latent components in each modality that are truly relevant to the other, effectively filtering out modality-specific information through a learnable masking mechanism.\n\nAt the core of the framework is a masked latent adaptation mechanism that applies learnable masks to selectively filter video and audio latent representations. This filtering is performed by modality-conditioned mask networks that identify the minimal subset of features in each modality that are predictive of the other. These masks are optimized jointly with adapter networks through a reconstruction loss combined with sparsity-inducing L1 regularization, ensuring that only truly cross-modal features are retained.\n\nTheoretically, the authors prove that this masked objective recovers the minimal sufficient set of shared latents under mild assumptions, improving both interpretability and robustness.\n\nIn practice, the system first generates a video from a text prompt using a pretrained video diffusion model. The resulting video latents are adapted and masked to extract audio-relevant information, which is then used—along with the text prompt—to condition an audio diffusion model. This cascaded generation design maintains modularity and allows efficient reuse of pretrained models.\n\nEmpirically, SAVA achieves state-of-the-art results on the VGGSound and AudioCaps datasets across multiple metrics (e.g., FVD, FAD, AVHScore, CAPSIM), significantly improving semantic alignment and temporal synchronization between audio and video streams.", "questions": "Q1: Line 79 refers to a superscript \"past\", which doesn't actually appear in eq 1.\n\nQ2: The relation between $Z$ and $X$ described in Eq 2 is unclear. Does this equation describe the VAE decoding step? Does $Z_V = Z^{t=0}_V$? If so, it is unclear why e.g. $X_A$ depends on $Z_C$ and not just on the final audio latents.\n\nQ3: The term \"video diffusion encoder\" is confusing. Does it refer to the video encoder (3D VAE) used by latent video diffusion models?\n\nQ4: It is not clear what \"indices of dimensions selected by the soft masks\" mean. (soft masks don't directly define a selection or subset)\n\nQ5: It is unclear how a latent after \"retaining only the selected dimensions\" can be decoded to provide e.g. $\\hat{x_v}$. Can the video VAE decode a partial latent? VAEs are typically convolutional.\n\nQ6: The term \"Mask Blocker\" in Fig 3 is not used anywhere else. If this is the masking-network, shouldn't it accept the latents and not the observed inputs?\n\nQ7: The notation $Q_{g_A}$ and $Q_{g_V}$ in eq. 7 are not clear. Also the relation between eq. 7 and eq. 8 where not clear to me. Do they describe the same loss? If so, why the change in notation?\n\nQ8: The main body of the method section describes a bidirectional method, where each modality is conditioned on projected and masked latents of the other modality, however, in practice, only uni-directional V-->A conditioning is used. Is there some experiment showing that the bi-directional approach can work?\n\nQ9: It is unclear if the video diffusion model is fine-tuned somehow specifically for the AV task (with some special loss), or just fine-tuned on the experiment dataset (as a regular text-to-video model).", "limitations": "There is some discussion about limitations in Appendix E.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "*Quality*\n\nStrengths:\n- Innovative approach to text-to-audio-video generation using modality-conditioned masking.\n- Theoretical justification for learning minimal cross-modal latents.\n\nWeaknesses:\n- Notation and terminology inconsistencies (e.g., unclear superscripts, ambiguous terms like \"video diffusion encoder\").\n- Equations lack clarity; relationships between variables and losses are not well-explained.\n- Decoder's ability to handle partially masked latents is not addressed.\n- Large gap between the described general bi-directional method to the actual tested method. In practice, the proposed method is a (text+video)-to-audio and not text-to-audio-video.\n\n*Originality*\n\nStrengths:\n- Novel use of modality-conditioned masking for selective cross-modal alignment in T2AV.\n\nWeaknesses:\n- Bidirectional conditioning is proposed but not empirically validated; only unidirectional (video-to-audio) is tested.\n\n*Clarity*\n\nStrengths:\n- Logical structure with supportive figures aiding comprehension.\n\nWeaknesses:\n- Inconsistent terminology (e.g., \"Mask Blocker\" not defined elsewhere).\n- Unclear notation in key equations; relationships between equations are not transparent.\n- Ambiguity in the interpretation and application of soft masks.\n- See questions below.\n\n*Significance*\n\nStrengths:\n- Addresses semantic and temporal misalignment in T2AV, a significant challenge in multimodal generation.\n- Potential applicability to broader multimodal tasks beyond T2AV.\n\nWeaknesses:\n- Claims regarding bidirectional conditioning lack experimental support.", "was_revised": true, "final_justification": "I have read the authors' rebuttal thoroughly and decided to keep my initial \"reject\" decision.\nThe authors agreed to my comments about the unclarity in the original description of the method, but in my opinion, a major revision is needed. Two examples are the discussion about decoding masked latents (the authors' answers were contradicting) and the fact the the \"inference\" stage is actually fine-tuning. Combined with the fact that the significance and novelty are not super high, I opt for rejecting.", "rebuttal": "We thank Reviewer **wB3p** for the thoughtful feedback and valuable suggestions. Below, we provide our detailed responses to the questions and comments raised.\n\n---\n\n**Q1: Fix superscript \"past\"**\n\nThank you for pointing this out. We have changed the notation in Equation 1 to use t−1 instead of the superscript “past.\n\n---\n\n**Q2: Clarification of Eq. 2**\n\nThank you for the thoughtful question. Equation 2 does not describe the VAE decoding step directly. Rather, it presents a conceptual generative model that reflects the causal structure between latent variables $(Z_V, Z_A, Z_C​)$ and the observed data $(X_V, X_A​)$. Specifically, it formalizes the assumption that the observed data is generated jointly from both modality-specific and shared latent factors.\n\nIn this formulation, $Z_V$​, $Z_A$​, and $Z_C$​ represent temporally evolving latent variables, not just their values at $t = 0$. That is, they refer to the full latent trajectories across time, and not a single time-step. The dependence of $X_A$​ on both $Z_A​$ and $Z_C$​ reflects the design choice that audio observations are generated from both audio-specific content $Z_A​$ and cross-modal (shared) content $Z_C$, for example, a barking sound depends on both the identity of the sound (modality-specific) and the event semantics (e.g., the presence of a dog, which is shared).\n\nThis abstraction justifies the use of our selective alignment approach, where only the shared component $Z_C​$ should be aligned across modalities. We will clarify this interpretation in the manuscript to avoid conflating it with the VAE decoding implementation.\n\n---\n\n**Q3: Term \"video diffusion encoder\"**\n\nYes, the term “video diffusion encoder” refers to the video encoder component used in latent video diffusion models (typically a 3D VAE) to map raw video frames into a lower-dimensional latent space. We will revise the manuscript to explicitly clarify this in order to avoid confusion.\n\n---\n\n**Q4: Indices of dimensions selected by the soft masks**\n\nThe soft masks in our model output continuous values in the range $[0,1]^d$, acting as attention-like weights over the latent dimensions. While these do not directly define a hard selection, we apply a thresholding operation to obtain binary masks that do. As described in Section 3 - cross-modal reconstruction, we compute sets which contain the indices of latent dimensions where the soft mask values equal 1 (after thresholding). These binary indices then define the selected subset of latent coordinates used in the masked latent representations. We will clarify this distinction and the terminology in the final version to avoid confusion.\n\n---\n\n**Q5: VAE decoding process**\n\nTo clarify, the masking is applied during the training stage only, where it serves to supervise the adapter network to learn which dimensions of the latent space are essential for cross-modal prediction. The actual decoding is not performed using a physically masked latent. Instead, the full latent is used during inference. The masking guides the learning of a projection that emphasizes cross-modal relevance, but does not constrain the decoder input to be literally sparse or zero-padded. This design ensures compatibility with convolutional VAE decoders, which typically expect full latent tensors. We will revise the manuscript to make this clearer.\n\n---\n\n**Q6: Mask Blocker Input**\n\nThank you for pointing this out. The masking network operates on the latent representations, not the observed inputs. The label “Mask Blocker” in Figure 3 was intended to refer to the masking network, but we acknowledge that the terminology and input arrows could cause confusion. We will update the figure and caption to accurately reflect that the mask network takes in the adapted latent embeddings (not raw inputs), aligning with the formal description in Section 3.\n\n---\n\n**Q7: Relation between Eq. 7 and Eq. 8**\n\nThank you for highlighting this. Equation 7 presents the abstract formulation of our training objective, where $Q_{g_A}​​$ and $Q_{g_V}$ denote the generative distributions parameterized by the respective decoders for audio and video. This notation is used to describe the conceptual framework underpinning our theoretical analysis in Section 4 and Section B in Appendix.\n\nIn contrast, Equation 8 appears in Section 3.1, which describes the practical implementation of our method. It introduces concrete instantiations of the reconstruction losses and explicitly includes components like the alignment loss and sparsity penalties. Although both equations describe the same overall objective, the change in notation reflects the shift from an abstract formalism (used for theoretical grounding) to an implementation-level formulation.\n\nWe believe that providing theoretical grounding is an important strength of our framework, and this structured exposition, moving from abstract principle to practical implementation, was chosen deliberately. That said, we recognize the potential for confusion and will revise the manuscript with clearer section headings and textual cues to explicitly link the two formulations.\n\n---\n\n**Q8: Clarification of unidirectional and bidirectional - (text+video)-to-audio and text-to-audio-video.**\n\nThe bidirectional alignment formulation described in the main method section serves two purposes: (1) it defines a symmetric training objective that allows the model to identify shared latent structure by cross-reconstructing both modalities, and (2) it provides theoretical grounding for the mask learning mechanism, as developed in Section 4.\n\nWe wish to clarify that the unidirectional generation process does not effect mutual or bidirecitional influence between the two modalities, we employ a masked latent alignment mechanism during training. This mechanism encourages the model to learn a shared cross-modal latent space by optimizing for both audio-to-video and video-to-audio reconstruction. As a result, although generation proceeds in a single direction at inference time, the model is trained to encode the mutual semantic structure of both modalities. This training design ensures that relevant information from both audio and video is captured in the shared latent space, thereby preserving semantic alignment without requiring bidirectional generation during deployment.\n\nFor practical consideration, visual content provides a more stable and structured prior for generating temporally aligned outputs. For example, the act of a drum being struck or a guitar being strummed is visually discrete and temporally grounded, providing strong cues for the expected acoustic outcome. Compared to audio, which may contain overlapping or ambiguous signals, visual representations often offer clearer event boundaries and object-specific dynamics. This structural clarity makes video a more reliable anchor modality when synthesizing aligned audio content. In addition, from a training perspective, finetuning the attention layers of a diffusion-based audio model to allow video-conditioning is substantially more efficient than modifying a large-scale video diffusion model, which further allows rapid adaptation to pretrained single-modal latent generative models.\n\nThat said, we have conducted an extended experiment on VGGSound+ to explore the bidirectional generation setup. We reverse the pipeline for video generation (audio generation pipeline is unchanged): raw matching audio is first generated using a text-conditioned AudioLDM [1] model, then encoded into a latent representation via the audio VAE. This latent is projected into the shared space using audio-to-video adapter. We then fine-tuned the CogVideo [2] architecture to support dual conditioning from both text and the projected audio latent. This allows the video generator to incorporate semantic signals from both modalities during generation.\n\n|        | FVD  $\\downarrow$     | FAV  $\\downarrow$     | AVHScore $\\uparrow$| CAVPSim $\\uparrow$|\n|----------------|----------------|----------------|----------------|----------------|\n| Uni-directional  | 662.9 | 5.49  |0.206|0.165|\n| Bi-directional  | 701.4 | 5.49  |0.217|0.183|\n\nOur findings show that this bidirectional setup results in improved cross-modal alignment, as measured by both AVHScore and CAVPSim. However, we also observe a slight degradation in video quality. Nonetheless, this result highlights a promising direction for future work, and we are actively exploring ways to refine this dual-conditioning scheme to achieve better trade-offs between alignment and generative fidelity.\n\n---\n\n**Q9: Fine-tuning Details**\n\nIn our framework, the video diffusion model is fine-tuned specifically for the audio-visual (AV) generation task, rather than as a standard text-to-video generator.\n\nConcretely, we introduce a cross-modal latent embedding $z_c$, learned via an audio-to-video adapter trained in Stage 1, which captures audio-visual alignment. During Stage 2, we fine-tune the video diffusion model with LoRA adapters applied to its cross-attention layers, enabling it to interpret this additional conditioning signal. The model is trained using the standard noise prediction loss from the diffusion objective. However, unlike regular text-to-video finetuning, our conditioning includes both text embeddings and the projected audio-derived latent $z_c$. This dual-conditioning setup allows the video generation process to be influenced by both modalities, making the fine-tuning AV-specific.\n\n---\nReferences:\n\n[1] Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W. and Plumbley, M.D., 2023, July. AudioLDM: Text-to-Audio Generation with Latent Diffusion Models. In International Conference on Machine Learning (pp. 21450-21474). PMLR.\n[2] Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y., Hong, W., Zhang, X., Feng, G. and Yin, D., CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer. In The Thirteenth International Conference on Learning Representations."}, {"review_id": "ZuD19pYbSL", "reviewer": "Reviewer_p3WX", "summary": "This paper presents a framework for generating audiovisual content from text. To address the issue of semantic misalignment between the generated audio and video, it proposes a novel solution. Specifically, the authors design a learnable masking mechanism to tackle the challenge of cross-modal semantic alignment, enabling the video features to better match the corresponding audio features.\nIn addition, the paper introduces a hierarchical generation approach. It first generates video content based on the input text, then leverages the proposed masking mechanism to produce video features that are semantically aligned with the audio. These features, together with the original text and other relevant conditions, are subsequently used to generate the audio. As a result, the final output achieves semantic consistency between audio and video.\nThe experiments are thorough and well-executed, demonstrating the effectiveness of the proposed method.", "questions": "1. Why does the framework adopt a video-first generation strategy, where the video is generated before being used as a condition for audio generation? Why not generate audio first and use it as a condition to generate video?\n\n2. In **Section 3**, it is mentioned that $Z_C \\subseteq \\mathbb{R}^d$. Does this imply that all $\\tilde{z}$ are $d$-dimensional? If so, how does this ensure temporal consistency?\n\n3. **Section 3** introduces $q_A$, but this component is not illustrated in **Figure 3**. Is $q_A$ similar to $q_V$, functioning as an adapter module?\n\n4. In **Figure 3**, does the \"Mask Blocker\" correspond to $M_V$ and $M_A$ described in the paper? Why are both the video and spectrogram used as inputs to the Mask Blocker?\n\n5. In the **Joint Optimization Objective** section of **Section 3**, the loss function is defined as $\\mathcal{L}(M_A, M_V, q_A, q_V, g_A, g_V)$. Does this imply that the parameters of $g_A$ and $g_V$ are also updated during training? Why not choose to fix their parameters instead?\n\n6. In **Section 3.1 (Stage II)**, what does the lightweight projection network $\\mathcal{P}_\\theta$ refer to specifically?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The learnable masking mechanism proposed in this paper is notably novel. The authors analyze the asymmetric nature of visual and auditory features, observing that not all visual information corresponds to auditory information, and vice versa. Based on this insight, they design the masking mechanism to align the semantic information between video and audio features. The mechanism filters out irrelevant feature dimensions through masking, and subsequently uses the masked video features to reconstruct the audio data, and the masked audio features to reconstruct the video data. This ensures that the model learns to align semantics across modalities, while also providing a certain degree of interpretability.\n\nMoreover, this approach can benefit other research areas as well, such as audio-driven video generation or generating appropriate background audio for video content.\n\nThe Weaknesses are: \n1. Some descriptions in the paper are unclear. For example:\n   * In **Section 3.1 (Stage I)**, the sentence “$\\ell_A$ and $\\ell_V$ are cross-modal reconstruction losses” lacks clarity. How exactly are the cross-modal reconstruction losses computed?\n\n   * In **Section 4 (Key Assumptions)**, only Assumptions 1–4 are listed. However, **line 198** refers to “Assumptions 4–5.” What is Assumption 5?\n\n2. It is recommended to align the notations used in the paper with the information presented in **Figure 3**. For instance, symbols such as $\\hat{z}_a$ and $\\tilde{z}_a$ could be annotated directly in the figure for clarity.\n\n3. It is suggested to provide a more detailed derivation or proof process for the claims made in **Section 4**.\n4. Some of the compared models have not undergone audio-video alignment, which compromises the fairness of the comparison.", "was_revised": true, "final_justification": "Thank you for the clarification. I raised my score.", "rebuttal": "We thank Reviewer **p3WX** for the thoughtful feedback and valuable suggestions. Below, we provide our detailed responses to the questions and comments raised.\n\n---\n\n**W1.1: Details of reconstruction loss**\n\nThe cross-modal reconstruction losses $\\ell_A$ and $\\ell_V​$ are computed by reconstructing each modality from the masked latent of the other modality, using the corresponding decoder. Specifically:\n$\\ell_A(x_a, g_A(M_V \\odot \\tilde{z}_v))$ is the loss between the ground-truth audio spectrogram $x_a​$ and the audio reconstructed from the masked video latent via the audio decoder. We compute this using mean squared error (MSE) on log-Mel spectrograms, which are widely used for audio reconstruction tasks.\n\n$\\ell_V(x_v, g_V(M_A \\odot \\tilde{z}_a))$ is the loss between the ground-truth video frames $x_v$​ and the video reconstructed from the masked audio latent via the video decoder. This is computed as frame-averaged MSE over the decoded video frames.\n\nThese reconstruction losses serve a dual purpose: (1) they ensure that the masked latent retains sufficient predictive information about the other modality, and (2) when combined with the sparsity constraint on the masks, they encourage the model to preserve only the minimal set of cross-modally relevant dimensions. We will clarify these implementation details in the revised manuscript for improved transparency.\n\n---\n\n**W1.2: Typo on line 198**\n\nThank you for catching this typo. The reference to “Assumptions 4–5” on line 198 is a typo. It should be to \"Assumption 4\". We will correct this in the revised manuscript.\n\n---\n\n**W2: Align notations in Figure 3**\n\nWe appreciate this helpful suggestion. We will update the Figure 3 in the revised manuscript to annotat key variables such as $z_v$, $z_a$, $M_V$​, and $M_A​$ directly to improve clarity between the text and visual illustration.\n\n---\n\n**W3: Derivation or proof for the claims made in Section 4.**\n\nThank you for the suggestion. As noted in the main text, the full derivation and formal proofs supporting the theoretical claims in Section 4 are included in *Appendix Section B* of the submission (placed in the supplementary material folder). This includes all relevant assumptions, formal definitions (e.g., minimum sufficient latents), and complete proofs of our key results such as:\n* Cross-entropy reduction to conditional entropy\n* Sufficientness of reconstruction\n* Sparsity-induced minimality\n* Global block-alignment and recovery\n\nThese results establish that our masked reconstruction objective provably identifies the minimal shared latent subspace under standard identifiability assumptions. We will make this connection more explicit in the main paper by adding cross-references to the relevant sections for improved clarity.\n\n---\n\n**W4: Audio-video alignment for baseline models**\n\nTo ensure fairness and consistency, we adopted the standard evaluation setting introduced in TAVDiffusion [1], which includes both alignment-aware models (e.g., TAVDiffusion [1], CoDi [2]) and baseline systems (e.g., Two-Streams, CasC-Diff) for benchmarking. These methods represent the current landscape of T2AV models, covering both joint and independent generation strategies.\n\nImportantly, our framework was evaluated under the same conditions:\n* Using a shared audio decoder for independent baselines (Two-Stream, CasC-Diff),\n* Applying the same evaluation metrics,\n* Generating outputs from identical text prompts and datasets,\n* And conducting inference in a cascaded setting where applicable.\n\nWe acknowledge that some cascaded models do not perform explicit audio-video alignment. *Their inclusion is intentional*: they serve as important baselines to highlight the effect of alignment mechanisms. In contrast, methods like TAVDiff and CoDi, which incorporate alignment strategies, offer direct comparisons to our alignment-aware approach.\n\nTo further strengthen the evaluation, we have conducted additional experiments incorporating recent alignment-aware models such as JavisDiT [3] and will include these results in the revised manuscript. We will also clearly annotate which models perform alignment to improve transparency in the baseline comparison.\n\nThus, while some baselines do not include alignment by design, we argue that their inclusion, alongside alignment-aware methods, offers a comprehensive and fair evaluation of our contributions.\n\n---\n\n**Q1: Video-first generation strategy**\n\nWe thank the reviewer for raising this important point. For practical consideration, visual content provides a more stable and structured prior for generating temporally aligned outputs. For example, the act of a drum being struck or a guitar being strummed is visually discrete and temporally grounded, providing strong cues for the expected acoustic outcome. Compared to audio, which may contain overlapping or ambiguous signals, visual representations often offer clearer event boundaries and object-specific dynamics. This structural clarity makes video a more reliable anchor modality when synthesizing aligned audio content. In addition, from a training perspective, finetuning the attention layers of a diffusion-based audio model to allow video-conditioning is substantially more efficient than modifying a large-scale video diffusion model, which further allows rapid adaptation to pretrained single-modal latent generative models.\n\n---\n\n**Q2: Temporal consistency**\n\nYes, all $\\tilde{z}$ vectors are d-dimensional latent representations. However, to preserve temporal structure, we do not operate on the entire video or audio as a single global representation. As mentioned in the paper, we split both video and audio into aligned time segments (Using the sampling strategy as in [3]), and compute one $\\tilde{z} \\in \\mathbb{R}^d$ per segment. This segment-level representation allows our framework to maintain local temporal consistency within each segment and encourages alignment across modalities in a temporally grounded way. Additionally, during inference, each video segment is independently mapped to an aligned latent that conditions audio generation for that segment, ensuring temporal synchronization between the two modalities across time. \n\n---\n\n**Q3: Audio Adapter**\n\n$q_A​$ serves the same functional role as $q_V$: it is a learnable adapter module that reparameterizes the output of the frozen audio encoder into a common latent space suitable for alignment. Both $q_V$​ and $q_A$ are lightweight projection networks trained to isolate the cross-modal semantic content. To avoid confusions, we will update the figure in the revised manuscript to explicitly include $q_A$​ alongside $q_V$​, and annotate both as adapter modules to improve consistency and clarity.\n\n---\n\n**Q4: Mask Blocker**\n\nYes, the “Mask Blocker” in Figure 3 corresponds to the learnable masking networks $M_V$​ and $M_A$​ introduced in Section 3. These networks produce soft masks that identify the dimensions of the latent representations that are most relevant for cross-modal alignment. As for the inputs: the masking functions are conditioned on both the video latent $\\tilde{z}_v$​ and the audio latent $\\tilde{z}_a$​, rather than the raw video or spectrogram. This design choice is intentional and discussed in Section 3: conditioning on both modalities enables the model to disambiguate context-dependent relationships, such as when the same video content could be paired with different types of audio (e.g., narration vs. ambient sound). Using both latent representations allows the mask network to more accurately isolate semantically aligned features across modalities. We will revise the figure and caption to reflect this more clearly, specifically, by relabeling the “Mask Blocker” as $M_V$/$M_A$​ and clarifying that it operates on latent embeddings rather than raw inputs.\n\n---\n\n**Q5: Joint optimization objective**\n\nThank you for this important question. In our framework, the decoders $g_A$​ and $g_V$ are not updated during Stage I training. They remain frozen and are used solely to compute the reconstruction losses that guide the learning of the adapters $(q_A, q_V​)$ and masking networks $(M_A, M_V​)$. The notation in the loss function was meant to indicate dependency on the decoder outputs, but we agree it may misleadingly suggest that $g_A​$ and $g_V​$ are trainable. We chose to freeze these decoders in order to (1) preserve the pre-learned generative priors of the pretrained diffusion models, and (2) maintain modularity and efficiency by avoiding full end-to-end retraining. We will clarify this point in the revised manuscript by explicitly stating that $g_A​$ and $g_V$​ are fixed during alignment training, and we will revise the loss notation to avoid ambiguity.\n\n---\n\n**Q6: Projection network**\n\n The projection network $P_\\theta$ used in Stage II corresponds to the adapter network $q_V$​ that was trained during Stage I. Specifically, after generating a video using the pretrained video diffusion model, we pass its latent representation $z^t_v$​ through the learned adapter $q_V$​ to obtain the aligned latent $\\tilde{z_v​}$, which is then used to condition the audio generation process. We refer to this trained adapter as $P_\\theta$ in the inference phase to emphasize its role as a fixed projection function, but it is indeed the same module as $q_V​$ learned during Stage I. \n\n---\nRerfernece:\n\n[1] Mao, Y., Shen, X., Zhang, J., Qin, Z., Zhou, J., Xiang, M., Zhong, Y. and Dai, Y., 2024, October. Tavgbench: Benchmarking text to audible-video generation. In Proceedings of the 32nd ACM International Conference on Multimedia (pp. 6607-6616).\n\n[2] Tang, Z., Yang, Z., Zhu, C., Zeng, M. and Bansal, M., 2023. Any-to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 36, pp.16083-16099.\n\n[3] Luo, S., Yan, C., Hu, C. and Zhao, H., 2023. Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. Advances in Neural Information Processing Systems, 36, pp.48855-48876."}, {"review_id": "o07PbR4BB8", "reviewer": "Reviewer_aSVy", "summary": "This paper tackles the challenge of synchronizing visual and audio features in text-\nto-audio-video generation. The key insight is that audio and visual elements don&#39;t\nalways align perfectly - sometimes audio events have no visual counterpart, and vice\nversa. The authors propose a cross-modal learnable masking mechanism that\nidentifies and aligns only the corresponding features between modalities. This\napproach generates more contextually consistent outputs compared to existing\nmethods that assume complete alignment between audio and visual elements.", "questions": " What is the specific function of the Adapter module. It’s unclear how\nmuch each adapter (vision and audio) effectively serves for learnable\nreparameterization or what semantic information each one captures. Two\npotential experiments could clarify this for the video adapter: (i) Modify the\nvalues of the video latent across different dimensions and observe how this\naffects the generated audio; (ii) Use text prompts that should generate\ndifferent videos but similar audio (e.g., \"A panda eating bamboo\"; vs. \"An\nelephant eating bamboo\") and analyze how the video latents differ in general\n(L2 distance), and across separate dimensions. This could potentially reveal\nwhether the adapter modules learn disentangled representations for different\nobject types or actions.\n\n How different loss terms contribute to overall quality. The method\nincludes several loss terms, but I wonder how qualitative results change\ndependant on different loss weightings (e.g., prioritizing alignment over\nsparsity).", "limitations": "--", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: \n\n The paper addresses an important gap in multimodal text-to-audio-video\ngeneration by recognizing that audiovisual features should only be partially\naligned, while existing work assumes full alignment. The authors achieve\nstate-of-the-art results in both multimodal alignment and generation quality.\n\n The masking mechanism presented could be easily integrated into existing\nencoder-decoder based architectures as a plug-and-play component.\n\n Their method is theoretically sound and intuitively reasonable.\n\nWeaknesses:\n\n More qualitative comparisons with TAVDiffusion are needed to showcase the\nquality of audiovisual synchronization. Qualitative results for different mask\nconfigurations could also be provided to strengthen their claims. Moreover,\nthe authors should include failure cases to show which types of prompts\nproduce less satisfactory results.\n\n The paper suffers from unclear presentation. Section 2 is redundant, only\nstating that visual, audio, and shared representations depend on previous\nstates, with no clear reference to this formulation later on in the article, and with almost no mention of the guidance mask. The method illustration could be clearer, as there are two separate adapters and masking modules\ntrained for both vision and audio, while the main figure (Figure 3) only illustrates the generation strategy.\n\n The technical implementation details of the Adapter modules are missing.", "was_revised": true, "final_justification": "The authors have promised to address my concerns in the final version and have cleared a few points. I maintain my supportive rating.", "rebuttal": "We thank Reviewer **aSVy** for the thoughtful feedback and valuable suggestions. Below, we provide our detailed responses to the questions and comments raised.\n\n---\n\n**W1: More qualitative comaprisons with TAVDiffusion**\n\nWe will include additional visualized results generated by our model and TAVDiffusion in the updated manuscript.\n\n---\n\n**W2: Presentation of Section 2 and Mask**\n\nWe appreciate the reviewer’s feedback on the presentation clarity and agree that clearer exposition would improve accessibility.\n\nSection 2 is intended to provide theoretical grounding for our method, particularly motivating the selective alignment mechanism via a causal latent structure. While this section is not directly operationalized in our implementation, it supports the theoretical analysis in Section 4, where we prove that our masked objective recovers the minimal set of shared latent factors. That said, we acknowledge that this connection could be made more explicit, and we will revise the manuscript to better link Section 2 with the training and alignment mechanism described in Section 3.\n\nRegarding the guidance mask and dual adapters: we agree that Figure 3 currently emphasizes the inference-time generation strategy and underrepresents the full training architecture. To address this, we will update the figure to:\n* Explicitly show the two adapter networks $q_V$​ and $q_A$​,\n* Depict the dual masking modules $M_V$​ and $M_A​$,\n* And annotate that these components are trained during Stage I.\n\nWe will also revise the figure caption and method description to clarify the distinct roles of these components and improve overall coherence across sections. \n\n---\n\n**W3 & Q1:**\n\nThe adapter modules $q_V​$ and $q_A$ are implemented as lightweight multilayer perceptrons (MLPs). Their function is to perform learnable reparameterization of the frozen encoder outputs into a shared latent space optimized for cross-modal alignment. Specifically, they help suppress modality-specific noise and enhance semantic factors that are predictive of the other modality.\n\nTo analyze the behavior of the video adapter, we conducted both suggested experiments:\n* Modified individual dimensions of the adapted video latent and observed the corresponding changes in the generated audio. Qualitative results show that certain dimensions are highly sensitive, affecting acoustic features, indicating that the adapter learns structured and interpretable features.\n\n* We mannually create a set of text prompts designed to generate different videos but similar audio content (e.g., “A brown panda eating bamboo” vs. “An black and white panda eating bamboo”; \"A dog barking in the snow\" vs. \"A dog barking under the sun\" ,.etc.), and report the mean cosine similarty (MCS) between feature after adaptation. Our analysis revealed that the masked aligned latents produced by the adapter are more similar (compared to direct aligned latents), suggesting that the masking effectively filters out visual identity and retains audio-relevant semantic actions like “eating bamboo.” \n\n    |Alignment        | MCS  | \n    |---------------|------------|\n    | w/ Mask     |   0.88       | \n    | wo/ Mask    |    0.74     | \n\nDue to rebuttal rules, we are unable to attach the figures here. However, we will include these analyses and corresponding figures in the revised manuscript and appendix. We appreciate the reviewer’s thoughtful feedback, which helped us better characterize and validate the role of the adapter modules.\n\n---\n\n**Q2:**\n\nThank you for raising this important point. Our full objective (Eq. 8) includes three components: (1) cross-modal reconstruction loss, (2) masked alignment loss between adapted latents $\\tilde{z}_v$ and $\\tilde{z}_a​$, and (3) sparsity regularization over the learned masks. Each term serves a distinct role:\n* Reconstruction loss ensures that the masked latent retains sufficient predictive information about the other modality\n* Masked alignment loss promotes geometric consistency across modalities in the adapted latent space.\n* Sparsity loss encourages the masking network to retain only essential dimensions, reducing overfitting to modality-specific noise.\n\nIn our experiments, we set the alignment loss weight $\\alpha = 0.1$ and sparsity regularization weight $\\lambda = 5.0$, based on a small-scale grid search. Empirically, we found that: Reducing $\\alpha$ too much weakens the latent coupling, leading to semantic drift between modalities (e.g., misaligned motion and sound). Excessively high $\\lambda$ over-prunes the mask. Disabling sparsity results in overly dense masks, which fail to isolate cross-modal signals and slightly reduce performance in AVHScore and CAVP similarity.\n\nWe will include a more detailed ablation table in the appendix showing how varying these weights affects both quantitative scores and qualitative outputs. Thank you for pointing out the need to clarify this aspect of our design."}]

Author rebuttal:

Meta-review: This paper introduces a three-stage masked-latent-alignment framework for text-to-audio-video (T2AV) generation that selectively aligns audio and video modalities via cross-modal masking, achieving improved semantic and temporal coherence. The reviewers agree on the (1) novelty and theoretical grounding of the selective masking strategy, (2) effectiveness in cross-modal semantic alignment, (3) modular design that supports cascaded generation and integration with pretrained diffusion models, and (4) state-of-the-art performance on standard benchmarks. However, they note (1) inconsistencies and ambiguities in the method's presentation (especially regarding decoder behavior, loss computation, and stage descriptions), (2) insufficient qualitative analysis and visualizations of outputs and failure cases, and (3) a lack of clarity in the architectural and training details that required extensive rebuttal clarification. The authors’ follow-up responses addressed the major concerns: they clarified the masking and reconstruction mechanisms, outlined the exact decoder behavior during training, acknowledged the framework as a three-stage pipeline rather than two, and proposed revisions to improve exposition, while also conducting additional experiments including bidirectional generation and new baselines. The AC thus leans to accept this submission.
Decision: Accept (poster)